Hi Sam!
I think I understand what you are asking …
Pytorch, roughly speaking, has a global random number generator that
is used for all random operations. So when you call multinomial(), it
“consumes” some random numbers so that future random operations
become different (but should be statistically equivalent). The “random
operations” in your training code could be things like random initialization
of model weights, Dropout operations, or a RandomSampler in a
DataLoader.
One approach would be to pre-calculate the multinomial values you will
need before setting your random seed and starting your training.
This is illustrated by this script:
import torch
torch.__version__
torch.random.manual_seed (2020)
torch.randn ((5,)) # desired reproducible result
torch.random.manual_seed (2020)
torch.ones(10).multinomial(num_samples=2, replacement=False) # this uses up some random numbers
torch.randn ((5,)) # therefore changing this result
torch.random.manual_seed (1010)
cached_multinomial = torch.ones(10).multinomial(num_samples=2, replacement=False) # cache some values for future use
torch.random.manual_seed (2020)
cached_multinomial # use cached values for whatever purpose
torch.randn ((5,)) # doesn't mess up desired result
Here is its output:
>>> import torch
>>> torch.__version__
'1.6.0'
>>>
>>> torch.random.manual_seed (2020)
<torch._C.Generator object at 0x7fe662956910>
>>> torch.randn ((5,)) # desired reproducible result
tensor([ 1.2372, -0.9604, 1.5415, -0.4079, 0.8806])
>>>
>>> torch.random.manual_seed (2020)
<torch._C.Generator object at 0x7fe662956910>
>>> torch.ones(10).multinomial(num_samples=2, replacement=False) # this uses up some random numbers
tensor([7, 9])
>>> torch.randn ((5,)) # therefore changing this result
tensor([-0.3136, 0.6418, 1.1961, 0.9936, 1.0911])
>>>
>>> torch.random.manual_seed (1010)
<torch._C.Generator object at 0x7fe662956910>
>>> cached_multinomial = torch.ones(10).multinomial(num_samples=2, replacement=False) # cache some values for future use
>>>
>>> torch.random.manual_seed (2020)
<torch._C.Generator object at 0x7fe662956910>
>>> cached_multinomial # use cached values for whatever purpose
tensor([2, 0])
>>> torch.randn ((5,)) # doesn't mess up desired result
tensor([ 1.2372, -0.9604, 1.5415, -0.4079, 0.8806])
You could also instantiate a non-global Generator object that you then
pass into your multinomial() calls so that global random numbers
won’t be consumed.
Best.
K. Frank