I have a pretty standard model. I need it to be reproducible, so I use a random seed. But when I insert a multinomial operation anywhere in the training code, e.g.,
torch.ones(10).multinomial(num_samples=2, replacement=False)
It changes the output/performance of the model.
Hi Sam!
I think I understand what you are asking …
Pytorch, roughly speaking, has a global random number generator that
is used for all random operations. So when you call multinomial(), it
“consumes” some random numbers so that future random operations
become different (but should be statistically equivalent). The “random
operations” in your training code could be things like random initialization
of model weights, Dropout operations, or a RandomSampler in a
DataLoader.
One approach would be to pre-calculate the multinomial values you will
need before setting your random seed and starting your training.
This is illustrated by this script:
import torch
torch.__version__
torch.random.manual_seed (2020)
torch.randn ((5,)) # desired reproducible result
torch.random.manual_seed (2020)
torch.ones(10).multinomial(num_samples=2, replacement=False) # this uses up some random numbers
torch.randn ((5,)) # therefore changing this result
torch.random.manual_seed (1010)
cached_multinomial = torch.ones(10).multinomial(num_samples=2, replacement=False) # cache some values for future use
torch.random.manual_seed (2020)
cached_multinomial # use cached values for whatever purpose
torch.randn ((5,)) # doesn't mess up desired result
Here is its output:
>>> import torch
>>> torch.__version__
'1.6.0'
>>>
>>> torch.random.manual_seed (2020)
<torch._C.Generator object at 0x7fe662956910>
>>> torch.randn ((5,)) # desired reproducible result
tensor([ 1.2372, -0.9604, 1.5415, -0.4079, 0.8806])
>>>
>>> torch.random.manual_seed (2020)
<torch._C.Generator object at 0x7fe662956910>
>>> torch.ones(10).multinomial(num_samples=2, replacement=False) # this uses up some random numbers
tensor([7, 9])
>>> torch.randn ((5,)) # therefore changing this result
tensor([-0.3136, 0.6418, 1.1961, 0.9936, 1.0911])
>>>
>>> torch.random.manual_seed (1010)
<torch._C.Generator object at 0x7fe662956910>
>>> cached_multinomial = torch.ones(10).multinomial(num_samples=2, replacement=False) # cache some values for future use
>>>
>>> torch.random.manual_seed (2020)
<torch._C.Generator object at 0x7fe662956910>
>>> cached_multinomial # use cached values for whatever purpose
tensor([2, 0])
>>> torch.randn ((5,)) # doesn't mess up desired result
tensor([ 1.2372, -0.9604, 1.5415, -0.4079, 0.8806])
You could also instantiate a non-global Generator object that you then
pass into your multinomial() calls so that global random numbers
won’t be consumed.
Best.
K. Frank
Thank you! Another option is to temporarily use numpy.choice in that call, but this would be better.
Do you know if numpy also does what pytorch does in that it also “consumes” some random numbers thus changing outputs with each extra call?
Hi Sam!
I’m not a numpy expert, but, yes, almost certainly. (You could easily run
a quick test.)
If various numpy distributions – normal, multinomial, etc. – don’t require
you to pass in a random-number-generator object, then they are drawing
random numbers from some global generator, and doing so will have the
effect you describe.
Best.
K. Frank
Got it. Is there a way I could extract the seed prior to the extra random operation so that I can set it back to that seed?
I noticed you set the seed to 1010 in your example. Was that line necessary? It seems like just resetting the seed to 2020 would be sufficient.
Hi Sam!
You don’t really want to do this, but if you decide that you do want to
do this, there are a couple of approaches. See the documentation for
torch.random for details.
As I understand it, you make calls to multinomial() during your training
process. If your calls to multinomial() are deterministic – called the
same number of times at the same point in your training process – then
your training process should be reproducible, notwithstanding the fact
that multinomial() consumes some random numbers. (If your calls
to multinomial() are not deterministic, why not?)
The problem with having multinomial() consume some random
numbers from the global generator and then resetting its state is that
your multinomial() values will now be correlated with the next set
of random values used by the next part of your training process. In
practice, this is very unlikely to have any problematic effect, but reusing
(pseudo) random numbers like this makes them no longer properly
(pseudo) random.
If you require that your training process be reproducible but you can’t,
for whatever reason, make your multinomial() calls deterministic,
then the standard approach would be to instantiate a separate
Generator object (seeded with an uncorrelated seed) for your
multinomial() calls (or you could pre-calculate your necessary
multinomial() values before seeding the global generator).
Calling manual_seed() prior to calling multinomial() is not required
to make the call to randn() reproducible. It is there to make the call
to multinomial() reproducible – certainly not necessary, but I put it
in for consistency.
Best.
K. Frank
It’s just that I need to compare it to a model that is the same in every way but does not call multinomial, so the randomness changes. For modularity reasons it’s hard to insert a manual random seed in the same place in both models, i.e., in the place corresponding to right after the multinomial call.
How would I go about doing this?
How can I pass a generator into multinomial? Can you provide an example? Multinomial doesn’t take a generator as input.
Is this okay?
rand_state = torch.random.get_rng_state()
samp = probas.multinomial(num_samples, replacement)
torch.random.set_rng_state(rand_state)
rand_state = torch.random.get_rng_state()
torch.random.manual_seed(torch.randn(1).data)
probas.multinomial(num_samples) # (temporarily) changes seed
torch.random.set_rng_state(rand_state)