After setting manual seed in PyTorch, if we apply dropouts in two scenarios on the same layers:
Scenario 1: 10% dropout on linear layer 6 with seed 2024
Scenario 2: 20% dropout on the same linear layer 6 with the same seed 2024
Will this always mean (deterministically) that in Scenario 2, the exact same neurons get dropped out as in Scenario 1 (for the 10%) + some additional neurons (for the remaining 10%)?
Or, does Scenario 2 randomly drop out neurons that may or may not be a part of the original 10% from Scenario 1?
You should assume the second behavior, i.e. no random numbers will be shared between these calls sampling different masks. You could see the same values but there would be no guarantee as e.g. the vectorization might differ between these calls.
It is plausible that your 20% dropout drops all of the same elements as your 10%
dropout and then some.
I’m not aware of any documentation that addresses this detail.
Yes. It is straightforward to test, and it appears to work as you suggest:
>>> import torch
>>> print (torch.__version__)
2.3.0
>>>
>>> t = torch.ones (1000000, 1)
>>>
>>> _ = torch.manual_seed (2024)
>>> d10 = torch.nn.Dropout1d (p = 0.10) (t)
>>>
>>> _ = torch.manual_seed (2024)
>>> d20 = torch.nn.Dropout1d (p = 0.20) (t)
>>>
>>> (d20[d10 == 0.0] != 0.0).sum() # number of elements that were dropped out in d10, but not in d20
tensor(0)
However, you shouldn’t rely on this behavior. It could change based on the size
of your tensor, on gpu vs. cpu, etc., and could well change in future versions.