Selectively applying random seed to different parts of the model

[ptrblck]

@Vignesh_Baskaran

The first one is: Ideally I would like to assign different random seeds to:

1. transformer dropout: random_seed(1)
2. linear dropout: random_seed(2)
3. linear initialization: random_seed(3)

You could set the seed using torch.manual_seed before running these operations. Note however that seeding the code before applying a random operation (such as using dropout) would use the same mask for the same input shape, so this operation isn’t “random” anymore.

@Vignesh_Baskaran

My second question is: How can I selectively apply random seed to any of them. To elaborate, I would like to do the following:

1. transformer dropout - Set random seed - So deterministic behaviour
2. linear dropout - No random seed - So random behaviour
3. linear initialization - Set random seed - So deterministic behaviour
4. Very importantly: Remove random seed so everything else following the model has non-deterministic behaviour

You cannot switch between deterministic and random behavior easily, since all operations using the pseudorandom number generator would use the already set seed.
E.g. if you set the seed in operation 1 (transformer dropout), the same drop mask will be samples, but also the following operations using the PRNG would use the same “random” numbers unless you reseed the code to a new (random) seed.