Selectively applying random seed to different parts of the model

1

I am trying to study the effect of random seeds on the dropout and linear initialization of the model. Here is a snippet of my model

class PoolerClassifier(nn.Module):
    def __init__(self, transformer: nn.Module, transformer_dropout: float, linear_dropout: float,
                 input_size: int, num_classes: int, seed: int):
        super(PoolerClassifier, self).__init__()
        self.transformer = transformer(transformer_dropout)  # Transformer dropout here
        self.linear_dropout = nn.Dropout(p=linear_dropout)  # Linear dropout here
        self.linear = nn.Linear(input_size, num_classes)  # Linear initialization here

    def forward(self, inputs):
        transformer_outputs = self.transformer(**inputs)
        vectors = self.linear_dropout(transformer_outputs.pooler_output)
        logits = self.linear(vectors)
        return logits

I have three questions:
The first one is: Ideally I would like to assign different random seeds to:

  1. transformer dropout: random_seed(1)
  2. linear dropout: random_seed(2)
  3. linear initialization: random_seed(3)

My second question is: How can I selectively apply random seed to any of them. To elaborate, I would like to do the following:

  1. transformer dropout - Set random seed - So deterministic behaviour
  2. linear dropout - No random seed - So random behaviour
  3. linear initialization - Set random seed - So deterministic behaviour
  4. Very importantly: Remove random seed so everything else following the model has non-deterministic behaviour

My third question is: Can I do something like this: nn.Dropout(random_state=12345)?

Thank you very much for your help.

2

You could set the seed using torch.manual_seed before running these operations. Note however that seeding the code before applying a random operation (such as using dropout) would use the same mask for the same input shape, so this operation isn’t “random” anymore.

You cannot switch between deterministic and random behavior easily, since all operations using the pseudorandom number generator would use the already set seed.
E.g. if you set the seed in operation 1 (transformer dropout), the same drop mask will be samples, but also the following operations using the PRNG would use the same “random” numbers unless you reseed the code to a new (random) seed.

1 Like
3

Thank you @ptrblck. I am going to try out your suggestions and write back to you. Thank you