Hi, can I use the same dropout object for multiple drop-out layers? And the same ReLU object? Or do these need to be created individually for each separate use in a layer?
You can definitely use the same ReLU activation, since it doesn’t have a specific state.
For dropout, I understand why it could not work, but the nn.Dropout module itself calls the functional API F.dropout at each forward call, so it would seem that each call randomizes the dropped weights, regardless of whether it’s several modules or just the one! See the source code here.
Thank you Thomas, for your answer, I think that is what I was looking for. Also, I usually directly call ReLUs in forward without declaring them in init but wasn’t sure if that is the best practice.
From my personal view,
Using different dropouts would be better.
Because using same dropout could cause to deactivate same connected neurons in different layers.
Different dropouts would randomly deactivate mitigating overfitting.
I would consider it is a red herring as calling into the same dropout module will still give different random samples.
Not re-using a single module is really a matter of cleanliness in expressing the structure as code.