Hi.
I’m wondering - if i use F.dropout in forward() method in my model, instead of nn.Dropout module defined in init - is pytorch using dropout while evaluating or not?
Both are completely equivalent in terms of applying dropout and even though the differences in usage are not that big, there are some reasons to favour the nn.Dropout over nn.functional.dropout:
Dropout is designed to be only applied during training, so when doing predictions or evaluation of the model you want dropout to be turned off.
The dropout module nn.Dropout conveniently handles this and shuts dropout off as soon as your model enters evaluation mode, while the functional dropout does not care about the evaluation / prediction mode.
Even though you can set functional dropout to training=False to turn it off, it is still not such a convenient solution like with nn.Dropout.
Also the drop rate is stored in the module, so you don’t have to save it in an extra variable. In larger networks you might want to create different dropout layers with different drop rates - here nn.Dropout may increase readability and can bear also some convenience when using the layers multiple times.
Finally, all modules which are assigned to your model are registered in your model. So you model class keeps track of them, that is why you can just turn off the dropout module by calling eval(). When using the functional dropout your model is not aware of it, thus it won’t appear in any summary.
Source: neural network - Using Dropout in Pytorch: nn.Dropout vs. F.dropout - Stack Overflow
2 Likes
Small addition to:
The right approach would be to use the internal self.training flag, which is manipulated by calling model.train() and model.eval() via:
x = F.dropout(x, p=self.p, training=self.training)
This would also enable and disable this dropout call if model.train() or model.eval() is called, respectively.
3 Likes