Actually, I want to implement ordered dropout as tackled in the fjord paper. The behavior of ordered dropout is dropping adjacent components of the model instead of random neurons. I am working on the Resnet18 model, and as a first step, I should override the forward method. Can someone give me some hints or ideas ?
Your idea to override the
forward method sounds right and you could either derive a custom class from the
ResNet class provided by
torchvision or copy-paste the code directly from here (you might need to fix some imports, too).
Once this is done, define your custom dropout layer as an
nn.Module and use it in your model.
Don’t forget to switch its behavior using the internal
self.training flag so that you can disable it after calling
model.eval() (assuming this dropout layer is also a no-op during evaluation).
Thank you sir for your answer