Actually, I want to implement ordered dropout as tackled in the fjord paper. The behavior of ordered dropout is dropping adjacent components of the model instead of random neurons. I am working on the Resnet18 model, and as a first step, I should override the forward method. Can someone give me some hints or ideas ?
Your idea to override the forward method sounds right and you could either derive a custom class from the ResNet class provided by torchvision or copy-paste the code directly from here (you might need to fix some imports, too).
Once this is done, define your custom dropout layer as an nn.Module and use it in your model.
Don’t forget to switch its behavior using the internal self.training flag so that you can disable it after calling model.eval() (assuming this dropout layer is also a no-op during evaluation).
Thank you sir for your answer