Inject dropout into resnet (or any other network)

So i want to inject dropout into a (pretrained) resnet, as i get pretty bad over-fitting. (for example add a dropout layer after each residual step)

I guess that i could simply monkey patch the resnet base class. However i wonder if there is easy way to achieve this?

1 Like

I think the cleanest approach would be to create your custom resnet class by deriving from the torchvision implementation and overriding the forward method.

1 Like

With Res net in particular the derivation is even easier as the actual layers are already implemented in a separate class that gets passed to a basic resnet.

I was hopping that there is a general approach, that i could apply to multiple models. The standard models typically don’t contain drop out, as they are usually trained with big datasets. But for small Training datasets which are pretty common in practice, dropout helps a lot. So having a function that would adds dropout before/after each relu would be very useful.

model_with_dropout = add_dropout(model, after=“relu”)

Alternatively to my proposed approach you could also use forward hooks and add dropout at some layers. I’m not sure, if it’s the best approach, but it might be similar to your mentioned approach.

Wouldn’t that also make the network apply dropout during testing/inference? Edit: maybe adding something like this.

It depends of course on your implementation, but you could use the linked training argument as:

model = models.resnet18()
model.fc.register_forward_hook(lambda m, inp, out: F.dropout(out, p=0.5, training=m.training))
1 Like

Hi @ptrblck, could you tell me where I should put that hook? In model.__init__ or model.forward or in training loop or somewhere else?

You could register the hook once after the model was initialized, which would be before the training loop.
Inside the __init__ would also work. I wouldn’t register it in the forward, as you would re-register a hook in each iteration.

1 Like