Dropout at test time in densenet

#1

I have fine-tuned the pre-trained densenet121 pytorch model with dropout rate of 0.2.

Now, is there any way I can use dropout while testing an individual image?

The purpose is to pass a single image multiple times through the learned network (with dropout) and calculate mean/variance on the outputs and do further analysis.

3 Likes
Using nn.Dropout2d at eval() time (for modelling uncertainty)
#2

you can set your whole network to .eval() mode, but then set your dropout layers to .train() mode.

You can use the apply function to achieve this for example:
http://pytorch.org/docs/master/nn.html#torch.nn.Module.apply

8 Likes
#3

Thanks. apply is a very handy function.

#4

Related to this, I have another question.

To achieve the same goal – that is to use dropout during testing–
I thought

  1. train a net with dropout and then
  2. during testing I could just set the net to .train(mode=True) and get the output with the same input for multiple runs without updating the network params after each run.

But even when I set the net to .train(), I got the same output during multiple runs with a single image.

Can you explain to me why my approach did not work ?

#5

can you provide a simple example on how to do it via apply to have dropout in the testing time?

#6

You can do something like this:
Suppose model has dropout layers.

def apply_dropout(m):
    if type(m) == nn.Dropout:
        m.train()

model.eval()

model.apply(apply_dropout)
6 Likes
#7

Or more generaly:

def enable_dropout(m):
  for m in model.modules():
    if m.__class__.__name__.startswith('Dropout'):
      m.train()
model.eval()
enable_dropout(model)
6 Likes
#8

Clean solution!

A typo edit suggestion:

def enable_dropout(m):
  for each_module in m.modules():
    if each_module.__class__.__name__.startswith('Dropout'):
      each_module.train()```
2 Likes
#9

Using apply is the recommended approach as setting the whole network to train mode will modify the statistics of any batch normalization layers you have in the network. Thanks for showing how to do it with that method!

#10

Just to add my two cents:

it only works when you have registered dropout layer during initialization with nn.Dropout layers. In case one uses functional dropout F.dropout(x,training = self.training) in the forwad() method as it is in densenet, such tuning off will not work. The only way to turn on the dropout during evaluation for me currently is to define the forward() method again by replacing the F.dropout(x,training = self.training) with F.dropout(x,training = True).