Dropout in LSTM during eval mode

1

Hi.

In pytorch implementation, LSTM takes droupout argument for its constructor, which determines the probability of dropout.

Any ideas on whether dropouts are ignored in evaluation mode? Ex) model.eval()

2

Yes, it should be disabled:

# no dropout
model = nn.LSTM(10, 10, 2, dropout=0.0)
x = torch.randn(10, 10, 10)
out1, _ = model(x)
out2, _ = model(x)
print((out1 - out2).abs().max())
> tensor(0., grad_fn=<MaxBackward1>)

# dropout
model = nn.LSTM(10, 10, 2, dropout=0.5)
x = torch.randn(10, 10, 10)
out1, _ = model(x)
out2, _ = model(x)
print((out1 - out2).abs().max())
> tensor(0.1524, grad_fn=<MaxBackward1>)

# dropout + eval
model.eval()
out1, _ = model(x)
out2, _ = model(x)
print((out1 - out2).abs().max())
> tensor(0., grad_fn=<MaxBackward1>)
3

How to activate dropouts in LSTM during eval mode?

4

Oh got it. Just don’t use eval mode. Just use torch.no_grad()