Dropout in LSTM during eval mode

Hi.

In pytorch implementation, LSTM takes droupout argument for its constructor, which determines the probability of dropout.

Any ideas on whether dropouts are ignored in evaluation mode? Ex) model.eval()

Yes, it should be disabled:

# no dropout
model = nn.LSTM(10, 10, 2, dropout=0.0)
x = torch.randn(10, 10, 10)
out1, _ = model(x)
out2, _ = model(x)
print((out1 - out2).abs().max())
> tensor(0., grad_fn=<MaxBackward1>)

# dropout
model = nn.LSTM(10, 10, 2, dropout=0.5)
x = torch.randn(10, 10, 10)
out1, _ = model(x)
out2, _ = model(x)
print((out1 - out2).abs().max())
> tensor(0.1524, grad_fn=<MaxBackward1>)

# dropout + eval
model.eval()
out1, _ = model(x)
out2, _ = model(x)
print((out1 - out2).abs().max())
> tensor(0., grad_fn=<MaxBackward1>)