The LSTM documentation suggests that it’s possible to make the LSTM deterministic if the right environment variable is set. I found that not to be the case. Minimal reproduction is included below. If this is intended behavior then the documentation should be updated so as not to mislead users.
lstm_test.py:
import torch
import torch.nn as nn
torch.use_deterministic_algorithms(True)
seed = 1
model= nn.LSTM(4, 4, 2, batch_first = True, dropout=0.5, bidirectional=True).cuda()
for i in range(3):
torch.manual_seed(seed)
torch.cuda.manual_seed(seed)
model.train()
data = torch.randn(4, 4)
if i > 0:
print(torch.equal(data, pre_data))
pre_data = data
data = data.cuda()
out, _ = model(data)
loss = out.sum()
print(i, loss.item())
Calling in bash results in the following:
> CUBLAS_WORKSPACE_CONFIG=:4096:2 python lstm_test.py
0 -3.0934510231018066
True
1 -2.6078720092773438
True
2 -2.4402637481689453
My env:
Ubuntu 18.04.6 LTS
torch.version = 1.12.1
torch.version.cuda = 11.3
sys.version = 3.9.13