The LSTM documentation suggests that it’s possible to make the LSTM deterministic if the right environment variable is set. I found that not to be the case. Minimal reproduction is included below. If this is intended behavior then the documentation should be updated so as not to mislead users.
import torch import torch.nn as nn torch.use_deterministic_algorithms(True) seed = 1 model= nn.LSTM(4, 4, 2, batch_first = True, dropout=0.5, bidirectional=True).cuda() for i in range(3): torch.manual_seed(seed) torch.cuda.manual_seed(seed) model.train() data = torch.randn(4, 4) if i > 0: print(torch.equal(data, pre_data)) pre_data = data data = data.cuda() out, _ = model(data) loss = out.sum() print(i, loss.item())
Calling in bash results in the following:
> CUBLAS_WORKSPACE_CONFIG=:4096:2 python lstm_test.py 0 -3.0934510231018066 True 1 -2.6078720092773438 True 2 -2.4402637481689453
Ubuntu 18.04.6 LTS
torch.version = 1.12.1
torch.version.cuda = 11.3
sys.version = 3.9.13