In different pytorch version, dropout performs differently(I set the same random seed)

nlp
1

Hi,
recently, I am migrating code from PyTorch 1.4 to PyTorch 1.7. I set the random seeds of these two pieces of code to 42. When I test my code, I find that torch.nn.Dropout in PyTorch 1.7 dropped different elements compared to torch.nn.Dropout in 1.4.

My function to set random seed:

def seed_all(num: int = 42):
    random.seed(num)
    np.random.seed(num)
    torch.manual_seed(num)
    if torch.cuda.device_count() > 0:
        torch.cuda.manual_seed_all(num)
        torch.backends.cudnn.deterministic = True
2

The internal implementations might have changed (e.g. vectorization could have been added) and I don’t think there is a guarantee for determinism between different PyTorch releases.

1 Like
3

Oh, I get it. Thx a lot