# Manual seed cannot make dropout deterministic on CUDA for Pytorch 1.0 preview version **URL:** \ **Category:** autograd\ **Created:** [October 15, 2018, 9:33am UTC](https://discuss.pytorch.org/t/manual-seed-cannot-make-dropout-deterministic-on-cuda-for-pytorch-1-0-preview-version/27281 "2018-10-15T09:33:20Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![wandering007](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/wandering007/32/3458_2.png) [@wandering007](https://discuss.pytorch.org/u/wandering007)\ **Post date:** [October 15, 2018, 9:33am UTC](https://discuss.pytorch.org/t/manual-seed-cannot-make-dropout-deterministic-on-cuda-for-pytorch-1-0-preview-version/27281/1 "2018-10-15T09:33:21Z")
The testing code is very simple: ```auto import torch import torch.nn as nn seed = 1 model= nn.Dropout(0.5) use_cuda = True for i in range(3): torch.manual_seed(seed) torch.cuda.manual_seed(seed) model.train() data = torch.randn(4, 4) if i > 0: print(torch.equal(data, pre_data)) pre_data = data if use_cuda: data = data.cuda() out = model(data) loss = out.sum() print(i, loss.item()) ``` For `torch-nightly-1.0.0.dev20181001`, the output is: ```auto 0 -5.580026626586914 True 1 -6.234308242797852 True 2 -8.840182304382324 ``` It means the random state is not entirely determined by the manual seed for dropout on CUDA (CPU operation is fine from my test). While the outputs of pytorch v0.4.1 are identical. I guess it is a bug of the preview version. ---
**Author:** ![albanD](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/32/215_2.png) [@albanD](https://discuss.pytorch.org/u/albanD)\ **Post date:** [October 15, 2018, 9:40am UTC](https://discuss.pytorch.org/t/manual-seed-cannot-make-dropout-deterministic-on-cuda-for-pytorch-1-0-preview-version/27281/2 "2018-10-15T09:40:52Z")
Hi, If you’re using the gpu, you need to set the cuda random see as well with [`torch.cuda.manual_seed()`](https://pytorch.org/docs/stable/cuda.html#torch.cuda.manual_seed). ---
**Author:** ![wandering007](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/wandering007/32/3458_2.png) [@wandering007](https://discuss.pytorch.org/u/wandering007)\ **Post date:** [October 15, 2018, 9:43am UTC](https://discuss.pytorch.org/t/manual-seed-cannot-make-dropout-deterministic-on-cuda-for-pytorch-1-0-preview-version/27281/3 "2018-10-15T09:43:21Z")
Oh, I thought `torch.cuda.manual_seed()` is merged into `torch.manual_seed()` since it works fine for Pytorch v0.4.1. ---
**Author:** ![albanD](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/32/215_2.png) [@albanD](https://discuss.pytorch.org/u/albanD)\ **Post date:** [October 15, 2018, 9:45am UTC](https://discuss.pytorch.org/t/manual-seed-cannot-make-dropout-deterministic-on-cuda-for-pytorch-1-0-preview-version/27281/4 "2018-10-15T09:45:21Z")
I don’t think this was ever a single function. I’ve never seen such change and I would really doubt such change would be reverted as it is very very not backward compatible. ---
**Author:** ![wandering007](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/wandering007/32/3458_2.png) [@wandering007](https://discuss.pytorch.org/u/wandering007)\ **Post date:** [October 15, 2018, 9:50am UTC](https://discuss.pytorch.org/t/manual-seed-cannot-make-dropout-deterministic-on-cuda-for-pytorch-1-0-preview-version/27281/5 "2018-10-15T09:50:38Z")
Sorry to tell you that it cannot be fixed by adding `torch.cuda.manual_seed(seed)` in the loop, updated in the snippet. ---
**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\ **Post date:** [October 15, 2018, 11:59am UTC](https://discuss.pytorch.org/t/manual-seed-cannot-make-dropout-deterministic-on-cuda-for-pytorch-1-0-preview-version/27281/6 "2018-10-15T11:59:56Z")
I thought the same, but [this information of reproducibility](https://pytorch.org/docs/master/notes/randomness.html#pytorch) might be misleading: > You can use [`torch.manual_seed()`](https://pytorch.org/docs/master/torch.html#torch.manual_seed) to seed the RNG for all devices (both CPU and CUDA) ---
**Author:** ![albanD](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/32/215_2.png) [@albanD](https://discuss.pytorch.org/u/albanD)\ **Post date:** [October 15, 2018, 12:28pm UTC](https://discuss.pytorch.org/t/manual-seed-cannot-make-dropout-deterministic-on-cuda-for-pytorch-1-0-preview-version/27281/7 "2018-10-15T12:28:35Z")
Interesting, after looking at the code, `torch.manual_seed` does call `torch.cuda.manual_seed_all()` 🙂 ---
**Author:** ![tom](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tom/32/3162_2.png) [@tom](https://discuss.pytorch.org/u/tom)\ **Post date:** [October 15, 2018, 3:52pm UTC](https://discuss.pytorch.org/t/manual-seed-cannot-make-dropout-deterministic-on-cuda-for-pytorch-1-0-preview-version/27281/8 "2018-10-15T15:52:20Z")
It has been there in one form or the other for quite a while: [https://github.com/pytorch/pytorch/pull/1762](https://github.com/pytorch/pytorch/pull/1762) Last I heard was that it was improved to “lazily” seeds all GPUs instead of doing it at call time. Apparently something doesn’t work with (cuda) manual\_seed, though. To decide between dropout and seeding as a source of error, I did two things: - It manual\_seed also doesn’t render bernoulli deterministic, so it’s not in dropout. - When using set\_rng\_state, you achieve reproducible random numbers. ```python import torch import torch.nn as nn seed = 1 use_cuda = True l = (torch.cuda.get_rng_state()) print("manual_seed - different each time") for i in range(3): torch.manual_seed(seed) torch.cuda.manual_seed(seed) lastl = l l = (torch.cuda.get_rng_state()) print ((l==lastl).all()) a = torch.bernoulli(torch.full((3,3), 0.5, device='cuda')) print (a) print("set_rng_state - same each time") for i in range(3): torch.cuda.set_rng_state(l) a = torch.bernoulli(torch.full((3,3), 0.5, device='cuda')) print (a) b = torch.nn.functional.dropout(torch.ones(3,3, device='cuda')) print (b) ``` If you mention me (t-vi on github) on a bug report, I’ll try to figure out what’s going wrong and produce a fix. Best regards Thomas ---
**Author:** ![albanD](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/32/215_2.png) [@albanD](https://discuss.pytorch.org/u/albanD)\ **Post date:** [October 15, 2018, 3:58pm UTC](https://discuss.pytorch.org/t/manual-seed-cannot-make-dropout-deterministic-on-cuda-for-pytorch-1-0-preview-version/27281/9 "2018-10-15T15:58:03Z")
Interestingly, changing the dropout layer to be inplace seems to make dropout deterministic. I guess unrelated fix then? ---
**Author:** ![tom](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tom/32/3162_2.png) [@tom](https://discuss.pytorch.org/u/tom)\ **Post date:** [October 15, 2018, 5:08pm UTC](https://discuss.pytorch.org/t/manual-seed-cannot-make-dropout-deterministic-on-cuda-for-pytorch-1-0-preview-version/27281/10 "2018-10-15T17:08:44Z")
Hm. I can’t see that inplace helps. ~~I think I have to correct my initial assessment and now thing that bernoulli is broken - dropout uses bernoulli. I think it has recently been moved to native, maybe something went wrong there~~. I’ll look a bit more. Edit: Turns out it was the seeding, which is now fixed in master. ---
**Author:** ![albanD](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/32/215_2.png) [@albanD](https://discuss.pytorch.org/u/albanD)\ **Post date:** [October 15, 2018, 6:02pm UTC](https://discuss.pytorch.org/t/manual-seed-cannot-make-dropout-deterministic-on-cuda-for-pytorch-1-0-preview-version/27281/11 "2018-10-15T18:02:21Z")
There is a special fused kernel for the non-inplace version that reproduces the logic from bernouilli. So if the bernouilli logic is broken, this one might be as well. The fused kernel is [here](https://github.com/pytorch/pytorch/blob/6dd71947ea39b321dbc7c357122c2d685e973e7d/aten/src/ATen/native/cuda/Dropout.cu) if you want to take a look at it. ---
**Author:** ![tom](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tom/32/3162_2.png) [@tom](https://discuss.pytorch.org/u/tom)\ **Post date:** [October 15, 2018, 7:43pm UTC](https://discuss.pytorch.org/t/manual-seed-cannot-make-dropout-deterministic-on-cuda-for-pytorch-1-0-preview-version/27281/12 "2018-10-15T19:43:39Z")
So I did some digging: To me it looks like manual\_seed, which ends up calling createGeneratorState only sets the MTGP32 states (=that of the RNG used in THC code), but not the seed + offset for Philox (=the RNG used in native code). [I filed a bug](https://github.com/pytorch/pytorch/issues/12669). @wandering007 Thank you for sharing this with us and providing the testing code! A reproducing example is gold. This should be fixed soon. As a workaround I would recommend using torch.cuda.set\_rng\_state.