Random seed that spans across devices


(jsn) #1

Sorry for throwing in multiple questions:

Does pytorch support setting same seed across all devices (CPUs/GPUs)? Do these devices all use the same PRNG? If not, are there any other ways to make layers like dropouts deterministic?


#2

torch.manual_seed(seed) will set the same seed on all devices.

However, setting the same seed on CPU and on GPU doesn’t mean they spew the same random number sequences – they both have fairly different PRNGs (for efficiency reasons).

As guidance, treat the PyTorch PRNGs as determinism helpers, but not so much as having PRNG consistency across device types.


(jsn) #3

Can I get deterministic results for layers like dropout just on GPUs with Pytorch?


#4

yes, if you fix the seed, dropout will be deterministic in pytorch, even on GPUs