Random seed that spans across devices

(jsn) #1

Sorry for throwing in multiple questions:

Does pytorch support setting same seed across all devices (CPUs/GPUs)? Do these devices all use the same PRNG? If not, are there any other ways to make layers like dropouts deterministic?


torch.manual_seed(seed) will set the same seed on all devices.

However, setting the same seed on CPU and on GPU doesn’t mean they spew the same random number sequences – they both have fairly different PRNGs (for efficiency reasons).

As guidance, treat the PyTorch PRNGs as determinism helpers, but not so much as having PRNG consistency across device types.

(jsn) #3

Can I get deterministic results for layers like dropout just on GPUs with Pytorch?


yes, if you fix the seed, dropout will be deterministic in pytorch, even on GPUs

(jsn) #5

How do I go about using the same PRNG on different PyTorch devices? Can I switch the backend for different layers? Also, what sort of backends are available in PyTorch for nn layers? CUDNN, MKL, torch (CPU) and torch (GPU)?