# Random seed initialization

**URL:** <https://discuss.pytorch.org/t/random-seed-initialization/7854>\
**Category:** Uncategorized\
**Created:** [September 26, 2017, 7:13am UTC](https://discuss.pytorch.org/t/random-seed-initialization/7854 "2017-09-26T07:13:45Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![Bixqu](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/958977/32.png) [@Bixqu](https://discuss.pytorch.org/u/Bixqu)\
**Post date:** [September 26, 2017, 7:13am UTC](https://discuss.pytorch.org/t/random-seed-initialization/7854/1 "2017-09-26T07:13:45Z")

</div>

I have a problem regarding a large variation in the result I get, by running my model multiple times. The exact same architecture and training gives anywhere from 91.5% to 93.4% accuracy on image classification (cifar 10).

The problem is that I don’t know how to use the torch random seed in order to get the better results, not the worse ones. I tried various values for the random seed, with:

```
torch.manual_seed(7)

```

and I get the lower bound of the results. Any ideas?

---

<div class="post-metadata">

**Author:** ![smth](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/smth/32/13_2.png) [@smth](https://discuss.pytorch.org/u/smth)\
**Post date:** [September 27, 2017, 5:34am UTC](https://discuss.pytorch.org/t/random-seed-initialization/7854/2 "2017-09-27T05:34:55Z")

</div>

if you are using GPU, you might also need to set `torch.cuda.manual_seed_all`.  
[http://pytorch.org/docs/master/cuda.html#random-number-generator](http://pytorch.org/docs/master/cuda.html#random-number-generator)

---

<div class="post-metadata">

**Author:** ![vsag](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vsag/32/3285_2.png) [@vsag](https://discuss.pytorch.org/u/vsag)\
**Post date:** [January 29, 2018, 9:54am UTC](https://discuss.pytorch.org/t/random-seed-initialization/7854/3 "2018-01-29T09:54:12Z")

</div>

@smth do we need to set torch.manual\_seed() and torch.cuda.manual\_seed\_all() or the second is enough? thanks.

---

<div class="post-metadata">

**Author:** ![smth](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/smth/32/13_2.png) [@smth](https://discuss.pytorch.org/u/smth)\
**Post date:** [January 29, 2018, 9:55am UTC](https://discuss.pytorch.org/t/random-seed-initialization/7854/4 "2018-01-29T09:55:19Z")

</div>

with the latest pytorch 0.3 version you only need to set torch.manual\_seed which will seed all devices

---

<div class="post-metadata">

**Author:** ![sid](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/5f9b8f/32.png) [@sid](https://discuss.pytorch.org/u/sid)\
**Post date:** [February 5, 2018, 12:41am UTC](https://discuss.pytorch.org/t/random-seed-initialization/7854/5 "2018-02-05T00:41:35Z")

</div>

Is the random number generator platform independent?

---

<div class="post-metadata">

**Author:** ![smth](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/smth/32/13_2.png) [@smth](https://discuss.pytorch.org/u/smth)\
**Post date:** [February 5, 2018, 7:09pm UTC](https://discuss.pytorch.org/t/random-seed-initialization/7854/6 "2018-02-05T19:09:46Z")

</div>

the CPU RNG is platform-independent. I am not sure about the CUDA RNG and what guarantees NVIDIA gives across GPU models, CUDA versions and platforms.

---

<div class="post-metadata">

**Author:** ![voxmenthe](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/voxmenthe/32/2739_2.png) [@voxmenthe](https://discuss.pytorch.org/u/voxmenthe)\
**Post date:** [March 29, 2018, 9:04am UTC](https://discuss.pytorch.org/t/random-seed-initialization/7854/7 "2018-03-29T09:04:27Z")

</div>

> [@smth](#):
>
> torch.cuda.manual\_seed\_all.

Funny, even though I have included both:

`torch.manual_seed(999)`

and

`if torch.cuda.is_available():`  
` torch.cuda.manual_seed_all(999)`

I am still getting inconsistent results, fluctuating 1-2% by re-running the model. I wonder why that could be?

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [March 29, 2018, 9:44am UTC](https://discuss.pytorch.org/t/random-seed-initialization/7854/8 "2018-03-29T09:44:11Z")

</div>

Could you try to add `torch.backends.cudnn.deterministic = True` to your code?  
CUDNN has some non-deterministic methods, so small fluctuations might come from this.

---

<div class="post-metadata">

**Author:** ![voxmenthe](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/voxmenthe/32/2739_2.png) [@voxmenthe](https://discuss.pytorch.org/u/voxmenthe)\
**Post date:** [March 31, 2018, 5:29am UTC](https://discuss.pytorch.org/t/random-seed-initialization/7854/10 "2018-03-31T05:29:36Z")

</div>

I added:

`torch.backends.cudnn.deterministic = True` in addition to:

`torch.manual_seed(999)` and

`if torch.cuda.is_available(): torch.cuda.manual_seed_all(999)`

but accuracy for same model/same data still varies considerably across runs. I’ve even tried duplicating the above in the code and even tried switching to the latest version of pytorch (3.1) but still getting the same variability in accuracy across runs for same model/same data. Weird.

---

<div class="post-metadata">

**Author:** ![amirkargar](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/8dc957/32.png) [@amirkargar](https://discuss.pytorch.org/u/amirkargar)\
**Post date:** [April 9, 2018, 4:20pm UTC](https://discuss.pytorch.org/t/random-seed-initialization/7854/11 "2018-04-09T16:20:05Z")

</div>

Hi,

I’m having the same issue, did you figure out a way to make the results consistent across runs?

Thanks,  
Amir

---

<div class="post-metadata">

**Author:** ![Darren\_Lee](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/f04885/32.png) [@Darren\_Lee](https://discuss.pytorch.org/u/Darren_Lee)\
**Post date:** [April 12, 2018, 3:09pm UTC](https://discuss.pytorch.org/t/random-seed-initialization/7854/12 "2018-04-12T15:09:43Z")

</div>

Hi,

Have you figured out how to make the results reproducible now?

Thanks,  
Darren

---

<div class="post-metadata">

**Author:** ![ivan-bilan](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ivan-bilan/32/3481_2.png) [@ivan-bilan](https://discuss.pytorch.org/u/ivan-bilan)\
**Post date:** [May 16, 2018, 6:34pm UTC](https://discuss.pytorch.org/t/random-seed-initialization/7854/13 "2018-05-16T18:34:50Z")

</div>

Same problem here, running on PyTorch 0.4. I am using RRelu though, even though I’ve set all the flags mentioned above, results differ by a margin of +/- 0.5% from run to run.

---

<div class="post-metadata">

**Author:** ![Yufeng\_Ma](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/yufeng_ma/32/4802_2.png) [@Yufeng\_Ma](https://discuss.pytorch.org/u/Yufeng_Ma)\
**Post date:** [May 17, 2018, 12:58am UTC](https://discuss.pytorch.org/t/random-seed-initialization/7854/14 "2018-05-17T00:58:55Z")

</div>

What’s the number of workers for your dataloader? The following post might be helpful for deterministic results.

> [@Deterministic/non-deterministic results with PyTorch](https://discuss.pytorch.org/t/deterministic-non-deterministic-results-with-pytorch/9087):
>
> Hi, I’ve had some trouble in reproducing results, and after reading a few posts there seem to be multiple causes for non-determinism, some which are expected. I’d like to check if I got it right. Issue 1 Dataloaders with multiple threads: They seem to be problematic, even if randoms seeds are set beforehand. [[ref](https://discuss.pytorch.org/t/non-reproducible-result-with-gpu/1831/4)]. Issue 2 cuDNN: Apparently cuDNN seems to have non-deterministic kernels [[ref1](https://discuss.pytorch.org/t/non-reproducible-result-with-gpu/1831/2)][[ref2](https://discuss.pytorch.org/t/non-deterministic-on-gpu/7355/2)] Issue 3 GPU: Apparently, some reductions are non-deterministic in GPU, even without cuDNN. …

---

<div class="post-metadata">

**Author:** ![wtsangmed](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/w/46a35a/32.png) [@wtsangmed](https://discuss.pytorch.org/u/wtsangmed)\
**Post date:** [September 4, 2018, 5:33pm UTC](https://discuss.pytorch.org/t/random-seed-initialization/7854/15 "2018-09-04T17:33:32Z")

</div>

Was following this post b/c I ran into same issues training an autoencoder. I don’t know if the OP has solved the problem. but I did a test last night on a AWS GPU and cuda on w/ the parameters below gave me consistent results.  
`torch.backends.cudnn.deterministic = True`  
`torch.manual_seed(999)`

Further I explicitly specify `model.eval()` after training when computing the decoders and encoders.

Alternatively when I have, below, the results were inconsistent.  
`torch.backends.cudnn.deterministic = True`  
`torch.cuda.manual_seed_all(999)`

As an above poster mentioned it seems as though torch.manual\_seed() applies to both cuda and cpu devices for the latest version. So if you’re not getting consistent result w/ torch.cuda.manual\_seed\_all, try just torch.manual\_seed. This may depend on the pytorch version you have installed…Hope this helps.

---

<div class="post-metadata">

**Author:** ![mutaku](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mutaku/32/7424_2.png) [@mutaku](https://discuss.pytorch.org/u/mutaku)\
**Post date:** [November 6, 2018, 3:30am UTC](https://discuss.pytorch.org/t/random-seed-initialization/7854/16 "2018-11-06T03:30:04Z")

</div>

Good info.

The docs also suggest setting: `torch.backends.cudnn.benchmark = False`

and remember that Numpy should be seeded as well.

–\> [Randomness [Docs]](https://pytorch.org/docs/master/notes/randomness.html)

---

<div class="post-metadata">

**Author:** ![Noam\_Salomonski](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/noam_salomonski/32/9359_2.png) [@Noam\_Salomonski](https://discuss.pytorch.org/u/Noam_Salomonski)\
**Post date:** [February 18, 2019, 11:26pm UTC](https://discuss.pytorch.org/t/random-seed-initialization/7854/17 "2019-02-18T23:26:07Z")

</div>

did anyone solve this yet?

---

<div class="post-metadata">

**Author:** ![isalirezag](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/i/fbc32d/32.png) [@isalirezag](https://discuss.pytorch.org/u/isalirezag)\
**Post date:** [March 17, 2019, 9:05pm UTC](https://discuss.pytorch.org/t/random-seed-initialization/7854/18 "2019-03-17T21:05:25Z")

</div>

Sounds like there is another question related [here](https://discuss.pytorch.org/t/how-to-get-deterministic-behavior/18177/11).

anyways, I think this can be a solution:

```auto
manualSeed = 1

np.random.seed(manualSeed)
random.seed(manualSeed)
torch.manual_seed(manualSeed)
# if you are suing GPU
torch.cuda.manual_seed(manualSeed)
torch.cuda.manual_seed_all(manualSeed)

torch.backends.cudnn.enabled = False 
torch.backends.cudnn.benchmark = False
torch.backends.cudnn.deterministic = True

```

also in the dataloader i set **num\_workers = 0**

based on [here](https://github.com/facebookresearch/maskrcnn-benchmark/issues/376)  
you also need to change `worker_init_fn` as :

```auto
def _init_fn():
    np.random.seed(manualSeed)
    

DataLoding = data.DataLoader(..., batch_size = ..., 
                             collate_fn = ..., 
                             num_workers =..., 
                             shuffle = ..., 
                             pin_memory = ...,
                             worker_init_fn=_init_fn)

```

I noticed if we dont do `torch.backends.cudnn.enabled = False` the results are very close, but some times not match 😯  
p.s. im using pytorch 1.0.1

---

<div class="post-metadata">

**Author:** ![ShangxuanWu](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shangxuanwu/32/12362_2.png) [@ShangxuanWu](https://discuss.pytorch.org/u/ShangxuanWu)\
**Post date:** [July 28, 2019, 2:56am UTC](https://discuss.pytorch.org/t/random-seed-initialization/7854/19 "2019-07-28T02:56:04Z")

</div>

Thanks!

**num\_workers = 0** and **torch.backends.cudnn.enabled = False** are the real thing that works! And I also see that if you train one step 10 times, only using **num\_workers = 0** we can get exactly same output 8 times and different output 2 times.

---

<div class="post-metadata">

**Author:** ![arnabsinha](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/arnabsinha/32/25300_2.png) [@arnabsinha](https://discuss.pytorch.org/u/arnabsinha)\
**Post date:** [June 9, 2020, 6:50am UTC](https://discuss.pytorch.org/t/random-seed-initialization/7854/20 "2020-06-09T06:50:34Z")

</div>

```auto
np.random.seed(0)
random.seed(0)
torch.manual_seed(0)
torch.cuda.manual_seed(0)
torch.cuda.manual_seed_all(0)
torch.backends.cudnn.enabled = False
torch.backends.cudnn.benchmark = False
torch.backends.cudnn.deterministic = True

```

and setting dataloader like the following:

```auto
torch.utils.data.DataLoader(training, shuffle = True, batch_size=BATCH_SIZE, worker_init_fn=np.random.seed(0),num_workers=0)

```

**WORKED FOR ME!**

I am using Pytorch version 1.0.0.

---

<div class="post-metadata">

**Author:** ![Tethys\_Sun](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tethys_sun/32/15895_2.png) [@Tethys\_Sun](https://discuss.pytorch.org/u/Tethys_Sun)\
**Post date:** [August 25, 2020, 4:01am UTC](https://discuss.pytorch.org/t/random-seed-initialization/7854/21 "2020-08-25T04:01:40Z")

</div>

I tried exactly same setting, even with torch.backends.cudnn.enabled =False, the results are not the same… Do you have any idea?

[Next page](https://discuss.pytorch.org/t/random-seed-initialization/7854.md?page=2)