What is the difference between torch.manual.seed() and torch.set_rng_state()?

vision
1

I have trained a stochastic model. I first saved the checkpoint file from that epoch and then loaded it back to my evaluation code.

Here is the first experiment I tried:

torch.manual.seed(0)
for i in range(10):   # Repeat the evaluation on this same model 10 times.
    for image, label in data_no_shuffle:
          preds = mymodel(image)
          ....
    print("Accuracy at {} is {}".format(i, accuracy))

The accuracy numbers are quite different in the above experiment.

The 2nd experiment I did is like this:

for i in range(10):   # Repeat the evaluation on this same model 10 times.
     torch.manual.seed(0)
     for image, label in data_no_shuffle:
          preds = mymodel(image)
           ....
     print("Accuracy at {} is {}".format(i, accuracy))

And I see the accuracy is a same value everytime, which makes sense.
However, if I do this:
prev_state = torch.random.get_rng_state()

for i in range(10):   # Repeat the evaluation on this same model 10 times.
    torch.random.set_rng_state(prev_state)
    for image, label in data_no_shuffle:
          preds = mymodel(image)
          ....
    print("Accuracy at {} is {}".format(i, accuracy))
    prev_state = torch.random.get_rng_state()

The accuray from the above experiment became random again.

So what is the difference between torch.manual.seed() and torch.random.set_rng_state()? If I were to reproduce result of a stochastic model multiple times in python code, which one should I use?

2

Shouldn’t you be doing:

rng_state = torch.random.get_rng_state()
for i in range(10):   # Repeat the evaluation on this same model 10 times.
    torch.random.set_rng_state(rng_state)
    for image, label in data_no_shuffle:
          preds = mymodel(image)
          ....
    print("Accuracy at {} is {}".format(i, accuracy))

What you are currently doing is a noop because you are getting the rng state after the state has been updated, and in the following iteration immediately setting the rng state to the value just saved

1 Like
3

Thanks for the guidance. I am not very familiar with these two functions.
What operation will cause rng_state to change?
Which one should I use to make the result reproducable (seed or rng_state)?

4

Doing any random operation, e.g. drop_out, rand, randn will advance the rng_state.

If you want to make your results reproduce across runs, e.g. you have a blah.py file and you run want running python blah.py two times in a row to produce the same result - you should use a seed.

If you don’t care about reproducibility between runs (in fact you might even want results to be different for each run), and all you care about is that each iteration of the loop computes the function and produces the same results - you can use getting and setting rng state as in the above.

1 Like
5

Thank yhou so much. Can I ask several questions about torch.manual_seed?
I am not sure if seed value 0 something special. I mean sometimes, I will see a seed equal to a long int number like “105481905719405” (Just a fake example, maybe it is not correct), instead of a integer like 0, 1, 2, 3.
So am I correct to say seed also changes over time? For example, you set it to 0 at the beginning, and later after 1 epoch, it becomes 127350985103900, and it will become something more different after 2 epochs.
Does the absolute value of seed mean anything? Can I say torch.manual_seed(0) and torch.manual_seed(1) are closer to each other, but torch.manual_seed(100) is far away from both of them?

6

Usually the seed is passed to some function which will use that seed to initialize the rng state. I’m not familiar with the inner workings of that function, but I don’t think you should be able to infer “closeness”.

So am I correct to say seed also changes over time?

It’s more correct to say that the rng state changes over time across iterations.