Different CIFAR10 performance on different version of torch

I tested the NASNet on cifar10 with the different torch versions. Surprisingly, 1.0.1.post2 has the highest performance, what is the difference of 1.0.1.post2 with other versions (PS: I also tested other architectures and have a similar result)

The detailed experiments are listed as follows:

Version Acc. training time (hours)
1.5.0 97.02 31.43
1.0.1 96.79 32.7025
1.0.1.post2 97.11 40.8297
1.1.0 96.86 42.3611
1.2.0 96.79 40.7019
1.3.0 96.86 34.0188
1.3.1 96.69 41.6177
1.4.0 97.02 34.2972

How many runs did you try and how large are the standard deviation and mean values for these versions? Single runs between different versions might not be bitwise accurate due to changes in the libs (CUDA, cudnn) as well as the PyTorch backend.

To generate exactly with the same results, I manually set all random seed to 2, including torch/torch coda/ python random seed/ and numpy random seed, every network is run with 4 times, each run have the same results. Actually, I want to know what is the difference between 1.0.1.post2 with other versions? Is there any files that summarize the changes?

The deterministic results are good for the same version, but I don’t think we can guarantee bitwise accurate results between different versions, so you would have to rerun the scripts with different seeds and check the std dev of the final results.

You could check the Release notes for a high-level overview or use the compare feature of GitHub.
Compare for 1.5.0 vs 1.0.1

1 Like

I use the same code on 10.2 and 0.4.1 the f1 value differents 0.25, and both of them are stable,