Randomness in weight init is not guaranteed on linux

1

Following this closed issue: np.random generates the same random numbers for each data batch · Issue #5059 · pytorch/pytorch (github.com)

We know that NumPy on Linux (in contrary to Windows) uses the same seed every time unless manually set. The issue was closed by adding a re-seeding to the DataLoader
(see [DataLoader] Add Numpy seeding to worker of DataLoader by ejguan · Pull Request #56488 · pytorch/pytorch (github.com))

However, as far as I can tell, the issue still exists in the weight initialization when creating a model or when calling reset_parameters() to reset the weights of a module.

Would it be possible to implement the seeding in the reset_parameters() methods?
Otherwise to ensure randomness across repeats on Linux, I think that a manual NumPy random seed should be forced.

2

The .reset_parameters() methods should not depend on numpy (and would break if numpy is not installed as an optional dependency). Seeding PyTorch should be enough to get reproducible results.
Could you post a code snippet showing the issue you are hitting, please?

3

Upon writing a minimized code snippet to reproduce this, I found that I had an import in my code to an old (and unused) python file that had torch.manual_seed(2021) under the imports lines (and not inside a function) which subsequently caused all following torch functions to use this seed :blush:
I think it’s a worthwhile cautionary tale so it may be of some value if anyone ever reads this.