Do we have anything similar to this for determinism in PyTorch. Seems very interesting and exciting approach to debug and discover sources of non-determinism in our pipelines and models?
I just skimmed through the presentation and it seems a few deterministic recipes are already implemented as described in the Reproducibility docs, while we still have some undeterministic modules due to atomic ops.
Are you looking for a specific feature presented in the talk or for general determinism?
Thanks for the reply.
Yes, somewhere in the talk he demonstrates how his tool allowed him to identify sources of determinism in his models. This is shown in the part where he compares hashes of different ops where red mean non-determinism introduced in that particular op or step.
Having sth like that would make debugging much better and allow for better understanding of the models.