Reproducibility of model using ctc loss

It would be great if there would be a full list of operations that might not get reproducible even by enabling cudnn.detereministic flag.
I am using torch.nn.CTCLoss with cpu the results are reproducible but with gpu and following setting the results from two experiments slowly get diverging. I just want to make sure there is no way for me to get reproducible training (loss values, parameter values) as long as I am using ctc loss.


    import numpy as np
    import os
    os.environ["PYTHONHASHSEED"] = "0"

    import random

    cudnn.deterministic = True
    cudnn.benchmark = False


If you use CuDNN, deterministic should work (but you need nightlies or CuDNN < 7.5(!)).
You need to follow the input conventions and probably check for the grad_fn to be the CuDNN one.
Or you could write the deterministic native backward.

Best regards


Thank you for your response. I have another question, I have installed pytorch with pip install torch==1.2.0 torchvision==0.4.0 does it come with cuDNN installation?
torch.backends.cudnn.version() gives me 7602 which make me believe that it has cudnn version 7.6 with cuda version ‘10.0.130’. If that’s the case, how can I change the cuDNN version?

Thank you for the help.