Reproducibility of model using ctc loss

1

It would be great if there would be a full list of operations that might not get reproducible even by enabling cudnn.detereministic flag.
I am using torch.nn.CTCLoss with cpu the results are reproducible but with gpu and following setting the results from two experiments slowly get diverging. I just want to make sure there is no way for me to get reproducible training (loss values, parameter values) as long as I am using ctc loss.

    torch.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)

    import numpy as np
    np.random.seed(seed)
    import os
    os.environ["PYTHONHASHSEED"] = "0"

    import random
    random.seed(seed)

    cudnn.deterministic = True
    cudnn.benchmark = False

Thanks

2

If you use CuDNN, deterministic should work (but you need nightlies or CuDNN < 7.5(!)).
You need to follow the input conventions and probably check for the grad_fn to be the CuDNN one.
Or you could write the deterministic native backward.

Best regards

Thomas

3

Thank you for your response. I have another question, I have installed pytorch with pip install torch==1.2.0 torchvision==0.4.0 does it come with cuDNN installation?
torch.backends.cudnn.version() gives me 7602 which make me believe that it has cudnn version 7.6 with cuda version ‘10.0.130’. If that’s the case, how can I change the cuDNN version?

Thank you for the help.