I am new about using CUDA. I am using the following code for seeding:
use_cuda = torch.cuda.is_available()
device = torch.device("cuda:0")
cudnn.deterministic = True
cudnn.benchmark = False
that is correct if I use this code for the network?
net = torch.nn.DataParallel(net)
cudnn.benchmark = True
I mean setting
cudnn.benchmark, for seeding
False and then for network
True. They are related?
cudnn.benchmark is a global option, so if you first disable and later enable it, it will be used again if you pass some input to your model, which might yield non-deterministic results.
Thank you for your response, @ptrblck. I am feeding input to my model, this means
cudnn.benchmark should be
True for the network?
If you would like to use the benchmark mode, then yes.
If activated, cudnn will perform some benchmarking internally using the current input shape and your model to determine the best performing algorithms to use for the operations.
This will most likely slow down the first iteration, but should generally yield better performance for the following iterations.
However, if you are dealing with varying input shapes, cudnn will benchmark each of them, thus slowing down your code.
Got it. Thank you very much for your help @ptrblck.