Cudnn.benchmark for the network

1

I am new about using CUDA. I am using the following code for seeding:

use_cuda = torch.cuda.is_available()
    if use_cuda:
        device = torch.device("cuda:0")
        torch.cuda.manual_seed(SEED)
        cudnn.deterministic = True
        cudnn.benchmark = False

that is correct if I use this code for the network?

if use_cuda:
        net.cuda()
        net = torch.nn.DataParallel(net)
        cudnn.benchmark = True

I mean setting cudnn.benchmark, for seeding False and then for network True. They are related?

2

cudnn.benchmark is a global option, so if you first disable and later enable it, it will be used again if you pass some input to your model, which might yield non-deterministic results.

3

Thank you for your response, @ptrblck. I am feeding input to my model, this means cudnn.benchmark should be True for the network?

4

If you would like to use the benchmark mode, then yes.
If activated, cudnn will perform some benchmarking internally using the current input shape and your model to determine the best performing algorithms to use for the operations.

This will most likely slow down the first iteration, but should generally yield better performance for the following iterations.

However, if you are dealing with varying input shapes, cudnn will benchmark each of them, thus slowing down your code.

1 Like
5

Got it. Thank you very much for your help @ptrblck.