What does torch.backends.cudnn.benchmark do?

18

What if you are training different networks in the same script exposed to the same input sizes?
I am training a single ResNet44 and after that an ensemble of 3 ResNets 18 on CIFAR10.

How do I manage the cudnn.benchmark = True?
First at the beggining of the ResNet44 training and then once again after the beggining of the first ResNet18?

Thanks!