How to determine if cudnn.benchmark is finished?

I need to measure the GPU memory consumption of my model and therefore have to wait until CUDNN’s benchmark has finished.

Is there any way to detect this easily (like cudnn.benchmark_finished == True or something similar) ?

I plotted the GPU Memory consumption over time with benchmarking enabled (orange) and disabled (blue).


Of course I could use this profile to “detect” when the benchmark has finished but it feels awfully hacky.


The benchmark will happen every time you call a convolution with inputs of different size.
So it never really “finishes” as if you try to do a convolution with new shape, it will benchmark that new shape to find the best algorithm.
Note that for a given shape, the benchmark will happen during the first call and when the call to conv returns (and you sync the device with torch.cuda.synchronize()), there won’t be any benchmark for that shape anymore.

Alright, that’s what I’ve thought.

So no way around measuring “convergence” manually, I guess!