# What does torch.backends.cudnn.benchmark do? **URL:** \ **Category:** Uncategorized\ **Created:** [August 8, 2017, 4:21pm UTC](https://discuss.pytorch.org/t/what-does-torch-backends-cudnn-benchmark-do/5936 "2017-08-08T16:21:17Z")\ **Posts on this page:** 20\ **Page:** 2
**Author:** ![Shubhankar](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/b9bd4f/32.png) [@Shubhankar](https://discuss.pytorch.org/u/Shubhankar)\ **Post date:** [January 13, 2020, 6:33pm UTC](https://discuss.pytorch.org/t/what-does-torch-backends-cudnn-benchmark-do/5936/25 "2020-01-13T18:33:39Z")
Is it possible to interact with CUDNN API from Pytorch. Following function returns the type of algorithm to be used defined by `CUDNN_CONVOLUTION_FWD_PREFER_FASTEST`: ```auto cudnnGetConvolutionForwardAlgorithm(cudnn, input_descriptor, kernel_descriptor, convolution_descriptor, output_descriptor, CUDNN_CONVOLUTION_FWD_PREFER_FASTEST, /*memoryLimitInBytes=*/0, &convolution_algorithm) ``` Or is it possible to use CUDNN logs somehow? ---
**Author:** ![albanD](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/32/215_2.png) [@albanD](https://discuss.pytorch.org/u/albanD)\ **Post date:** [January 13, 2020, 7:29pm UTC](https://discuss.pytorch.org/t/what-does-torch-backends-cudnn-benchmark-do/5936/26 "2020-01-13T19:29:25Z")
cc @ptrblck who is more experimented with cudnn 🙂 ---
**Author:** ![Shubhankar](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/b9bd4f/32.png) [@Shubhankar](https://discuss.pytorch.org/u/Shubhankar)\ **Post date:** [January 13, 2020, 7:30pm UTC](https://discuss.pytorch.org/t/what-does-torch-backends-cudnn-benchmark-do/5936/27 "2020-01-13T19:30:06Z")
> [@How to get (print) convolution\_algorithm chosen by CUDNN autotune?](https://discuss.pytorch.org/t/how-to-get-print-convolution-algorithm-chosen-by-cudnn-autotune/66554): > > How to get the convolution algorithm chosen by CUDNN autotune in pytorch and how to manually define it later? Is it possible to interact with CUDNN API from inside Pytorch. Following [CUDNN function](http://www.goldsborough.me/cuda/ml/cudnn/c++/2017/10/01/14-37-23-convolutions_with_cudnn/) returns the type of algorithm to be used defined by CUDNN\_CONVOLUTION\_FWD\_PREFER\_FASTEST: cudnnGetConvolutionForwardAlgorithm(cudnn, input\_descriptor, kernel\_descriptor, convoluti… ---
**Author:** ![Shubhankar](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/b9bd4f/32.png) [@Shubhankar](https://discuss.pytorch.org/u/Shubhankar)\ **Post date:** [January 15, 2020, 3:30pm UTC](https://discuss.pytorch.org/t/what-does-torch-backends-cudnn-benchmark-do/5936/28 "2020-01-15T15:30:28Z")
FYI This issue is still open on [github](https://github.com/pytorch/pytorch/issues/18536) ---
**Author:** ![mohit117](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mohit117/32/13447_2.png) [@mohit117](https://discuss.pytorch.org/u/mohit117)\ **Post date:** [December 29, 2020, 6:12am UTC](https://discuss.pytorch.org/t/what-does-torch-backends-cudnn-benchmark-do/5936/29 "2020-12-29T06:12:38Z")
Should this be enabled:- - During training? - During inference? or - Or both. I care only about inference speed. Thankyou ---
**Author:** ![albanD](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/32/215_2.png) [@albanD](https://discuss.pytorch.org/u/albanD)\ **Post date:** [December 30, 2020, 10:21am UTC](https://discuss.pytorch.org/t/what-does-torch-backends-cudnn-benchmark-do/5936/30 "2020-12-30T10:21:34Z")
If your input always have the same size, you should enable it all the time. But it will influence which algorithm cudnn is using only while the flag is enabled. So setting it during training does not influence inference in any way. ---
**Author:** ![veritas](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/veritas/32/11018_2.png) [@veritas](https://discuss.pytorch.org/u/veritas)\ **Post date:** [January 20, 2021, 12:05pm UTC](https://discuss.pytorch.org/t/what-does-torch-backends-cudnn-benchmark-do/5936/31 "2021-01-20T12:05:22Z")
@fmassa @ptrblck Hello. I would like to ask a few questions about the behavior of `torch.backends.cudnn.benchmark = True`. 1. Does the mini-batch size matter? Many people say that benchmarking uses the same cache if image input size is the same. However, I have not found a clear explanation of whether changing batch size is OK. 2. How many caches can it manage? For example, I might have two types of input: 224x224 and 320x320. Would changing between the two types of images constantly require additional benchmarking or would there be two separate caches? Thank you in advance for your replies! ---
**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\ **Post date:** [January 20, 2021, 11:22pm UTC](https://discuss.pytorch.org/t/what-does-torch-backends-cudnn-benchmark-do/5936/32 "2021-01-20T23:22:21Z")
1. Yes, the batch size matters, as the `ConvolutionParams` will be stored from [here](https://github.com/pytorch/pytorch/blob/24fd84313f010822ceb62166b46b9ca61d902fc9/aten/src/ATen/native/cudnn/ConvShared.h#L18-L32). 2. It’s using a [`std::unordered_map`](https://github.com/pytorch/pytorch/blob/24fd84313f010822ceb62166b46b9ca61d902fc9/aten/src/ATen/native/cudnn/Conv_v7.cpp#L77) with the mentioned `ConvolutionParams`, so no additional benchmarking would be required for these two shapes once they are already profiled. ---
**Author:** ![Michael-H777](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/michael-h777/32/36056_2.png) [@Michael-H777](https://discuss.pytorch.org/u/Michael-H777)\ **Post date:** [March 20, 2021, 8:24pm UTC](https://discuss.pytorch.org/t/what-does-torch-backends-cudnn-benchmark-do/5936/33 "2021-03-20T20:24:18Z")
hello guys, I have a quick question about the `torch.backends.cudnn.benchmark = True` When you say the `input_size` cannot change, does that apply to each convolution layer? I have a UNet design using dense blocks. Since in a block, input for each layer is different, does that mean I cannot use `torch.backends.cudnn.benchmark = True` ? Is there any workaround for dense block so that I can use `torch.backends.cudnn.benchmark = True` ? Thanks in advance 🙂 ---
**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\ **Post date:** [March 21, 2021, 7:34am UTC](https://discuss.pytorch.org/t/what-does-torch-backends-cudnn-benchmark-do/5936/34 "2021-03-21T07:34:55Z")
> [@Michael-H777](#): > > When you say the `input_size` cannot change, does that apply to each convolution layer? The input shape can change, but each new input shape will rerun `cudnnFind` to find the fastest kernel for this shape (for all layers with a new input shape) and will add these kernels to a cache. > [@Michael-H777](#): > > I have a UNet design using dense blocks. Since in a block, input for each layer is different, does that mean I cannot use `torch.backends.cudnn.benchmark = True` ? No, you can use it, but each new input shape will cause a slowdown once. ---
**Author:** ![Martin\_JH](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/martin_jh/32/36569_2.png) [@Martin\_JH](https://discuss.pytorch.org/u/Martin_JH)\ **Post date:** [October 18, 2021, 8:57am UTC](https://discuss.pytorch.org/t/what-does-torch-backends-cudnn-benchmark-do/5936/35 "2021-10-18T08:57:30Z")
How can we use in c++? ---
**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\ **Post date:** [October 18, 2021, 9:01am UTC](https://discuss.pytorch.org/t/what-does-torch-backends-cudnn-benchmark-do/5936/36 "2021-10-18T09:01:52Z")
`at::globalContext().setBenchmarkCuDNN(true);` should work. ---
**Author:** ![divinho](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/9dc877/32.png) [@divinho](https://discuss.pytorch.org/u/divinho)\ **Post date:** [May 26, 2023, 2:46pm UTC](https://discuss.pytorch.org/t/what-does-torch-backends-cudnn-benchmark-do/5936/37 "2023-05-26T14:46:24Z")
what is the default value? ---
**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\ **Post date:** [May 26, 2023, 5:05pm UTC](https://discuss.pytorch.org/t/what-does-torch-backends-cudnn-benchmark-do/5936/38 "2023-05-26T17:05:46Z")
By default `benchmark` is set to `False`. ---
**Author:** ![Apdullah\_Yayik](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/apdullah_yayik/32/61010_2.png) [@Apdullah\_Yayik](https://discuss.pytorch.org/u/Apdullah_Yayik)\ **Post date:** [June 3, 2023, 8:07pm UTC](https://discuss.pytorch.org/t/what-does-torch-backends-cudnn-benchmark-do/5936/39 "2023-06-03T20:07:41Z")
What about using it at CPU, can it still lead a positive effect? ---
**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\ **Post date:** [June 3, 2023, 8:52pm UTC](https://discuss.pytorch.org/t/what-does-torch-backends-cudnn-benchmark-do/5936/40 "2023-06-03T20:52:23Z")
No, `cuDNN` is used on NVIDIA GPUs only. ---
**Author:** ![Paloha](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/ad7895/32.png) [@Paloha](https://discuss.pytorch.org/u/Paloha)\ **Post date:** [February 22, 2024, 12:35am UTC](https://discuss.pytorch.org/t/what-does-torch-backends-cudnn-benchmark-do/5936/41 "2024-02-22T00:35:25Z")
Hello all, I would like to report/mention that I am experiencing out of memory issues when I am already tight on VRAM and then set `torch.backends.cudnn.benchmark = True`. Profiling VRAM usage on smaller data shows that after setting`torch.backends.cudnn.benchmark = True`, there is a spike of VRAM usage at the beginning. After it drops, the overall footprint is still a bit higher than compared to what I measure with `torch.backends.cudnn.benchmark = False`. To put this in numbers, peak VRAM usage is ~7GiB with `False`. Peak with `True` is 19GiB and it eventually settles on 10GiB. The execution time in my case is 25% shorter. Without further digging into this, I assume the increase in VRAM is the price that needs to be paid to get faster execution. Is that correct or could the increased VRAM be avoided somehow? In case not, it might be good to at least document it. Thanks ---
**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\ **Post date:** [February 22, 2024, 1:50am UTC](https://discuss.pytorch.org/t/what-does-torch-backends-cudnn-benchmark-do/5936/42 "2024-02-22T01:50:12Z")
In `benchmark` mode `cuDNN` will profile different algorithms with different memory requirements. If the GPU is running out of memory, the error will be caught [here](https://github.com/pytorch/pytorch/blob/fff9d98e58cda4ad67fef6d0739f684ca7d4bbec/aten/src/ATen/native/cudnn/Conv_v8.cpp#L653), cleared, and the algorithms deselected. If the cuDNN algorithm was already selected and did not cause an OOM, the memory footprint could increase depending on the algorithm, and next operation might run OOM. You can set the max. allowed cuDNN workspace via `CUDNN_CONV_WSCAP_DBG`. ---
**Author:** ![Paloha](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/ad7895/32.png) [@Paloha](https://discuss.pytorch.org/u/Paloha)\ **Post date:** [February 24, 2024, 12:52am UTC](https://discuss.pytorch.org/t/what-does-torch-backends-cudnn-benchmark-do/5936/43 "2024-02-24T00:52:16Z")
Thanks for your reply. I tried setting the env variable `CUDNN_CONV_WSCAP_DBG=38000` (My cudnn version is 8.2.1.32). I am testing on A100-40GB GPU and the [documentation](https://docs.nvidia.com/deeplearning/cudnn/archives/cudnn-831/pdf/cuDNN-Release-Notes.pdf) says the value should be set in MiB. However, I still get OOM. Did you anticipate that limiting the cudnn workspace would help? I figured that doing the benchmarking on smaller input to figure out the best algorithm, and then just directly setting that algorithm without further benchmarking could help avoiding the VRAM usage peak. But I did not find out how to do this even after digging further into this [question](https://discuss.pytorch.org/t/how-to-get-print-convolution-algorithm-chosen-by-cudnn-autotune/66554) and this [git issue](https://github.com/pytorch/pytorch/issues/18536). ---
**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\ **Post date:** [February 24, 2024, 3:40pm UTC](https://discuss.pytorch.org/t/what-does-torch-backends-cudnn-benchmark-do/5936/44 "2024-02-24T15:40:23Z")
> [@Paloha](#): > > Did you anticipate that limiting the cudnn workspace would help? It depends where the OOM was raised. cuDNN itself will not run OOM in its benchmarking as explained before. However, reducing its memory usage could avoid future OOMs but of course depends on the workload. You could set the env variable to 0 and check if you are still running OOM. [Previous page](https://discuss.pytorch.org/t/what-does-torch-backends-cudnn-benchmark-do/5936.md?page=1)