# Enabling cuDNN slows down training process

**URL:** <https://discuss.pytorch.org/t/enabling-cudnn-slows-down-training-process/151266>\
**Category:** Uncategorized\
**Created:** [May 10, 2022, 5:18am UTC](https://discuss.pytorch.org/t/enabling-cudnn-slows-down-training-process/151266 "2022-05-10T05:18:35Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![zachnguyen03](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zachnguyen03/32/47312_2.png) [@zachnguyen03](https://discuss.pytorch.org/u/zachnguyen03)\
**Post date:** [May 10, 2022, 5:18am UTC](https://discuss.pytorch.org/t/enabling-cudnn-slows-down-training-process/151266/1 "2022-05-10T05:18:35Z")

</div>

Hi! I am trying to train a model using Pytorch and I found that enabling cuDNN would speed up the training process. However, training got significantly slowed down instead. Can anyone help me with this issue? I performed some minor tests and got this result:

```auto
Enable Cudnn
Conv: 0.5449938774108887
Conv backward: 0.1148219108581543
---------------------------
Disable Cudnn
Conv: 0.01213216781616211
Conv backward: 0.13735365867614746
---------------------------

```

Here is the code snippet that I used for testing:

```python
import torch
import time
def run():
    in_c = 10
    out_c = 15
    kernel = 3
    padding = 1
    inp = torch.rand(512, in_c, 128, 128, requires_grad=True).cuda()
    conv = torch.nn.Conv2d(in_c, out_c, kernel, padding=padding, bias=False).cuda()
    torch.cuda.synchronize()
    start = time.time()
    out = conv(inp)
    torch.cuda.synchronize()
    print("Conv: ", time.time() - start)
    start = time.time()
    out.sum().backward()
    torch.cuda.synchronize()
    print("Conv backward: ", time.time() - start)
    print('---------------------------')
if __name__ ==' __main__':
    torch.backends.cudnn.enabled=True
    print('Enable Cudnn')
    run()
    torch.backends.cudnn.enabled=False
    print('Disable Cudnn')
    run()

```

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [May 17, 2022, 9:01pm UTC](https://discuss.pytorch.org/t/enabling-cudnn-slows-down-training-process/151266/2 "2022-05-17T21:01:05Z")

</div>

I cannot reproduce the issue using warmup iterations and a mean over multiple profiling iterations.  
I would also recommend to use `torch.utils.benchmark` to profile CUDA workloads, as it would add warmup iterations, add synchronizations, and execute the workload until a time threshold is met.

Results in a 3090 using the latest nightly:

```python
Enable Cudnn
Conv: 0.35583066940307617
Conv backward: 14.73327088356018
---------------------------
Disable Cudnn
Conv: 1.5492074489593506
Conv backward: 16.762986421585083
---------------------------

```