cuDNN version incompatibility

vision
1

Hie, I built a simple CNN architecture when I try to run it, I get the error
RuntimeError: GET was unable to find an engine to execute this computation

Then I tried to find cudnn version using torch.backends.cudnn.version() which throwed me another error
RuntimeError: cuDNN version incompatibility: PyTorch was compiled against (8, 5, 0) but found runtime version (8, 1, 0). PyTorch already comes bundled with cuDNN. One option to resolving this error is to ensure PyTorch can find the bundled cuDNN.Looks like your LD_LIBRARY_PATH contains incompatible version of cudnnPlease either remove it from the path or install cudnn (8, 5, 0).

But conda says my cudnn version is 8.5.0.96

image
image833×107 4.6 KB

A simple Linear Model works fine in GPU, but the CNN doesn’t. Below is the simple CNN I tried which throws me the error.

layer =nn.Conv2d(in_channels=1,
                  out_channels=10,
                  kernel_size=3,
                  stride=1,
                  padding=1).to(device)

rand_image_tensor = torch.randn(size=(1, 1, 28, 28)).to(device)

layer(rand_image_tensor).shape

PyTorch Version - 2.0.1+cu117
OS - WSL2
Ubuntu Version in WSL2:
image

2

As the error message explains an older locally installed cuDNN version is found and loaded, so you could:

  • uninstall the locally installed cuDNN version
  • remove it from the LD_LIBRARY_PATH
  • or preload the cuDNN version installed from PyPI.
1 Like
3

I am not sure, but for WSL it was not recommended to install cudnn inside WSL so I had to install it in Windows(I’m using Windows 11). So I really don’t understand what you mean in the first point, if you could please explain the 2nd and 3rd point.

I am inside the .bashrc file, I see 3 lines
export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH export LD_LIBRARY_PATH=/usr/local/cuda/include:$LD_LIBRARY_PATH export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda/extras/CUPTI/lib64
are these the lines you’re talking about, if so, what should I do?

4

Hie, @ptrblck

If I remove the line,

export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH export

from .bashrc file, the moment I run the Conv2d code the Jupyter kernel crashes.

5

I simply uninstalled Cuda and re-installed it, that fixed it.