RuntimeError: GET was unable to find an engine to execute this computation

I’m currently adapting an existing model to push it on Replicate which implies building a container to push it onto their servers. They use a tool called cog that is basically an abstraction around Docker.

When running the model within the container, I get the errors below.

I tried to run the container elsewhere and also on Replicate and I get the same errors so clearly something is wrong with my container. I think it might be because of a wrong version of PyTorch or another package.

Could anybody help figure this out please? Thanks!

root@e9a4ecf8be50:/src# python --config configs/text.yaml prompt="a photo of an icecream" save_path=icecream
Number of points at initialisation :  5000
[INFO] loading SD...
Loading pipeline components...: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 6/6 [00:01<00:00,  3.31it/s]
[INFO] loaded SD!
  0%|                                                                                                                                                           | 0/500 [00:00<?, ?it/s]Could not load library Error: /usr/lib/x86_64-linux-gnu/ undefined symbol: _ZTIN10cask_cudnn14BaseKernelInfoE, version
Could not load library Error: /usr/lib/x86_64-linux-gnu/ undefined symbol: _ZTIN10cask_cudnn14BaseKernelInfoE, version
Could not load library Error: /usr/lib/x86_64-linux-gnu/ undefined symbol: _ZTIN10cask_cudnn14BaseKernelInfoE, version
Could not load library Error: /usr/lib/x86_64-linux-gnu/ undefined symbol: _ZTIN10cask_cudnn14BaseKernelInfoE, version
  0%|                                                                                                                                                           | 0/500 [00:01<?, ?it/s]
Traceback (most recent call last):
  File "/src/", line 902, in <module>
  File "/src/", line 878, in train
  File "/src/", line 258, in train_step
  File "/root/.pyenv/versions/3.11.7/lib/python3.11/site-packages/torch/", line 492, in backward
  File "/root/.pyenv/versions/3.11.7/lib/python3.11/site-packages/torch/autograd/", line 251, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: GET was unable to find an engine to execute this computation

Here’s some debug info about my current configuration:

docker --version

Docker version 24.0.7, build afdd53b

docker container: nvcc --version

root@e9a4ecf8be50:/src# nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Fri_Jan__6_16:45:21_PST_2023
Cuda compilation tools, release 12.0, V12.0.140
Build cuda_12.0.r12.0/compiler.32267302_0

docker container: nvidia-smi

root@e9a4ecf8be50:/src# nvidia-smi
Sun Dec 10 19:59:51 2023
| NVIDIA-SMI 525.147.05   Driver Version: 525.147.05   CUDA Version: 12.0     |
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|   0  Tesla T4            On   | 00000000:00:1E.0 Off |                    0 |
| N/A   26C    P8    14W /  70W |      0MiB / 15360MiB |      0%      Default |
|                               |                      |                  N/A |

docker container: pytorch version

>>> import torch
>>> torch.__version__


Distributor ID:	Ubuntu
Description:	Ubuntu 22.04.3 LTS
Release:	22.04
Codename:	jammy

cog.yaml (contains packages, python version, cuda version, etc.)

  gpu: true
  cuda: "12.1"
    - "libgl1-mesa-glx"
    - "libegl1-mesa-dev"
  python_version: "3.10"
    - "tqdm"
    - "rich"
    - "ninja"
    - "numpy"
    - "pandas"
    - "scipy"
    - "scikit-learn"
    - "matplotlib"
    - "opencv-python"
    - "imageio"
    - "imageio-ffmpeg"
    - "omegaconf"
    - "torch==2.1.0"
    - "einops"
    - "plyfile"
    - "pygltflib"
    - "dearpygui"
    - "huggingface_hub"
    - "diffusers"
    - "accelerate"
    - "transformers"
    - "xatlas"
    - "trimesh"
    - "PyMCubes"
    - "pymeshlab"
    - "rembg[gpu,cli]"
    - "git clone --recursive"
    - "pip install ./diff-gaussian-rasterization"
    - "pip install git+"
    - "pip install git+"
    - "pip install git+"
    - "pip install git+"
    - "echo 'READY.'"

It seems your cuDNN installation is broken or you are trying to mix different cuDNN versions. If you are using the PyTorch binaries, you could uninstall your locally installed cuDNN package or remove it from the library path so it won’t be loaded.

1 Like

It seems like cuDNN is provided by the cog utility provided by Replicate. I just found out that they had issue with that in the past β†’ RuntimeError: cuDNN version incompatibility Β· Issue #815 Β· replicate/cog Β· GitHub

I don’t really know how I could install a different version of cuDNN since it seems to be part of the base image they provide but I’ll look into it. At least this gives me a good clue to follow. Thanks!

You don’t need to install CUDA or cuDNN locally if you are using the PyTorch binaries, as they already ship with these dependencies. Only a properly installed NVIDIA driver is needed.

1 Like

But what if I’m running PyTorch from inside a Docker container? I assume, the version of PyTorch should be compatible with the version of the drivers installed on the host machine. I double checked all versions and made sure they were compatible in between each others. I’m probably missing out on something :face_with_monocle:

So basically it means that my version of PyTorch is wrong and I’d need a version that’s compatible with the drivers only?

I looked up every compatibility matrices I could find and it seems like it should all be compatible together? (You can see all the versions in my OP)

Thanks for your help. Greatly appreciated!

That’s right. You would need to use a properly installed NVIDIA driver, but don’t need a locally installed CUDA toolkit or cuDNN, since these are shipped as dependencies in the PyTorch binaries.
Your locally installed CUDA toolkit (including cuDNN) would be used if you build PyTorch from source or custom CUDA extensions.

No, it means that your locally installed CUDA toolkit and cuDNN conflicts with the PyTorch binaries. Either rebuild PyTorch from source if you explicitly want to use this CUDA toolkit and cuDNN, or delete them (or remove them from the LD_LIBRARY_PATH).


It seems you are right! I removed cuDNN from the container (apt-get remove libcudnn8) and the error disappeared :partying_face: so it was a conflict between what’s on the container and what’s bundled inside PyTorch!

Now my model almost runs entirely but I encountered the following error which I’m researching about at the moment.

[F glutil.cpp:338] eglInitialize() failed
Aborted (core dumped)

Different story though. I’m glad to finally understand what was happening in my OP.

Thanks so much for your help, you’re a real life saver!

1 Like

Good to hear it’s working now!
If I’m not mistaken the new error is related to OpenGL, so unsure if you are trying to visualize something inside your container, but visualizations might not be supported.

1 Like