# CUDA Error in Docker Container

**URL:** <https://discuss.pytorch.org/t/cuda-error-in-docker-container/195909>\
**Category:** torch.package / torch::deploy\
**Created:** [January 22, 2024, 8:34pm UTC](https://discuss.pytorch.org/t/cuda-error-in-docker-container/195909 "2024-01-22T20:34:35Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Tanay\_Baswa](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/t/91b2a8/32.png) [@Tanay\_Baswa](https://discuss.pytorch.org/u/Tanay_Baswa)\
**Post date:** [January 22, 2024, 8:34pm UTC](https://discuss.pytorch.org/t/cuda-error-in-docker-container/195909/1 "2024-01-22T20:34:35Z")

</div>

test\_cuda.py:

```auto
import torch

def check_cuda():
    print("Is CUDA available in PyTorch:", torch.cuda.is_available())
    if torch.cuda.is_available():
        print("Number of CUDA devices:", torch.cuda.device_count())
        for i in range(torch.cuda.device_count()):
            print("CUDA Device #{}: {}".format(i, torch.cuda.get_device_name(i)))

if __name__ == " __main__":
    check_cuda()

```

Dockerfile:

```auto
FROM python:3.8

RUN pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118

COPY test_cuda.py /test_cuda.py

CMD ["python3", "/test_cuda.py"]

```

Machine: AzureStandard NCC24ads A100 v4 (24 vcpus, 220 GiB memory)  
OS: Ubuntu 20.04  
nvidia-smi:

```auto
Mon Jan 22 20:30:30 2024
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.10.12 Driver Version: 470.10.12 CUDA Version: 11.4 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA A100-PCI... On | 00000001:00:00.0 Off | 0 |
| N/A 28C P0 42W / 300W | 10MiB / 81251MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| No running processes found |
+-----------------------------------------------------------------------------+

```

If I start a python shell and run torch.cuda.is\_available(), I receive True.  
If I run “sudo docker run --rm --gpus all ubuntu nvidia-smi”, I get the expected output.

When I try to run a tensorflow image, it detects and uses the GPU.

But when I try to run my test dockerfile with torch, I receive this error:

```auto
/usr/local/lib/python3.8/site-packages/torch/cuda/init.py:138: UserWarning: CUDA initialization: 
Unexpected error from cudaGetDeviceCount(). 
Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? 
Error 801: operation not supported (Triggered internally at ../c10/cuda/CUDAFunctions.cpp:108.)
return torch._C._cuda_getDeviceCount() > 0

```

I have tried to change driver versions, install fabric manager, and dcgm, all to no avail. It seems to be specifically when running torch in a container.

I think this is different to other issues, which have different Error codes, like 803, or 804.

Any help, including general info on running pytorch in a container, would be appreciated.

Thanks!

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [January 22, 2024, 9:04pm UTC](https://discuss.pytorch.org/t/cuda-error-in-docker-container/195909/2 "2024-01-22T21:04:17Z")

</div>

Are you able to run any other PyTorch docker container from [docker.hub](https://hub.docker.com/r/pytorch/pytorch/tags) or [NGC](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorch)?

---

<div class="post-metadata">

**Author:** ![Tanay\_Baswa](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/t/91b2a8/32.png) [@Tanay\_Baswa](https://discuss.pytorch.org/u/Tanay_Baswa)\
**Post date:** [January 22, 2024, 9:17pm UTC](https://discuss.pytorch.org/t/cuda-error-in-docker-container/195909/3 "2024-01-22T21:17:47Z")

</div>

Running the example container:

> **[PyTorch | NVIDIA NGC](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorch)**
>
> PyTorch is a GPU accelerated tensor computational framework. Functionality can be extended with common Python libraries such as NumPy and SciPy. Automatic differentiation is done with a tape-based system at the functional and neural network layer...

Gives me the same error:

ERROR: The NVIDIA Driver is present, but CUDA failed to initialize. GPU functionality will not be available.  
[[Operation not supported (error 801)]]