NotImplementedError: Could not run 'torchvision::nms' with arguments from the 'CUDA' backend. This could be because the operator doesn't exist for this backend,

Hi, when I try to run Yolov5 I got this error:

NotImplementedError: Could not run ‘torchvision::nms’ with arguments from the ‘CUDA’ backend. This could be because the operator doesn’t exist for this backend, or was omitted during the selective/custom build process (if using custom build). If you are a Facebook employee using PyTorch on mobile, please visit Internal Login for possible resolutions. ‘torchvision::nms’ is only available for these backends: [CPU, QuantizedCPU, BackendSelect, Named, ADInplaceOrView, AutogradOther, AutogradCPU, AutogradCUDA, AutogradXLA, UNKNOWN_TENSOR_TYPE_ID, AutogradMLC, Tracer, Autocast, Batched, VmapMode].

Is there any solution for that? I tried various of Pytorch-Cuda version combinations like 1.9.0-10.2 , 1.9.0-11.1 , 1.7-10.2 etc. but none of them worked?

I guess you might not have installed torchvision with the CUDA runtime? If so, try to install the latest torchvision release using the same CUDA runtime as used in your PyTorch installation.

pip3 install torch==1.9.0+cu102 torchvision==0.10.0+cu102 torchaudio===0.9.0 -f https://download.pytorch.org/whl/torch_stable.html I used command like this, I think I installed it with cuda.

1 Like

It works on my setup using the torchvision binaries with the CUDA runtime and fails with the CPU-only package:

import torch
import torchvision

device = 'cuda'
boxes = torch.tensor([[0., 1., 2., 3.]]).to(device)
scores = torch.randn(1).to(device)
iou_thresholds = 0.5

print(torchvision.ops.nms(boxes, scores, iou_thresholds))

With CUDA:

pip install torchvision==0.10.0+cu111 -f https://download.pytorch.org/whl/torch_stable.html

python tmp.py 
>tensor([0], device='cuda:0')

Without CUDA:

pip uninstall torchvision
pip install torchvision==0.10.0+cpu -f https://download.pytorch.org/whl/torch_stable.html

python tmp.py
Traceback (most recent call last):
  File "tmp.py", line 10, in <module>
    print(torchvision.ops.nms(boxes, scores, iou_thresholds))
  File "/opt/conda/envs/tmp/lib/python3.8/site-packages/torchvision/ops/boxes.py", line 35, in nms
    return torch.ops.torchvision.nms(boxes, scores, iou_threshold)
NotImplementedError: Could not run 'torchvision::nms' with arguments from the 'CUDA' backend. This could be because the operator doesn't exist for this backend, or was omitted during the selective/custom build process (if using custom build). If you are a Facebook employee using PyTorch on mobile, please visit https://fburl.com/ptmfixes for possible resolutions. 'torchvision::nms' is only available for these backends: [CPU, QuantizedCPU, BackendSelect, Named, ADInplaceOrView, AutogradOther, AutogradCPU, AutogradCUDA, AutogradXLA, UNKNOWN_TENSOR_TYPE_ID, AutogradMLC, Tracer, Autocast, Batched, VmapMode].
2 Likes

When I changed the IDE, it worked. Probably I have more than one enviremont that I was not aware of it. Thank you for answering

Hi @ptrblck,

Does the standard command on the website

conda install pytorch torchvision torchaudio cudatoolkit=10.2 -c pytorch

install torchvision with CUDA? Or do we need to do it explicitly as shown (+cu102) to avoid this error message?

This command will install torchvision with CUDA support as seen during the installation:

...
The following packages will be downloaded:
...
torchvision-0.11.3         |       py38_cu102        30.1 MB  pytorch
1 Like

Hi @ptrblck
If I use ResNet50 everything works, but if I switch to fasterrsnn_resnet50_fpn (Works if I run on CPU). It produces the CUDA error (same as author of the post has).
What should I do?
Thank you in advance.

See this post: you are most likely using the CPU-only torchvision binary.

No, it was not the case, I solved the problem by installing torchaudio and earlier cuda version.