RuntimeError: CUDNN_STATUS_ARCH_MISMATCH. two k20 use different

Traceback (most recent call last):
File “05.py”, line 123, in
outputs = resnet(images)
File “/home/amax/anaconda2/lib/python2.7/site-packages/torch/nn/modules/module.py”, line 224, in call
result = self.forward(*input, **kwargs)
File “05.py”, line 90, in forward
out = self.conv(x)
File “/home/amax/anaconda2/lib/python2.7/site-packages/torch/nn/modules/module.py”, line 224, in call
result = self.forward(*input, **kwargs)
File “/home/amax/anaconda2/lib/python2.7/site-packages/torch/nn/modules/conv.py”, line 254, in forward
self.padding, self.dilation, self.groups)
File “/home/amax/anaconda2/lib/python2.7/site-packages/torch/nn/functional.py”, line 52, in conv2d
return f(input, weight, bias)
RuntimeError: CUDNN_STATUS_ARCH_MISMATCH

±----------------------------------------------------------------------------+
| NVIDIA-SMI 375.39 Driver Version: 375.39 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla K20c Off | 0000:08:00.0 Off | Off |
| 32% 44C P8 16W / 225W | 0MiB / 5061MiB | 0% Default |
±------------------------------±---------------------±---------------------+
| 1 Quadro 600 Off | 0000:81:00.0 On | N/A |
| 30% 44C P12 N/A / N/A | 465MiB / 964MiB | 2% Default |
±------------------------------±---------------------±---------------------+
| 2 Tesla K20c Off | 0000:82:00.0 Off | Off |
| 30% 33C P8 15W / 225W | 0MiB / 5061MiB | 0% Default |
±------------------------------±---------------------±---------------------+

±----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| No running processes found |
±----------------------------------------------------------------------------+

when I use GPU 0, it’s alright. But use GPU 2 make the mistake above. I don’t know why?? My system is Ubuntu 14.04

it’s the quadro 600 tghat’s the problem.

Try this:

CUDA_VISIBLE_DEVICES="0,2" python [your script.py]

@smth,

I have same problem. You meant problem comes form Quadro Series? I have Quadro 2000.

I already trained and tested my model on Quadro P5000 and just want to run inference on new machine with Quadro 2000.

Thank you

Quadro series in general is not a problem. Quadro 600 has compute capability that is too low for PyTorch to run. Quadro P2000 or M2000 should be fine, but Quadro K2000 is too old.

1 Like

Thank you :slight_smile: