When I try to train FasterRCNN in newly released torchvision 0.3, I run into a PTX JIT compilation failed error.
Traceback (most recent call last):
File "train_rcnn.py", line 124, in <module>
loss_dict = model(images, targets)
File "/opt/anaconda3/envs/nuscenes/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in __call__
result = self.forward(*input, **kwargs)
File "/home/danielkang92/nuscenes/code/FasterRCNN/model.py", line 19, in forward
return self.rcnn(images, targets)
File "/opt/anaconda3/envs/nuscenes/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in __call__
result = self.forward(*input, **kwargs)
File "/opt/anaconda3/envs/nuscenes/lib/python3.7/site-packages/torchvision/models/detection/generalized_rcnn.py", line 48, in forward
features = self.backbone(images.tensors)
File "/opt/anaconda3/envs/nuscenes/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in __call__
result = self.forward(*input, **kwargs)
File "/opt/anaconda3/envs/nuscenes/lib/python3.7/site-packages/torch/nn/modules/container.py", line 92, in forward
input = module(input)
File "/opt/anaconda3/envs/nuscenes/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in __call__
result = self.forward(*input, **kwargs)
File "/opt/anaconda3/envs/nuscenes/lib/python3.7/site-packages/torchvision/models/_utils.py", line 58, in forward
x = module(x)
File "/opt/anaconda3/envs/nuscenes/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in __call__
result = self.forward(*input, **kwargs)
RuntimeError: /pytorch/torch/csrc/jit/fuser/cuda/fused_kernel.cpp:202: a PTX JIT compilation failed
My environment is as follows :
Collecting environment information...
PyTorch version: 1.1.0
Is debug build: No
CUDA used to build PyTorch: 9.0.176
OS: Debian GNU/Linux 9.8 (stretch)
GCC version: (Debian 6.3.0-18+deb9u1) 6.3.0 20170516
CMake version: Could not collect
Python version: 3.7
Is CUDA available: Yes
CUDA runtime version: 10.0.130
GPU models and configuration: GPU 0: Tesla P100-PCIE-16GB
Nvidia driver version: 410.72
cuDNN version: Could not collect
Versions of relevant libraries:
[pip3] intel-numpy==1.15.1
[pip3] numpy==1.16.3
[pip3] torch==1.1.0
[pip3] torchvision==0.3.0
[conda] torch 1.1.0 pypi_0 pypi
[conda] torchvision 0.3.0 pypi_0 pypi
Would appreciate any help on this!