RuntimeError: cublas runtime error : the GPU program failed to execute at /opt/conda/conda-bld/pytorch_1556653183467/work/aten/src/THC/THCBlas.cu:259

Do you know how I can fix this error? I am following all the instructions as shown in this git repo:

full log accessible here https://pastebin.com/raw/TCGxksDt
also https://github.com/pytorch/pytorch/issues/49116

(base) mona@mona:~/research$ git clone https://github.com/yinyunie/Total3DUnderstanding.git
Cloning into 'Total3DUnderstanding'...
remote: Enumerating objects: 206, done.
remote: Counting objects: 100% (206/206), done.
remote: Compressing objects: 100% (181/181), done.
remote: Total 206 (delta 31), reused 192 (delta 20), pack-reused 0
Receiving objects: 100% (206/206), 4.23 MiB | 19.59 MiB/s, done.
Resolving deltas: 100% (31/31), done.
(base) mona@mona:~/research$ cd Total3DUnderstanding/
(basRuntimeError: cublas runtime error : the GPU program failed to execute at /opt/conda/conda-bld/pytorch_1556653183467/work/aten/src/THC/THCBlas.cu:259e) mona@mona:~/research/Total3DUnderstanding$ conda env create -f environment.yml
Collecting package metadata (repodata.json): done
Solving environment: done

Downloading and Extracting Packages
Installing pip dependencies: / Ran pip subprocess with arguments:
['/home/mona/anaconda3/envs/Total3D/bin/python', '-m', 'pip', 'install', '-U', '-r', '/home/mona/research/Total3DUnderstanding/condaenv.usbz06he.requirements.txt']
Pip subprocess output:

done
#
# To activate this environment, use
#
#     $ conda activate Total3D
#
# To deactivate an active environment, use
#
#     $ conda deactivate

(base) mona@mona:~/research/Total3DUnderstanding$ conda activate Total3D
(Total3D) mona@mona:~/research/Total3DUnderstanding$ python main.py configs/total3d.yaml --mode demo --demo_path demo/inputs/1
Loading configurations.
{'method': 'TOTAL3D', 'resume': False, 'finetune': True, 'weight': ['out/pretrained_models/pretrained_model.pth'], 'seed': 123, 'device': {'use_gpu': True, 'gpu_ids': '0', 'num_workers': 2}, 'data': {'dataset': 'sunrgbd', 'split': 'data/sunrgbd/splits', 'tmn_subnetworks': 2, 'face_samples': 1, 'with_edge_classifier': True}, 'model': {'layout_estimation': {'method': 'PoseNet', 'loss': 'PoseLoss'}, 'object_detection': {'method': 'Bdb3DNet', 'loss': 'DetLoss'}, 'mesh_reconstruction': {'method': 'DensTMNet', 'loss': 'ReconLoss'}}, 'optimizer': {'method': 'Adam', 'lr': '1e-4', 'betas': [0.9, 0.999], 'eps': '1e-08', 'weight_decay': '1e-04'}, 'scheduler': {'patience': 5, 'factor': 0.5, 'threshold': 0.01}, 'train': {'epochs': 400, 'phase': 'joint', 'freeze': ['mesh_reconstruction'], 'batch_size': 2}, 'test': {'phase': 'joint', 'batch_size': 2}, 'demo': {'phase': 'joint'}, 'log': {'vis_path': 'out/total3d/2020-12-09T15:00:36.822598/visualization', 'save_results': True, 'vis_step': 100, 'print_step': 50, 'path': 'out/total3d/2020-12-09T15:00:36.822598'}, 'config': 'configs/total3d.yaml', 'mode': 'demo', 'demo_path': 'demo/inputs/1'}
Data save path: out/total3d/2020-12-09T15:00:36.822598
Loading device settings.
GPU mode is on.
GPU Ids: 0 used.
Loading model.
Downloading: "https://download.pytorch.org/models/resnet34-333f7ec4.pth" to /home/mona/.cache/torch/checkpoints/resnet34-333f7ec4.pth
100.0%
Downloading: "https://download.pytorch.org/models/resnet18-5c106cde.pth" to /home/mona/.cache/torch/checkpoints/resnet18-5c106cde.pth
100.0%
Begin to finetune from the existing weight.
Loading checkpoint from out/pretrained_models/pretrained_model.pth.
set() subnet missed.
Weights for finetuning loaded.
----------------------------------------------------------------------------------------------------
Loading data.
Traceback (most recent call last):
  File "main.py", line 38, in <module>
    demo.run(cfg)
  File "/home/mona/research/Total3DUnderstanding/demo.py", line 147, in run
    est_data = net(data)
  File "/home/mona/anaconda3/envs/Total3D/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/mona/research/Total3DUnderstanding/models/total3d/modules/network.py", line 67, in forward
    lo_centroid_result, lo_coeffs_result = self.layout_estimation(data['image'])
  File "/home/mona/anaconda3/envs/Total3D/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/mona/anaconda3/envs/Total3D/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 150, in forward
    return self.module(*inputs[0], **kwargs[0])
  File "/home/mona/anaconda3/envs/Total3D/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/mona/research/Total3DUnderstanding/models/total3d/modules/layout_estimation.py", line 63, in forward
    cam = self.fc_1(x)
  File "/home/mona/anaconda3/envs/Total3D/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/mona/anaconda3/envs/Total3D/lib/python3.6/site-packages/torch/nn/modules/linear.py", line 92, in forward
    return F.linear(input, self.weight, self.bias)
  File "/home/mona/anaconda3/envs/Total3D/lib/python3.6/site-packages/torch/nn/functional.py", line 1406, in linear
    ret = torch.addmm(bias, input, weight.t())
RuntimeError: cublas runtime error : the GPU program failed to execute at /opt/conda/conda-bld/pytorch_1556653183467/work/aten/src/THC/THCBlas.cu:259
(Total3D) mona@mona:~/research/Total3DUnderstanding$ python
Python 3.6.10 |Anaconda, Inc.| (default, May  8 2020, 02:54:21) 
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.cuda.is_available()
True
>>> torch.__version__
'1.1.0'
>>> quit()
(Total3D) mona@mona:~/research/Total3DUnderstanding$ bat environment.yml 
───────┬───────────────────────────────────────────────────────────────────────────────────────────────────────
       │ File: environment.yml
───────┼───────────────────────────────────────────────────────────────────────────────────────────────────────
   1   │ name: Total3D
   2   │ channels:
   3   │   - pytorch
   4   │   - defaults
   5   │ dependencies:
   6   │   - _libgcc_mutex=0.1
   7   │   - blas=1.0
   8   │   - bzip2=1.0.8
   9   │   - ca-certificates=2020.1.1
  10   │   - certifi=2020.6.20
  11   │   - cffi=1.14.0
  12   │   - cudatoolkit=9.0
  13   │   - curl=7.69.1
  14   │   - expat=2.2.6
  15   │   - freetype=2.9.1
  16   │   - future=0.18.2
  17   │   - geos=3.8.0
  18   │   - hdf4=4.2.13
  19   │   - hdf5=1.10.4
  20   │   - icu=58.2
  21   │   - intel-openmp=2020.1
  22   │   - jpeg=9b
  23   │   - jsoncpp=1.8.4
  24   │   - krb5=1.17.1
  25   │   - ld_impl_linux-64=2.33.1
  26   │   - libcurl=7.69.1
  27   │   - libedit=3.1.20181209
  28   │   - libffi=3.3
  29   │   - libgcc-ng=9.1.0
  30   │   - libgfortran-ng=7.3.0
  31   │   - libnetcdf=4.6.1
  32   │   - libogg=1.3.2
  33   │   - libpng=1.6.37
  34   │   - libssh2=1.9.0
  35   │   - libstdcxx-ng=9.1.0
  36   │   - libtheora=1.1.1
  37   │   - libtiff=4.1.0
  38   │   - libvorbis=1.3.6
  39   │   - libxml2=2.9.10
  40   │   - lz4-c=1.8.1.2
  41   │   - mkl=2020.1
  42   │   - mkl-service=2.3.0
  43   │   - mkl_fft=1.0.15
  44   │   - mkl_random=1.1.1
  45   │   - ncurses=6.2
  46   │   - ninja=1.9.0
  47   │   - numpy=1.18.1
  48   │   - numpy-base=1.18.1
  49   │   - olefile=0.46
  50   │   - openssl=1.1.1g
  51   │   - pandas=1.0.5
  52   │   - pillow=7.1.2
  53   │   - pip=20.0.2
  54   │   - pycparser=2.20
  55   │   - python=3.6.10
  56   │   - python-dateutil=2.8.1
  57   │   - pytorch=1.1.0
  58   │   - pytz=2020.1
  59   │   - pyyaml=5.3.1
  60   │   - readline=8.0
  61   │   - scipy=1.4.1
  62   │   - setuptools=47.1.1
  63   │   - shapely=1.7.0
  64   │   - six=1.15.0
  65   │   - sqlite=3.31.1
  66   │   - tbb=2020.0
  67   │   - tk=8.6.8
  68   │   - torchvision=0.3.0
  69   │   - vtk=8.2.0
  70   │   - wheel=0.34.2
  71   │   - xz=5.2.5
  72   │   - yaml=0.1.7
  73   │   - zlib=1.2.11
  74   │   - zstd=1.3.7
  75   │   - pip:
  76   │     - cycler==0.10.0
  77   │     - jellyfish==0.8.2
  78   │     - kiwisolver==1.2.0
  79   │     - matplotlib==3.2.2
  80   │     - opencv-python==4.2.0.34
  81   │     - pyparsing==2.4.7
  82   │     - seaborn==0.10.1
  83   │

I have:

$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Sun_Jul_28_19:07:16_PDT_2019
Cuda compilation tools, release 10.1, V10.1.243



$ nvidia-smi
Wed Dec  9 15:14:27 2020       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.80.02    Driver Version: 450.80.02    CUDA Version: 11.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  GeForce RTX 2070    Off  | 00000000:01:00.0 Off |                  N/A |
| N/A   49C    P8    10W /  N/A |   3121MiB /  7982MiB |     11%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      1364      G   /usr/lib/xorg/Xorg                816MiB |
|    0   N/A  N/A      1797      G   /usr/bin/gnome-shell              516MiB |
|    0   N/A  N/A      3284      G   /usr/lib/firefox/firefox            2MiB |
|    0   N/A  N/A      3506      G   /usr/lib/firefox/firefox            2MiB |
|    0   N/A  N/A      4545      G   /usr/lib/firefox/firefox            2MiB |
|    0   N/A  N/A      7443      G   /usr/lib/firefox/firefox            2MiB |
|    0   N/A  N/A     37638      G   /usr/lib/firefox/firefox            2MiB |
|    0   N/A  N/A     37787      G   /usr/lib/firefox/firefox            2MiB |
|    0   N/A  N/A     69220      G   /usr/lib/firefox/firefox            2MiB |
|    0   N/A  N/A     74559      G   /usr/lib/firefox/firefox            2MiB |
|    0   N/A  N/A     77168      G   ...AAAAAAAAA= --shared-files      136MiB |
|    0   N/A  N/A     77506      C   ...mona/anaconda3/bin/python     1621MiB |
+-----------------------------------------------------------------------------+

Double post from here.

1 Like