Do you know how I can fix this error? I am following all the instructions as shown in this git repo:
full log accessible here https://pastebin.com/raw/TCGxksDt
also https://github.com/pytorch/pytorch/issues/49116
(base) mona@mona:~/research$ git clone https://github.com/yinyunie/Total3DUnderstanding.git
Cloning into 'Total3DUnderstanding'...
remote: Enumerating objects: 206, done.
remote: Counting objects: 100% (206/206), done.
remote: Compressing objects: 100% (181/181), done.
remote: Total 206 (delta 31), reused 192 (delta 20), pack-reused 0
Receiving objects: 100% (206/206), 4.23 MiB | 19.59 MiB/s, done.
Resolving deltas: 100% (31/31), done.
(base) mona@mona:~/research$ cd Total3DUnderstanding/
(basRuntimeError: cublas runtime error : the GPU program failed to execute at /opt/conda/conda-bld/pytorch_1556653183467/work/aten/src/THC/THCBlas.cu:259e) mona@mona:~/research/Total3DUnderstanding$ conda env create -f environment.yml
Collecting package metadata (repodata.json): done
Solving environment: done
Downloading and Extracting Packages
Installing pip dependencies: / Ran pip subprocess with arguments:
['/home/mona/anaconda3/envs/Total3D/bin/python', '-m', 'pip', 'install', '-U', '-r', '/home/mona/research/Total3DUnderstanding/condaenv.usbz06he.requirements.txt']
Pip subprocess output:
done
#
# To activate this environment, use
#
# $ conda activate Total3D
#
# To deactivate an active environment, use
#
# $ conda deactivate
(base) mona@mona:~/research/Total3DUnderstanding$ conda activate Total3D
(Total3D) mona@mona:~/research/Total3DUnderstanding$ python main.py configs/total3d.yaml --mode demo --demo_path demo/inputs/1
Loading configurations.
{'method': 'TOTAL3D', 'resume': False, 'finetune': True, 'weight': ['out/pretrained_models/pretrained_model.pth'], 'seed': 123, 'device': {'use_gpu': True, 'gpu_ids': '0', 'num_workers': 2}, 'data': {'dataset': 'sunrgbd', 'split': 'data/sunrgbd/splits', 'tmn_subnetworks': 2, 'face_samples': 1, 'with_edge_classifier': True}, 'model': {'layout_estimation': {'method': 'PoseNet', 'loss': 'PoseLoss'}, 'object_detection': {'method': 'Bdb3DNet', 'loss': 'DetLoss'}, 'mesh_reconstruction': {'method': 'DensTMNet', 'loss': 'ReconLoss'}}, 'optimizer': {'method': 'Adam', 'lr': '1e-4', 'betas': [0.9, 0.999], 'eps': '1e-08', 'weight_decay': '1e-04'}, 'scheduler': {'patience': 5, 'factor': 0.5, 'threshold': 0.01}, 'train': {'epochs': 400, 'phase': 'joint', 'freeze': ['mesh_reconstruction'], 'batch_size': 2}, 'test': {'phase': 'joint', 'batch_size': 2}, 'demo': {'phase': 'joint'}, 'log': {'vis_path': 'out/total3d/2020-12-09T15:00:36.822598/visualization', 'save_results': True, 'vis_step': 100, 'print_step': 50, 'path': 'out/total3d/2020-12-09T15:00:36.822598'}, 'config': 'configs/total3d.yaml', 'mode': 'demo', 'demo_path': 'demo/inputs/1'}
Data save path: out/total3d/2020-12-09T15:00:36.822598
Loading device settings.
GPU mode is on.
GPU Ids: 0 used.
Loading model.
Downloading: "https://download.pytorch.org/models/resnet34-333f7ec4.pth" to /home/mona/.cache/torch/checkpoints/resnet34-333f7ec4.pth
100.0%
Downloading: "https://download.pytorch.org/models/resnet18-5c106cde.pth" to /home/mona/.cache/torch/checkpoints/resnet18-5c106cde.pth
100.0%
Begin to finetune from the existing weight.
Loading checkpoint from out/pretrained_models/pretrained_model.pth.
set() subnet missed.
Weights for finetuning loaded.
----------------------------------------------------------------------------------------------------
Loading data.
Traceback (most recent call last):
File "main.py", line 38, in <module>
demo.run(cfg)
File "/home/mona/research/Total3DUnderstanding/demo.py", line 147, in run
est_data = net(data)
File "/home/mona/anaconda3/envs/Total3D/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
result = self.forward(*input, **kwargs)
File "/home/mona/research/Total3DUnderstanding/models/total3d/modules/network.py", line 67, in forward
lo_centroid_result, lo_coeffs_result = self.layout_estimation(data['image'])
File "/home/mona/anaconda3/envs/Total3D/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
result = self.forward(*input, **kwargs)
File "/home/mona/anaconda3/envs/Total3D/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 150, in forward
return self.module(*inputs[0], **kwargs[0])
File "/home/mona/anaconda3/envs/Total3D/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
result = self.forward(*input, **kwargs)
File "/home/mona/research/Total3DUnderstanding/models/total3d/modules/layout_estimation.py", line 63, in forward
cam = self.fc_1(x)
File "/home/mona/anaconda3/envs/Total3D/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
result = self.forward(*input, **kwargs)
File "/home/mona/anaconda3/envs/Total3D/lib/python3.6/site-packages/torch/nn/modules/linear.py", line 92, in forward
return F.linear(input, self.weight, self.bias)
File "/home/mona/anaconda3/envs/Total3D/lib/python3.6/site-packages/torch/nn/functional.py", line 1406, in linear
ret = torch.addmm(bias, input, weight.t())
RuntimeError: cublas runtime error : the GPU program failed to execute at /opt/conda/conda-bld/pytorch_1556653183467/work/aten/src/THC/THCBlas.cu:259
(Total3D) mona@mona:~/research/Total3DUnderstanding$ python
Python 3.6.10 |Anaconda, Inc.| (default, May 8 2020, 02:54:21)
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.cuda.is_available()
True
>>> torch.__version__
'1.1.0'
>>> quit()
(Total3D) mona@mona:~/research/Total3DUnderstanding$ bat environment.yml
───────┬───────────────────────────────────────────────────────────────────────────────────────────────────────
│ File: environment.yml
───────┼───────────────────────────────────────────────────────────────────────────────────────────────────────
1 │ name: Total3D
2 │ channels:
3 │ - pytorch
4 │ - defaults
5 │ dependencies:
6 │ - _libgcc_mutex=0.1
7 │ - blas=1.0
8 │ - bzip2=1.0.8
9 │ - ca-certificates=2020.1.1
10 │ - certifi=2020.6.20
11 │ - cffi=1.14.0
12 │ - cudatoolkit=9.0
13 │ - curl=7.69.1
14 │ - expat=2.2.6
15 │ - freetype=2.9.1
16 │ - future=0.18.2
17 │ - geos=3.8.0
18 │ - hdf4=4.2.13
19 │ - hdf5=1.10.4
20 │ - icu=58.2
21 │ - intel-openmp=2020.1
22 │ - jpeg=9b
23 │ - jsoncpp=1.8.4
24 │ - krb5=1.17.1
25 │ - ld_impl_linux-64=2.33.1
26 │ - libcurl=7.69.1
27 │ - libedit=3.1.20181209
28 │ - libffi=3.3
29 │ - libgcc-ng=9.1.0
30 │ - libgfortran-ng=7.3.0
31 │ - libnetcdf=4.6.1
32 │ - libogg=1.3.2
33 │ - libpng=1.6.37
34 │ - libssh2=1.9.0
35 │ - libstdcxx-ng=9.1.0
36 │ - libtheora=1.1.1
37 │ - libtiff=4.1.0
38 │ - libvorbis=1.3.6
39 │ - libxml2=2.9.10
40 │ - lz4-c=1.8.1.2
41 │ - mkl=2020.1
42 │ - mkl-service=2.3.0
43 │ - mkl_fft=1.0.15
44 │ - mkl_random=1.1.1
45 │ - ncurses=6.2
46 │ - ninja=1.9.0
47 │ - numpy=1.18.1
48 │ - numpy-base=1.18.1
49 │ - olefile=0.46
50 │ - openssl=1.1.1g
51 │ - pandas=1.0.5
52 │ - pillow=7.1.2
53 │ - pip=20.0.2
54 │ - pycparser=2.20
55 │ - python=3.6.10
56 │ - python-dateutil=2.8.1
57 │ - pytorch=1.1.0
58 │ - pytz=2020.1
59 │ - pyyaml=5.3.1
60 │ - readline=8.0
61 │ - scipy=1.4.1
62 │ - setuptools=47.1.1
63 │ - shapely=1.7.0
64 │ - six=1.15.0
65 │ - sqlite=3.31.1
66 │ - tbb=2020.0
67 │ - tk=8.6.8
68 │ - torchvision=0.3.0
69 │ - vtk=8.2.0
70 │ - wheel=0.34.2
71 │ - xz=5.2.5
72 │ - yaml=0.1.7
73 │ - zlib=1.2.11
74 │ - zstd=1.3.7
75 │ - pip:
76 │ - cycler==0.10.0
77 │ - jellyfish==0.8.2
78 │ - kiwisolver==1.2.0
79 │ - matplotlib==3.2.2
80 │ - opencv-python==4.2.0.34
81 │ - pyparsing==2.4.7
82 │ - seaborn==0.10.1
83 │
I have:
$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Sun_Jul_28_19:07:16_PDT_2019
Cuda compilation tools, release 10.1, V10.1.243
$ nvidia-smi
Wed Dec 9 15:14:27 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.80.02 Driver Version: 450.80.02 CUDA Version: 11.0 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 GeForce RTX 2070 Off | 00000000:01:00.0 Off | N/A |
| N/A 49C P8 10W / N/A | 3121MiB / 7982MiB | 11% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 1364 G /usr/lib/xorg/Xorg 816MiB |
| 0 N/A N/A 1797 G /usr/bin/gnome-shell 516MiB |
| 0 N/A N/A 3284 G /usr/lib/firefox/firefox 2MiB |
| 0 N/A N/A 3506 G /usr/lib/firefox/firefox 2MiB |
| 0 N/A N/A 4545 G /usr/lib/firefox/firefox 2MiB |
| 0 N/A N/A 7443 G /usr/lib/firefox/firefox 2MiB |
| 0 N/A N/A 37638 G /usr/lib/firefox/firefox 2MiB |
| 0 N/A N/A 37787 G /usr/lib/firefox/firefox 2MiB |
| 0 N/A N/A 69220 G /usr/lib/firefox/firefox 2MiB |
| 0 N/A N/A 74559 G /usr/lib/firefox/firefox 2MiB |
| 0 N/A N/A 77168 G ...AAAAAAAAA= --shared-files 136MiB |
| 0 N/A N/A 77506 C ...mona/anaconda3/bin/python 1621MiB |
+-----------------------------------------------------------------------------+