kmkurn
April 7, 2018, 3:44am
1
Hi,
Sorry if this shouldn’t belong here in the forum, but I have this error
from torch._C import *
E ImportError: libcudart.so.8.0: cannot open shared object file: No such file or directory
when running tests with Pytorch inside Travis CI. Here’s my .travis.yml file:
language: python
before_install:
- wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda.sh
- bash ~/miniconda.sh -b -p $HOME/miniconda
- export PATH="$HOME/miniconda/bin:$PATH"
- conda create -y -n some_env python=3.6
- source activate some_env
- conda install -y -c pytorch pytorch=0.3.0
install:
- pip install -r requirements.txt
- pip install .
script: pytest --cov
after_success: coveralls
The build fails when executing pytest --cov. This was fine ~3 months ago. I have tried putting unset LD_LIBRARY_PATH (as suggested here ) at the last step of before_install but it doesn’t work.
smth
April 7, 2018, 3:16pm
2
try adding conda library path to LD_LIBRARY_PATH:
export LD_LIBRARY_PATH="$HOME/miniconda/lib:$LD_LIBRARY_PATH"
That’s where libcudart.so.8.0 should be located.
kmkurn
April 7, 2018, 3:49pm
3
Hi, thank you for replying.
I tried that (after export PATH) but the error is still there. It seems there’s no libcudart.so.8.0 in $HOME/miniconda/lib. Here’s the complete list of files:
drwxr-xr-x 2 travis travis 236 Apr 7 15:42 engines
drwxr-xr-x 2 travis travis 158 Apr 7 15:42 itcl4.1.0
lrwxrwxrwx 1 travis travis 16 Apr 7 15:42 libasan.so -> libasan.so.4.0.0
lrwxrwxrwx 1 travis travis 16 Apr 7 15:42 libasan.so.4 -> libasan.so.4.0.0
-rwxr-xr-x 2 travis travis 7337600 Oct 6 2017 libasan.so.4.0.0
lrwxrwxrwx 1 travis travis 18 Apr 7 15:42 libatomic.so -> libatomic.so.1.2.0
lrwxrwxrwx 1 travis travis 18 Apr 7 15:42 libatomic.so.1 -> libatomic.so.1.2.0
-rwxr-xr-x 2 travis travis 161216 Oct 6 2017 libatomic.so.1.2.0
-rw-r--r-- 1 travis travis 4454732 Apr 7 15:42 libcrypto.a
lrwxrwxrwx 1 travis travis 18 Apr 7 15:42 libcrypto.so -> libcrypto.so.1.0.0
-rwxr-xr-x 1 travis travis 2627408 Apr 7 15:42 libcrypto.so.1.0.0
-rw-r--r-- 2 travis travis 375120 Sep 17 2017 libedit.a
-rwxr-xr-x 1 travis travis 967 Apr 7 15:42 libedit.la
lrwxrwxrwx 1 travis travis 17 Apr 7 15:42 libedit.so -> libedit.so.0.0.56
lrwxrwxrwx 1 travis travis 17 Apr 7 15:42 libedit.so.0 -> libedit.so.0.0.56
-rwxr-xr-x 2 travis travis 247224 Sep 17 2017 libedit.so.0.0.56
-rw-r--r-- 2 travis travis 50024 Nov 8 21:42 libffi.a
-rwxr-xr-x 1 travis travis 922 Apr 7 15:42 libffi.la
lrwxrwxrwx 1 travis travis 15 Apr 7 15:42 libffi.so -> libffi.so.6.0.4
lrwxrwxrwx 1 travis travis 15 Apr 7 15:42 libffi.so.6 -> libffi.so.6.0.4
-rwxr-xr-x 2 travis travis 40864 Nov 8 21:42 libffi.so.6.0.4
lrwxrwxrwx 1 travis travis 10 Apr 7 15:42 libform.a -> libformw.a
lrwxrwxrwx 1 travis travis 15 Apr 7 15:42 libform.so -> libformw.so.6.0
-rw-r--r-- 2 travis travis 144262 Nov 10 21:52 libformw.a
lrwxrwxrwx 1 travis travis 15 Apr 7 15:42 libformw.so -> libformw.so.6.0
lrwxrwxrwx 1 travis travis 15 Apr 7 15:42 libformw.so.6 -> libformw.so.6.0
-rwxr-xr-x 2 travis travis 88424 Nov 10 21:52 libformw.so.6.0
-rw-r--r-- 2 travis travis 132 Oct 6 2017 libgcc_s.so
-rw-r--r-- 2 travis travis 714800 Oct 6 2017 libgcc_s.so.1
lrwxrwxrwx 1 travis travis 16 Apr 7 15:42 libgomp.so -> libgomp.so.1.0.0
lrwxrwxrwx 1 travis travis 16 Apr 7 15:42 libgomp.so.1 -> libgomp.so.1.0.0
-rwxr-xr-x 2 travis travis 818632 Oct 6 2017 libgomp.so.1.0.0
-rw-r--r-- 2 travis travis 163462 Nov 10 21:58 libhistory.a
lrwxrwxrwx 1 travis travis 17 Apr 7 15:42 libhistory.so -> libhistory.so.7.0
lrwxrwxrwx 1 travis travis 17 Apr 7 15:42 libhistory.so.7 -> libhistory.so.7.0
-rwxr-xr-x 1 travis travis 149360 Apr 7 15:42 libhistory.so.7.0
lrwxrwxrwx 1 travis travis 15 Apr 7 15:42 libitm.so -> libitm.so.1.0.0
lrwxrwxrwx 1 travis travis 15 Apr 7 15:42 libitm.so.1 -> libitm.so.1.0.0
-rwxr-xr-x 2 travis travis 859472 Oct 6 2017 libitm.so.1.0.0
lrwxrwxrwx 1 travis travis 16 Apr 7 15:42 liblsan.so -> liblsan.so.0.0.0
lrwxrwxrwx 1 travis travis 16 Apr 7 15:42 liblsan.so.0 -> liblsan.so.0.0.0
-rwxr-xr-x 2 travis travis 1991168 Oct 6 2017 liblsan.so.0.0.0
-rw-r--r-- 2 travis travis 299356 Nov 7 22:04 liblzma.a
-rwxr-xr-x 1 travis travis 942 Apr 7 15:42 liblzma.la
lrwxrwxrwx 1 travis travis 16 Apr 7 15:42 liblzma.so -> liblzma.so.5.2.3
lrwxrwxrwx 1 travis travis 16 Apr 7 15:42 liblzma.so.5 -> liblzma.so.5.2.3
-rwxr-xr-x 2 travis travis 173408 Nov 7 22:04 liblzma.so.5.2.3
lrwxrwxrwx 1 travis travis 10 Apr 7 15:42 libmenu.a -> libmenuw.a
lrwxrwxrwx 1 travis travis 15 Apr 7 15:42 libmenu.so -> libmenuw.so.6.0
-rw-r--r-- 2 travis travis 71350 Nov 10 21:52 libmenuw.a
lrwxrwxrwx 1 travis travis 15 Apr 7 15:42 libmenuw.so -> libmenuw.so.6.0
lrwxrwxrwx 1 travis travis 15 Apr 7 15:42 libmenuw.so.6 -> libmenuw.so.6.0
-rwxr-xr-x 2 travis travis 42312 Nov 10 21:52 libmenuw.so.6.0
lrwxrwxrwx 1 travis travis 13 Apr 7 15:42 libncurses.a -> libncursesw.a
lrwxrwxrwx 1 travis travis 15 Apr 7 15:42 libncurses++.a -> libncurses++w.a
lrwxrwxrwx 1 travis travis 18 Apr 7 15:42 libncurses.so -> libncursesw.so.6.0
-rw-r--r-- 2 travis travis 156880 Nov 10 21:52 libncurses++w.a
-rw-r--r-- 2 travis travis 428898 Nov 10 21:52 libncursesw.a
lrwxrwxrwx 1 travis travis 18 Apr 7 15:42 libncursesw.so -> libncursesw.so.6.0
lrwxrwxrwx 1 travis travis 18 Apr 7 15:42 libncursesw.so.6 -> libncursesw.so.6.0
-rwxr-xr-x 2 travis travis 252768 Nov 10 21:52 libncursesw.so.6.0
lrwxrwxrwx 1 travis travis 11 Apr 7 15:42 libpanel.a -> libpanelw.a
lrwxrwxrwx 1 travis travis 16 Apr 7 15:42 libpanel.so -> libpanelw.so.6.0
-rw-r--r-- 2 travis travis 28834 Nov 10 21:52 libpanelw.a
lrwxrwxrwx 1 travis travis 16 Apr 7 15:42 libpanelw.so -> libpanelw.so.6.0
lrwxrwxrwx 1 travis travis 16 Apr 7 15:42 libpanelw.so.6 -> libpanelw.so.6.0
-rwxr-xr-x 2 travis travis 17544 Nov 10 21:52 libpanelw.so.6.0
-rwxr-xr-x 1 travis travis 22817750 Apr 7 15:42 libpython3.6m.a
lrwxrwxrwx 1 travis travis 20 Apr 7 15:42 libpython3.6m.so -> libpython3.6m.so.1.0
lrwxrwxrwx 1 travis travis 20 Apr 7 15:42 libpython3.6m.so.1 -> libpython3.6m.so.1.0
-rwxr-xr-x 1 travis travis 3571200 Apr 7 15:42 libpython3.6m.so.1.0
lrwxrwxrwx 1 travis travis 20 Apr 7 15:42 libquadmath.so -> libquadmath.so.0.0.0
lrwxrwxrwx 1 travis travis 20 Apr 7 15:42 libquadmath.so.0 -> libquadmath.so.0.0.0
-rwxr-xr-x 2 travis travis 752952 Oct 6 2017 libquadmath.so.0.0.0
-rw-r--r-- 2 travis travis 1353950 Nov 10 21:58 libreadline.a
lrwxrwxrwx 1 travis travis 18 Apr 7 15:42 libreadline.so -> libreadline.so.7.0
lrwxrwxrwx 1 travis travis 18 Apr 7 15:42 libreadline.so.7 -> libreadline.so.7.0
-rwxr-xr-x 1 travis travis 959288 Apr 7 15:42 libreadline.so.7.0
-rw-r--r-- 2 travis travis 1164344 Jan 30 15:25 libsqlite3.a
-rwxr-xr-x 1 travis travis 993 Apr 7 15:42 libsqlite3.la
lrwxrwxrwx 1 travis travis 19 Apr 7 15:42 libsqlite3.so -> libsqlite3.so.0.8.6
lrwxrwxrwx 1 travis travis 19 Apr 7 15:42 libsqlite3.so.0 -> libsqlite3.so.0.8.6
-rwxr-xr-x 2 travis travis 971904 Jan 30 15:25 libsqlite3.so.0.8.6
-rw-r--r-- 2 travis travis 816186 Dec 7 16:03 libssl.a
lrwxrwxrwx 1 travis travis 15 Apr 7 15:42 libssl.so -> libssl.so.1.0.0
-rwxr-xr-x 2 travis travis 537368 Dec 7 16:03 libssl.so.1.0.0
lrwxrwxrwx 1 travis travis 19 Apr 7 15:42 libstdc++.so -> libstdc++.so.6.0.24
lrwxrwxrwx 1 travis travis 19 Apr 7 15:42 libstdc++.so.6 -> libstdc++.so.6.0.24
-rwxr-xr-x 2 travis travis 8188008 Oct 6 2017 libstdc++.so.6.0.24
-rwxr-xr-x 1 travis travis 2036224 Apr 7 15:42 libtcl8.6.so
-rwxr-xr-x 2 travis travis 8260 Nov 8 20:10 libtclstub8.6.a
lrwxrwxrwx 1 travis travis 11 Apr 7 15:42 libtinfo.a -> libtinfow.a
lrwxrwxrwx 1 travis travis 16 Apr 7 15:42 libtinfo.so -> libtinfow.so.6.0
-rw-r--r-- 1 travis travis 403628 Apr 7 15:42 libtinfow.a
lrwxrwxrwx 1 travis travis 16 Apr 7 15:42 libtinfow.so -> libtinfow.so.6.0
lrwxrwxrwx 1 travis travis 16 Apr 7 15:42 libtinfow.so.6 -> libtinfow.so.6.0
-rwxr-xr-x 1 travis travis 250824 Apr 7 15:42 libtinfow.so.6.0
-rwxr-xr-x 2 travis travis 1572608 Nov 8 20:11 libtk8.6.so
-rwxr-xr-x 2 travis travis 6056 Nov 8 20:11 libtkstub8.6.a
lrwxrwxrwx 1 travis travis 16 Apr 7 15:42 libtsan.so -> libtsan.so.0.0.0
lrwxrwxrwx 1 travis travis 16 Apr 7 15:42 libtsan.so.0 -> libtsan.so.0.0.0
-rwxr-xr-x 2 travis travis 5113208 Oct 6 2017 libtsan.so.0.0.0
lrwxrwxrwx 1 travis travis 17 Apr 7 15:42 libubsan.so -> libubsan.so.0.0.0
lrwxrwxrwx 1 travis travis 17 Apr 7 15:42 libubsan.so.0 -> libubsan.so.0.0.0
-rwxr-xr-x 2 travis travis 1878328 Oct 6 2017 libubsan.so.0.0.0
lrwxrwxrwx 1 travis travis 18 Apr 7 15:42 libyaml-0.so.2 -> libyaml-0.so.2.0.5
-rwxr-xr-x 2 travis travis 131576 Nov 13 21:53 libyaml-0.so.2.0.5
-rw-r--r-- 2 travis travis 171088 Nov 13 21:53 libyaml.a
-rwxr-xr-x 1 travis travis 959 Apr 7 15:42 libyaml.la
lrwxrwxrwx 1 travis travis 18 Apr 7 15:42 libyaml.so -> libyaml-0.so.2.0.5
-rw-r--r-- 2 travis travis 126634 Nov 7 22:03 libz.a
lrwxrwxrwx 1 travis travis 14 Apr 7 15:42 libz.so -> libz.so.1.2.11
lrwxrwxrwx 1 travis travis 14 Apr 7 15:42 libz.so.1 -> libz.so.1.2.11
-rwxr-xr-x 2 travis travis 101152 Nov 7 22:04 libz.so.1.2.11
drwxr-xr-x 2 travis travis 4096 Apr 7 15:42 pkgconfig
drwxr-xr-x 35 travis travis 8192 Apr 7 15:42 python3.6
drwxr-xr-x 2 travis travis 52 Apr 7 15:42 sqlite3.20.0
drwxr-xr-x 5 travis travis 39 Apr 7 15:42 tcl8
drwxr-xr-x 6 travis travis 250 Apr 7 15:42 tcl8.6
-rw-r--r-- 1 travis travis 8165 Apr 7 15:42 tclConfig.sh
-rw-r--r-- 2 travis travis 773 Nov 8 20:10 tclooConfig.sh
drwxr-xr-x 2 travis travis 112 Apr 7 15:42 tdbc1.0.5
drwxr-xr-x 2 travis travis 75 Apr 7 15:42 tdbcmysql1.0.5
drwxr-xr-x 2 travis travis 73 Apr 7 15:42 tdbcodbc1.0.5
drwxr-xr-x 2 travis travis 81 Apr 7 15:42 tdbcpostgres1.0.5
lrwxrwxrwx 1 travis travis 17 Apr 7 15:42 terminfo -> ../share/terminfo
drwxr-xr-x 2 travis travis 69 Apr 7 15:42 thread2.8.1
drwxr-xr-x 6 travis travis 4096 Apr 7 15:42 tk8.6
-rw-r--r-- 1 travis travis 4334 Apr 7 15:42 tkConfig.sh
smth
April 8, 2018, 1:50am
4
for some reason cudatoolkit isn’t being installed in your CI setup. I dont know why, it is listed as a dependency for pytorch=0.3.0. Try:
conda update -y conda # get latest version of tool
#manually install cudatoolkit maybe
conda install -y cudatoolkit cudnn
kmkurn
April 8, 2018, 7:54am
5
I finally solved the issue. The cause is two-fold:
The correct conda library path is $HOME/miniconda/envs/some_env/lib because I created a virtual env named some_env for everything. So, I needed to add export LD_LIBRARY_PATH="$HOME/miniconda/envs/some_env/lib:$LD_LIBRARY_PATH.
cudatoolkit was in fact installed as pytorch’s dependency, but the version was wrong. Installing only pytorch 0.3.0 will install cudatoolkit 9.0 as dependency, while pytorch itself is compiled with cudatoolkit 8.0 (as indicated by the build string py36_cuda8.0.61_cudnn7.0.3h37a80b5_4). So, I needed to explicitly specify cudatoolkit version: conda install -y -c pytorch pytorch=0.3.0 cudatoolkit=8.0
The complete .travis.yml file is:
language: python
before_install:
- wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda.sh
- bash ~/miniconda.sh -b -p $HOME/miniconda
- export PATH="$HOME/miniconda/bin:$PATH"
- conda update -y conda
- conda create -y -n some_env python=3.6
- source activate some_env
- conda install -y -c pytorch pytorch=0.3.0 cudatoolkit=8.0 # specify cudatoolkit version explicitly
- export LD_LIBRARY_PATH="$HOME/miniconda/envs/some_env/lib:$LD_LIBRARY_PATH" # add conda library path
install:
- pip install -r requirements.txt
- pip install .
script: pytest --cov
after_success: coveralls
Now the build succeeds again. Thanks for pointing me to the right direction
P.S. I find it weird that cudatoolkit 9.0 is installed instead of 8.0 which pytorch 0.3.0 is compiled with. Why is that?
smth
April 9, 2018, 1:38am
6
P.S. I find it weird that cudatoolkit 9.0 is installed instead of 8.0 which pytorch 0.3.0 is compiled with. Why is that?
This is my mistake. I specified dependency on cudatoolkit, but not an exact version.
I’m working on fixing up the 0.3.1 packages.
1 Like
