[solved] PyTorch 0.3.0 ImportError at runtime

Hello,

Have been using PyTorch 0.2.0 for CNNs with no issues. Wonderful DL framework.

Conda unistalled PyTorch 0.2.0
Installed PyTorch 0.3.0 using

conda install pytorch torchvision -c pytorch

which installed

pytorch:     0.3.0-py36_cuda8.0.61_cudnn7.0.3h37a80b5_4 pytorch
torchvision: 0.2.0-py36h17b6947_1                       pytorch

First ran the trivial test program as for PyTorch 0.2.0

import torch


if __name__ == "__main__":

    x = torch.rand(5, 3)

    y = torch.rand(5, 3)

    if torch.cuda.is_available():
        x = x.cuda()
        y = y.cuda()
        x + y

    print('x', x)
    print('y', y)
    print('x + y', x + y)

The following error message was returned.

File “/home/audrius/projects/manifold/radiomics/PMH_Metser_Project_1_NSCLC/dl_cnn/pytorch/models/test_pytorch.py”, line 10, in
import torch

File “/home/audrius/anaconda3/lib/python3.6/site-packages/torch/init.py”, line 56, in
from torch._C import *

ImportError: /home/audrius/anaconda3/lib/python3.6/site-packages/torch/_C.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _ZNK2at4Type10arange_outERNS_6TensorENS_6ScalarE

At a bit of a loss as to what this means and how to address it.

System:

Linux: Ubuntu 17.10 with kernel 4.13.0
GeForce GTX 1080 Ti
Nvidia driver 384.90
CUDA 8.0
cuDNN 7.0 for CUDA 8.0
Anaconda Python 3.6

Any insight into this error message would be appreciated.

I tried to reproduce this in a docker image with 17.10 but it runs perfectly so far:

Linux d57c82e0d23d 4.4.0-45-generic #66~14.04.1-Ubuntu SMP Wed Oct 19 15:05:38 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

Can you make sure that your conda package has this installed?

libstdcxx-ng              7.2.0                h7a57d05_2

Thanks for your suggestion. Checked and found

# All requested packages already installed.
# packages in environment at /home/audrius/anaconda3:
#
libstdcxx-ng              7.2.0                h7a57d05_2

What is the output of:

ldd /home/audrius/anaconda3/lib/python3.6/site-packages/torch/_C.cpython-36m-x86_64-linux-gnu.so

The output is

ldd /home/audrius/anaconda3/lib/python3.6/site-packages/torch/_C.cpython-36m-x86_64-linux-gnu.so
	linux-vdso.so.1 =>  (0x00007ffffdffe000)
	libshm.so => /home/audrius/anaconda3/lib/python3.5/site-packages/torch/lib/libshm.so (0x00007fc05a716000)
	libcudart.so.8.0 => /home/audrius/anaconda3/lib/libcudart.so.8.0 (0x00007fc05a4ae000)
	libnvToolsExt.so.1 => /home/audrius/anaconda3/lib/libnvToolsExt.so.1 (0x00007fc05a2a4000)
	libcudnn-a2b758a6.so.7.0.3 => /home/audrius/anaconda3/lib/python3.6/site-packages/torch/lib/libcudnn-a2b758a6.so.7.0.3 (0x00007fc04d138000)
	libTH.so.1 => /home/audrius/anaconda3/lib/python3.5/site-packages/torch/lib/libTH.so.1 (0x00007fc04ca80000)
	libTHS.so.1 => /home/audrius/anaconda3/lib/python3.5/site-packages/torch/lib/libTHS.so.1 (0x00007fc04c851000)
	libTHNN.so.1 => /home/audrius/anaconda3/lib/python3.5/site-packages/torch/lib/libTHNN.so.1 (0x00007fc04c500000)
	libATen.so.1 => /home/audrius/anaconda3/lib/python3.5/site-packages/torch/lib/libATen.so.1 (0x00007fc04be4f000)
	libTHC.so.1 => /home/audrius/anaconda3/lib/python3.5/site-packages/torch/lib/libTHC.so.1 (0x00007fc03f9a0000)
	libTHCS.so.1 => /home/audrius/anaconda3/lib/python3.5/site-packages/torch/lib/libTHCS.so.1 (0x00007fc03f653000)
	libTHCUNN.so.1 => /home/audrius/anaconda3/lib/python3.5/site-packages/torch/lib/libTHCUNN.so.1 (0x00007fc03badc000)
	libnccl.so.1 => /home/audrius/anaconda3/lib/python3.5/site-packages/torch/lib/libnccl.so.1 (0x00007fc038e00000)
	libgcc_s.so.1 => /home/audrius/anaconda3/lib/libgcc_s.so.1 (0x00007fc038bee000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fc0389cf000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fc0385ef000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fc05c37f000)
	libstdc++.so.6 => /home/audrius/anaconda3/lib/libstdc++.so.6 (0x00007fc0382b5000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fc0380b1000)
	librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fc037ea9000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fc037b53000)
	libmkl_gf_lp64.so => /home/audrius/anaconda3/lib/libmkl_gf_lp64.so (0x00007fc03706d000)
	libmkl_gnu_thread.so => /home/audrius/anaconda3/lib/libmkl_gnu_thread.so (0x00007fc0359af000)
	libmkl_core.so => /home/audrius/anaconda3/lib/libmkl_core.so (0x00007fc03381c000)
	libgomp.so.1 => /home/audrius/anaconda3/lib/libgomp.so.1 (0x00007fc0335f9000)
	libcublas.so.8.0 => /home/audrius/anaconda3/lib/libcublas.so.8.0 (0x00007fc0305b1000)
	libcusparse.so.8.0 => /home/audrius/anaconda3/lib/libcusparse.so.8.0 (0x00007fc02da91000)

it looks like an issue with your anaconda (or your $LD_LIBRARY_PATH). Basically your default anaconda environment has some libraries under /home/audrius/anaconda3/lib/python3.5 and some under /home/audrius/anaconda3/lib/python3.6. My guess is older libs are under /home/audrius/anaconda3/lib/python3.5 and are mixing up with newer libs under /home/audrius/anaconda3/lib/python3.6.

I suggest maybe reinstalling your anaconda or checking your environment variable $LD_LIBRARY_PATH

echo $LD_LIBRARY_PATH

and unsetting it with:

unset LD_LIBRARY_PATH
1 Like

My guess is older libs are under /home/audrius/anaconda3/lib/python3.5 and are mixing up with newer libs under /home/audrius/anaconda3/lib/python3.6.

This was indeed the issue. Had updated from Python 3.5 to 3.6 at one point.
LD_LIBRARY_PATH was pointing to both current /home/audrius/anaconda3/lib/python3.6 and old /home/audrius/anaconda3/lib/python3.5 paths.
Reinstalling anaconda was the solution. PyTorch 0.3.0 is now running.

Thank you for your prompt response and assistance. Much appreciated by this linux newbie.

Hi, @smth, I also got a error report like Audrius-St ,

>>> import torch
python: symbol lookup error: /datah/gyhu/soft/Anaconda3/envs/env3/lib/python3.6/site-packages/torch/_C.cpython-36m-x86_64-linux-gnu.so: undefined symbol: THInferNumThreads

and when I run ldd /datah/gyhu/soft/Anaconda3/envs/env3/lib/python3.6/site-packages/torch/_C.cpython-36m-x86_64-linux-gnu.so. I got output in the following:


        linux-vdso.so.1 =>  (0x00007ffffb3e8000)
        libshm.so => /datah/gyhu/soft/Anaconda3/envs/env3/lib/libshm.so (0x00007f88e2c5c000)
        libcudart.so.8.0 => /datah/gyhu/soft/Anaconda3/envs/env3/lib/libcudart.so.8.0 (0x00007f88e29f3000)
        libnvToolsExt.so.1 => /datah/gyhu/soft/Anaconda3/envs/env3/lib/libnvToolsExt.so.1 (0x00007f88e27e9000)
        libcudnn-a2b758a6.so.7.0.3 => /datah/gyhu/soft/Anaconda3/envs/env3/lib/python3.6/site-packages/torch/lib/libcudnn-a2b758a6.so.7.0.3 (0x00007f88d567d000)
        libTH.so.1 => /datah/gyhu/soft/Anaconda3/envs/env3/lib/libTH.so.1 (0x00007f88d50c9000)
        libTHS.so.1 => /datah/gyhu/soft/Anaconda3/envs/env3/lib/libTHS.so.1 (0x00007f88d4e95000)
        libTHNN.so.1 => /datah/gyhu/soft/Anaconda3/envs/env3/lib/libTHNN.so.1 (0x00007f88d4b8b000)
        libATen.so.1 => /datah/gyhu/soft/Anaconda3/envs/env3/lib/python3.6/site-packages/torch/lib/libATen.so.1 (0x00007f88d437c000)
        libTHC.so.1 => /datah/gyhu/soft/Anaconda3/envs/env3/lib/python3.6/site-packages/torch/lib/libTHC.so.1 (0x00007f88c2e60000)
        libTHCS.so.1 => /datah/gyhu/soft/Anaconda3/envs/env3/lib/python3.6/site-packages/torch/lib/libTHCS.so.1 (0x00007f88c2a71000)
        libTHCUNN.so.1 => /datah/gyhu/soft/Anaconda3/envs/env3/lib/python3.6/site-packages/torch/lib/libTHCUNN.so.1 (0x00007f88bdd7d000)
        libnccl.so.1 => /datah/gyhu/soft/Anaconda3/envs/env3/lib/libnccl.so.1 (0x00007f88bb721000)
        libgcc_s.so.1 => /datah/gyhu/soft/Anaconda3/envs/env3/lib/libgcc_s.so.1 (0x00007f88bb50b000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f88bb2cb000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f88baf01000)
        /lib64/ld-linux-x86-64.so.2 (0x0000558f58421000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f88bacf9000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f88ba9ef000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f88ba7eb000)
        libstdc++.so.6 => /datah/gyhu/soft/Anaconda3/envs/env3/lib/./libstdc++.so.6 (0x00007f88ba4d5000)
        libopenblas.so.0 => /datah/gyhu/soft/Anaconda3/envs/env3/lib/./libopenblas.so.0 (0x00007f88b9491000)
        libgomp.so.1 => /datah/gyhu/soft/Anaconda3/envs/env3/lib/./libgomp.so.1 (0x00007f88b9282000)
        libcublas.so.8.0 => /datah/gyhu/soft/Anaconda3/envs/env3/lib/libcublas.so.8.0 (0x00007f88b6239000)
        libcurand.so.8.0 => /datah/gyhu/soft/Anaconda3/envs/env3/lib/libcurand.so.8.0 (0x00007f88b22c2000)
        libcusparse.so.8.0 => /datah/gyhu/soft/Anaconda3/envs/env3/lib/libcusparse.so.8.0 (0x00007f88af7a2000)
        libmkl_gf_lp64.so => /datah/gyhu/soft/Anaconda3/envs/env3/lib/libmkl_gf_lp64.so (0x00007f88aecbb000)
        libmkl_gnu_thread.so => /datah/gyhu/soft/Anaconda3/envs/env3/lib/libmkl_gnu_thread.so (0x00007f88ad5fd000)
        libmkl_core.so => /datah/gyhu/soft/Anaconda3/envs/env3/lib/libmkl_core.so (0x00007f88ab46a000)
        libgfortran.so.3 => /datah/gyhu/soft/Anaconda3/envs/env3/lib/././libgfortran.so.3 (0x00007f88ab160000)

It seems that I did not mix up python3.5 & 3.6 , so what the problem in me? Thank you in advance!

@guyue_hu your problem is an older libTH.so.1 is being loaded, most likely because of $LD_LIBRARY_PATH.

What is output of:

echo $LD_LIBRARY_PATH