Failed when trying to install PyTorch from source

1

I’ve got an ubuntu17.04 (64bit), the newest anaconda for a python of 3.6.2.
When I run
python setup.py install
(no CUDA support),
I got something like this:

/home/thomas/anaconda3/compiler_compat/ld: cannot find -lpthread
/home/thomas/anaconda3/compiler_compat/ld: cannot find -lc
collect2: error: ld returned 1 exit status
error: command ‘gcc’ failed with exit status 1

What shall I do to solve this issue?
Thanks for your help.

2

There’s an issue open for this, here: https://github.com/pytorch/pytorch/issues/3050
My understanding is that the gcc provided by conda doesn’t build pytorch (I’m assuming you installed that with conda install gcc.

For now, you can do the following:

  1. conda uninstall gcc
  2. Install a gcc on your system. Preferably 4.9. Mine is 4.8.5 and it works as well.
  3. Try to python setup.py install. I think what’ll happen is that something will complain that it can’t find the right compilers, at which point you can symlink your local system gcc and g++ to the respective places.
1 Like
3

Thanks for your fast advice, but I’m afraid I’m not facing an issue like yours…
In fact, the output of gcc --version on my linux is exactly gcc (GCC) 4.8.5

4

Did you install a gcc via conda? (conda list | grep gcc) That one takes precedence over your system gcc for building pytorch.
If not, then could I see a larger chunk of the output? You could dump it into a pastebin or gist.

5

Well, didn’t.
But even if I ran conda install gcc, and made sure that this time conda list | grep gcc returned a proper result
gcc 4.8.5 libgcc-ng 7.2.0
The next attempt for python setup.py install still failed with the same exception message just as above.

The whole chunk is bigger than the byte limit here. I wonder how to upload a .txt file for this.

6

I was setting up pytorch fork in a new conda env and run into te same issue.

I had gcc-4.9 already installed. What worked for me was

conda create --name pytorch-dev python=3.6
source activate pytorch-dev


export CMAKE_PREFIX_PATH="$(dirname $(which conda))/../" # [anaconda root directory]

# Install basic dependencies
conda install numpy pyyaml mkl setuptools cmake cffi

# Add LAPACK support for the GPU
conda install -c soumith magma-cuda80 # or magma-cuda75 if CUDA 7.5


export CMAKE_C_COMPILER=/usr/bin/gcc-4.9
export CMAKE_CXX_COMPILER=/usr/bin/gcc-4.9

NO_CUDA=1 DEBUG=1 python setup.py clean
NO_CUDA=1 DEBUG=1 python setup.py build develop

This works. I was not attempting to perform full installation with CUDA and DEBUG=0 yet

Edit This workaround works with CUDA and no DEBUG as well

1 Like
7

Thanks for your advice, and I’ve tried it twice after that.
Unfortunately, it returned a good result succeeded in installing, as well as a

import torch
Traceback (most recent call last):
File “”, line 1, in
File “/home/thomas/anaconda3/envs/pytorch-dev/lib/python3.6/site-packages/torch/init.py”, line 53, in
from torch._C import *
ImportError: libmkl_intel_lp64.so: cannot open shared object file: No such file or directory

when I try to import it in my codes.
I googled it, and just found a similar issue like this https://github.com/pytorch/pytorch/issues/936. But the solutions are not suitable for me.

8

For what it’s worth, I recommend trying this in a clean virtual (or physical) machine. I haven’t tried to build PyTorch recently in Ubuntu 17.04, but I did recently struggle with Ubuntu 16.04, and I can confirm that the following sequence works:

Install anaconda 5
Activate it (eg “. anaconda3/bin/activate” - don’t forget the dot before the command)

sudo apt-get install g++ cmake
conda install numpy pyyaml mkl setuptools cffi
git clone https://github.com/pytorch/pytorch.git
cd pytorch
python setup.py install

Close your terminal and open a new one.
You should now be able to import pytorch

2 Likes
9

I just built PyTorch from source with Ubuntu 17.04, GCC 6.3, CUDA 9.0, MKL 2018, and cuDNN 7.

It works :100: without problems. Thanks PyTorch team!

1 Like
10

You need to set your environment variables for library paths properly. Look up LD_LIBRARY_PATH and CMAKE_PREFIX_PATH and set them correctly. Then build the project from source again.

11

It didn’t work for me, just with a same output as “cannot find -lpthread, etc.”.

12

I ran sudo find / -name "mkl", and it returned
/home/thomas/anaconda3/pkgs/mkl-service-1.1.2-py36h17a0993_4/lib/python3.6/site-packages/mkl
/home/thomas/anaconda3/pkgs/mkl-service-1.1.2-py36_3/lib/python3.6/site-packages/mkl
/home/thomas/anaconda3/lib/python3.6/site-packages/mkl

so I wrote export LD_LIBRARY_PATH="/home/thomas/anaconda3/lib/python3.6/site-packages/mkl" in ~/.bashrc.
But still, nothing changed. Only to return the same ImportError: libmkl_intel_lp64.so: cannot open shared object file: No such file or directory as before.

13

That is your CMAKE path.

export CMAKE_PREFIX_PATH="/home/thomas/anaconda3"

LD Library path is usually more complicated, with 3-5 different paths in it. “/usr/lib:/home/thomas/anaconda3/lib:etc:etc”

On a side note, since you are new to PyTorch it might be much quicker to just do conda install pytorch -c soumith and try it out

14

Well, I’m in China and this means that my approach through conda install pytorch -c soumith can be easily cut down for every second, and also means that if not so, the speed of download should be a poor 3Kb/s or so. I mean - here in China, ways for conda-ing packages usually is to just fetch it from the tsinghua mirrors: http://mirrors.tuna.tsinghua.edu.cn/help/anaconda/ But, well, it seems that it is not being maintained by now.
So installing from git sources is the only way to me.

1 Like
15

Ouch I had no idea download speeds were so slow in China!

You can build code inside a Docker image: it is super easy and can be done with a single command.
docker build -t pytorch .

16

Sorry for a late reply,
I’ve tried for the docker, but it seems that thousands of attempts before have ruined my linux storage, and that when the Docker image asks for a 4G or so space, my NEW-INSTALLED ubuntu refused that.
Alright, have to say goodbye then…

17

If you can find a VPS beyond the GREAT WALL, you can make it by the following way.

Install Anaconda on your VPS
Create virtual python envs (e.g. conda create -n py3_pytorch python=3.6 )
Activate your virtual envs and install pytorch
Find the directory of your envs (maybe it will be here ~/anaconda2/envs/py3_pytorch) and copy the whole directory to you own machine and place it in the dicectory your_machine/anaconda2/envs/

Then you can use it by directly activate the virtual envs (source activate py3_pytorch)

18

Ecauchy,i have the same problem with you ,have you solve it?
Than you very much
/home/wyy/anaconda3/compiler_compat/ld: cannot find -lm
/home/wyy/anaconda3/compiler_compat/ld: cannot find -lpthread
/home/wyy/anaconda3/compiler_compat/ld: cannot find -lc
collect2: error: ld returned 1 exit status

19

Hello, I also have the same problem too. Does anyone above solve this question? Thanks for sharing very much!