How to build after making code changes in cpp files in ATen/native

C++
1

I have successfully build pytorch. And the changes in python files are reflected instantly as I have build using

NO_CUDA=1 python3 setup.py develop

But, I am making changes in the cpp files and I am trying to build it using this

python3 setup.py clean
NO_CUDA=1 python3 setup.py develop

and it failes with this error

subprocess.calledprocesserror: command '['ninja', 'install']' returned non-zero exit status 1.

Also, its taking about 20mins to reach to this point, any way of doing this faster? I just have to build the native files.

2

You should use ccache, it will change your life!

You won’t need to do python3 setup.py clean before rebuilding, build PyTorch again using ccache, then make your changes, then build it again straight away.

1 Like
3

I am on Ubuntu, and I am planning to install ccache using this

# install ccache
sudo apt install ccache

# update symlinks and create/re-create nvcc link
sudo /usr/sbin/update-ccache-symlinks
sudo ln -s /usr/bin/ccache /usr/lib/ccache/nvcc

# config: cache dir is ~/.ccache, conf file ~/.ccache/ccache.conf

# max size of cache
ccache -M 25Gi  # -M 0 for unlimited
# unlimited number of files
ccache -F 0

# deploy (and add to ~/.bashrc for later)
export PATH="/usr/lib/ccache:$PATH"

After intallation, I should just run

python3 setup.py develop

to build, right?

4

I’d recommend you to run precisely the script in the ccache link above, I always do this without thinking and it works like a charm.

Specifically, in the terminal, I go to the PyTorch folder, I copy & paste all the code between then and fi in the ccache link, then I run export PATH=~/ccache/lib:$PATH. It works a 100% of times, in several machines, I never spent any second trying to understand why :smile:!

BTW, whenever you close the terminal, you should run export PATH=~/ccache/lib:$PATH again for ccache to work.

1 Like