I’ve been contributing to Pytorch and had few queries regarding building it. Building Pytorch is resource intensive.
1, Do we need to build Pytorch everytime when we do git pull ? I noticed that even though the compiled binaries are still there, using the build command again starts the compilation again (but not all).
DEBUG=1 USE_DISTRIBUTED=0 USE_MKLDNN=0 USE_CUDA=0 BUILD_TEST=0 USE_FBGEMM=0 USE_NNPACK=0 USE_QNNPACK=0 USE_XNNPACK=0 python setup.py develop
- Are there more flags that like the ones above that can reduce the compilation time and resource required ? I am using
ninja
as mentioned in the README,
How do we rebuild the changed functions only ? I am using ccache also.
I’m using python setup.py develop
to only build what is needed after setting up ccache and ninja.
python setup.py develop
builds everything as per CONTRIBUTING.MD
. I’m currently using this
DEBUG=1 USE_DISTRIBUTED=0 USE_MKLDNN=0 USE_CUDA=0 BUILD_TEST=0 USE_FBGEMM=0 USE_NNPACK=0 USE_QNNPACK=0 USE_XNNPACK=0 python setup.py develop
My query is for example if a breaking change is introduced in Pytorch, for example here,
AttributeError: module 'torch' has no attribute 'true_divide'
true_divide
will become part of 1.5. But now after importing torch, it throws this error. So instead of building again, how can I only build the change that has been recently introduced. So now when I run develop
command, it compiles [124/620]
620 other binaries also which means some things are getting compiled again even though they have been compiled.
If true_divide
was used in other methods (and files), they need to be rebuilt as well. Same goes for other depending files thereof.
That’s my question. What do I do to rebuild only the changes ? Is there some automatic way of compiling only the changes ?
Yes, as I said python setup.py develop
should do the job. At least it’s working every time for me.
Okay, thanks will try.
I’m doing with this command
DEBUG=1 USE_DISTRIBUTED=0 USE_MKLDNN=0 USE_CUDA=0 BUILD_TEST=0 USE_FBGEMM=0 USE_NNPACK=0 USE_QNNPACK=0 USE_XNNPACK=0 python setup.py develop
It builds other stuff also, but this is a subset of what you shared.