I want to build torch v1.12.1 from source and create wheel with all the dependencies included into the wheel file . Additionally, to force the inclusion of all the Cuda and Cudnn -related stuff from my system into the wheel file. Similar task goes for torchvision v0.13.1.
I have Ubuntu 22.04 with Nvidia-driver-535 and Cuda Toolkit 11.2+Cudnn 8.1 installed system-wide.
conda create --name pytorch-build python=3.7
conda activate pytorch-build
conda install astunparse numpy ninja pyyaml mkl mkl-include setuptools cmake cffi typing_extensions future six requests dataclasses
conda install -c pytorch magma-cuda112
cd ~
git clone --recursive https://github.com/pytorch/pytorch
cd ~/pytorch
git checkout v1.12.1
git submodule sync
git submodule update --init --recursive --jobs 0
export USE_CUDA=1 USE_CUDNN=1 USE_MKLDNN=1
cd ~/pytorch
python setup.py sdist bdist_wheel
as a result, the file ‘torch-1.12.0a0+git664058f-cp37-cp37m-linux_x86_64.whl’ has been created. The size of the file is 250.4mb.
conda deactivate pytorch-build
conda create --name new-environment python=3.7
conda activate new-environment
pip install ~/pytorch/dist/torch-1.12.0a0+git664058f-cp37-cp37m-linux_x86_64.whl
Processing ./pytorch/dist/torch-1.12.0a0+git664058f-cp37-cp37m-linux_x86_64.whl
Collecting typing-extensions (from torch==1.12.0a0+git664058f)
Downloading typing_extensions-4.7.1-py3-none-any.whl.metadata (3.1 kB)
Downloading typing_extensions-4.7.1-py3-none-any.whl (33 kB)
Installing collected packages: typing-extensions, torch
Successfully installed torch-1.12.0a0+git664058f typing-extensions-4.7.1
‘pip install …/torch-1.12.0a0+git664058f-cp37-cp37m-linux_x86_64.whl’ is collecting typing-extensions (from torch==1.12.0a0+git664058f) as well as is downloading typing_extensions-4.7.1-py3-none-any.whl (33 kB).
Will it be safe to install, prior to compiling and building, all three typing packages as follows?
conda install typing-extensions typing_extensions typing
How to force ‘python setup.py sdist bdist_wheel’ to include all the dependencies into the wheel file ‘torch-1.12.0a0+git664058f-cp37-cp37m-linux_x86_64.whl’?
How to force ‘pip install *.whl’ not to look for (and not to download) the dependencies if they already included with the wheel file?
Moreover, how can I force the inclusion of all the Cuda and Cudnn -related stuff from my system into the wheel file ‘torch-*.whl’?
Next, in the original ‘pytorch-build’ conda enviroment, I want to build torchvision:
conda activate pytorch-build
conda install libpng libjpeg-turbo -c pytorchhttps://discuss.pytorch.org/t/torch-tf-spacy-more-gpu-python-3-7-ubuntu-22-04
conda install pillow
cd ~
git clone https://github.com/pytorch/vision.git
cd vision
git checkout v0.13.1
python setup.py sdist bdist_wheel
as a result, the file ‘torchvision-0.13.1a0+bddbd7e-cp37-cp37m-linux_x86_64.whl’ has been created. The size of the file is 19mb.
conda deactivate pytorch-build
conda activate new-environment ### with Python 3.7
pip install ~/vision/dist/torchvision-0.13.1a0+bddbd7e-cp37-cp37m-linux_x86_64.whl
Processing ./vision/dist/torchvision-0.13.1a0+bddbd7e-cp37-cp37m-linux_x86_64.whl
Requirement already satisfied: typing-extensions in ./miniconda3/envs/py37/lib/python3.7/site-packages (from torchvision==0.13.1a0+bddbd7e) (4.7.1)
Collecting numpy (from torchvision==0.13.1a0+bddbd7e)
Downloading numpy-1.21.6-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (15.7 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 15.7/15.7 MB 11.5 MB/s eta 0:00:00
Collecting requests (from torchvision==0.13.1a0+bddbd7e)
Downloading requests-2.31.0-py3-none-any.whl.metadata (4.6 kB)
Requirement already satisfied: torch in ./miniconda3/envs/py37/lib/python3.7/site-packages (from torchvision==0.13.1a0+bddbd7e) (1.12.0a0+git664058f)
Collecting pillow!=8.3.*,>=5.3.0 (from torchvision==0.13.1a0+bddbd7e)
Downloading Pillow-9.5.0-cp37-cp37m-manylinux_2_28_x86_64.whl (3.4 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.4/3.4 MB 10.1 MB/s eta 0:00:00
Collecting charset-normalizer<4,>=2 (from requests->torchvision==0.13.1a0+bddbd7e)
Downloading charset_normalizer-3.3.2-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (33 kB)
Collecting idna<4,>=2.5 (from requests->torchvision==0.13.1a0+bddbd7e)
Downloading idna-3.4-py3-none-any.whl (61 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.5/61.5 kB 15.9 MB/s eta 0:00:00
Collecting urllib3<3,>=1.21.1 (from requests->torchvision==0.13.1a0+bddbd7e)
Downloading urllib3-2.0.7-py3-none-any.whl.metadata (6.6 kB)
Requirement already satisfied: certifi>=2017.4.17 in ./miniconda3/envs/py37/lib/python3.7/site-packages (from requests->torchvision==0.13.1a0+bddbd7e) (2022.12.7)
Downloading requests-2.31.0-py3-none-any.whl (62 kB)
Downloading charset_normalizer-3.3.2-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (136 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 136.8/136.8 kB 14.9 MB/s eta 0:00:00
Downloading urllib3-2.0.7-py3-none-any.whl (124 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 124.2/124.2 kB 19.9 MB/s eta 0:00:00
Installing collected packages: urllib3, pillow, numpy, idna, charset-normalizer, requests, torchvision
Successfully installed charset-normalizer-3.3.2 idna-3.4 numpy-1.21.6 pillow-9.5.0 requests-2.31.0 torchvision-0.13.1a0+bddbd7e urllib3-2.0.7
Why ‘pip install’ wants to download the above dependencies?
How to force ‘python setup.py sdist bdist_wheel’ to include all the dependencies as well as Cuda+Cudnn into the wheel file?
How to force ‘pip install *.whl’ not to look for (and not to download) the dependencies if they already included with the wheel file?
This topic is the continuation of https://discuss.pytorch.org/t/torch-tf-spacy-more-gpu-python-3-7-ubuntu-22-04