I received the task to build Pytorch (Pytorch) from Source to see if it makes our processing faster. I never did this, and i am not a expert using Docker so i managed to piece together a Dockerfile to build from source:
# Base image with Python 3.11 slim
FROM python:3.11-slim
# Set environment variables
ENV GUNICORN_WORKERS=1
# Install miniconda
# conda at end of path in order to use OS python in container
ENV PATH="${PATH}:/root/miniconda3/bin"
ARG PATH="${PATH}:/root/miniconda3/bin"
# Install system dependencies required for building PyTorch
RUN apt-get update && apt-get install -y --no-install-recommends \
build-essential \
# cmake \
# ninja-build \
curl \
libopenblas-dev \
liblapack-dev \
libx11-dev \
wget \
git \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
RUN wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh \
&& mkdir /root/.conda \
&& bash Miniconda3-latest-Linux-x86_64.sh -b \
&& rm -f Miniconda3-latest-Linux-x86_64.sh \
&& conda update conda \
&& conda install cmake ninja setuptools
ENV CMAKE_PREFIX_PATH=${CONDA_PREFIX:-"$(dirname $(which conda))/../"}
# Install additional dependencies for Python
# RUN pip install --no-cache-dir --upgrade pip setuptools wheel
RUN pip install --no-cache-dir --upgrade pip wheel
# Set the working directory
WORKDIR /usr/src/app
# # Create and activate a virtual environment (optional)
# RUN python -m venv env \
# && . env/bin/activate
# Clone PyTorch source code
RUN git clone --recursive https://github.com/pytorch/pytorch.git \
&& cd pytorch \
&& make triton
# COPY ./pytorch ./pytorch
# RUN cd pytorch
# Install Python dependencies for PyTorch
RUN pip install -r pytorch/requirements.txt
# Install additional libraries like MKL and MAGMA (adjust based on your system)
# RUN pip install mkl-static mkl-include
# Build PyTorch
WORKDIR /usr/src/app/pytorch
RUN python setup.py develop
When i it gets to python setup.py develop it breaks with the following error:
269.3 ninja: build stopped: subcommand failed.
269.4 Building wheel torch-2.7.0a0+git2efa98d
269.4 -- Building version 2.7.0a0+git2efa98d
269.4 cmake -GNinja -DBUILD_PYTHON=True -DBUILD_TEST=True -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/usr/src/app/pytorch/torch -DCMAKE_PREFIX_PATH=/usr/local/lib/python3.11/site-packages;/usr/src/app/env: -DPython_EXECUTABLE=/usr/local/bin/python -DTORCH_BUILD_VERSION=2.7.0a0+git2efa98d -DUSE_NUMPY=True /usr/src/app/pytorch
269.4 cmake --build . --target install --config Release
------
1 warning found (use docker --debug to expand):
- UndefinedVar: Usage of undefined variable '$CMAKE_PREFIX_PATH' (line 6)
Dockerfile:46
--------------------
44 | # Build PyTorch
45 | WORKDIR /usr/src/app/pytorch
46 | >>> RUN python setup.py develop
47 |
48 | # Optional: Clean up unnecessary files to reduce image size
--------------------
ERROR: failed to solve: process "/bin/sh -c python setup.py develop" did not complete successfully: exit code: 1'