Building from source in docker: Memory use requirments?

6

Well for the meantime I’ll dump it in this thread in case it comes up in anyone’s search. But I’ll try find a better place for it later.

In VSCode there is really useful feature for [VSCode container development](https://code.visualstudio.com/docs/remote/containers). This can be really useful for making sure your builds and python environment are reproducible.

After reading the doc, you’ll see that you need two things to setup a remote container env

  1. Dockerfile
  2. .devcontainer.json

It took me some time to set these up, so I’m sharing some workable versions below. Note that I see this up for working on functorch (i.e wanted latest pytorch built from source), so minor modifications may be needed for what you want to do.

(note, in addition I STRONGLY suggest mounting a container with a volume, so the data you’re working on can live between restarts, see this)

.devcontainer.json

{
	"name": "torchfunc",
	"build": {
		"dockerfile": "Dockerfile",
	},
	"runArgs": [ "--cap-add=SYS_PTRACE", "--security-opt", "seccomp=unconfined"],
	"settings": {},
	"extensions": [
		"ms-vscode.cpptools",
		"ms-vscode.pylance"
	],
	"remoteUser": "root"
}

Dockerfile (based on that from pytorch repo)

# syntax = docker/dockerfile:experimental
#
# NOTE: To build this you will need a docker version > 18.06 with
#       experimental enabled and DOCKER_BUILDKIT=1
#
#       If you do not use buildkit you are not going to have a good time
#
#       For reference:
#           https://docs.docker.com/develop/develop-images/build_enhancements/
ARG BASE_IMAGE=ubuntu:18.04
ARG PYTHON_VERSION=3.8

FROM ${BASE_IMAGE} as dev-base
RUN --mount=type=cache,id=apt-dev,target=/var/cache/apt \
    apt-get update && apt-get install -y --no-install-recommends \
        build-essential \
        ca-certificates \
        ccache \
        cmake \
        curl \
        git \
        libjpeg-dev \
        libpng-dev && \
    rm -rf /var/lib/apt/lists/*
RUN /usr/sbin/update-ccache-symlinks
RUN mkdir /opt/ccache && ccache --set-config=cache_dir=/opt/ccache
ENV PATH /opt/conda/bin:$PATH

FROM dev-base as conda
ARG PYTHON_VERSION=3.8
RUN curl -fsSL -v -o ~/miniconda.sh -O  https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh  && \
    chmod +x ~/miniconda.sh && \
    ~/miniconda.sh -b -p /opt/conda && \
    rm ~/miniconda.sh && \
    /opt/conda/bin/conda install -y python=${PYTHON_VERSION} conda-build pyyaml numpy ipython&& \
    /opt/conda/bin/conda clean -ya

RUN mkdir -p /code
RUN git clone https://github.com/pytorch/pytorch.git /code/pytorch
RUN cd /code/pytorch && git submodule sync && git submodule update --init --recursive --jobs 0 && pip install -r requirements.txt
RUN cd /code/pytorch && MAX_JOBS=4 python setup.py install