Building from source in docker: Memory use requirments?

Hi all,

I am trying to setup a dev environment for pytorch (actually functorch), and having battled with some C++ version/library problems I figured developing inside of a docker container would be a simple way to go.

However, I’m trying to build a container and compile pytorch inside of it and getting errors like:

c++: internal compiler error: Killed (program cc1plus)
at about 85%

My guess is that this is due to memory constraints in my container - but they’re already quite high (10GB). I am trying to understand if that is expected, should I be able to compile from source with less RAM?

You might need to limit the number of jobs via MAX_JOBS=n python setup.py install in case you are running our of host memory.

Aha that makes a lot of sense. Thanks a lot I’ll give it a try.

Thanks for the help, all is well now.

Would there be a useful place for me to put instructions for running a dev container for pytorch (using vscode)? I don’t run a blog but it might be useful for others.

I’m sure a lot of users would appreciate your instructions and findings. The forum might be one place or any blogging platform.

Well for the meantime I’ll dump it in this thread in case it comes up in anyone’s search. But I’ll try find a better place for it later.

In VSCode there is really useful feature for [VSCode container development](https://code.visualstudio.com/docs/remote/containers). This can be really useful for making sure your builds and python environment are reproducible.

After reading the doc, you’ll see that you need two things to setup a remote container env

  1. Dockerfile
  2. .devcontainer.json

It took me some time to set these up, so I’m sharing some workable versions below. Note that I see this up for working on functorch (i.e wanted latest pytorch built from source), so minor modifications may be needed for what you want to do.

(note, in addition I STRONGLY suggest mounting a container with a volume, so the data you’re working on can live between restarts, see this)

.devcontainer.json

{
	"name": "torchfunc",
	"build": {
		"dockerfile": "Dockerfile",
	},
	"runArgs": [ "--cap-add=SYS_PTRACE", "--security-opt", "seccomp=unconfined"],
	"settings": {},
	"extensions": [
		"ms-vscode.cpptools",
		"ms-vscode.pylance"
	],
	"remoteUser": "root"
}

Dockerfile (based on that from pytorch repo)

# syntax = docker/dockerfile:experimental
#
# NOTE: To build this you will need a docker version > 18.06 with
#       experimental enabled and DOCKER_BUILDKIT=1
#
#       If you do not use buildkit you are not going to have a good time
#
#       For reference:
#           https://docs.docker.com/develop/develop-images/build_enhancements/
ARG BASE_IMAGE=ubuntu:18.04
ARG PYTHON_VERSION=3.8

FROM ${BASE_IMAGE} as dev-base
RUN --mount=type=cache,id=apt-dev,target=/var/cache/apt \
    apt-get update && apt-get install -y --no-install-recommends \
        build-essential \
        ca-certificates \
        ccache \
        cmake \
        curl \
        git \
        libjpeg-dev \
        libpng-dev && \
    rm -rf /var/lib/apt/lists/*
RUN /usr/sbin/update-ccache-symlinks
RUN mkdir /opt/ccache && ccache --set-config=cache_dir=/opt/ccache
ENV PATH /opt/conda/bin:$PATH

FROM dev-base as conda
ARG PYTHON_VERSION=3.8
RUN curl -fsSL -v -o ~/miniconda.sh -O  https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh  && \
    chmod +x ~/miniconda.sh && \
    ~/miniconda.sh -b -p /opt/conda && \
    rm ~/miniconda.sh && \
    /opt/conda/bin/conda install -y python=${PYTHON_VERSION} conda-build pyyaml numpy ipython&& \
    /opt/conda/bin/conda clean -ya

RUN mkdir -p /code
RUN git clone https://github.com/pytorch/pytorch.git /code/pytorch
RUN cd /code/pytorch && git submodule sync && git submodule update --init --recursive --jobs 0 && pip install -r requirements.txt
RUN cd /code/pytorch && MAX_JOBS=4 python setup.py install