I am trying to setup a dev environment for pytorch (actually functorch), and having battled with some C++ version/library problems I figured developing inside of a docker container would be a simple way to go.
However, I’m trying to build a container and compile pytorch inside of it and getting errors like:
c++: internal compiler error: Killed (program cc1plus)
at about 85%
My guess is that this is due to memory constraints in my container - but they’re already quite high (10GB). I am trying to understand if that is expected, should I be able to compile from source with less RAM?
Would there be a useful place for me to put instructions for running a dev container for pytorch (using vscode)? I don’t run a blog but it might be useful for others.
Well for the meantime I’ll dump it in this thread in case it comes up in anyone’s search. But I’ll try find a better place for it later.
In VSCode there is really useful feature for [VSCode container development](https://code.visualstudio.com/docs/remote/containers). This can be really useful for making sure your builds and python environment are reproducible.
After reading the doc, you’ll see that you need two things to setup a remote container env
Dockerfile
.devcontainer.json
It took me some time to set these up, so I’m sharing some workable versions below. Note that I see this up for working on functorch (i.e wanted latest pytorch built from source), so minor modifications may be needed for what you want to do.
(note, in addition I STRONGLY suggest mounting a container with a volume, so the data you’re working on can live between restarts, see this)
# syntax = docker/dockerfile:experimental
#
# NOTE: To build this you will need a docker version > 18.06 with
# experimental enabled and DOCKER_BUILDKIT=1
#
# If you do not use buildkit you are not going to have a good time
#
# For reference:
# https://docs.docker.com/develop/develop-images/build_enhancements/
ARG BASE_IMAGE=ubuntu:18.04
ARG PYTHON_VERSION=3.8
FROM ${BASE_IMAGE} as dev-base
RUN --mount=type=cache,id=apt-dev,target=/var/cache/apt \
apt-get update && apt-get install -y --no-install-recommends \
build-essential \
ca-certificates \
ccache \
cmake \
curl \
git \
libjpeg-dev \
libpng-dev && \
rm -rf /var/lib/apt/lists/*
RUN /usr/sbin/update-ccache-symlinks
RUN mkdir /opt/ccache && ccache --set-config=cache_dir=/opt/ccache
ENV PATH /opt/conda/bin:$PATH
FROM dev-base as conda
ARG PYTHON_VERSION=3.8
RUN curl -fsSL -v -o ~/miniconda.sh -O https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh && \
chmod +x ~/miniconda.sh && \
~/miniconda.sh -b -p /opt/conda && \
rm ~/miniconda.sh && \
/opt/conda/bin/conda install -y python=${PYTHON_VERSION} conda-build pyyaml numpy ipython&& \
/opt/conda/bin/conda clean -ya
RUN mkdir -p /code
RUN git clone https://github.com/pytorch/pytorch.git /code/pytorch
RUN cd /code/pytorch && git submodule sync && git submodule update --init --recursive --jobs 0 && pip install -r requirements.txt
RUN cd /code/pytorch && MAX_JOBS=4 python setup.py install