Sevelopment work in Docker?

Hello; a few getting started questions. Post is a bit rambling; so I numbered some questions; but any general advice on how to structure a development environment is most welcome.

In other projects; I do my development work in Docker. I find it nice to have the dockerfile to “document” the environment; and then docker run ... pytorch-dev /bin/bash and from there tend to interactively work in the container. 1.) Is that somewhat common for pytorch; or do most rely on the env management offered by Conda?

I started with the dockerfile given here - Building from source in docker: Memory use requirments? - #6 by Padarn_Wilson - which is pretty similar to the Dockerfile in the repo.

But running Coda in Docker seems to have some considerations

conda activate

CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'.
```https://pythonspeed.com/articles/activate-conda-dockerfile/ has some pointers on using Conda; so perhaps either of the suggested options would be worth investigating.

printf '#include <stdio.h>\nint main() { printf("Hello World");}'|clang -x c -; ./a.out
bash: clang: command not found
bash: ./a.out: No such file or directory

2.) Are these errors happening because the Conda environment isn’t active; so the required executables are not being found?

The below command did run through without any errors; so I do think the setup is quite close:

CMAKE_PREFIX_PATH="$(dirname $(which conda))/../" python setup.py develop

Trying to run python tests; I get the below

Traceback (most recent call last):
  File "test/run_test.py", line 19, in <module>
    from torch.testing._internal.common_utils import (
  File "/home/brent/repos/pytorch/torch/testing/_internal/common_utils.py", line 46, in <module>
    import expecttest
ModuleNotFoundError: No module named 'expecttest'

3.) If I try to stick with docker; I guess I could consider dropping using conda; since I could do multiple builds in different docker images.

Again; any pointers will be much appreciated!
Thanks,Brent

Hey @Brent1, I don’t have answers to all of your questions but I can give my perspective based on the workflow I’m using.

I didn’t intend to use conda (my thought was the same as your 3rd question), so there are some things missing that are in the original Dockerfile like

ENV PATH /opt/conda/bin:$PATH

I guess this is why you’re getting the conda setup error.

For the expecttest problem, this I’m not clear on at all, in my Dockerfile I have pip install -r requirements.txt, which should install this.

How are you triggering the tests? I just tried pytorch/test_torch.py at master · pytorch/pytorch · GitHub in my dev container and it was okay (but I haven’t rebuilt the container in a little while).

  1. You could install the conda binaries or pip wheels inside a docker container, so these approaches are not mutually exclusive.

  2. However, it seems you cannot create a new conda env or activate it, so maybe you would need to run conda init or conda init bash first in your container?
    I guess clang is also not installed in the docker container, so you could add it to the Dockerfile and rebuild it.

  3. Yes, I would prefer to use an “isolated” environment via a docker container (unless I’m debugging multiple binaries).

Thank you for the help; I’ve been able to get the tests to run. I could hopefully work out the answers to the below eventually on my own; I’ve still got a ways to go on my learning curve with the various components in use; so just posting here for completeness.

Padarn_Wilson “I didn’t intend to use conda”

Are the conda commands in the dockerfile in building-from-source-in-docker “just” to be able to use conda packages - such as conda-build; pyyaml; numpy; ipython - but not to use an actual conda environment?

pip install

After running

pip install -r requirements.txt
pip install pytest

python test/run_test.py starts running without errors. YAY. I’m running it from a shell which is connected to the running container.

Sorry I wasn’t notified of your reply!

Are the conda commands in the dockerfile in building-from-source-in-docker “just” to be able to use conda packages - such as conda-build; pyyaml; numpy; ipython - but not to use an actual conda environment?

Right, I don’t have a conda environment separate from the global (or user) python stup.