DDP is affected by code modification

[Jacfger]

I was training a model using DDP and I wished to train multiple instance with some difference in configuration. However, when I tried to checkout to another branch, the running instance suddenly report an error and stopped. The error also only existed in the newly checkout branch, indicating the process actually read the modified code. Is that supposed to happen and how exactly does this work? I don’t quite understand…

I saw there’s a similar question but unfortunately no one had answer.

With the PyTorch Distributed training model, code changes affect running applications

I noticed two strange things, First, when I used Distributed for training, I modified the code. After a while, the application reported an error when RUNNING because of my modification. Second, I added print to the code, and after a while, the program that was already running will start to print accordingly

[ptrblck]

The behavior seems to depend on the OS or rather the method to fork or spawn the processes as described here.