Synchronization for sharing/updating shared model state dict across multi-process

[wwiiiii (Jeong TaeYeong)]

Hello, while I’m doing a RL project, I need some way to share model state dict across multi-processes.

I found the A3C implementation which has this feature, but I have some questions about it.

1. In the above repo, the author defined model instance on CPU shared memory, and then all the sub processes also share that. In each subprocess, it has its own model instance on GPU, and it loads the state from globally shared CPU model instance without any lock.
   So my question is, Is it guranteed that it’s safe to access parameters of the model on CPU shared memory simultaneously in multiple processes without any lock or synchronization?
   I’m pretty sure it’s fine since model parameters are treated as read-only variable here, but I just want to double check it.

2. The next question is my main interest. At here, the author simply copies the gradient of model in sub-process to shared model without any lock mechanism, and also optimization is done without synchronization. How come is it possible? Why isn’t everything messed up when multiple processes overwrite the gradient of shared model without any lock mechanism?

If I missed out some points, please let me know

[wwiiiii (Jeong TaeYeong)]

Okay, I found there’s already well documented answer at https://pytorch.org/docs/stable/notes/multiprocessing.html.
And also with hogwild!-spirit, it’s fine to overwrite gradient from other process.

So the only thing I wonder now is that how the updating shared model parameter could be done without any lock mechanism. Are the add_ or addcdiv_ atomic operation?

[Yanzzzzz]

I have the same question, and I think it’s not atomic operatino.

[neoql (Qianli Neo)]

I found a possible answer. @Yanzzzzz @wwiiiii

Grad sharing problem in a3c

yeah it really there just to make sure initializing goes smoothly. async updates without locks mean they updates from the parallel processes are done asynchronously which means they are run on their own clock and the different process threads can actually make updates at the same time which can lead to bad updates. The reason they do this even though there is risk of bad updates being made is that updates are done much faster which speeds up training as the process of acquiring and releasing lo…

And I found another related answer

Model update with "share_memory" need lock protection

TL;DR. Maybe a lock is needed for optimizer.step() while using multiprocessing. The example provided in [MULTIPROCESSING BEST PRACTICES] (https://pytorch.org/docs/stable/notes/multiprocessing.html) is to update parameters without lock protection. import torch.multiprocessing as mp from model import MyModel def train(model): # Construct data_loader, optimizer, etc. for data, labels in data_loader: optimizer.zero_grad() loss_fn(model(data), labels).backward() o…