Centralized learning-decentralized execution clarification (engineering perspective on PPO algo)

[Kimonili]

So in general, there will be N=number_of_agents critic and actor networks. The centralized learning appears in the fact that each critic network recieves state information from the other agents and then updates its weights?

@iffiX

sample an action from all agent policies P(x)_m, m \in {0, n} where n is the number of agents

So here all the agents make an action simultaneously.

@iffiX

then you can exchange the state information between all agents

Given the action each agent made in the previous step, they transision to a new state. The state of each agent will be shared to all the agents. So basically, every agent will be aware of the position of every agent.

@iffiX

then parallelly use critics to evaluate values

Here the critics are evaluating its agent’s actions but being aware of the new dynamics of the environments? (after getting informed of the relative position of every agent)

@iffiX

then parallely update critics, and finally parallely update actors using values produced by critics.

This step is related to the previous one so the question remains the same.

@iffiX

The simplest (with medium performance) way to implement this is using a threadpool, eg: the one from joblib or multiprocessing module, if your model is small. A more complex way would be using torch.jit.fork_ , which is even better than using the threadpool because it completely avoids the GIL problem (about 50% faster than threadpool). ProcessPool is only recommended if your model is on aa GPU and very large (like using a resnet or something of similar complexity and parameter size, there are many caveats in using a ProcessPool and I don’t recommend it.

I believe then that the best choice is torch.jit.fork_ , which I understood that can run in CPU as well.

I am really sorry for the questions, I just want to be sure I understand this correctly!

Thank you so much!