I try to build a distributed RL with PyTorch RPC, but I got a problem that I can’t use my GPU power fully. Here is my code structure. They are two parts, learner and actor.
class Learner: def __init__(self): self.policy = Policy() self.buffer = container() run_multiple_actor() def learning_loop(self): "backward & optimize" while True: get_trans_from_buffer() computing_loss() backward() optimizer.step() def get_action(self, state): action = self.policy(state) insert_to_buffer(state,action) return action class Actor: def __init__(self): self.env = env self.learner_rref = rref def act_loop(self): while True: state = self.env.step(action) action = self.learner_rref.run_sync().get_action(state)
For learner, after initating, it runs in
learning_loop. For actor, after initating, it runs in
act_loop and call Learner’s
get_action from remote.
The question is that could
get_action threads run simultaneously on GPU? If they can, tt seems that as long as I run enough actors, I fully use my GPU power. However, after adding serveal actors, the volatile of GPU stop inscreasing and stays in a low level(eg. 20% or 30%). And I don’t think it’s a problem of my CPU cores. I have enough CPU to make all actors run simultaneously.
Could anyone point out what’s the problem of my code. I am new to PyTorch PRC, help me please.