Hello!
I try to build a distributed RL with PyTorch RPC, but I got a problem that I can’t use my GPU power fully. Here is my code structure. They are two parts, learner and actor.
class Learner:
def __init__(self):
self.policy = Policy()
self.buffer = container()
run_multiple_actor()
def learning_loop(self):
"backward & optimize"
while True:
get_trans_from_buffer()
computing_loss()
backward()
optimizer.step()
def get_action(self, state):
action = self.policy(state)
insert_to_buffer(state,action)
return action
class Actor:
def __init__(self):
self.env = env
self.learner_rref = rref
def act_loop(self):
while True:
state = self.env.step(action)
action = self.learner_rref.run_sync().get_action(state)
For learner, after initating, it runs in learning_loop. For actor, after initating, it runs in act_loop and call Learner’s get_action from remote.
The question is that could get_action threads run simultaneously on GPU? If they can, tt seems that as long as I run enough actors, I fully use my GPU power. However, after adding serveal actors, the volatile of GPU stop inscreasing and stays in a low level(eg. 20% or 30%). And I don’t think it’s a problem of my CPU cores. I have enough CPU to make all actors run simultaneously.
Could anyone point out what’s the problem of my code. I am new to PyTorch PRC, help me please.