How to use ParallelEnv?

reinforcement-learning
1

I finished this guide and in the end there’s a note at the end that I can use ParallelEnv env to speed up data collection. I couldn’t find any examples or information about this besides documentation page. So I tried to just change TransformedEnv to ParallelEnv in cod. But when I try to use data collector it throws me errors

2

The syntax is

ParallelEnv(num_of_p_envs: int, env_constructor: Callable[[], EnvBase], mp_start_method: str="spawn")

if you want to spawn one single env (alternatively you can pass a list of constructors as second argument)
Example

from torchrl.envs import GymEnv, ParallelEnv
import functools
make_env = functools.partial(GymEnv, "CartPole-v1")
num_envs = 2

env = ParallelEnv(num_envs, make_env)
assert env.batch_size = (num_envs, )
print(env.rollout(10, break_when_any_done=False))

The doc is here
https://pytorch.org/rl/reference/generated/torchrl.envs.ParallelEnv.html

3

Sorry for late reply, I had a busy month

Thanks to you and docs I figured how to create ParallelEnv and do random rollouts. But how do I use it with an actual policy? In my case the policy is naturally made to work with one environment (in: number of features, out: number of actions). But when I try to execute something like this:

env.rollout(10, policy=policy_module)

I get dimensions mismatch error because rollout has different shape. I’m not sure what to do, I can’t access rollout code, and designing policy module specifically for ParallelEnvs seems wrong

4

Can you give an example or a snippet of how you’re building the policy? Or the error you’re getting?
In general if your policy can look at your observation regardless of the batch size you should be fine.