I’m working on a PPO implementation. It is a RL algorithm in which we constraint the policy to be updated in the neighborhood of the previous policy.
To do so, the algorithm relies on a ratio that evaluates the differences of probabilities between old and new policy.
So, in my implementation I:
- Compute the log prob coming from the old policy
- Compute the log prob coming from the new policy
- Evaluate the ratio
- Clone the new policy into the old one
- Update the new policy
I realize that my ratio is always absolutely equal to one. Could it be because I copy the network using this method:
for p_source, p_target in zip(self.parameters(), clone.parameters()): p_target.data = p_source.data