Pretrained loaded but the performance worse at beginning

reinforcement-learning
1

I am trying reinforcement learning using pytorch,
and when I load a pretrained model as initial,
the performance is worse at begin, and it become better very fast

this is a plot the shows the performance drop when resume.

image
image740×528 11.1 KB

I save my model like this post suggest:
and I didn’t delete any layer of my model

 save_checkpoint({
            'epoch': epoch + 1,
            'state_dict': model.state_dict(),
            'optimizer' : optimizer.state_dict(),
      })

def save_checkpoint(state, filename='checkpoint.pth.tar'):
    torch.save(state, filename)

and resume model like this:

def load_pretrained(model, pretrained_dict):
    model_dict = model.state_dict()
    # filter out unmatch dict 
    pretrained_dict = {k: v for k, v in pretrained_dict.items() if k in model_dict}
    model_dict.update(pretrained_dict)
    model.load_state_dict(pretrained_dict)

if checkpoint_file is not None:
    print('loading checkpoint_file {}'.format(checkpoint_file))
    cp = torch.load(checkpoint_file)
    load_pretrained(optimizer, cp['optimizer'])
    load_pretrained(state_dict, cp['state_dict'])
    trained_episodes = cp['epoch']

Is that normal that need some time to let the performance recover?

2

This looks weird to me… I’m not sure what happened.

What do you know away in pretrained_dict = {k: v for k, v in pretrained_dict.items() if k in model_dict}?

3

Ok, I figured out why.

I’m training DQN, and there have two network (q and target_q) when training.
in the training process, only update one of the net (q), and the other net (target_q) will periodically copy parameters from the q net.

The performance problem is because I only save the q net parameters, after I save the both net, the issue is solved.

1 Like
4

Hey,

I have trained model for robotic arm in Gazebo simulation. When I am loading that model, I can see all
trained weights in place but new loaded model does not produce same successful runs.What could be the reason.

DRQN Network is similar to

Am I missing something?

Thanks in Advance