Using a pretrained model on another dataset

1

Hi everyone :slight_smile:

I have a pretrained model that I trained on ImageNet data. I stored the model.state_dict() and the optimizer.state_dict() in a checkpoint file.

Now I would like to train a model on VGGFace2 data with the pretrained weights of the model trained on ImageNet to speed up training. This means that I have to change some of the layers in my model. I have seen that I can add the strict=False flag when I load the model.state_dict() to only add the weights to the layers that are present in both the old and the new model. But there is no such option for the optimizer.state_dict().

How can I resume training with both the model.state_dict() and optimizer.state_dict() on another model than they were trained on?

Any help is very much appreciated!

All the best,
snowe

2

Have you checked
optimizer.load_state_dict(pathload[])?

I think for most of the optimizer you donā€™t need to load their state.

3

Hi @CedricLy, thank you for your response!

What do you mean with optimizer.load_state_dict(pathload[])?

But as discussed here, I think you are right. You only need to load the optimizer state if you want to continue training on the same data with the same network. Otherwise the model.state_dict() is sufficient.

4 
model = TheModelClass(*args, **kwargs)
optimizer = TheOptimizerClass(*args, **kwargs)

checkpoint = torch.load(PATH)
model.load_state_dict(checkpoint['model_state_dict'])
optimizer.load_state_dict(checkpoint['optimizer_state_dict'])
epoch = checkpoint['epoch']
loss = checkpoint['loss']

model.eval()
# - or -
model.train()

This is from a tutorial how to load if you want to continue a training.
Check this link for more details.
Link
Does this already solve your issue?

5

I am familiar with how to resume training and loading the model.state_dict() and the optimizer.state_dict() :slight_smile:

My problem was that I had to load the state_dicts into another network. Whereas for the model.state_dict() you can pass the strict=False flag to tell PyTorch to only load the parameters of the layers that are present in both the old and the new network, there is no such option for the optimizer.state_dict(). However, apparently this is not a problem since you donā€™t need the optimizer.state_dict() if you donā€™t continue on the same network.

Essentially, my use case does not really fall under the category ā€˜resume trainingā€™ but more so ā€˜start training from scratchā€™ but with better initial parameters compared to random initialisation.

Hope that makes sense!

6

Ah I see.

So theoretically it would be interesting to see what the ā€œoptimizer stateā€ consists of and if the state is dependent on the parameters, which it tries to train.
If the optimizer does not depent on the parameters there will be no issue with loading the current state with different parameters.
If it is dependent, it is questionable if the dependence is separated for each parameter.
In this case it may be possible to load the state of the optimizer without all parameters.

In the case of the optimizer having the second derivative saved in its state, it will not be possible. Or at least I would not know how .