Saving and loading a model in Pytorch?

43

If your optimizer has internal parameters, e.g. Adam, then yes, you should also store it.

44

Iā€™m forking the official word level language modelling. I canā€™t find an explicit optimizer here. All Iā€™ve is a loss.backward(). Iā€™m not able to reproduce the results by saving the model like explained here.

45

In this tutorial the weight updates are performed manually in this line of code.
Since you donā€™t have internal estimates you donā€™t have to store anything regarding the optimization.

46

I saved the model which performed the following graph.

loss_iter_graph%20copy
loss_iter_graph%20copy3000Ɨ2500 470 KB

On reproducing the results with the model saved, my test error is,

End of training | test loss 17.28 | test ppl 32104905.14

Generated result is gibberish.

Something is terribly wrong. Iā€™m not sure where I should check :confused:

47

Something looks fishy. Could you create a new thread and post your complete issue there?
It would also be easier to debug, if you could post your code so that we can have a look.

1 Like
48

I started a new thread at here.

49

Hi! I have a problem with loading my model. Iā€™m training VGG19 on cifar10 in colab, when I load it in colab it is OK but when I load it on my laptop with same code it gives error. Theyā€™re both python3 and trained with cuda.
Error:

a
a786Ɨ367 45.6 KB

Save code

def save_checkpoint(state, filename):
    torch.save({'state_dict': net.state_dict(),
                'optimizer': optimizer.state_dict(),
                }, filename)

Load

checkpoint = torch.load('./vgg19_200.pth')
net.load_state_dict(checkpoint['state_dict'])
optimizer.load_state_dict(checkpoint['optimizer'])
1 Like
50

hello Iā€™m trying to save my adam optimizer,

but why whenever I load it, the state_dict is always different
if I restart my environment?

Iā€™ve also make my own thread here, Saved model have higher loss

thank you

51

no such thing as mistake or understand or not, think any is ok

52

Is there a way to save and load models from s3 directly?

53

Have you solved this problem? I have encountered this problem, I donā€™t know where it is wrong.

54

Yes, if you use StringIO you can create a file stream, write your model state to it, then push that to s3.

What I additionally do is use joblib to add compression and pickle after writing to the stream, push that to s3, then unload with joblib back to a file stream object and read the model state back into a model object to resume.

55

Itā€™s not necessary, you can use .copy() itā€™ll work fine too. :slight_smile:

56

He just mean when you need to evaluate or infer.

57

Hi, I am new to pytorch and was wondering how to create the model class for a trained pytorch model. I wish to use that to save and load the model for serving it with flask.

58

MyModel.eval() insert before or after state_dict method ? i think is after,beacuse after load parameters and freeze some parameters,right ?

59

Yes, you are right! Put MyModel.eval() after loading state_dict.

60

Hi,

So a sort of related question but in the context of the saving the optimizer. Is saving/loading the full optimizer object, the same as saving/loading only the optimizerā€™s state_dict() when resuming training? Aside from the obvious that saving only the state_dict(), saves memoryā€¦

61

In my case, the results are correct without eval() but not with it.

I am trained Gener. Adv. Net.

63

Wow. I did not think of that, but this also worked for me and I have no idea why.