Save model or save the best model?

vision
1

I see it in many different PyTorch tutorials. There are two types of methods to save models.

For example, a model is trained using train/validation/test (k-fold cross-validation).

The first method is that after training/validation is completed, then save the model (no epoch accuracy and best accuracy comparison).

The second method is that during the validation process, save the model where the validation accuracy is the highest.

Which method is the right/recommended/better one?

2

Not going to pretend that I know the definitive answer here, but just two thoughts to consider:

  • The more you think overfitting is a risk (either because you don’t have enough training data given your particular model design choice, or for any other reason) the more you should favor saving the “best” model. This is because overfitting will cause actual performance to start to drop after enough training has occurred (as the model starts to train on the particulars of your training data, and not on its desired generalizable characteristics) and you want to make sure you use a version of the model from before the falling-off in performance.
  • The less validation data you have, the more “noisy” your validation accuracy is, and therefore the less you should trust the “best” model.

Just a couple of crude rules of thumb.