Usually, we separate input data into three, which are training set, validation set, test set in deep learning.
For each epoch, I want to do the best way to get a better model using validation set.
For example, for each epoch, after finishing learning with training set, I can select the model parameter which has the lowest loss w.r.t. validation set by saving the model parameter each time when the loss w.r.t. validation set has lower value than the
lowest loss by then.
Also, I can select the lowest validation loss while using different validation set each time
by mixing training set and validation set and randomly selecting some fraction of the mixture as a new
validation set
Is this the best or at least recommended way to get a better or the best model parameter???
I will appreciate if you can help me with this problem
Thank you
Yes, this is a common technique and is called early stopping. You would store the model’s state_dict for the lowest validation loss (or highest validation accuracy).
No, you shouldn’t do it, since mixing the training and validation set would give you a better “validation” loss than you would expect to see on “unseen” data. After your training is finished, the “validation” loss (which is not really the validation loss anymore) would most likely be much lower than the test loss.
Dear ptrblck:
Thank you so much for your kind answer!!!
Now I understand the validation process much better
May I ask you one more question? Actually it may be the same question again.
(Sorry if it is the same question…)
Actually, what I wanted to do is as follow:
I separate input data into two parts, training set and test set.
And for each epoch, I randomly jumble training set and separate it into two parts,
and I train the model using the first part and, after that, evaluate the model using the second part by calculating validation loss with this second part and trace when it has the minimum validation loss
and when finishing epochs, get the model parameter at that epoch having lowest validation loss.
Why I wanted to do this is, because I randomly jumble training set and select newly created validation set inside the training set each epoch and hope it may make sense theoretically…
No, you should not do that because your model will be biased. It means that in epoch 1 some of the data will be used to train, but then in the second epoch that data may be part of the validation. The model will perform very well on that data then as it has seen it before, which you do not want. The validation set should always be data that the model has never seen during training.
What you seem to want is something like k-fold cross-validation. You can Google it, there’s quite a lot of information about it on the internet.
Thank you, Bram.
Now I understand fully.
Validation should not be a part of training…(which I didn’t know correctly)
whereas cross validation is permitted…
Have a nice day and see you again
Thank you and
sorry that I can select only one answer(solution) to my question
I have a slightly different question on the same topic. Say I find out out that epoch 25 gives the minimum loss on validation set. I was wondering if I should train another model using training + validation data with 25 epochs and call this my final model?
I want to do this so that I am able to use maximum possible data for training.