I’m wonder why we don’t use model.eval() command in training_step method of the “LightningModule”
def training_step(self, batch, batch_idx):
x, y = batch
pred = self(x) ### but our model is in training mode now …
I’m wonder why we don’t use model.eval() command in training_step method of the “LightningModule”
def training_step(self, batch, batch_idx):
x, y = batch
pred = self(x) ### but our model is in training mode now …
There is two parts to this.
training_step
is about training, so it seems natural that the model is in training mode,Best regards
Thomas
Thanks for clarification, @tom !
Does Litghtning do the same for “on_train_epoch_end” callback or I have to set model.eval() manually?
Which of pre-defined callbacks provided with model.eval “under the hood”?
is it correct that training loss calculating using model in training mode?
I don’t know, you could find out yourself with assert not model.training
(for eval mode) or so.
The rule of thumb would be that training is set up with train and validation/testing with eval, but if you want this more detailed, I would probably check with in the lightning discussions.
I think the training loss is just computed during trianing, yes.
I guess PL authors took care of switching between eval/train mode within pre defined callbacks… But problem is that when I try to predict test data in “on_fit_end” callback without using model.eval() it gives me different result than predicting outside training routine (and of course using model.eval in advance), thats why I’m wonder If ‘on_fit_end’ callback provided with model.eval … but I guess I chose inappropriate forum
However, it should be emphasized that reporting the loss in train
mode might be incorrect if for example the model uses Dropout
. In that case, the training loss will be underestimated.