Pure vocabulary question.
How would you define and differentiate those 3 terms : Loss, Loss Function, Criterion
Is the “Loss Function” part of the larger ensemble “Criterion”, and “Loss” the value resulting from the “Loss Function” ?
I see that documentation and tutorials often talks about loss functions, but in practice code snippet often use the term criterion instead, as the variable pointing the loss function. But this word criterion is never discussed when introducing loss functions.
Are there criterions which are not loss function ?
Is “criterion” widely understood or should I always prefer “loss function” when speaking about it ?
Side question: Metrics, Evaluation Metric.
For instance, Accuracy is one of those metrics.
Is loss part of the metrics, but not the evaluation metrics, as opposed to accuracy ?
As you note, this is not completely distinct. “criterion” is typically a callable (function or nn.Module instance) that computes the loss (value), “loss function” makes this explicit in the name. “loss” is - in my experience - more often used for the loss value but sometimes (for example in naming the loss Modules) it is also used for the loss function or even the class whose instance computes the loss function.
I think criterion is relatively widely understood, I have not seen ones that were not loss functions.
Loss typically is differentiable, while metrics don’t need to be. I would consider metrics to comprise both losses and non-differentiable metrics (such as accuracy). It is completely OK to use validation loss as a evaluation metric, but you cannot differentiate accuracy and so you cannot do gradient descent on it.