Loss function shape

Hi, What is the meaning of this jumps in the loss function?
I am decreasing the learning rate at iterations [300, 600, 900, 1200]

Screenshot 2020-09-09 at 07.22.00

Hi, @lycaenidae. Some questions/comments, but fair warning… I’m not very experienced with pytorch. :wink:

  • What is the scale of your plot?
  • Are you calculating loss as an instantaneous value? A moving average?
  • What is your use case? Without some context, it’s tough to tell what could be going on.
  • Are you shuffling the samples in your training set? If not, your instantaneous loss values could definitely jump around like that. Datasets are sometimes “clumpy”. Imagine hitting your model with a large “clump” of a novel class after you’ve been training for a while.