Validation Loss Not Decreasing, Local Minima

vision
1

I have been training a model with nearly 2L images and the validation loss seems to get stuck at 0.8. I tried many ways:

  1. Changed activation function
  2. Used weighted CE instead of normal CE since the dataset is imbalanced.
  3. Tried changing optimizers and lr_schedulers
    But nothing worked. I hope so it getting stuck with local minima. Is there anything else that I should try??

Thanks in advance,
Thirumalai Kumar

2

Without seeing which strategies you already followed (your data you initial model), there is a variety of things you can follow, using more recently proposed activation functions (e.g. gelu) and optimizer etc.
Maybe you can provide more info about what you already tried.
You can also compare your own result to results obtained by others (e.g. literature) maybe you already reached the minimum feasible loss.

Another thing I would emphasize is that there is a difference between validation loss and validation metric. Usually, validation loss is loss function applied to validation data whereas validation metric would be a metric (appropriate for your problem ) applied to validation data.

For instance, if you were to experiment of cifar32 using NLL as loss function and accuracy as metric. Then, you will see the NLL for validation data will start increasing even though validation accuracy keep increasing (at a slower rate than initially) (There is a diverse set of ideas and papers on why this occurs)
Maybe, you can check validation metric see if it is improving

3

actually it is the validation loss which is not decreasing after it reaches 0.8