What to do when two classes need different lr?

Hi guys,

I’ve been stuck on this problem for a few days now.

There’s 5 classes to predict. When my lr = 0.00001, the accuracy of class 0 is 99% and class 1 is 0%.

However if I remove class 0 and change the lr to 0.0001, class 1 goes up to 99% accuracy.

So I need a different lr for each class. I’ve never read about this problem, what should I do?

Is it possible to start with a high lr and once the accuracy of class 1 is above 98% stop considering it, decrease the lr to 0.000001 and only optimize for class 0?