That probably doesn’t what you think it does.
Some things you could look at / share for debugging
- What’s the imbalance between the classes? What’s happening? Does the net always predict a large class? Generally, looking at per-class accuracies / the confusion matrix is a good idea.
- I would usually try to modify the dataset to give the classes an equal number of times by duplicating the minority classes. Similar to here: Multiclass classification: Skewed data problem and labeling
Best regards
Thomas