Reasons for class mismatch high accuracy?

My image is classified into 4 distinct classes but I wrongly have been outputting 20 possible classes, but the thing is my 20 class output was a much more accurate (94%) model then the 4 class (74%) and the image segmentation was looked better? How in the heck? I even checked pixel values and the my target is unique values 0-3 and my prediction is 0-20. I was wondering if anyone could shed some light

Thank you

This sounds counter-intuitive to the higher overall accuracy or are the “additionally” predicted class indices quite sparse and do not lower the accuracy a lot?
I would have assumed your model would learn to predict only the used target classes, as it would never get any signal from classes 5-20 so unsure why the model is learning to predict them at all.