I don’t know the dataset, but your hunch sounds very plausible. In my experience, it can be tricky to compensate class imbalance through the loss function. In our book we use cropping to reduce the imbalance in the sampling (a long time ago, someone had a similar issue).
Best regards
Thomas