mIOU decreases with increasing epoch

I am training a model for semi-supervised semantic segmentation using resnet as backbone and feature pyramid network as decoder .Cityscapes dataset is used. As epoch increases, mean IOU and class average accuracy decreases but overall accuracy increases. Cross entropy loss also decreases. I have kept learning rate very small 1e-5. What might be the reason behind this?
Epoch1: CE Loss : 3.07788 Overall ACC: 0.8059 | Mean ACC: 0.3714 | mIoU: 0.2459
Epoch2: CE Loss : 2.74041 Overall ACC: 0.8323 | Mean ACC: 0.3008 | mIoU: 0.2322
Epoch3: CE Loss : 2.54747 Overall ACC: 0.8435 | Mean ACC: 0.2959 | mIoU: 0.2302
Epoch4: CE Loss : 2.50084 Overall ACC: 0.8490 | Mean ACC: 0.2842 | mIoU: 0.2173

Here’s the code for calculating accuracy and mean iou

def _fast_hist(label_true, label_pred, n_class):
    mask = (label_true >= 0) & (label_true < n_class) # Exclude unlabelled data.
    hist = np.bincount(n_class * label_true[mask] + label_pred[mask],\
                       minlength=n_class ** 2).reshape(n_class, n_class)
    
    return hist


def scores(label_trues, label_preds, n_class):
    hist = np.zeros((n_class, n_class))
    for lt, lp in zip(label_trues, label_preds):
        hist += _fast_hist(lt.flatten(), lp.flatten(), n_class)
    return hist


def get_result_metrics(histogram):
    tp = np.diag(histogram)
    fp = np.sum(histogram, 0) - tp
    fn = np.sum(histogram, 1) - tp 

    iou = tp / (tp + fp + fn)
    prc = tp / (tp + fn) 
    opc = np.sum(tp) / np.sum(histogram)

    result = {"iou": iou,
              "mean_iou": np.nanmean(iou),
              "precision_per_class (per class accuracy)": prc,
              "mean_precision (class-avg accuracy)": np.nanmean(prc),
              "overall_precision (pixel accuracy)": opc}


    return result
type or paste code here

Hi Rakibul!

The mismatch between your class-average and “overall” accuracies suggests
that you have an unbalanced dataset. How many classes do you have, and
what are the pixel counts (or percentages) for each class? Be alert to the
possibility that your “background” class could be highly overrepresented.

If your dataset is unbalanced, consider using CrossEntropyLoss’s weight
constructor argument to account for the imbalance.

Best.

K. Frank