Different MAE values from predict() versus actual calculated form data

Hi all,

This is my first post here. Hopefully will get some quick feedback. I’m working on crowd counting and density estimation on images using (ShanghaiTech dataset) and using MSE as loss and MAE as metric function. I’m calculating both MAE in train(), validate() and predict() functions as follow (showing predict() function for reference):

def predict(model, dataloader, device):
    Make prediction from a pytorch model 
    # set model to evaluate model
    test_images = torch.tensor([], device=device)
    test_gt     = torch.tensor([], device=device)
    test_pred   = torch.tensor([], device=device)
    epoch_mae   = 0.0
    # deactivate autograd engine and reduce memory usage and speed up computations
    with torch.no_grad():
        for i, data in tqdm(enumerate(dataloader), total=int(len(dataloader)/dataloader.batch_size)):
            img, gt = data[0].to(device), data[1].to(device)  # Read a single batch
            outputs        = model(img)  # predict the output
            # Metric
            actual_count = gt.sum()
            pred_count   = outputs.sum()
            epoch_mae     += abs(actual_count - pred_count) # sum over all batches (try .data.sum())
            # Output to return
            test_images  = torch.cat((test_images, img), 0)  # concatenate images in this batch to 'images'
            test_gt      = torch.cat((test_gt, gt), 0)
            test_pred    = torch.cat((test_pred, outputs), 0)
        # Calc MAE
        mae  = epoch_mae/len(dataloader.dataset)
        # print(f'mae: {mae}')
    return test_images, test_gt, test_pred, mae

When run predict() function over test set, I got MAE = 10.6043. And that’s correct as this MAE was calculated as lowest value at which I saved the model during training and then reloaded the saved model before running predict().

But then when I plot prediction performance as follow, I get a higher MAE.

  1. Find actual and predicted counts for test set.
test_actual_count  = []
test_pred_count    = []
## Test predictions
for i in range(0, len(test_dm)): # test_ds_unbatch.as_numpy_iterator():

# Lists to Tensors
test_actual_count  = torch.tensor(test_actual_count) #.detach().cpu()
test_pred_count    = torch.tensor(test_pred_count) #.detach().cpu()

# Calculate MAE
test_mae = abs(test_actual_count - test_pred_count).sum()/len(test_dl.dataset)

The value of ‘test_mae’ calculated in the last line above is 61. Is this strange or am I doing something wrong? (By the way looking into the scatter plot of actual_count vs pred_count, it seems that the value of “test_mae” calculated in the last line is correct, but something is wrong with the value returned by predict().