Hi all,
This is my first post here. Hopefully will get some quick feedback. I’m working on crowd counting and density estimation on images using (ShanghaiTech dataset) and using MSE as loss and MAE as metric function. I’m calculating both MAE in train(), validate() and predict() functions as follow (showing predict() function for reference):
def predict(model, dataloader, device):
'''
Make prediction from a pytorch model
'''
# set model to evaluate model
model.eval()
test_images = torch.tensor([], device=device)
test_gt = torch.tensor([], device=device)
test_pred = torch.tensor([], device=device)
epoch_mae = 0.0
# deactivate autograd engine and reduce memory usage and speed up computations
with torch.no_grad():
for i, data in tqdm(enumerate(dataloader), total=int(len(dataloader)/dataloader.batch_size)):
img, gt = data[0].to(device), data[1].to(device) # Read a single batch
outputs = model(img) # predict the output
# Metric
actual_count = gt.sum()
pred_count = outputs.sum()
epoch_mae += abs(actual_count - pred_count) # sum over all batches (try .data.sum())
# Output to return
test_images = torch.cat((test_images, img), 0) # concatenate images in this batch to 'images'
test_gt = torch.cat((test_gt, gt), 0)
test_pred = torch.cat((test_pred, outputs), 0)
# Calc MAE
mae = epoch_mae/len(dataloader.dataset)
# print(f'mae: {mae}')
return test_images, test_gt, test_pred, mae
When run predict() function over test set, I got MAE = 10.6043. And that’s correct as this MAE was calculated as lowest value at which I saved the model during training and then reloaded the saved model before running predict().
But then when I plot prediction performance as follow, I get a higher MAE.
- Find actual and predicted counts for test set.
test_actual_count = []
test_pred_count = []
## Test predictions
for i in range(0, len(test_dm)): # test_ds_unbatch.as_numpy_iterator():
test_actual_count.append(torch.sum(test_dm[i,:]))
test_pred_count.append(torch.sum(test_dm_h[i,:]))
# Lists to Tensors
test_actual_count = torch.tensor(test_actual_count) #.detach().cpu()
test_pred_count = torch.tensor(test_pred_count) #.detach().cpu()
# Calculate MAE
test_mae = abs(test_actual_count - test_pred_count).sum()/len(test_dl.dataset)
The value of ‘test_mae’ calculated in the last line above is 61. Is this strange or am I doing something wrong? (By the way looking into the scatter plot of actual_count vs pred_count, it seems that the value of “test_mae” calculated in the last line is correct, but something is wrong with the value returned by predict().