Importing COCO format label with segmentation mask


I am working on writing my own dataloader with labels provided in COCO json format. The task is to perform semantic segmentation. I would like to return the annotation list (segmentation mask) from the custom class that I have written but the mask for images can be of different size. When I load the dataset and iterate through the loader, I get the following error:
“RuntimeError: each element in list of batch should be of equal size”

I understand that this is because the segmentation mask for each image has different number of vertices.
How may I be able to solve this? is there a work-around?