Why custom collate_fn is very slow as compared to ToTensor() mehtod

1

Hi,
I am trying to train and check accuracy of my network, in my case if I train using ToTensor function it very fast, almost takes 5 minutes for one epoch, but when i want to evaluate my network and use custom_collate function to read labels which may be empty sometimes, it becomes too slow,
I am using 4 workes.
even if i load the same batch same data with default collate it is very fast but when i use custom collate it bexomes 4 times slower

def pred_collate_fn(data):
file_name = [item[cfg.SAMPLE_FILENAME] for item in data]
pc = [item[cfg.SAMPLE_POINT_CLOUD] for item in data]
voxel_grid = [item[cfg.SAMPLE_VOXEL_GRID].reshape(1, item[cfg.SAMPLE_VOXEL_GRID].shape[0], item[cfg.SAMPLE_VOXEL_GRID].shape[1], item[cfg.SAMPLE_VOXEL_GRID].shape[2]) for item in data]
gt_objectness_map = [item[‘gt_objectness_map’] for item in data]
gt_coordinates_map = [item[‘gt_coordinate_map’] for item in data]
gt_labels = [item[cfg.LABEL_FILE] for item in data]
image = [item[cfg.SAMPLE_IMAGE] for item in data]

return [
    torch.FloatTensor(voxel_grid),  # -- Voxel Rep
    torch.FloatTensor(gt_objectness_map), # -- Objectness
    torch.FloatTensor(gt_coordinates_map),   # -- coordinates
    file_name,
    np.array(pc),
    np.array(gt_labels),
    np.array(image)
]
1 Like