Hi everyone! I am kind of new in deep learning and pytorch. That’s why I’m having some issues and thought to address them here…
I need to create a dataset object for loading the data afterwards as a set of labeled bags (Multiple instance learning). Each bag will contain 10 images. And to label the bags (directories), I created a CSV file with two columns - one holding the directory name and the other some certain value as a label (regression task). How can I code the dataset object, such that when I provide it to the DataLoader, my data will be loaded bag-wise?
I could not find any piece of code or any link which could help me further. Any help would be much appreciated!
Could you explain the use case of creating the bags a bit more, please?
If I understand it correctly, you would like to sample from specific indices to create these bags?
Would you like to return a data tensor in [batch_size, 10, ...] or would 10 already be the batch size?
Thank you for your answer @ptrblck! Sorry for not explaining it more concisely. Actually, my batch size should be 10, meaning that it will have 10 bags per batch. And the bags will contain a specific number of images, let’s say 15. Each bag is labelled by a number, from a CSV file.
Is this according to you more reasonable? Or you think sampling from loaded images to create bags makes more sense…
Hi, I am doing the same project with medical data.
The images are super large with a resolution of 5000*5000, to solve this I made patches of it and saved in folders. Then created a bag for each folder images
my bag has more than 187 images which I couldn’t in the gpu memory. Is there any other way to solve it ?
I’m not fully understand what “bad” means in this use case.
If your data still doesn’t fit into the GPU memory, you might need to reduce the patch sizes further.
Hi @ptrblck actually bag means set of images. Like in HISTOPATHOLOGY one works with very large images called WSIs.
In pathology WSI is given a label. It’s now has become standard in histology to make bags of images for each WSI and give them a weak label. It’s weakly supervised.
What’s the major issues is image model work with (b,c,h,w). The MIL thing works with (1,num-patches,c,h,w) the major disadvantage is they work on 1-batch size.
People are here in this might want to work with multiple batch size as far as I know there are not definate ways around it but grad accumulation can be one of the methods to address multibatch training
