How to create 3D custom dataloader

[saint_codded]

I am trying to create a custom dataloader for 3D data in pytorch. I have chunked data of size (10,1,10,512,512) meaning (N, C, D, H, W).
Any idea?..please assist

[ptrblck]

This tutorial shows you an example on how to write a custom Dataset. Did you already check it and if so, are you stuck somewhere?

[saint_codded]

Thank you so much for your response, i could do that for 2D data. However, i am currently working on 3D lung CT scans, i am actually aiming to train a classification model using a 3D CNN.
I was able to stack the entire data in a numpy array. For example, for 10 paients with 10 CT slices each, i stacked it into an array of (10, 1,10,512,512)=(N,C,D,H,W)
I could also do a train/validation split. But i am stuck on how i will match each patient data to the corresponding label of either normal or diseased and how i could join both data and label to form a data loader. The labels are in a csv file as either 0 or 1.
I hope you understand. And am sorry this is already looking like a long essay.
Thank you for taking time to assist me with this.

[J_Johnson]

You can build a custom dataloader, if you plan to store the data in GPU or CPU RAM. And then just create an index that you can shuffle and use to get the batches. Here is an example:

Use case for loading the entire dataset into RAM

Just a suggestion: don’t use the vanilla dataloader for this. If you can fit the data all into ram, just turn it into a tensor for each set of data, shape being: self.dataset.size()=([data_index, ...]) Where each data_index is a sample. Then create an indexer, either in NumPy or Pytorch. Here is an example for NumPy. rng=np.random.default_rng() ... self.data_index=np.arange(self.dataset.size()[0]) Then define a shuffler class you can call whenever you want to shuffle the index: def shuffl…

[saint_codded]

Thank you @J_Johnson. Well, I am a beginner so working my way through your suggestion to fit my situation seems a bit hectic. This is the code after chunking the CT scans of 10 patients into a NumPy array. Now my problem is how I will match each patient to a label of either 0 or 1 and create a dataloader to train a network in Pytorch. your assistance would be much appreciated.
Thank you

[J_Johnson]

The trick is you want to keep your data and labels unshuffled. You only shuffle the index and apply it to both. Suppose this time the 4th value of the index is 8 after shuffling. By inserting that index into the 0 dim of both the data and labels, we get back the 8th data sample and 8th label.

In fact, you could probably just copy the example I sent you, throw it into a class, and make an instance of it. It should already generalize to 3D data, as long as your samples are in dim 0.

[saint_codded]

Thank you again. I think it is my novice self not knowing what to do. I tried to apply the code you suggested, here is what I got. only if you are not tired of me already!!!

[J_Johnson]

You need a class wrapper to instantiate your data, labels, and the self reference.

Can you copy your code here? I’ll edit it later. Not in front of a computer now.