# Custom Dataset class for medical image patches

**URL:** <https://discuss.pytorch.org/t/custom-dataset-class-for-medical-image-patches/144940>\
**Category:** vision\
**Created:** [February 24, 2022, 4:23pm UTC](https://discuss.pytorch.org/t/custom-dataset-class-for-medical-image-patches/144940 "2022-02-24T16:23:33Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![AmayaGS](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/c37758/32.png) [@AmayaGS](https://discuss.pytorch.org/u/AmayaGS)\
**Post date:** [February 24, 2022, 4:23pm UTC](https://discuss.pytorch.org/t/custom-dataset-class-for-medical-image-patches/144940/1 "2022-02-24T16:23:33Z")

</div>

Hi,

I’m trying to define a custom Dataset for my medical images. For each patient, I have a histopathology Whole Slide Images (WSI). I preprocess each image, obtaining a variable number of 224 x 224 patches and keep them in a folder with the patient ID. The label for each patch in a given patient folder is the label assigned to the WSI. The data directory structure is as follows:

![image](https://discuss.pytorch.org/uploads/default/original/3X/b/b/bbf9cd420a150e8f3a4eb3b459c93a29307b32b0.png)

I want to write a custom Dataset that will load each patch, with the correct label and feed it into DataLoader, in this case treating each patch as independent. My current implementation return a list of images and list of labels, as so:

> class histoDataset(Dataset):
> 
> ```
> def __init__ (self, main_dir, csv_file, transform):
> 
> self.main_dir = main_dir 
> self.transform = transform 
> self.CSV = pd.read_csv(csv_file) 
> self.folder_id_code = self.CSV['Patient image ID'].tolist() 
> self.image_diagnosis = self.CSV['Classification.based.on.CD20'].tolist()
> self.classes = list(dict.fromkeys(self.image_diagnosis))
> self.label = self.CSV['Classification.based.on.CD20'].apply(self.classes.index)
> self.label_dict = dict(zip(self.label, self.classes))
> self.folders = [f.path for f in os.scandir(self.main_dir)]
> 
> def __len__ (self):
> return len(self.folders)
> 
> def __getitem__ (self, idx): 
> image_list = []
> image_labels = []
> self.folder_loc = os.path.join(self.main_dir, self.folder_id_code[idx])
> self.folder_files = [f.path for f in os.scandir(self.folder_loc)]
> for image in self.folder_files:
> image = Image.open(image) 
> image_tensor = self.transform(image) 
> image_label = self.label[idx] 
> image_list.append(image_tensor)
> image_labels.append(image_label)
>         
> return image_list, image_labels
> 
> ```

After inputting the resulting histoDataset into DataLoder and running this:

> sample\_batch = next(iter(df\_loader))  
> sample\_images, sample\_labels = sample\_batch  
> print\_images = sample\_images[0:10]  
> print\_labels = sample\_labels[0:10]

to check the images and labels, I get the following error:

```auto
Traceback (most recent call last):

  File "C:\Users\AmayaGS\Documents\PhD\Data\Dataset_loader.py", line 248, in <module>
    sample_batch = next(iter(df_loader))

  File "C:\ProgramData\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 521, in __next__
    data = self._next_data()

  File "C:\ProgramData\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 561, in _next_data
    data = self._dataset_fetcher.fetch(index) # may raise StopIteration

  File "C:\ProgramData\Anaconda3\lib\site-packages\torch\utils\data\_utils\fetch.py", line 47, in fetch
    return self.collate_fn(data)

  File "C:\ProgramData\Anaconda3\lib\site-packages\torch\utils\data\_utils\collate.py", line 84, in default_collate
    return [default_collate(samples) for samples in transposed]

  File "C:\ProgramData\Anaconda3\lib\site-packages\torch\utils\data\_utils\collate.py", line 84, in <listcomp>
    return [default_collate(samples) for samples in transposed]

  File "C:\ProgramData\Anaconda3\lib\site-packages\torch\utils\data\_utils\collate.py", line 82, in default_collate
    raise RuntimeError('each element in list of batch should be of equal size')

RuntimeError: each element in list of batch should be of equal size

```

Could someone please point out how to correctly implement the histoDataset and why I am getting this error? Thanks!

---

<div class="post-metadata">

**Author:** ![Dazitu616](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/dazitu616/32/39023_2.png) [@Dazitu616](https://discuss.pytorch.org/u/Dazitu616)\
**Post date:** [February 25, 2022, 6:10am UTC](https://discuss.pytorch.org/t/custom-dataset-class-for-medical-image-patches/144940/2 "2022-02-25T06:10:55Z")

</div>

Maybe something is wrong in the `transform` of your dataset. Can you post it here?

---

<div class="post-metadata">

**Author:** ![AmayaGS](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/c37758/32.png) [@AmayaGS](https://discuss.pytorch.org/u/AmayaGS)\
**Post date:** [February 25, 2022, 9:17am UTC](https://discuss.pytorch.org/t/custom-dataset-class-for-medical-image-patches/144940/3 "2022-02-25T09:17:25Z")

</div>

This is how I use my `transform`

> import torchvision  
> import torchvision.transforms as transforms

> transforms = transforms.Compose([  
> transforms.Resize((224, 224)),  
> transforms.ToTensor(),  
> transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))  
> ])

> df = histoDataset(main\_dir, csv\_file, transforms)

> df\_loader = torch.utils.data.DataLoader(df, batch\_size=10, shuffle=True, num\_workers=0, drop\_last=False)

---

<div class="post-metadata">

**Author:** ![Dazitu616](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/dazitu616/32/39023_2.png) [@Dazitu616](https://discuss.pytorch.org/u/Dazitu616)\
**Post date:** [February 25, 2022, 4:52pm UTC](https://discuss.pytorch.org/t/custom-dataset-class-for-medical-image-patches/144940/4 "2022-02-25T16:52:57Z")

</div>

The transform looks good to me. The error message basically says that some of the loaded data are of different shapes. I think it should be `image_list` not `image_labels`.  
Is it possible that `self.folder_files` can have different number of items when loading with different `idx`? For example, sometimes a folder contains 2 images, while sometimes it contains 4. Then the loaded data shape will be [2, 3, 224, 224] and [4, 2, 224, 224], which causes the problem.

---

<div class="post-metadata">

**Author:** ![aiGeekProgrammer](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/aigeekprogrammer/32/47179_2.png) [@aiGeekProgrammer](https://discuss.pytorch.org/u/aiGeekProgrammer)\
**Post date:** [February 26, 2022, 1:28pm UTC](https://discuss.pytorch.org/t/custom-dataset-class-for-medical-image-patches/144940/5 "2022-02-26T13:28:11Z")

</div>

I’m not sure if you structure this data loading correctly.  
If I understand this right, the number of patches may be different from patient to patient.  
Consequently, your _image\_list, image\_labels_ variables that you return from _ **getitem** _ are of different length.  
That might be the reason why you get: _RuntimeError: each element in list of batch should be of equal size_  
As I don’t have this data and the code running in my notebook, this is just a vague hypothesis.

**Did you consider getitem returning 1 patch only?**