Wrong with dataloader?

File “/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py”, line 224, in init
sampler = RandomSampler(dataset, generator=generator)
File “/usr/local/lib/python3.6/dist-packages/torch/utils/data/sampler.py”, line 96, in init
“value, but got num_samples={}”.format(self.num_samples))
ValueError: num_samples should be a positive integer value, but got num_samples=0

Hi. Can you please check that the path to dataset is correct ? I had a similar issue and providing the correct path to dataset solved that.

How can I make the generator give the data I expect, can you give me some guidance?

Hi. So, I don’t think the issue is related to the generator. The generator here is basically a PyTorch generator object that manages the state of the algorithm which produces pseudo random numbers like which device to use (cpu/gpu), state of the generator, setting the seed etc. https://pytorch.org/docs/stable/generated/torch.Generator.html
For your problem, you just need to provide the right path to the dataset.
Can you provide a bit more details here like what is the generator in your case and what kind of dataset are you using?

I am using some pictures labeled as voc dataset (JPEGImages and SegmentationClass), and I want to use it in another model(image and mask).I put it directly into the file.How does the image convert into mask? is there any format?

9 sampledataset
my dataset and the annotated road crack image database that used in the model

Hi. So the images that you have in 1, 2, 3, 4 are basically the segmentation masks. The difference in their color represents different class/label.
It looks like that the model that you are using right now does handle the case when the mask is rgb instead of grayscale.
Just change the mask color from grayscale to rgb if your image has 3 channels.
I hope this gives some clarity.

1 Like

I change maskcolormode to rgb but the image still not load in,
to apply this model what else should I do, can you give me some guidance?

May the reason be that only 4 images can not be divided into the train and val dataset in the network?
So I try part of Voc in this model, but the size of image is not the same, how should I make them the same?

Hi. Yes the number of images could be a reason depending on how you are loading them. To make the images to the same size, you can use a custom transform something like this:

# Transforms for DataLoader
class Resize(object):
    """ Resize Image and/or Masks """

    def __init__(self, height, width):

            height (int): resize to height
            width (int): resize to width

        self.height = height
        self.width = width

    def __call__(self, sample):

            sample (dict): input image/mask pair

            dict: image/mask pair as ndarray

        image, mask = sample['image'], sample['mask']
        dim = (self.width, self.height)
        image = image.resize(dim)
        mask = mask.resize(dim)

        return {'image': image,
                'mask': mask}

NOTE that the images in the above Resize transform are loaded using PIL.Image.

Don’t forget to convert the above images to tensor by either using the ToTensor transform as below:

# Convert Image from Numpy to Tensor
class ToTensor(object):
    """ Convert ndarray to Tensor """

    def __call__(self, sample):

            sample (dict): input image/mask pair

            torch Tensor: image/mask tensor

        image, mask = sample['image'], sample['mask']

        image = np.asarray(image)
        mask = np.asarray(mask)

        if len(image.shape) == 3:
            image = image.transpose(2, 0, 1)

        if len(mask.shape) == 2:
            mask = mask.reshape((1,) + mask.shape)

        if len(image.shape) == 2:
            image = image.reshape((1,) + image.shape)

        return {'image': torch.from_numpy(np.array(image)),
                'mask': torch.from_numpy(np.array(mask))}

Then you can setup these transforms:

transforms.Compose([Resize(330, 422), ToTensor()])

The reason for writing these transforms instead of using the built in ones is due to image as feature and another image as label and we need to apply transforms to both so as to get a image/mask pair.

Hope this helps :slight_smile:

Hi. So, you need to change your dataloader from that GitHub repository as below to make it work with the VOC segmentation dataset.

import torch
import torchvision
from torch.utils.data import Dataset
from PIL import Image
import glob
import numpy as np

class SegmentationDataset(Dataset):
    """Segmentation Dataset"""

    def __init__(self, root_dir: str, image_dir: str, mask_dir: str,
                 transform=None, seed: int = None, fraction: float = None,
                 subset: str = None, imagecolormode: str = 'rgb',
                 maskcolormode: str = 'rgb'):

            root_dir (str): dataset dir path
            image_dir (str): input image dir name
            mask_dir (str): mask image dir name
            transform: PyTorch data transform
            seed (int): random seed for reproducibility
            fraction (float): dataset train/test split percentage
            subset (str): subset from existing dataset
            imagecolormode (str): input image color mode
            maskcolormode (str): input mask color mode


        self.color_dict = {'rgb': 1, 'grayscale': 0}
        assert (imagecolormode in ['rgb', 'grayscale'])
        assert (maskcolormode in ['rgb', 'grayscale'])

        self.imagecolorflag = self.color_dict[imagecolormode]
        self.maskcolorflag = self.color_dict[maskcolormode]
        self.root_dir = root_dir
        self.transform = transform
        if not fraction:
            # UPDATE: Get the Segmentation Masks Before Images
            self.mask_names = sorted(
                glob.glob(os.path.join(self.root_dir, mask_dir, '*')))
            # UPDATE: Get images with the names in the mask_names list but with updated path and '.jpg' extension
            self.image_names = sorted(
                os.path.join(self.root_dir, image_dir, fname.split('/')[4].split('.png')[0] + '.jpg')
                for fname in self.mask_names)
            assert (subset in ['Train', 'Test'])
            self.fraction = fraction

            # UPDATE: Get the Segmentation Masks Before Images
            self.mask_list = np.array(
                sorted(glob.glob(os.path.join(self.root_dir, mask_dir, '*'))))
            # UPDATE: Get images with the names in the mask_names list but with updated path and '.jpg' extension
            self.image_list = np.array(
                sorted(os.path.join(self.root_dir, image_dir, fname.split('/')[4].split('.png')[0] + '.jpg')
                       for fname in self.mask_list))
            if seed:
                indices = np.arange(len(self.image_list))
                self.image_list = self.image_list[indices]
                self.mask_list = self.mask_list[indices]
            if subset == 'Train':
                self.image_names = self.image_list[:int(
                    np.ceil(len(self.image_list) * (1 - self.fraction)))]
                self.mask_names = self.mask_list[:int(
                    np.ceil(len(self.mask_list) * (1 - self.fraction)))]
                self.image_names = self.image_list[int(
                    np.ceil(len(self.image_list) * (1 - self.fraction))):]
                self.mask_names = self.mask_list[int(
                    np.ceil(len(self.mask_list) * (1 - self.fraction))):]

    def __getitem__(self, idx):

            idx (int): index of input image

            dict: image and mask image

        img_name = self.image_names[idx]
        mask_name = self.mask_names[idx]

        image = Image.open(img_name)
        mask = Image.open(mask_name)

        sample = {'image': image, 'mask': mask}

        if self.transform:
            sample = self.transform(sample)

        return sample

    def __len__(self):

        Returns: length of dataset

        return len(self.image_names)

Now you can call the above dataset class as follows:

dataset = SegmentationDataset(root_dir='./VOCdevkit/VOC2012/', image_dir='JPEGImages', mask_dir='SegmentationClass', seed=100, fraction=0.1, subset='Train')

and everything should work. You can test the image/mask pairs using the following code:

import matplotlib.pyplot as plt

for i,data in enumerate(dataset):
    image, mask = data['image'], data['mask']
    show_image_mask(image, mask)
    if i > 5:

Hope this helps :slight_smile:

So grateful for you help and awesome code!
But the train not begin by running the command

May it be that the train and test dataset not splited successfully?

plt.imshow might work, but currently you are passing an invalid image path, thus the image cannot be loaded.

it not work within a loop when the image path is right, it just give no feedback

Ok. So I instantiated the dataset line just to explain the concept and test it out, but you don’t need that. Just make changes to the SegmentationDataset class and that’s it. Nothing else needs to be changed in your code from that GitHub repository. It has the code to instantiate the dataset and split into train and test sets in the get_dataloader_single_folder function.

Also up there, plt.imshow() takes only one image at a time. So if you just do plt.imshow(image), that should work.

So grateful. But when I only change imageFolder to image_dir in SegDataset it gives me TypeError of init(), but if I change it to imageFolder , SegDataset can not works well alone, what cause the problem and how should I define them?

Thanks you very much to teach me the original function, I conduct a breakpoints debugging in colab and the image has been read, but something still wrong with the dataloader, can you give me some suggestions?

The original problem appears and this time for more specific…but I cannot understand this…