# Questions about Dataloader and Dataset

**URL:** <https://discuss.pytorch.org/t/questions-about-dataloader-and-dataset/806>\
**Category:** Uncategorized\
**Created:** [March 1, 2017, 9:19am UTC](https://discuss.pytorch.org/t/questions-about-dataloader-and-dataset/806 "2017-03-01T09:19:16Z")\
**Posts on this page:** 17\
**Page:** 1

<div class="post-metadata">

**Author:** ![chenyuntc](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/chenyuntc/32/299_2.png) [@chenyuntc](https://discuss.pytorch.org/u/chenyuntc)\
**Post date:** [March 1, 2017, 9:19am UTC](https://discuss.pytorch.org/t/questions-about-dataloader-and-dataset/806/1 "2017-03-01T09:19:17Z")

</div>

I run my dataloader like this:

```python
dataset= ImageFolder('/home/x/data/pre/train/',
                      transform=transforms.Compose([transforms.Scale(opt.image_size),
                                              transforms.RandomCrop(opt.image_size) ,
                                              transforms.ToTensor(),
                                              transforms.Normalize([0.5]*3,[0.5]*3)
                                             ]))
 
dataloader=t.utils.data.DataLoader(dataset,opt.batch_size,True,num_workers=opt.workers)

```

but some images are corrupted, and it raised an error(an PIL error in `default_loader: Image.open(path).convert('RGB')`

my temporary fix is modifying [/torchvision/datasets/folder.py Line 65](https://github.com/pytorch/vision/blob/master/torchvision/datasets/folder.py#L65) to

```python
        try:
            img = self.loader(os.path.join(self.root, path))
        except Exception as e:
            index = index - 1 if index > 0 else index + 1 
            return self. __getitem__ (index)

```

Is there a better way to solve this, i.e. modifying the code of dataloader to load another Image when exception or write a new loader.  
or does it work if I simply return None when caught an exception?

---

<div class="post-metadata">

**Author:** ![smth](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/smth/32/13_2.png) [@smth](https://discuss.pytorch.org/u/smth)\
**Post date:** [March 1, 2017, 11:45am UTC](https://discuss.pytorch.org/t/questions-about-dataloader-and-dataset/806/2 "2017-03-01T11:45:28Z")

</div>

you can look at the default\_collate function if it handles `None` or not.  
Or you can give a custom collate function to your `DataLoader` which will handle `None`:

> <https://github.com/pytorch/pytorch/blob/master/torch/utils/data/dataloader.py#L32>

> <https://github.com/pytorch/pytorch/blob/master/torch/utils/data/dataloader.py#L61-L84>

---

<div class="post-metadata">

**Author:** ![apaszke](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/apaszke/32/21_2.png) [@apaszke](https://discuss.pytorch.org/u/apaszke)\
**Post date:** [March 1, 2017, 11:49am UTC](https://discuss.pytorch.org/t/questions-about-dataloader-and-dataset/806/3 "2017-03-01T11:49:08Z")

</div>

The DataLoader doesn’t support recovery from errors in datasets because it would be too complex to add and keep the guarantee that it will **always return the batches in the exact same order** , as the sampler generated. It’s not even clear what’s the behaviour that user expects (some people might want to be notified about the error, others don’t).

---

<div class="post-metadata">

**Author:** ![chenyuntc](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/chenyuntc/32/299_2.png) [@chenyuntc](https://discuss.pytorch.org/u/chenyuntc)\
**Post date:** [March 1, 2017, 1:50pm UTC](https://discuss.pytorch.org/t/questions-about-dataloader-and-dataset/806/4 "2017-03-01T13:50:44Z")

</div>

thanks @smth@apaszke, that really makes me have deeper comprehension of dataloader.

At first I try:

```python
def my_loader(path):
    try: 
        return Image.open(path).convert('RGB')
    except Exception as e:
        print e
def my_collate(batch):
    "Puts each data field into a tensor with outer dimension batch size"
    batch = filter (lambda x:x is not None, batch)
    return default_collate(batch)

dataset= ImageFolder('/home/x/train/',
            transform=transforms.Compose([transforms.ToTensor()]),
            loader = my_loader)
dataloader=t.utils.data.DataLoader(dataset,4,True,collate_fn=my_collate)

```

it raise exception, because `transforms` in `dataset` can’t handle None

so then I try this：

```auto
def my_collate(batch):
    batch = filter (lambda x:x is not None, batch)
    return default_collate(batch)
class MyImageFolder(ImageFolder):
    __init__ = ImageFolder. __init__
    def __getitem__ (self, index):
        try: 
            return super(MyImageFolder, self). __getitem__ (index)
        except Exception as e:
            print e

dataset= MyImageFolder('/home/x/train/', transform = transforms.Compose([transforms.ToTensor()..]) )
dataloader=t.utils.data.DataLoader(dataset, 4, True, collate_fn=my_collate)

```

not so pythonic, but it works.  
and I think the best way maybe just cleaning the data.

---

<div class="post-metadata">

**Author:** ![PermanentPon](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/permanentpon/32/2904_2.png) [@PermanentPon](https://discuss.pytorch.org/u/PermanentPon)\
**Post date:** [May 9, 2018, 10:18am UTC](https://discuss.pytorch.org/t/questions-about-dataloader-and-dataset/806/5 "2018-05-09T10:18:05Z")

</div>

To solve this particular problem with corrupted images you can just add two lines before your code:

```auto
from PIL import ImageFile
ImageFile.LOAD_TRUNCATED_IMAGES = True

```

as suggested [here](https://stackoverflow.com/a/23575424/1719231)

---

<div class="post-metadata">

**Author:** ![Sun\_ShiJie](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/c89c15/32.png) [@Sun\_ShiJie](https://discuss.pytorch.org/u/Sun_ShiJie)\
**Post date:** [May 19, 2018, 10:47am UTC](https://discuss.pytorch.org/t/questions-about-dataloader-and-dataset/806/6 "2018-05-19T10:47:15Z")

</div>

Hi, Chen. Does the batch size decrease when use this way to filter invalid data.

---

<div class="post-metadata">

**Author:** ![msamogh](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/msamogh/32/6861_2.png) [@msamogh](https://discuss.pytorch.org/u/msamogh)\
**Post date:** [October 11, 2018, 9:25am UTC](https://discuss.pytorch.org/t/questions-about-dataloader-and-dataset/806/7 "2018-10-11T09:25:36Z")

</div>

I wrote a PyTorch library that allows you to do exactly this - drop samples, handle bad items, use Transforms as Filters (and more). **Check out [nonechucks](https://github.com/msamogh/nonechucks).**

Using nonechucks, your code would look something like this:

```python
dataset= ImageFolder('/home/x/data/pre/train/',
                      transform=transforms.Compose([transforms.Scale(opt.image_size),
                                              transforms.RandomCrop(opt.image_size) ,
                                              transforms.ToTensor(),
                                              transforms.Normalize([0.5]*3,[0.5]*3)
                                             ]))

import nonechucks as nc
dataset = nc.SafeDataset(dataset)
dataloader = nc.SafeDataLoader(dataset,opt.batch_size,True,num_workers=opt.workers)

# You can now use `dataloader` as though it was a regular DataLoader without
# having to worry about the bad samples!

```

Feel free to check out the documentation on the [Github page](https://github.com/msamogh/nonechucks)!

---

<div class="post-metadata">

**Author:** ![nimning](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/n/bcef8e/32.png) [@nimning](https://discuss.pytorch.org/u/nimning)\
**Post date:** [November 10, 2018, 10:41pm UTC](https://discuss.pytorch.org/t/questions-about-dataloader-and-dataset/806/8 "2018-11-10T22:41:12Z")

</div>

> [@chenyuntc](#):
>
> default\_collate

How you get the internal function ‘default\_collate’? It is within the ‘dataloder.py’.

---

<div class="post-metadata">

**Author:** ![tom](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tom/32/3162_2.png) [@tom](https://discuss.pytorch.org/u/tom)\
**Post date:** [November 10, 2018, 10:52pm UTC](https://discuss.pytorch.org/t/questions-about-dataloader-and-dataset/806/9 "2018-11-10T22:52:04Z")

</div>

It’s imported under that module, just not directly under `.data`:

```python
import torch
import torch.utils.data
print(torch.utils.data.dataloader.default_collate)

```

Best regards

Thomas

---

<div class="post-metadata">

**Author:** ![zanaa](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/z/e19b73/32.png) [@zanaa](https://discuss.pytorch.org/u/zanaa)\
**Post date:** [December 15, 2018, 6:56pm UTC](https://discuss.pytorch.org/t/questions-about-dataloader-and-dataset/806/10 "2018-12-15T18:56:02Z")

</div>

Note: for python 3, replace

`batch = filter(lambda x : x is not None, batch)`

with

`batch = list(filter(lambda x : x is not None, batch))`

---

<div class="post-metadata">

**Author:** ![Shivam\_Chandhok](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shivam_chandhok/32/10384_2.png) [@Shivam\_Chandhok](https://discuss.pytorch.org/u/Shivam_Chandhok)\
**Post date:** [March 25, 2019, 12:18pm UTC](https://discuss.pytorch.org/t/questions-about-dataloader-and-dataset/806/11 "2019-03-25T12:18:32Z")

</div>

![36%20PM](https://discuss.pytorch.org/uploads/default/original/2X/a/ae8628a2aab8a0db0a7ea5eb256957acf707e0a8.png) ![46%20PM](https://discuss.pytorch.org/uploads/default/original/2X/3/36f43ad452a09ae4ca32b0a2d3ae30c751ad7480.png) ![54%20PM](https://discuss.pytorch.org/uploads/default/original/2X/e/ead78cdaf9cad77c8566254e352f1ff79180fcc4.png)

I am getting an error:

samples = collate\_fn([dataset[i] for i in batch\_indices])  
TypeError: ‘DataLoader’ object does not support indexing

What am I doing wrong.

---

<div class="post-metadata">

**Author:** ![weiwei](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/w/f9ae1b/32.png) [@weiwei](https://discuss.pytorch.org/u/weiwei)\
**Post date:** [June 7, 2019, 11:44am UTC](https://discuss.pytorch.org/t/questions-about-dataloader-and-dataset/806/12 "2019-06-07T11:44:55Z")

</div>

hi, i have similar problem about share memory, could you help me to solve the problem ? thanks very much, the topic link is :

> [@Can we disable share memory usage in pytorch?](https://discuss.pytorch.org/t/can-we-disable-share-memory-usage-in-pytorch/47342):
>
> since i am not able to adjust the share memory usage in the remote server, can we disable share memory usage in pytorch. the same experiment run with tensorflow without shm size problem, so i just want to find a solution for this problem.

---

<div class="post-metadata">

**Author:** ![Martin36](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/dc4da7/32.png) [@Martin36](https://discuss.pytorch.org/u/Martin36)\
**Post date:** [July 14, 2021, 7:13pm UTC](https://discuss.pytorch.org/t/questions-about-dataloader-and-dataset/806/13 "2021-07-14T19:13:48Z")

</div>

This does not work if the batch size is 1, since `default_collate` will be called with an empty list in that case. The same thing could happen if the batch size is larger and all the data samples in the batch size are corrupt (highly unlikely though).

Is there a way to get this to work when the batch size is only 1?

---

<div class="post-metadata">

**Author:** ![Keyv\_Krmn](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/keyv_krmn/32/13975_2.png) [@Keyv\_Krmn](https://discuss.pytorch.org/u/Keyv_Krmn)\
**Post date:** [July 28, 2021, 11:35pm UTC](https://discuss.pytorch.org/t/questions-about-dataloader-and-dataset/806/14 "2021-07-28T23:35:08Z")

</div>

I faced the same issue. my hot fix was to return a datapoint drawn randomly from the dataset.  
so basically, catch the error and return a valid batch with a single datapoint.  
It can happen in other cases where the batch\_size \> 1,but last batch has only 1 datapoint.

---

<div class="post-metadata">

**Author:** ![xu555](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/x/f08c70/32.png) [@xu555](https://discuss.pytorch.org/u/xu555)\
**Post date:** [December 26, 2021, 2:55am UTC](https://discuss.pytorch.org/t/questions-about-dataloader-and-dataset/806/15 "2021-12-26T02:55:40Z")

</div>

from torch.utils.data.dataloader import default\_collate

---

<div class="post-metadata">

**Author:** ![Syzygianinfern0](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/syzygianinfern0/32/46517_2.png) [@Syzygianinfern0](https://discuss.pytorch.org/u/Syzygianinfern0)\
**Post date:** [February 10, 2022, 1:57pm UTC](https://discuss.pytorch.org/t/questions-about-dataloader-and-dataset/806/16 "2022-02-10T13:57:54Z")

</div>

Did you find any fix?

Edit: I found a fix [here](https://stackoverflow.com/a/67583699/8878627). It has many other relevant answers as well.

---

<div class="post-metadata">

**Author:** ![Anirudh257](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/anirudh257/32/66663_2.png) [@Anirudh257](https://discuss.pytorch.org/u/Anirudh257)\
**Post date:** [January 8, 2024, 8:58pm UTC](https://discuss.pytorch.org/t/questions-about-dataloader-and-dataset/806/17 "2024-01-08T20:58:16Z")

</div>

I also found a fix inspired by [here](https://stackoverflow.com/questions/57815001/pytorch-collate-fn-reject-sample-and-yield-another/69578320#69578320).

```auto
def collate_fn_replace_corrupted(batch, dataset):
    if batch[0] is None:
        batch = [dataset[randint(0, len(dataset) - 1)]]
        return collate_fn_replace_corrupted(batch, dataset)

    return torch.utils.data.dataloader.default_collate(batch)

```

```auto
  collate_fn = functools.partial(collate_fn_replace_corrupted, dataset = dataset)

```