# Dataloader re-initialize dataset after each iteration?

**URL:** <https://discuss.pytorch.org/t/dataloader-re-initialize-dataset-after-each-iteration/32658>\
**Category:** vision\
**Created:** [December 20, 2018, 11:07am UTC](https://discuss.pytorch.org/t/dataloader-re-initialize-dataset-after-each-iteration/32658 "2018-12-20T11:07:53Z")\
**Posts on this page:** 18\
**Page:** 1

<div class="post-metadata">

**Author:** ![Zhang\_Chi](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zhang_chi/32/1582_2.png) [@Zhang\_Chi](https://discuss.pytorch.org/u/Zhang_Chi)\
**Post date:** [December 20, 2018, 11:07am UTC](https://discuss.pytorch.org/t/dataloader-re-initialize-dataset-after-each-iteration/32658/1 "2018-12-20T11:07:54Z")

</div>

Hi  
I write a dataset class, which has a dictionary called image\_pool. Each time the **getitem** function is called, I will first check whether the image exists in the pool. If not, load from the disk and save it into the pool. However, i find that, in the second iteration the dictionary becomes empty and so on in all later iterations. What is the reason?

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [December 20, 2018, 11:10am UTC](https://discuss.pytorch.org/t/dataloader-re-initialize-dataset-after-each-iteration/32658/2 "2018-12-20T11:10:47Z")

</div>

If you use multiple workers an inplace manipulation of your `Dataset` won’t be saved as far as I know, since each worker will get a copy of your `Dataset`.  
We had a similar discussion about caching some data and I’ve created a small example using shared arrays [here](https://discuss.pytorch.org/t/dataloader-resets-dataset-state/27960/4?u=ptrblck).  
Maybe this could also help in your case.

---

<div class="post-metadata">

**Author:** ![Zhang\_Chi](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zhang_chi/32/1582_2.png) [@Zhang\_Chi](https://discuss.pytorch.org/u/Zhang_Chi)\
**Post date:** [December 21, 2018, 3:43am UTC](https://discuss.pytorch.org/t/dataloader-re-initialize-dataset-after-each-iteration/32658/3 "2018-12-21T03:43:22Z")

</div>

Hi, thank you so much for you reply. However, in my project, the images are of different sizes which can not be saved as arrays with same shape. Do you have any suggestions?

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [December 21, 2018, 12:19pm UTC](https://discuss.pytorch.org/t/dataloader-re-initialize-dataset-after-each-iteration/32658/4 "2018-12-21T12:19:07Z")

</div>

That’s an interesting use case. You are right, variable sized input won’t work with my first approach.  
However, luckily Python provides some implementation for a shared dict.  
Here is a small example using it:

```python
from multiprocessing import Manager

import torch
from torch.utils.data import Dataset, DataLoader

class MyDataset(Dataset):
    def __init__ (self, shared_dict, length):
        self.shared_dict = shared_dict
        self.length = length
        
    def __getitem__ (self, index):
        if index not in self.shared_dict:
            print('Adding {} to shared_dict'.format(index))
            self.shared_dict[index] = torch.tensor(index)
        return self.shared_dict[index]
        
    def __len__ (self):
        return self.length

# Init
manager = Manager()
shared_dict = manager.dict()
dataset = MyDataset(shared_dict, length=100)

loader = DataLoader(
    dataset,
    batch_size=10,
    num_workers=6,
    shuffle=True,
    pin_memory=True
)

# First loop will add data to the shared_dict
for x in loader:
    print(x)

# The second loop will just get the data
for x in loader:
    print(x)

```

Would that work for you? I guess you are using a custom collate function to create your batch?

---

<div class="post-metadata">

**Author:** ![Zhang\_Chi](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zhang_chi/32/1582_2.png) [@Zhang\_Chi](https://discuss.pytorch.org/u/Zhang_Chi)\
**Post date:** [December 23, 2018, 7:29am UTC](https://discuss.pytorch.org/t/dataloader-re-initialize-dataset-after-each-iteration/32658/5 "2018-12-23T07:29:51Z")

</div>

Hi, thank you very much for your kind reply. It seems the dictionary can not store PIL image, but can sore numpy array and Tensor. I prefer saving PIL because the inbuilt image transformations mainly work for PIL image. A stupid solution is transform it to numpy array when saving it, and transform it back to PIL when loading it. Any better idea?

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [December 23, 2018, 2:03pm UTC](https://discuss.pytorch.org/t/dataloader-re-initialize-dataset-after-each-iteration/32658/6 "2018-12-23T14:03:47Z")

</div>

Are you sure the `dict` throws this error?  
It seems to work on my machine.  
I guess the `DataLoader`’s `default_collate` method throws an error like

> TypeError: batch must contain tensors, numbers, dicts or lists

If you really want to return `PIL.Images`, you could write your own `collate_fn`.  
Usually you would apply the transformation in ` __getitem__ ` and store the processed sample.  
Wouldn’t that work

---

<div class="post-metadata">

**Author:** ![Zhang\_Chi](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zhang_chi/32/1582_2.png) [@Zhang\_Chi](https://discuss.pytorch.org/u/Zhang_Chi)\
**Post date:** [December 24, 2018, 5:17am UTC](https://discuss.pytorch.org/t/dataloader-re-initialize-dataset-after-each-iteration/32658/7 "2018-12-24T05:17:08Z")

</div>

it works now. thank you so much

---

<div class="post-metadata">

**Author:** ![Zhang\_Chi](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zhang_chi/32/1582_2.png) [@Zhang\_Chi](https://discuss.pytorch.org/u/Zhang_Chi)\
**Post date:** [December 24, 2018, 5:55am UTC](https://discuss.pytorch.org/t/dataloader-re-initialize-dataset-after-each-iteration/32658/8 "2018-12-24T05:55:59Z")

</div>

hi, the problem is like this. When I manually call getitem function, it works fine. But when I use dataloader, it breaks down.  
Bascially, I save the PIL images in the shared dict. Every time I call getitem,i take one from the dictionary and apply transformation (random scale, crop, ToTensor) to it and return . what is the problem?

```auto
Traceback (most recent call last):
  File "/home/zhangchi/pycharm-2017.3.4/helpers/pydev/pydev_run_in_console.py", line 53, in run_file
    pydev_imports.execfile(file, globals, locals) # execute the script
  File "/home/zhangchi/pycharm-2017.3.4/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/home/zhangchi/Dropbox/iccv/data/train/dataset_mask_train_memory.py", line 211, in <module>
    for data in trainloader:
  File "/home/zhangchi/anaconda3/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 267, in __next__
    return self._process_next_batch(batch)
  File "/home/zhangchi/anaconda3/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 301, in _process_next_batch
    raise batch.exc_type(batch.exc_msg)
multiprocessing.managers.RemoteError: 
---------------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/zhangchi/anaconda3/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 55, in _worker_loop
    samples = collate_fn([dataset[i] for i in batch_indices])
  File "/home/zhangchi/anaconda3/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 55, in <listcomp>
    samples = collate_fn([dataset[i] for i in batch_indices])
  File "/home/zhangchi/Dropbox/iccv/data/train/dataset_mask_train_memory.py", line 129, in __getitem__
    support_rgb,support_mask=self.load_pil(support_name,sample_class)
  File "/home/zhangchi/Dropbox/iccv/data/train/dataset_mask_train_memory.py", line 97, in load_pil
    self.shared_dict[(name, sample_class)] = rgb, mask
  File "<string>", line 2, in __setitem__
  File "/home/zhangchi/anaconda3/lib/python3.6/multiprocessing/managers.py", line 772, in _callmethod
    raise convert_to_error(kind, result)
multiprocessing.managers.RemoteError: 
---------------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/zhangchi/anaconda3/lib/python3.6/multiprocessing/managers.py", line 228, in serve_client
    request = recv()
  File "/home/zhangchi/anaconda3/lib/python3.6/multiprocessing/connection.py", line 251, in recv
    return _ForkingPickler.loads(buf.getbuffer())
AttributeError: Can't get attribute 'JpegImageFile' on <module 'PIL.JpegImagePlugin' from '/home/zhangchi/anaconda3/lib/python3.6/site-packages/PIL/JpegImagePlugin.py'>
---------------------------------------------------------------------------
---------------------------------------------------------------------------

```

---

<div class="post-metadata">

**Author:** ![Zhang\_Chi](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zhang_chi/32/1582_2.png) [@Zhang\_Chi](https://discuss.pytorch.org/u/Zhang_Chi)\
**Post date:** [December 24, 2018, 6:05am UTC](https://discuss.pytorch.org/t/dataloader-re-initialize-dataset-after-each-iteration/32658/9 "2018-12-24T06:05:05Z")

</div>

I modify your example a little bit to show the problem.

```auto
from multiprocessing import Manager
import torchvision
import torch
from torch.utils.data import Dataset, DataLoader
from PIL import Image

class MyDataset(Dataset):
    def __init__ (self, shared_dict, length):
        self.shared_dict = shared_dict
        self.length = length
        self.scale=torchvision.transforms.Resize([321,321])
        self.ToTensor=torchvision.transforms.ToTensor()
    def __getitem__ (self, index):
        if index not in self.shared_dict:
            print('Adding {} to shared_dict'.format(index))
            self.shared_dict[index] = Image.open('test_img.png')#torch.FloatTensor([index])

        return self.ToTensor(self.scale(self.shared_dict[index]))
    def __len__ (self):
        return self.length

# Init
manager = Manager()
shared_dict = manager.dict()
dataset = MyDataset(shared_dict, length=100)

loader = DataLoader(
    dataset,
    batch_size=2,
    num_workers=6,
    shuffle=True,
    pin_memory=True
)

# First loop will add data to the shared_dict
for x in loader:
    
    pass

# The second loop will just get the data
for x in loader:
    pass

```

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [December 24, 2018, 6:07am UTC](https://discuss.pytorch.org/t/dataloader-re-initialize-dataset-after-each-iteration/32658/10 "2018-12-24T06:07:19Z")

</div>

Would it work if you apply both transformations on the image before saving it to the dict, such that the tensor will be stored instead of the image?

---

<div class="post-metadata">

**Author:** ![Zhang\_Chi](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zhang_chi/32/1582_2.png) [@Zhang\_Chi](https://discuss.pytorch.org/u/Zhang_Chi)\
**Post date:** [December 24, 2018, 6:10am UTC](https://discuss.pytorch.org/t/dataloader-re-initialize-dataset-after-each-iteration/32658/11 "2018-12-24T06:10:58Z")

</div>

if tensor is saved in the dict, it works well, but the transformations are random (crop, scale). it does not make sense to save the transformed ones. ☹

---

<div class="post-metadata">

**Author:** ![Zhang\_Chi](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zhang_chi/32/1582_2.png) [@Zhang\_Chi](https://discuss.pytorch.org/u/Zhang_Chi)\
**Post date:** [December 24, 2018, 6:23am UTC](https://discuss.pytorch.org/t/dataloader-re-initialize-dataset-after-each-iteration/32658/12 "2018-12-24T06:23:54Z")

</div>

it’s weird that if num\_worker is 1, it is ok

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [December 24, 2018, 7:11am UTC](https://discuss.pytorch.org/t/dataloader-re-initialize-dataset-after-each-iteration/32658/13 "2018-12-24T07:11:47Z")

</div>

Thanks for debugging.  
I think the easiest way would be to store the data as a numpy array (or tensor) and convert it back to a `PIL.Image` for the random transformations.

---

<div class="post-metadata">

**Author:** ![Zhang\_Chi](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zhang_chi/32/1582_2.png) [@Zhang\_Chi](https://discuss.pytorch.org/u/Zhang_Chi)\
**Post date:** [December 24, 2018, 7:12am UTC](https://discuss.pytorch.org/t/dataloader-re-initialize-dataset-after-each-iteration/32658/14 "2018-12-24T07:12:42Z")

</div>

Maybe. I will try. Thank you for your help.

---

<div class="post-metadata">

**Author:** ![Rohan\_Rathore](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rohan_rathore/32/37896_2.png) [@Rohan\_Rathore](https://discuss.pytorch.org/u/Rohan_Rathore)\
**Post date:** [October 3, 2021, 6:20pm UTC](https://discuss.pytorch.org/t/dataloader-re-initialize-dataset-after-each-iteration/32658/15 "2021-10-03T18:20:29Z")

</div>

Hi @ptrblck , I have the same issue. I was thinking if it would be good to do the following:  
use `functools.lru_cache( )` on ` __getitem__ ` and iterate over the dataloader once so that all getitem are cached. Now increase the `num_workers` from 0 to some value, and expect the cached data is shared across the processes.

Will this work ?

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [October 3, 2021, 8:25pm UTC](https://discuss.pytorch.org/t/dataloader-re-initialize-dataset-after-each-iteration/32658/16 "2021-10-03T20:25:51Z")

</div>

I would see a few problematic points with this approach, which you might have already thought about:

- usually you are transforming the data in ` __getitem__ `, which would yield slightly different results in each call for the same sample. This would be a cache miss and you would most likely never hit a cached sample. Are you thus only using static transformations? If so, why wouldn’t preloading the entire dataset into a single tensor work?
- “so that all getitem are cached” → this would mean that you are able to preload the entire dataset (which is not possible if the dataset is too large for your host RAM or you just don’t want to waste it to store the data). If so, how large is the data?
- “cached data is shared” → I don’t know how the `lru_cache` works internally, but to share data between multiple processes you would need to use shared object types to be able to use it for IPC.

---

<div class="post-metadata">

**Author:** ![Rohan\_Rathore](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rohan_rathore/32/37896_2.png) [@Rohan\_Rathore](https://discuss.pytorch.org/u/Rohan_Rathore)\
**Post date:** [October 4, 2021, 6:18am UTC](https://discuss.pytorch.org/t/dataloader-re-initialize-dataset-after-each-iteration/32658/17 "2021-10-04T06:18:02Z")

</div>

Oh ! I see. The ram size isn’t sufficient to load the whole data, I missed that. Thanks anyway.

---

<div class="post-metadata">

**Author:** ![halah](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/halah/32/43438_2.png) [@halah](https://discuss.pytorch.org/u/halah)\
**Post date:** [October 29, 2021, 10:40pm UTC](https://discuss.pytorch.org/t/dataloader-re-initialize-dataset-after-each-iteration/32658/18 "2021-10-29T22:40:28Z")

</div>

`functools.lru_cache() ` actually will not work because of the issue mentioned in this [thread](https://discuss.pytorch.org/t/dataloader-resets-dataset-state/27960).