# Shared Dict errors in pytorch dataloader

**URL:** <https://discuss.pytorch.org/t/shared-dict-errors-in-pytorch-dataloader/101941>\
**Category:** vision\
**Created:** [November 6, 2020, 10:05pm UTC](https://discuss.pytorch.org/t/shared-dict-errors-in-pytorch-dataloader/101941 "2020-11-06T22:05:04Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![Muhammad\_Inam](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/muhammad_inam/32/30719_2.png) [@Muhammad\_Inam](https://discuss.pytorch.org/u/Muhammad_Inam)\
**Post date:** [November 6, 2020, 10:05pm UTC](https://discuss.pytorch.org/t/shared-dict-errors-in-pytorch-dataloader/101941/1 "2020-11-06T22:05:05Z")

</div>

I have implemented a custom dataset using pytorch. I am stating the sample code below:

class MyDataset(Dataset):  
def **init** (self, shared\_dict, length):  
self.shared\_dict = shared\_dict  
self.length = length

```
def __getitem__ (self, index):
#Check share dict for the index as the key, if key exists return the value.(function then returns from here)

ELSE
#Heavy pre-processing of images happen here that consumes a lot of time
#Add the pre-processed data to the shared dict with the key set as the index.
#return the pre-processed data.

```

Since I wanted to save the time taken to pre-process the data in every epoch, I used the following approach:

> [@Dataloader re-initialize dataset after each iteration?](https://discuss.pytorch.org/t/dataloader-re-initialize-dataset-after-each-iteration/32658/4):
>
> That’s an interesting use case. You are right, variable sized input won’t work with my first approach. However, luckily Python provides some implementation for a shared dict. Here is a small example using it: from multiprocessing import Manager import torch from torch.utils.data import Dataset, DataLoader class MyDataset(Dataset): def \_\_init\_\_(self, shared\_dict, length): self.shared\_dict = shared\_dict self.length = length def \_\_getitem\_\_(self, index): …

During the training process multiple errors rise including:  
Broken Pipe error occurring on the for loop of data loader enumerator (already tried including this for loop in python’s main function scope)  
Workers quitting unexpectedly , Memory Buffer Errors, etc…  
Tracing the errors always lead towards multiprocessing.manager libraries.

Training configuration:  
number of workers = 40  
batch size = 150

Hardware Specs:  
GPU: Nvidia A100  
CPU: EPYC 128 cores processor  
RAM: 1 TB  
HDD/SDD: Its not a limitation either

Software Specs:  
Pytorch: latest version,  
CUDA 11,  
OS: Ubuntu

Dataset:  
Consists of images and json files making up a size of 17GB.

NOTE: Code works perfectly without shared dict.