# Data cached in a global variable for DataLoader is cleared for different epoches!

**URL:** <https://discuss.pytorch.org/t/data-cached-in-a-global-variable-for-dataloader-is-cleared-for-different-epoches/33840>\
**Category:** Uncategorized\
**Created:** [January 5, 2019, 8:23am UTC](https://discuss.pytorch.org/t/data-cached-in-a-global-variable-for-dataloader-is-cleared-for-different-epoches/33840 "2019-01-05T08:23:47Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![af3b04f0e6ef68be2e0e](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/af3b04f0e6ef68be2e0e/32/11352_2.png) [@af3b04f0e6ef68be2e0e](https://discuss.pytorch.org/u/af3b04f0e6ef68be2e0e)\
**Post date:** [January 5, 2019, 8:23am UTC](https://discuss.pytorch.org/t/data-cached-in-a-global-variable-for-dataloader-is-cleared-for-different-epoches/33840/1 "2019-01-05T08:23:47Z")

</div>

I am doing a CV project with training data stored in Google Drive and executed on the Google Colab.  
For the efficiency of loading images, which are downloaded from Google Drive, I created a cache for the images have been loaded.

Here’s the code:

```auto
imgcache = {}

def get_img_by_url(url):
    file_id = url.split('open?id=')[1]
    
    global imgcache
    if file_id in imgcache:
        return imgcache[file_id]
    else:
        downloaded = drive.CreateFile({'id': file_id})
        downloaded.FetchContent()
        img_stream = Image.open(downloaded.content)
        imgcache[file_id] = img_stream
        return img_stream 

```

The get\_img\_by\_url will be called in the _getitem_ of my custom Dataset function.

This function works fine with no forked workers ( num\_workers = 0 ), however, with more workers added, I found the “imgcache” got cleared everytime the program enters a new epoch ( the len of imgcache goes to 0 again ).

Can anyone tell me how this happened?

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [January 5, 2019, 12:31pm UTC](https://discuss.pytorch.org/t/data-cached-in-a-global-variable-for-dataloader-is-cleared-for-different-epoches/33840/2 "2019-01-05T12:31:45Z")

</div>

Have a look at [this thread](https://discuss.pytorch.org/t/dataloader-resets-dataset-state/27960/4?u=ptrblck) for some information on caching with multiple workers using shared arrays.