# Is there any way in which we can speed up the dataloader

**URL:** <https://discuss.pytorch.org/t/is-there-any-way-in-which-we-can-speed-up-the-dataloader/197192>\
**Category:** vision\
**Created:** [February 16, 2024, 9:03am UTC](https://discuss.pytorch.org/t/is-there-any-way-in-which-we-can-speed-up-the-dataloader/197192 "2024-02-16T09:03:29Z")\
**Posts on this page:** 1\
**Showing post:** 2

<div class="post-metadata">

**Author:** ![alepistola](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alepistola/32/67425_2.png) [@alepistola](https://discuss.pytorch.org/u/alepistola)\
**Post date:** [February 21, 2024, 5:56pm UTC](https://discuss.pytorch.org/t/is-there-any-way-in-which-we-can-speed-up-the-dataloader/197192/2 "2024-02-21T17:56:50Z")

</div>

I had and am having the same problem as you, using 2048x2048 images on Google Colab and analyzing the training process I noticed that the bottleneck is actually fetching the data from GDrive.

Things I have adopted temporarily are:

- set the device globally :

```auto
device = "cuda" if torch.cuda.is_available() else "cpu"
torch.device(device)

```

- follow this rule to calculate the num\_workers:  
`NUM_WORKERS = 4 * os.cpu_count()`

I believe, however, that to get maximum performance, you need to load all the data into memory beforehand, but I couldn’t find anything online.  
To speed up loading, an HDF5 dataset should be used and then a data loader wrapper created.  
Here is some useful information:

> **[GitHub - mvsjober/pytorch-hdf5](https://github.com/mvsjober/pytorch-hdf5)**
>
> Contribute to mvsjober/pytorch-hdf5 development by creating an account on GitHub.

> [@DataLoader, when num\_worker \>0, there is bug](https://discuss.pytorch.org/t/dataloader-when-num-worker-0-there-is-bug/25643/15?fbclid=IwAR2jFrRkKXv4PL9urrZeiHT_a3eEn7eZDWjUaQ-zcLP6BRtMO7e0nMgwlKU):
>
> so then it seems to be a problem relating to hdf5 in general in combination with multirpocessing rather than with Pytorch itself? If so, it would be interesting so see, how this issue evolves when using the C++ frontend

---

_[View the full topic](https://discuss.pytorch.org/t/is-there-any-way-in-which-we-can-speed-up-the-dataloader/197192)._