# DataLoader, when num\_worker \>0, there is bug

**URL:** <https://discuss.pytorch.org/t/dataloader-when-num-worker-0-there-is-bug/25643>\
**Category:** Uncategorized\
**Created:** [September 21, 2018, 6:29am UTC](https://discuss.pytorch.org/t/dataloader-when-num-worker-0-there-is-bug/25643 "2018-09-21T06:29:58Z")\
**Posts on this page:** 1\
**Showing post:** 16

<div class="post-metadata">

**Author:** ![piojanu](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/piojanu/32/4141_2.png) [@piojanu](https://discuss.pytorch.org/u/piojanu)\
**Post date:** [February 5, 2019, 11:42am UTC](https://discuss.pytorch.org/t/dataloader-when-num-worker-0-there-is-bug/25643/16 "2019-02-05T11:42:04Z")

</div>

So I investigated it further and in deed opening HDF5 introduces **huge overhead**. I’ve tested it on this code: [https://github.com/piojanu/World-Models](https://github.com/piojanu/World-Models) (my implementation of the World Models (further WM) paper, the memory training is written in PyTorch). _Note: the code I link here doesn’t have multiprocessing data preloading capabilities, I test it in the private repo._  
I use the [Pyflame](http://pyflame.readthedocs.io) profiler to profile the WM’s memory module training for 30s with sampling every 1ms on HW: Intel® Core™ i7-7700 CPU @ 3.60GHz with GeForce GTX 1060 6GB.

## Experiments:

1. With data loading **in main process** (DataLoader’s `num_worker` = 0) and opening hdf5 file **each time** in ` __getitem__ `:
  - _Batches per second: ~0,18_
  - Most of the time data is being loaded, above 70% of the profiling time.
  - **Opening the hdf5 file takes 20% of the profiling time!**
  - Then we have data preprocessing and mem copy in last 10% of the profiling time.
  - Training one layer LSTM on the GPU is so fast, that the profiler didn’t catch it.

2. With data loading **in main process** (DataLoader’s `num_worker` = 0) and opening hdf5 file **once** in ` __getitem__ `:
  - _Batches per second: ~2_
  - Still most of the time data is being loaded, ~90% of the profiling time.
  - **There is no overhead from opening the hdf5 file of course** , that’s why larger proportion of time went to loading the data.
  - Profiler was able to catch couple of samples of LSTM training, still below 1% of the profiling time.

3. With data loading **in worker processes** (DataLoader’s `num_worker` = 4) and opening hdf5 file **once** in ` __getitem__ `:
  - _Batches per second: ~5,1_
  - **There is no overhead from opening the hdf5 file and loading data is successfully covered with GPU execution**. DataLoader’s ` __next__ ` operation (getting next batch) in main process takes below 1% of the profiling time and we have full utilisation of GTX1060! Win 😉

## My recommendations:

- Use HDF5 in version 1.10 (better multiprocessing handling),
- Because an opened HDF5 file isn’t pickleable and to send Dataset to workers’ processes it needs to be serialised with pickle, you can’t open the HDF5 file in ` __init__ `. Open it in ` __getitem__ ` and **store as the singleton!**. Do not open it each time as it introduces huge overhead.
- Use `DataLoader` with `num_workers` \> 0 (reading from hdf5 (i.e. hard drive) is slow) and `batch_sampler` (random access to hdf5 (i.e. hard drive) is slow).

## Sample code:

```auto
class H5Dataset(torch.utils.data.Dataset):
    def __init__ (self, path):
        self.file_path = path
        self.dataset = None
        with h5py.File(self.file_path, 'r') as file:
            self.dataset_len = len(file["dataset"])

    def __getitem__ (self, index):
        if self.dataset is None:
            self.dataset = h5py.File(self.file_path, 'r')["dataset"]
        return self.dataset[index]

    def __len__ (self):
        return self.dataset_len

```

---

_[View the full topic](https://discuss.pytorch.org/t/dataloader-when-num-worker-0-there-is-bug/25643)._