Hdf5 a data format for pytorch

[piojanu (Piotr Januszewski)]

Well I don’t think that the problem is:

@Carsten_Ditzel

hdf5, even in version 1.10 does not support multiple process read

It does support it. The problem is that opened hdf5 file can’t be pickled as I mentioned here:

@piojanu

Because an opened HDF5 file isn’t pickleable and to send Dataset to workers’ processes it needs to be serialised with pickle, you can’t open the HDF5 file in __init__ . Open it in __getitem__ and store as the singleton! . Do not open it each time as it introduces huge overhead.

Work around proposed here: DataLoader, when num_worker >0, there is bug - #16 by piojanu works perfectly fine for me for really big datasets.

Also, I don’t think there is a silver bullet for this problem. What to use is specific to a use case.