It looks like HDF5 has some concurrency issues. My suggestion of using it is probably not appropriate when you use several workers. I often use one worker because my networks are computationally heavy and I’m not limited by the data iterator. Perhaps you should try other approaches like zarr (http://zarr.readthedocs.io/en/stable/) which have been designed to be thread-safe.
2 Likes