What's the best way to load large data?

2

lmdb can use multiple workers for loading, have you tried that?
See how I use lmdb for my LSUN dataset here: https://github.com/pytorch/vision/blob/master/torchvision/datasets/lsun.py#L19-L20

I keep readers at 1, but I use multiple workers to load from lsun (and hence lmdb):

2 Likes