I found the idea of lazy loading is similar term like load on demand, or load single item (image) on demand to be more precise.
Load in here means load it from non GPU memory to GPU, which is what I was most interested.
This further means, writing a custom dataset and overwriting __getitem__ as described in here.
If we are writing a custom Dataset why do we need a Subset which is also a custom Dataset?
If we will load a single item, and we will do that for every item in the batch, wouldn’t that be slower, comparing to loading a batch from once (DataLoader).
Which leads to the quest when is the best time to load to GPU, in DataSet or in DataLoader?