Is it possible to read data on the fly in DataLoader? I have a huge dataset and I can’t just get all the data. It’s my first question. And second question: how to unfold it in two variables:
for input, target in dataset:
Have a look at the Data Loading tutorial.
Basically your data can be lazily loaded in Dataset's __getitem__ method, if you for example provide the image paths.
In __getitem__ you should return two tensors, i.e. the data and the corresponding target.
By wrapping the Dataset into a DataLoader multiple worker can be used to load and preprocess the data.
Your for loop will automatically work, if you use for input, target in loader:.
Here is a better explanation.
@ptrblck you are absolutely right. DataLoader handles all the headache. Thank you very much for your response.