How to load batch of 365*3000 data in a memory efficient way?

I have a dataset that for each sample, it is a 365 * 3000 pandas/numpy array after preprocessing. And they are sparse. The label or target is binary for each sample. I have around 500K of these samples. Apparently I can not load the the 500K * 365 * 3000 data in the memory. So my plan is to save the numpy array to disk like an image and load them back during training using batch, so I will be fine on the memory side. However, I think it’s waste of time by doing the I/O. Do you think is there better solution out there? If I have to save the file, will the torch.sparse help?

Thank you.

edited for error correction

I’m not sure how to implement your data loading logic using sparse, but for your first suggestion you could create a custom Dataset and load the samples lazily.
Using a DataLoader the batches can be preloaded while your GPU is busy with the training step.

Have a look at the data loading tutorial to create your Dataset.