I have some data which is thrice large as my system’s RAM. I need to run some Deep Learning models using pytorch. Could you please advise how can I use torch data loaders (or alternative) in this scenario?
Assume my data is stored as subfolders inside parent directory as below.
Transaction_Data/ ---Customer1/ --- day1 --- day2 . --- dayN ---Customer2/ --- day1 --- day2 . --- dayN ---CustomerN/ --- day1 --- day2 . --- dayN
Lets assume these are clean data and each customer is like an individual data frame (days represent rows).
I want to load these data in batches (probably 5 customers at once - can fit into memory) and train DL model using torch. What is the efficient way to load these?
I need to iterate over these data and run DL models. Should I be using custom dataset? Any pointers would be really appreciated.
Thank you very much in advance!