Hello,
I am trying to setup up my dataloader to sample from multiple csv files within a directory, each of which contains a variable number of samples.
- Each sample is a row in one of the csv files
- Each file is too big to load in as a single batch (~50k)
- Each file contains a different number of samples (between 30k-60k)
- There are several thousand csv files in the folder
- The entire training set is to large too hold in memory (around 200M samples)
I have looked at some of the examples on this forum and they include using torchvision.datafolder, however I think that assumes that each csv file contains a single sample, which does not apply to my situation.