I want to store my trainset as many little files, because if each sample stored as on file, it’s too many. but when I override the getitem , the [idx] is the index of files, not the samples. so how to override the Dataset Class to load several files .
You just need to create a list that maps an index to a file and to a sample, for this you will have to iterate over the complete dataset once, and build this map at the same time.
You will also have to write the def __getitem__(self, index): in the same way.
Is it possible to share any code ?