How to create 3D custom dataloader

[J_Johnson]

You can build a custom dataloader, if you plan to store the data in GPU or CPU RAM. And then just create an index that you can shuffle and use to get the batches. Here is an example:

Use case for loading the entire dataset into RAM

Just a suggestion: don’t use the vanilla dataloader for this. If you can fit the data all into ram, just turn it into a tensor for each set of data, shape being: self.dataset.size()=([data_index, ...]) Where each data_index is a sample. Then create an indexer, either in NumPy or Pytorch. Here is an example for NumPy. rng=np.random.default_rng() ... self.data_index=np.arange(self.dataset.size()[0]) Then define a shuffler class you can call whenever you want to shuffle the index: def shuffl…