I have a dataset of size [40000, 512, 512] images (type np.int16) and I want to create a Dataloader for segmentation network.
Although I have 200 Nifti images of 512x512x200 images, Since I am creating a 2D network, it would be efficient to shuffle between subject slices of different subjects as well.
I had the following thoughts:
i) Create multiple tensors and load them through dataloader.
ii) Load image every single time and the slice and send the data(slow and useless and throws away a lot of data)
iii) Create individual slices, store them as .npz and then use it (very very space consuming and very useless)
Can someone help me understanding what to do?
I can share thoughts and code for the dataloader that I made.