If i am having say quite a large dataset with around 50k images, will it be efficient to store the paths in a dictionary and read them in the custom dataloader? Or is CSV a better option?
A dataset, probably?
This depends on lots of factors (not lest how you intend to read CSVs), but to be honest, I would expect the time for either to be very small compared to the rest of the training. So my advice not worry about it until it becomes the bottleneck.
Okay, thanks @tom for the reply, unfortunately its a custom dataset, so i wont be able to sshare it. Thanks 