Loading ImageNet data from zip files

data
1

Hi everyone,

I am going to download and store ImageNet training set on the server in my lab. The examples on the internet require to extract all sub folders in the data root in advance. However, due to a memory limitation on the amount of data samples, I can not extract sub folders from ImageNet directory in my server. My question is that whether it is possible to implement custom data loader that only extracts a sub folder in case it wants to load a sample from that zip directory. if so, I would really appreciate any help, explanation, guidance or sharing any code snippet. My question is opposite to the solution where all data are extracted in advance.

Thank you.

2

You could check e.g. this approach to unzip the desired folder(s) during the actual loading, but I would also expect to see a large performance hit.
In any case you could experiment with it in a custom Dataset and see how well this approach of re-extracting the same folder would work.