I want to find a data structure to save intermediate results. Numpy and Pickle is not an option because I cannot load the data in chunks.
The data is images, but saving
.jpeg etc… is a loss of information.
How can I save these intermediate results and which enables me to load them memory efficient with a data loader?
I don’t quite understand the “cannot load the data in chunks” part, but given that I guess
torch.save is not an option.
You could use a lossless compression format such as PNG.
I think that: Using instead of pickle or
torch.load. It is better to use
tf.records, e.g. from the PyTorch DALI plugin.
What do you think about that?
Yes, DALI is certainly an option and compatible with PyTorch. I don’t know enough about
tf.records to say if it’s needed or not.
Well, I think that
tf.records are able to store the tuples (images, tuples). And load it batch-wise on the GPU. So that the RAM is not heavily used.
At the moment I use
torch.load, which has the drawback just loads files at once and if the files very huge, my RAM is too small.
I havnt seen anything comparable in the pytorch lib.