How to save intermediate results and load it later with a datalaoder again?

I want to find a data structure to save intermediate results. Numpy and Pickle is not an option because I cannot load the data in chunks.

The data is images, but saving .jpeg etc… is a loss of information.

How can I save these intermediate results and which enables me to load them memory efficient with a data loader?

I don’t quite understand the “cannot load the data in chunks” part, but given that I guess is not an option.

You could use a lossless compression format such as PNG.

I think that: Using instead of pickle or It is better to use tf.records, e.g. from the PyTorch DALI plugin.

What do you think about that?

Yes, DALI is certainly an option and compatible with PyTorch. I don’t know enough about tf.records to say if it’s needed or not.

Well, I think that tf.records are able to store the tuples (images, tuples). And load it batch-wise on the GPU. So that the RAM is not heavily used.

At the moment I use and torch.load, which has the drawback just loads files at once and if the files very huge, my RAM is too small.

I havnt seen anything comparable in the pytorch lib.