How to save intermediate results and load it later with a datalaoder again?

vision
#1

I want to find a data structure to save intermediate results. Numpy and Pickle is not an option because I cannot load the data in chunks.

The data is images, but saving .jpeg etc… is a loss of information.

How can I save these intermediate results and which enables me to load them memory efficient with a data loader?

#2

I don’t quite understand the “cannot load the data in chunks” part, but given that I guess torch.save is not an option.

You could use a lossless compression format such as PNG.

#3

I think that: Using instead of pickle or torch.save/torch.load. It is better to use tf.records, e.g. from the PyTorch DALI plugin.

What do you think about that?

#4

Yes, DALI is certainly an option and compatible with PyTorch. I don’t know enough about tf.records to say if it’s needed or not.

#5

Well, I think that tf.records are able to store the tuples (images, tuples). And load it batch-wise on the GPU. So that the RAM is not heavily used.

At the moment I use torch.save and torch.load, which has the drawback just loads files at once and if the files very huge, my RAM is too small.

I havnt seen anything comparable in the pytorch lib.