How to save intermediate results and load it later with a datalaoder again?

[jS5t3r (Peter Lorenz)]

I want to find a data structure to save intermediate results. Numpy and Pickle is not an option because I cannot load the data in chunks.

The data is images, but saving .jpeg etc… is a loss of information.

How can I save these intermediate results and which enables me to load them memory efficient with a data loader?

[ptrblck]

@jS5t3r

Numpy and Pickle is not an option because I cannot load the data in chunks.

I don’t quite understand the “cannot load the data in chunks” part, but given that I guess torch.save is not an option.

@jS5t3r

The data is images, but saving .jpeg etc… is a loss of information.

You could use a lossless compression format such as PNG.