Cache/store transformed input to speed up data loading

data
1

Hey there,

The inputs to my dataloader are audio files (wav.), which during the preprocessing go through stft and other transformations. The spectrograms are then fed into the network.

Instead of computing stft and performing the same transformations at each epoch, I would like to cache the transformed inputs either in RAM or on disk. What’s a more efficient way of doing it in pytorch?

After searching online, I found that people cache those transformed as numpy, which I don’t know if is slower than saving as tensors. Others use tfrecord and custom pytorch dataloader.

Many thanks in advance!

2

Assuming you don’t use ay random transformations you could try to save the tensors directly via torch.save and check how much memory it would take of your SSD.
During training you could write a custom Dataset to load each tensor and pass it to the model.

3

Aside from ptrblck’s suggestion, you can also try TorchData which provides a few build-in caching functionalities (in-memory or disk).