DataLoader/ImageFolder slow with very low CPU usage

If your data is stored an a HDD, I doubt you can do a lot to avoid the IO bottleneck besides maybe reducing the batch size. Based on your explanation it seems that your code is IO bound so even pre-calculating the preprocessing would not change anything.
An SSD would provide a speedup, if that’s an option.

Also, have a look at this thread about a similar issue. Maybe you could try some suggestions like using an HDF5 file etc.