You can check out torchdata which introduces DataPipe as replacement for torch's Dataset. The built-in DataPipes have some of the functionalities you need (in-memory cache, on-disk cache, shuffle).
We do not have a DataPipe for prefetching yet, you can create your own or try using the built-in prefetching feature of DataLoader.