I have a pytorch tensor that is size of 5000000x100; it takes forever to load it; any way to do it more efficiently?
Take a look at the h5py library. It lets you store and load hdf5 files in chunks and you can do many cool things with it. You can save the tensor using it and load it. But I dont think you load a file stored with
torch.save() (wich uses pickle) using the h5py library.