Memory mapped Tensor?

#1

Hello,

I was wondering whether something like numpy.memmap exists in PyTorch? Or should I just save as a numpy array and use numpy.memmap instead, because conversion to torch tensors is cheap?

Numpy: https://docs.scipy.org/doc/numpy-1.13.0/reference/generated/numpy.memmap.html

2 Likes
#2

You can use torch.Storage.from_file (docs):

e.g.

x = torch.IntTensor(torch.IntStorage.from_file('file.bin', size=100))
1 Like
Loading a Tensor from file in batches
#3

Is there a way to store the data to memory mapped files as well. Does Pytorch have a method for this?