mbp28
October 22, 2017, 11:34pm
1
Hello,
I was wondering whether something like numpy.memmap exists in PyTorch? Or should I just save as a numpy array and use numpy.memmap instead, because conversion to torch tensors is cheap?
Numpy: https://docs.scipy.org/doc/numpy-1.13.0/reference/generated/numpy.memmap.html
2 Likes
colesbury
October 22, 2017, 11:46pm
2
You can use torch.Storage.from_file (docs ):
e.g.
x = torch.IntTensor(torch.IntStorage.from_file('file.bin', size=100))
1 Like
Is there a way to store the data to memory mapped files as well. Does Pytorch have a method for this?
vmoens
October 6, 2022, 8:49am
4
Check this feature from TorchRL if you’re interested in it:
# Copyright (c) Meta Platforms, Inc. and affiliates.
#
# This source code is licensed under the MIT license found in the
# LICENSE file in the root directory of this source tree.
from __future__ import annotations
import functools
import os
import tempfile
from typing import Any, Callable, List, Optional, Tuple, Union, Dict
import numpy as np
import torch
from torchrl._utils import prod
from torchrl.data.tensordict.utils import _getitem_batch_size
from torchrl.data.utils import (
DEVICE_TYPING,
INDEX_TYPING,
show original