Hey, I’m simply trying to save a vector of LibTorch (C++) tensors to file and then load those tensors back into PyTorch (Python) for post-processing reasons.
On the C++ side, I have the following sample code:
const auto new_tensor = torch::rand({2, 3, 4});
const auto new_tensor2 = torch::rand({1, 125, 13, 13});
torch::save({new_tensor, new_tensor2}, "tensor_vector.pt");
I then copy the generated file into my python project directory and call the following script:
import torch
tensors = torch.load('./tensor_vector.pt')
Which returns the following python error message:
Traceback (most recent call last):
File "C:\Users\admin\.conda\envs\py37_torch\lib\tarfile.py", line 189, in nti
n = int(s.strip() or "0", 8)
ValueError: invalid literal for int() with base 8: 'ZZZZZZZZ'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\admin\.conda\envs\py37_torch\lib\tarfile.py", line 2297, in next
tarinfo = self.tarinfo.fromtarfile(self)
File "C:\Users\admin\.conda\envs\py37_torch\lib\tarfile.py", line 1093, in fromtarfile
obj = cls.frombuf(buf, tarfile.encoding, tarfile.errors)
File "C:\Users\admin\.conda\envs\py37_torch\lib\tarfile.py", line 1035, in frombuf
chksum = nti(buf[148:156])
File "C:\Users\admin\.conda\envs\py37_torch\lib\tarfile.py", line 191, in nti
raise InvalidHeaderError("invalid header")
tarfile.InvalidHeaderError: invalid header
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\admin\.conda\envs\py37_torch\lib\site-packages\torch\serialization.py", line 524, in _load
return legacy_load(f)
File "C:\Users\admin\.conda\envs\py37_torch\lib\site-packages\torch\serialization.py", line 448, in legacy_load
with closing(tarfile.open(fileobj=f, mode='r:', format=tarfile.PAX_FORMAT)) as tar, \
File "C:\Users\admin\.conda\envs\py37_torch\lib\tarfile.py", line 1589, in open
return func(name, filemode, fileobj, **kwargs)
File "C:\Users\admin\.conda\envs\py37_torch\lib\tarfile.py", line 1619, in taropen
return cls(name, mode, fileobj, **kwargs)
File "C:\Users\admin\.conda\envs\py37_torch\lib\tarfile.py", line 1482, in __init__
self.firstmember = self.next()
File "C:\Users\admin\.conda\envs\py37_torch\lib\tarfile.py", line 2309, in next
raise ReadError(str(e))
tarfile.ReadError: invalid header
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:/Users/admin/PycharmProjects/TestProject/importLibtorchTensor.py", line 2, in <module>
tensors = torch.load('./tensor_vector.pt')
File "C:\Users\admin\.conda\envs\py37_torch\lib\site-packages\torch\serialization.py", line 368, in load
return _load(f, map_location, pickle_module)
File "C:\Users\admin\.conda\envs\py37_torch\lib\site-packages\torch\serialization.py", line 528, in _load
raise RuntimeError("{} is a zip archive (did you mean to use torch.jit.load()?)".format(f.name))
RuntimeError: ./tensor_vector.pt is a zip archive (did you mean to use torch.jit.load()?)
Additionally, If I instead try to use the jit torch package to load:
import torch
tensors = torch.jit.load('./tensor_vector.pt')
Then I get the following error:
Traceback (most recent call last):
File "C:/Users/admin/PycharmProjects/TestProject/importLibtorchTensor.py", line 2, in <module>
tensors = torch.jit.load('./tensor_vector.pt')
File "C:\Users\admin\.conda\envs\py37_torch\lib\site-packages\torch\jit\__init__.py", line 132, in load
torch._C.import_ir_module(module_lookup, f, map_location)
RuntimeError: INVALID_ARGUMENT:tensors[1]: Cannot find field. (deserialize at ..\torch\csrc\jit\import.cpp:95)
(no backtrace available)
Is there any way to save a C++ Libtorch tensor and load it in as Python Pytorch tensor? I don’t need to save/load entire models, just specific tensors.