Serialization issue

1

Hi,

I am loading a large serialized file (about 4.2GB) and encountered some error. With a smaller dataset, it was ok.
Any hints about this? Thank you.

`

Traceback (most recent call last):
File “train.py”, line 509, in
main()
File “train.py”, line 409, in main
dataset = torch.load(opt.data)
File “Y:\user\v\pytorch\Anaconda3\lib\site-packages\torch\serialization.py”, line 229, in load
return _load(f, map_location, pickle_module)
File “Y:\user\v\pytorch\Anaconda3\lib\site-packages\torch\serialization.py”, line 384, in _load
deserialized_objects[key]._set_from_file(f, offset)
RuntimeError: storage has wrong size: expected 77201408 got 13

`

2

I meet the similar question,have you solved your problem?
loading pretrained model models/resnet-152-kinetics.pth
Traceback (most recent call last):
File “main.py”, line 40, in
model, parameters = generate_model(opt)
File “/home/zhaoyan/self_3D-ResNets-PyTorch_8/model.py”, line 106, in generate_model
pretrain = torch.load(opt.pretrain_path)
File “/usr/local/lib/python2.7/dist-packages/torch/serialization.py”, line 261, in load
return _load(f, map_location, pickle_module)
File “/usr/local/lib/python2.7/dist-packages/torch/serialization.py”, line 416, in _load
deserialized_objects[key]._set_from_file(f, offset)
RuntimeError: storage has wrong size: expected -5033425001016997258 got 512

3

Unfortunately no.
I changed to online data processing.

4

I’ve also encountered this error when trying to restore a large seq2seq model on a CPU machine after moving it from GPU.

RuntimeError Traceback (most recent call last)
in ()
4 encoder, decoder = init_encoder_decoder(CCO_KMER_VOCAB_SZ, CCO_GO_VOCAB_SZ)
5
----> 6 encoder, decoder, _ = load_encoder_decoder_weights(encoder, decoder, “/tmp/seq2go-CCO-latest.tar”)

in load_encoder_decoder_weights(encoder, decoder, resume_path)
20 if os.path.isfile(resume_path):
21 print("=> loading checkpoint ‘%s’" % resume_path)
—> 22 checkpoint = torch.load(resume_path, map_location=lambda storage, loc: storage)
23 epoch = checkpoint[‘epoch’]
24 encoder.load_state_dict(checkpoint[‘encoder’])

~/development/prot2vec/virtualenv/lib/python3.6/site-packages/torch/serialization.py in load(f, map_location, pickle_module)
229 f = open(f, ‘rb’)
230 try:
–> 231 return _load(f, map_location, pickle_module)
232 finally:
233 if new_fd:

~/development/prot2vec/virtualenv/lib/python3.6/site-packages/torch/serialization.py in _load(f, map_location, pickle_module)
384 for key in deserialized_storage_keys:
385 assert key in deserialized_objects
–> 386 deserialized_objects[key]._set_from_file(f, offset)
387 offset = None
388

RuntimeError: storage has wrong size: expected -5162093521787232627 got 1194000