Torch.save triggers error when saving model

I am getting the following error code:

Traceback (most recent call last):
File “/home/arams/anaconda3/lib/python3.8/site-packages/torch/serialization.py”, line 372, in save
_save(obj, opened_zipfile, pickle_module, pickle_protocol)
File “/home/arams/anaconda3/lib/python3.8/site-packages/torch/serialization.py”, line 478, in _save
zip_file.write_record(‘data.pkl’, data_value, len(data_value))
OSError: [Errno 28] No space left on device

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/home/arams/anaconda3/lib/python3.8/site-packages/torch/serialization.py”, line 373, in save
return
File “/home/arams/anaconda3/lib/python3.8/site-packages/torch/serialization.py”, line 259, in exit
self.file_like.write_end_of_file()
RuntimeError: [enforce fail at inline_container.cc:274] . unexpected pos 64 vs 0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “train.py”, line 499, in
run()
File “train.py”, line 309, in call
self.validate(current_epoch, validation_loader)
File “train.py”, line 253, in validate
self.save_network(prev_score, current_score, epoch, current_lr)
File “train.py”, line 141, in save_network
torch.save(state_dict, savepath)
File “/home/arams/anaconda3/lib/python3.8/site-packages/torch/serialization.py”, line 374, in save
_legacy_save(obj, opened_file, pickle_module, pickle_protocol)
File “/home/arams/anaconda3/lib/python3.8/site-packages/torch/serialization.py”, line 214, in exit
self.file_like.close()
OSError: [Errno 28] No space left on device
terminate called after throwing an instance of ‘c10::Error’
what(): [enforce fail at inline_container.cc:274] . unexpected pos 64 vs 0
frame #0: c10::ThrowEnforceNotMet(char const*, int, char const*, std::string const&, void const*) + 0x47 (0x7f78209ae0e7 in /home/arams/anaconda3/lib/python3.8/site-packages/torch/lib/libc10.so)
frame #1: + 0x1dd5620 (0x7f786668c620 in /home/arams/anaconda3/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
frame #2: + 0x1dd17f3 (0x7f78666887f3 in /home/arams/anaconda3/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
frame #3: caffe2::serialize::PyTorchStreamWriter::writeRecord(std::string const&, void const*, unsigned long, bool) + 0xa9 (0x7f786668cf99 in /home/arams/anaconda3/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
frame #4: caffe2::serialize::PyTorchStreamWriter::writeEndOfFile() + 0xe1 (0x7f786668dad1 in /home/arams/anaconda3/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
frame #5: caffe2::serialize::PyTorchStreamWriter::~PyTorchStreamWriter() + 0x115 (0x7f786668e2c5 in /home/arams/anaconda3/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
frame #6: + 0x734373 (0x7f786d879373 in /home/arams/anaconda3/lib/python3.8/site-packages/torch/lib/libtorch_python.so)
frame #7: + 0x366610 (0x7f786d4ab610 in /home/arams/anaconda3/lib/python3.8/site-packages/torch/lib/libtorch_python.so)
frame #8: + 0x36787e (0x7f786d4ac87e in /home/arams/anaconda3/lib/python3.8/site-packages/torch/lib/libtorch_python.so)

frame #21: __libc_start_main + 0xf3 (0x7f789382d0b3 in /lib/x86_64-linux-gnu/libc.so.6)

Aborted (core dumped)

A bit unsure how to fix this. It says no OSError: [Errno 28] No space left on device, but when I comment out torch.save, the programs works fine.

The error message shows that your device doesn’t have enough space so that you won’t be able to save anything on it.
Create new space on it by deleting unwanted files and rerun the script.

1 Like