I use single machine with 4 gpus to train my models, and then I save paramaters of the model with torch.save(aemodel.state_dict(), file). And when I use aemodel.load_state_dict(torch.load(output_path + 'ae.pkl', map_location='cpu')) to load its paramaters, error occurs: Traceback (most recent call …

This error might be raised, if you were using multiple processes (e.g. via DistributedDataParallel) and didn’t guard the storing of the checkpoint against multiple writers. You could use the ImageNet example to only use the first process to store the checkpoint.

RuntimeError: storage has wrong size

Brando_Miranda (MirandaAgent) February 28, 2022, 11:48pm 4

this was fixed for me once I uploaded the checkpoint to the cluster again.