# Multi-GPU torch.load() out of memory

**URL:** <https://discuss.pytorch.org/t/multi-gpu-torch-load-out-of-memory/117775>\
**Category:** autograd\
**Created:** [April 11, 2021, 9:10am UTC](https://discuss.pytorch.org/t/multi-gpu-torch-load-out-of-memory/117775 "2021-04-11T09:10:31Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![tcsn\_wty](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tcsn_wty/32/11956_2.png) [@tcsn\_wty](https://discuss.pytorch.org/u/tcsn_wty)\
**Post date:** [April 11, 2021, 9:10am UTC](https://discuss.pytorch.org/t/multi-gpu-torch-load-out-of-memory/117775/1 "2021-04-11T09:10:31Z")

</div>

I was training a model with 1 GPU device and just now figured out how to train with 2 GPU devices. Therefore I paused the training and resume after adding in lines of code to use 2 GPUs. Firstly, loading the checkpoint would cause torch.load() out of memory no matter I use 1 GPU or 2 GPUs. Then I followed some posts to first load the check point to CPU and delete checkpoint, and the model was able to resume on 1 GPU.

```auto
checkpoint = torch.load('checkpoint/ckpt.t7', map_location=torch.device("cpu"))
model.load_state_dict(checkpoint['state'])
del checkpoint
torch.cuda.empty_cache()

```

However, when I tried to use 2 GPUs it caused a cuda out of memory error. I wasn’t able to find any related posts specifically regarding OOM regarding multi-GPUs especially it worked just fine with single GPU. I wonder if there are any tricks playing with multi-GPU when loading checkpoint?