# Multi gpu inference pytorch

**URL:** <https://discuss.pytorch.org/t/multi-gpu-inference-pytorch/137679>\
**Category:** Uncategorized\
**Created:** [November 24, 2021, 10:48am UTC](https://discuss.pytorch.org/t/multi-gpu-inference-pytorch/137679 "2021-11-24T10:48:36Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![marcel\_Gibier1](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/marcel_gibier1/32/44240_2.png) [@marcel\_Gibier1](https://discuss.pytorch.org/u/marcel_Gibier1)\
**Post date:** [November 24, 2021, 10:48am UTC](https://discuss.pytorch.org/t/multi-gpu-inference-pytorch/137679/1 "2021-11-24T10:48:36Z")

</div>

I trained a model in multigpu thanks to accelerate from HuggingFace [https://github.com/huggingface/accelerate](https://github.com/huggingface/accelerate)

Nevertheless in inference I can’t see the use of my two gpus. I have done

```auto
   generator = Generator().to(device)
    # Load weights
    checkpoint = torch.load(weights_dir, map_location="cpu")
    generator.load_state_dict(checkpoint['gen_state_dict'])
    generator = accelerator.prepare(generator)
    generator.eval()

How would you do it for inference? 

Thanks !
```

---

<div class="post-metadata">

**Author:** ![eqy](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/eqy/32/46233_2.png) [@eqy](https://discuss.pytorch.org/u/eqy)\
**Post date:** [November 25, 2021, 1:52am UTC](https://discuss.pytorch.org/t/multi-gpu-inference-pytorch/137679/2 "2021-11-25T01:52:24Z")

</div>

I’m not familiar with accelerator but why prevents the same approach from being used at inference time? For example, just using the same accelerator workflow but removing the gradient computation and setting the model to eval mode?