# PyTorch inference with multiple images

**URL:** <https://discuss.pytorch.org/t/pytorch-inference-with-multiple-images/109022>\
**Category:** vision\
**Created:** [January 16, 2021, 9:37am UTC](https://discuss.pytorch.org/t/pytorch-inference-with-multiple-images/109022 "2021-01-16T09:37:19Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![tominator](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/t/bb73d2/32.png) [@tominator](https://discuss.pytorch.org/u/tominator)\
**Post date:** [January 16, 2021, 9:37am UTC](https://discuss.pytorch.org/t/pytorch-inference-with-multiple-images/109022/1 "2021-01-16T09:37:19Z")

</div>

I wrote an API to serve my PyTorch Object Detection Model. I followed the [Tutorial](https://pytorch.org/tutorials/intermediate/flask_rest_api_tutorial.html) provided on the PyTorch website. Th aim is to send the API an image and to get back the result (classes, bounding boxes, etc.). For the prediction of only one image at time it works like a charm.

Now I want to extend it. The aim is to send multiple images to the API and to get back the results (in my case 5 images). I want to get the results as fast as possible. In a [Tensorflow Tutorial](https://www.tensorflow.org/tfx/serving/performance#batch_size) I read that batching is a way to increase the performance. Is it better to do the predictions sequentially or to return the predictions for multiple images at once (batching)? What is the better and faster method? I run the inference on a GPU.

---

<div class="post-metadata">

**Author:** ![1chimaruGin](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/1chimarugin/32/27435_2.png) [@1chimaruGin](https://discuss.pytorch.org/u/1chimaruGin)\
**Post date:** [January 16, 2021, 11:41am UTC](https://discuss.pytorch.org/t/pytorch-inference-with-multiple-images/109022/2 "2021-01-16T11:41:12Z")

</div>

In my case, I used python `concurrent.futures.ProcessPoolExecutor` and `concurrent.futures.ThreadPoolExecutor` to inference 4 images at once.

But, this [one](https://discuss.pytorch.org/t/how-to-load-images-for-inference-in-batch/33329/2) might help you.

---

<div class="post-metadata">

**Author:** ![tominator](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/t/bb73d2/32.png) [@tominator](https://discuss.pytorch.org/u/tominator)\
**Post date:** [January 19, 2021, 7:54am UTC](https://discuss.pytorch.org/t/pytorch-inference-with-multiple-images/109022/3 "2021-01-19T07:54:54Z")

</div>

@1chimaruGin thanks for your reply. I will try your first suggestion. I also found the second entry, but it didn’t really help to reduce the inference time.

---

<div class="post-metadata">

**Author:** ![1chimaruGin](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/1chimarugin/32/27435_2.png) [@1chimaruGin](https://discuss.pytorch.org/u/1chimaruGin)\
**Post date:** [January 19, 2021, 8:55am UTC](https://discuss.pytorch.org/t/pytorch-inference-with-multiple-images/109022/4 "2021-01-19T08:55:17Z")

</div>

> [@tominator](#):
>
> to reduce the inference time.

Then, use TensorRT. It might help you a lot.