Infer multiple torch models on a single GPU

[Tuyen_Vo_Quang]

• I am currently trying to infer 2 torch models on the same GPU, but my observation is that if 2 of them run at the same time in 2 different threads, the inference time is much larger than running them individually.
• Is there any way to make use of single GPU for running multiple models in parallel?

Reference:

Inference Multiple Models on a Single GPU

I am trying to build a system and i need to do inference on 60 segmentation models at same time ( Same models but different inputs). I wonder if this is possible to do on pytorch? I am not sure about what kinda system i need to use for this, i am planning to use 4X RTX8000 and if it is not enough i can use two system with 4X RTX8000 each or with a better gpu. Would i lose too much performance because of using multiple models? How many models i can put on a gpu, what the performance of it depen…

[marksaroufim]

The best approach here is to use an inferencing framework which handles this problem for you pytorch/serve is an example I work and can help answer any questions