Running inference for 3 GPT2 models concurrently is slower than sequentially. How to improve?

[trianxy]

Hey team,

I have an NVidia GPU with 24 GB RAM (g5.xlarge on AWS) and 3 differently fine-tuned GPT2 models. Unfortunately, running them concurrently is slower (45ms for 1 token each) than running them sequentially (38ms for 1 token each).

I expected running them concurrently to be close to 3x faster because size of GPU memory is not a problem and GPU utilisiation also seems to be always below 33%.

Any ideas or packages I should try out?

Here is a minimal example (plese note: below I attempt to run them concurrently using threads, but I also did the same using multiprocessing with 3 different Python scripts and got similar results):

import sys, os, threading
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

ARCHITECTURE = "gpt2"
DEVICE = "cuda"

tokenizer = AutoTokenizer.from_pretrained(ARCHITECTURE, use_fast=True)
model1 = AutoModelForCausalLM.from_pretrained(ARCHITECTURE).to(DEVICE)
model2 = AutoModelForCausalLM.from_pretrained(ARCHITECTURE).to(DEVICE)
model3 = AutoModelForCausalLM.from_pretrained(ARCHITECTURE).to(DEVICE)
inputs = tokenizer(["George Washington is"], return_tensors="pt").to(DEVICE)

### RUN SEQUENTIALLY: takes 38ms (timed with %%timeit and other methods)
output1 = model1(**inputs)
output2 = model2(**inputs)
output3 = model3(**inputs)

### RUN CONCURRENTLY WITH THREADS: takes 45 ms 
thread1 = threading.Thread(target=model1, kwargs=inputs)
thread2 = threading.Thread(target=model2, kwargs=inputs)
thread3 = threading.Thread(target=model3, kwargs=inputs)
thread1.start()
thread2.start()
thread3.start()
thread1.join()
thread2.join()
thread3.join()

## RUN CONCURRENTLY WITH STREAMS: also takes ~45ms
p1 = torch.cuda.Stream()
p2 = torch.cuda.Stream()
p3 = torch.cuda.Stream()
with torch.cuda.stream(p1):
    output1 = model1(**inputs)
with torch.cuda.stream(p2):
    output2 = model2(**inputs)
with torch.cuda.stream(p3):
    output3 = model3(**inputs)
p1.synchronize()
p2.synchronize()
p3.synchronize()

There are many related posts, but none seemed to provide a solution. I hope that as time has progressed, new solutions might have been found?

Some pointers

Split Single GPU

What about starting three web servers with different ports?

Infer multiple torch models on a single GPU

I am currently trying to infer 2 torch models on the same GPU, but my observation is that if 2 of them run at the same time in 2 different threads, the inference time is much larger than running them individually. Is there any way to make use of single GPU for running multiple models in parallel? Reference:

Deploying multiple PyTorch models on a single GPU

Hello PyTorch community, Suppose I have 10 different PyTorch models (classification, detection, embedding) and 10 GPUs. I would like to serve real-time image traffic on these models. We can assume a uniform traffic distribution for each model. What is the most efficient (low latency, high throughput) way? Deploy all 10 models onto each and every GPU (10 models on all 10 GPUs). This will probably incur context switch and cache miss costs. Memory management might also be costlier. Latency will …

Multiple models on a single GPU

Does running multiple copies of a model (like resnet18) on the same GPU have any benefits like parallel execution? I would like to find the loss/accuracy on two different datasets and I was wondering if it can done more efficiently on a single GPU. I am currently evaluating the datasets sequentially. Thanks!

Run multiple independent models on single GPU

Hola! Thanks for your answer! I can’t use independent Python kernen because I want the models to interact after training session. And there are a lot of training sessions involved. So I need to train them in a single script. Why does multiprocessing not work with CUDA? What library can I use to try threading?

Multiprocessing best practices — PyTorch 2.1 documentation → multiprocessing difficult on CUDA
Multiple replicas of the model on same GPU?

Not helpful

Inference Multiple Models on a Single GPU

I am trying to build a system and i need to do inference on 60 segmentation models at same time ( Same models but different inputs). I wonder if this is possible to do on pytorch? I am not sure about what kinda system i need to use for this, i am planning to use 4X RTX8000 and if it is not enough i can use two system with 4X RTX8000 each or with a better gpu. Would i lose too much performance because of using multiple models? How many models i can put on a gpu, what the performance of it depen…

Run two nets in parallel on single machine with multiple GPUs

Hi, I want to run 2 different networks that take the same input in parallel, with multiple gpus. The main purpose is to achieve inference speed up. So I did: inputs = torch.randn(1, 3, 256, 256) inputs1 = inputs.to('cuda:0') inputs2 = inputs.to('cuda:1') model1 = model1.to('cuda:0') model2 = model2.to('cuda:1') s1 = torch.cuda.Stream() s2 = torch.cuda.Stream() torch.cuda.synchronize() i = 0 time_spent = [] while i < 1000: with torch.no_grad…

Run n models in parallel on single GPU

I want to train n models (per n, I have f times t data points). I can load all data onto a single GPU. I assign the dataloader batches and each batch gets a number of minibatches. Each minibatch holds the data to train one model (one n). The data per n is rather small, but the number of models is large. The ‘problem’ that I am facing is that the batches are executed sequentially rather than in parallel on the single GPU. Is there a way to parallelize the batches on the single GPU to ensure scali…

Using CUDA multiprocessing with single GPU

This page outlines that the multiprocessing module can be used with CUDA: http://pytorch.org/docs/master/notes/multiprocessing.html However CUDA generally has issues running multiple processes in paralell on one GPU: Do these limitations apply to the pytorch multiprocessing module? Thanks in advance

Running multiple instances of same program in single GPU

I have a program with a simple dense layer model, which takes around 2GB/24GB of GPU memory (3090). I want to run multiple instance of the same program , with differet config files, with the same GPU. I used to be able to do it on other machines, but getting the following error RuntimeError: CUDA error: all CUDA-capable devices are busy or unavailable. CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect. Should I make any cha…

Multithreading on a single GPU using Single Model for Inferencing

Hello, I want to load the PyTorch model on a single GPU and perform multiple inferences through multi-threading on a single CUDA device. Is it possible, and if so, could you guide me on how to do so? I want to return inference results as well. The reason is I have noticed that not complete GPU power is utilized in single inference, so I thought through multithreading I would be able to 100% utilize my CUDA device

Multiple models inference time on the same GPU

AFAIK GPUs are asynchronous machines so you shouldn’t need to manage multiple processes loading to the GPU, just load it as normal and let PyTorch and NVIDIA handle how to make it efficient

Parallelism with single GPU (streams?)

Hi, I have a single GPU which is not fully used by my model. For this reason, I’d like to parallelise multiple forward calls on the same GPU. I’d prefer, if possible, to avoid multiprocessing. As I understand, cuda streams should help me on this, but I’ve not been able to get performance improvements, as if the streams were executed sequentially anyway. Can I obtain parallelism on the same GPU by using cuda streams without multiprocessing?

Is it possbile using a single gpu to predict multi images in parallelization？

Hi guys, in order to saving the inference time, I expect to use one gpu card to predict multi images concurrently, could someone give me some suggestion？many thanks. By the way, I have tried to used more than one batch to predict, it will be ok. However, I want to use multiprocessing to predict multi images in parallel. Kind regards

How to parallelize evaluation on a single GPU

Hi everyone. I have a nn.Module object which has been written such that it can take one data point (sample) at a time instead of a batch and can make some calculations about it. The weights for this model are trained and I don’t plan to update them so I plan to use this mainly just for evaluation. It is non-trivial for me to further vectorize the code to permit evaluation of multiple sentences as a batch. I still have a good chunk of GPU memory as well as processing power left when I evaluate o…

Split Single GPU

Is there any way to split single GPU and use a single GPU as multiple GPUs? For example, we have 2 different ResNet18 model and we want to forward pass these two models in parallel just in one GPU (with enough memory, e.g., 12Gb). I mean that the forward pass of these two models runs in parallel and concurrent in just one GPU.

Multiprocessing inside forward on single GPU

Hello, Did u solve the problem

[ptrblck]

You are most likely running into the GIL in your approach. Have you also profiled the code to see the actual execution performed in the GPU?

[trianxy]

Thanks a lot for the pointers @ptrblck !

The above code might run into the GIL, but I also tried the same using multiple fully independent Python scripts (where GIL shouldn’t be an issue), and still had the same slow performance.

I am now running profiling deep-dives using

from torch.profiler import profile, record_function, ProfilerActivity
with profile(activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA], record_shapes=True, profile_memory=False) as prof:
    with record_function("model_inference"):
        model1(**inputs)
print(prof.key_averages().table(sort_by="cpu_time_total", row_limit=10))

Regarding profiling, would you suggest other tools?