How to run two models in both CPU and GPU concurrently?

1

I have two pre-trained ResNet18 models and I’m trying to run these models concurrently in both CPU and GPU to achieve maximum throughput.

2

Well, you should only pass one of them to the GPU in this case, don’t think you have to do anything else.

Though, I think it’d still b faster if you use GPU for both of them.

3

Thank you, but I’m trying to run these models heterogeneously. Does PyTorch support this type of compatibility?

4

As @TinfoilHat0 suggested, you could just call both models as:

model0 = MyModel().to('cuda')
model1 = MyModel()

x0 = torch.randn(1, 1).to('cuda') 
x1 = torch.randn(1, 1)

out0 = model0(x0) # calls asynchronously to CUDA
out1 = model1(x1)