How to run two models in both CPU and GPU concurrently?

I have two pre-trained ResNet18 models and I’m trying to run these models concurrently in both CPU and GPU to achieve maximum throughput.

Well, you should only pass one of them to the GPU in this case, don’t think you have to do anything else.

Though, I think it’d still b faster if you use GPU for both of them.

Thank you, but I’m trying to run these models heterogeneously. Does PyTorch support this type of compatibility?

As @TinfoilHat0 suggested, you could just call both models as:

model0 = MyModel().to('cuda')
model1 = MyModel()

x0 = torch.randn(1, 1).to('cuda') 
x1 = torch.randn(1, 1)

out0 = model0(x0) # calls asynchronously to CUDA
out1 = model1(x1)