Hello,
I have a model that combines the inference of two models, as the following:
class model_all(nn.Module):
def __init__(self,dense,shuffle):
super(model_all,self).__init__()
self.dense = dense
self.shuffle = shuffle
def forward(self,input):
self.dense(input)
self.shuffle(input)
I am expecting the inference of the two models to happen in a parallel since cuda is running asynchronously. In other words,
model_all_inference_time < (dense_inference_time + shuffle_infernce_time).
However, what I get is
model_all inference_time ~= (dense_inference_time + shuffle_infernce_time)
When I measured inference time of model_all using this block of code output was 0.0228.
dummy = torch.ones((1,3,256,512)).to(device)
t = []
for _ in range(1000):
tic = time.perf_counter()
model(dummy)
torch.cuda.synchronize()
toc = time.perf_counter()
t.append(toc-tic)
print(sum(t)/1000)
Then I measured self.shuffle inference time by commenting self.dense(input) in forward function and output was 0.0069. Next, I measured self.dense inference time by commenting self.shuffle(input) and output was 0.0161. So, we can see that 0.0161+0.0069~=0.0228 which shows that models inference wasn’t in parallel but sequential .
Also, GPU utilization doesn’t exceed 50% with model_all inference. But with self.dense infernce it reaches 65%.
Is there a way to run the sub two models in parallel?
I am using pytorch 1.1.0, cuda 9 and nvidia TITAN Xp.
You can use this minimal code to reproduce the issue. Thank you!
import torch
import torch.nn as nn
import time
import torchvision
device = torch.device('cuda')
class model_all(nn.Module):
def __init__(self,dense,shuffle):
super(model_all,self).__init__()
self.dense = dense
self.shuffle = shuffle
def forward(self,input):
self.dense(input)
self.shuffle(input)
dense = torchvision.models.densenet121(pretrained=False).to(device)
shuffle = torchvision.models.shufflenet_v2_x0_5(pretrained=False).to(device)
model = model_all(dense,shuffle)
dummy = torch.ones((1,3,256,512)).to(device)
t = []
for _ in range(1000):
tic = time.perf_counter()
model(dummy)
torch.cuda.synchronize()
toc = time.perf_counter()
t.append(toc-tic)
print(sum(t)/1000)