TorchScript Model inference slow in python

When testing if the prediction is correct after the model is transferring to script model, I found that the inference time is twice longer than pytorch model.

# pytorch 1.4, python 3.7
out1 = model(input) # forward time: 0.05 sec
ScriptModel = torch.jit.script(model, input)
out2 = ScriptModel(input) # forward time: 0.12 sec

The first pass does some extra work (details in the link below), if you warm up ScriptModel by calling it a few times first the time should improve.

As the snippet below, script model actually get slower than average time 0.12sec in the first iteration in for loop as @driazati mentioned above . However, after the first iteration, the rest is ~ 0.12 sec/img and is still twice longer than pytorch model.
Put more information here: batch_size=1

pytorch 1.4, python 3.7
# pytorch model
for i in range(image_number):
    out1 = model(input) # forward time: 0.05 sec

#torchscript model
ScriptModel = torch.jit.script(model, input)
for i in range(image_number):
    out2 = ScriptModel(input) # forward time: 0.12 sec if i > 0, 3 sec if i==0

If you are using the native lstm, it is normal that the native one is faster than touchscript. Also, according to here, increasing the batch size can reduce the difference in performance. And according to my own experience, increasing the hidden size can also reduce the difference.