Hello Everyone, Hope you are having a great day.
For the past several weeks, I have been trying to get our models to run faster either using the MKLDNN or the Quantization. However, both of them seems to be failing for some reasons.
Today, I found out MKLDNN also doesnt work! it seems not to have any effect on runtime performance. Its even some times, slower than normal cpu mode!
The issue exists both on 1.3.1 and 1.4.0.
You may try the following snippet and see how it performs for you.
Running this on my local machine and also Google Colab, resulted the same and they were both slow as snail , in some cases slower than the normal (default) cpu mode!
Run this on a supporting platform such as Linux (e.g. ubuntu 18.04)
import torch
print(f'Pytorch version : {torch.__version__}')
print(*torch.__config__.show().split("\n"), sep="\n")
from torchvision import models
from torch.utils import mkldnn as mkldnn_utils
import time
def forward(net, use_mkldnn=False, iteration=1, batch_size=10):
net.eval()
batch = torch.rand(batch_size, 3,224,224)
if use_mkldnn:
net = mkldnn_utils.to_mkldnn(net)
batch = batch.to_mkldnn()
start_time = time.time()
for i in range(iteration):
net(batch)
return time.time() - start_time
net = models.resnet18(False)
iter_cnt = 100
batch_size = 1
no_mkldnn = forward(net, False, iter_cnt, batch_size)
with_mkldnn = forward(net, True, iter_cnt, batch_size)
print(f"time-normal: {no_mkldnn:.4f}s")
print(f"time-mkldnn: {with_mkldnn:.4f}s")
print(f"mkldnn is {with_mkldnn/no_mkldnn:.2f}x slower!")
Any help is greatly appreciated.