Why is MKLDNN slower than normal cpu mode

Hello Everyone, Hope you are having a great day.
For the past several weeks, I have been trying to get our models to run faster either using the MKLDNN or the Quantization. However, both of them seems to be failing for some reasons.
Today, I found out MKLDNN also doesnt work! it seems not to have any effect on runtime performance. Its even some times, slower than normal cpu mode!
The issue exists both on 1.3.1 and 1.4.0.
You may try the following snippet and see how it performs for you.
Running this on my local machine and also Google Colab, resulted the same and they were both slow as snail , in some cases slower than the normal (default) cpu mode!

Run this on a supporting platform such as Linux (e.g. ubuntu 18.04)

import torch
print(f'Pytorch version : {torch.__version__}')
print(*torch.__config__.show().split("\n"), sep="\n")

from torchvision import models
from torch.utils import mkldnn as mkldnn_utils
import time 

def forward(net, use_mkldnn=False, iteration=1, batch_size=10):
  net.eval()
  batch = torch.rand(batch_size, 3,224,224)
  if use_mkldnn:
    net = mkldnn_utils.to_mkldnn(net)
    batch = batch.to_mkldnn()

  start_time = time.time()
  for i in range(iteration):
      net(batch)
  return time.time() - start_time

net = models.resnet18(False)
iter_cnt = 100
batch_size = 1

no_mkldnn   = forward(net, False, iter_cnt, batch_size)
with_mkldnn = forward(net, True,  iter_cnt, batch_size)

print(f"time-normal: {no_mkldnn:.4f}s")
print(f"time-mkldnn: {with_mkldnn:.4f}s")
print(f"mkldnn is {with_mkldnn/no_mkldnn:.2f}x slower!")

Any help is greatly appreciated.

It might depend on your architecture, but I see a speedup:

time-normal: 2.7962s
time-mkldnn: 1.5869s
mkldnn is 0.57x slower!

for an Intel(R) Core(TM) i7-5820K CPU @ 3.30GHz.

1 Like

Compared to results from here : its really nothing ! I ran this some time before and got around 600x faster inference on mkldnn version!
I remember running it couple of times as well!