I installed pytorch on several machines, from source and from conda and I am getting different execution times for matrix multiplication. All installs are with Anaconda 4.3.0, python 3.6. However I can’t figure out if pytorch is using MKL or OpenBLAS or other backend. Right now the macOS install is the fastest despite the machine having the slowest CPU.
The reason I ran these tests is because I noticed a severe slowdown (~10 times slower) of a multiprocessing RL algo I am working on when executed on the Linux machines.
On the Linux machines torch seems to be using only a single thread when doing the multiplication, as opposed to macOS. Even though torch.get_num_threads() return the correct no of threads on each system.
###Results:
macOS: Sierra, CPU: Intel i7-4870HQ (8) @ 2.50GHz, 16GB RAM, GeForce GT 750M. Installed from sources.
Allocation: 5.921
Torch Blas: 7.277
Numpy Blas: 7.841
Torch cuBlas: 0.205
Ubuntu 16.10, CPU: Intel i7-4720HQ (8) @ 3.60GHz, 16GB RAM, GeForce GTX 960M. Installed from sources.
Allocation: 4.030
Torch Blas: 21.112
Numpy Blas: 21.82
Torch cuBlas: 0.121
CentOS 7.2, CPU: Intel Xeon E5-2640v4 (40) @ 2.40GHz, 16GB RAM, Titan X. Installed both from source and with conda. Also ran the test with python 3.5 and pytorch built from sources.
Allocation: 4.557
Torch Blas: 19.646
Numpy Blas: 20.155
Torch cuBlas: 0.155
Finally, this is the output of np.__config__.show() on all the machines:
openblas_lapack_info:
NOT AVAILABLE
lapack_opt_info:
define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
include_dirs = ['/opt/anaconda3/include']
library_dirs = ['/opt/anaconda3/lib']
libraries = ['mkl_intel_lp64', 'mkl_intel_thread', 'mkl_core', 'iomp5', 'pthread']
blas_mkl_info:
...
blas_opt_info:
...
lapack_mkl_info:
...
The code I am using:
import time
import torch
import numpy
torch.set_default_tensor_type("torch.FloatTensor")
w = 5000
h = 40000
is_cuda = torch.cuda.is_available()
start = time.time()
a = torch.rand(w,h)
b = torch.rand(h,w)
a_np = a.numpy()
b_np = b.numpy()
if is_cuda:
a_cu = a.cuda()
b_cu = b.cuda()
allocation = time.time()
print("Allocation ", allocation - start)
c = a.mm(b)
th_blas = time.time()
print("Torch Blas ", th_blas - allocation)
c = a_np.dot(b_np)
np_blas = time.time()
print("Numpy Blas ", np_blas - th_blas)
if is_cuda:
c = a_cu.mm(b_cu)
cu_blas = time.time()
print("Torch cuBlas ", cu_blas - np_blas)
print("Final", time.time() - start)
edit: For comparison here are the results of the same script on Lua Torch on the last machine from above:
Allocation: 4.426
Torch Blas: 2.777
Torch cuBlas: 0.097
At this point I am more inclined to believe my linux pytorch installs are using a BLAS fallback. Hoping this isn’t Python’s overhead… 