Is it possible to trace calls to the shared library routines in libAten
from within python while pytorch models are running,
or does this have to happen outside via something like dtrace/strace?
thanks
I wasn’t able to do anything from within python,
but for mac os x (and something similar should work for linux),
i used dtrace and the script: dapptrace that comes with it.
To find the libraries loaded by torch:
export DYLD_PRINT_LIBRARIES=YES
>>> import torch
dyld: loaded: /Users/t/anaconda3/lib/python3.6/lib-dynload/math.cpython-36m-darwin.so
dyld: loaded: /Users/t/anaconda3/lib/python3.6/site-packages/numpy/core/multiarray.cpython-36m-darwin.so
…
_dyld: loaded: /Users/t/anaconda3/lib/python3.6/site-packages/torch/C.cpython-36m-darwin.so
dyld: loaded: /Users/t/anaconda3/lib/python3.6/site-packages/torch/lib/libshm.dylib
dyld: loaded: /Users/t/anaconda3/lib/python3.6/site-packages/torch/lib/libTH.1.dylib
dyld: loaded: /Users/t/anaconda3/lib/python3.6/site-packages/torch/lib/libTHS.1.dylib
dyld: loaded: /Users/t/anaconda3/lib/python3.6/site-packages/torch/lib/libTHPP.1.dylib
dyld: loaded: /Users/t/anaconda3/lib/python3.6/site-packages/torch/lib/libTHNN.1.dylib
dyld: loaded: /Users/t/anaconda3/lib/python3.6/site-packages/torch/lib/libATen.1.dylib
dyld: loaded: /Users/t/anaconda3/lib/python3.6/site-packages/torch/lib/…/…/…/…/libmkl_intel_lp64.dylib
…
then, attach to the process, something like:
sudo dapptrace -p 1558 -u libTH.1 libTHS.1 libTHPP.1 libTHNN.1 libATen.1
then python statements can be traced in terms of the library calls:
>>> b=torch.randn(5,5)
>>> a=torch.randn(5,5)
>>> c=torch.mm(a,b)
dyld: loaded: /Users/t/anaconda3/lib/libmkl_avx2.dylib
results in dtrace output like:
_-> THFloatTensor_addmm(0x7F9F9B61B400, 0x7F9F9B61B400, 0x7F9F9B704DB0) _
_-> THFloatBlas_gemm(0x6E, 0x6E, 0x5) _
<- THFloatBlas_gemm = 328
<- THFloatTensor_addmm = 1770