PyTorch inference on tensors of a particular size cause Illegal Instruction (core dumped) on Jetson Nano

1

Hi there,

I’m trying to run webcam feed based inference on a pretrained pytorch model on the Jetson Nano. When I run a frame through the network I get the

Illegal instruction (core dumped)

error message. I can reliably reproduce this error using the following minimal script:

import torch
import torch.nn as nn

lstm = nn.LSTM(63, 70, 1, True)
x = torch.randn(1,14,63)
lstm(x)

The dimensions of the model and the input are chosen to fit the data I’m using. In reality the model should perform inference on tensors of size (1,175,63). Interestingly running the following:

import torch
import torch.nn as nn

lstm = nn.LSTM(63, 70, 1, True)
for i in range(175):
     x = torch.randn(1,i,63)
     lstm(x)

produces output until the size of the second dimension of x reaches 14 at which point the core dumped error message occurs. Any ideas what’s happening here? I’ve read that torch has some issues with old CPUs but seeing as it seems to be working up to a certain point I doubt this was an issue. I tested this by trying out some solutions posted here which involved entering

export OPENBLAS_CORETYPE=ARMV8

before running the script in the terminal. I’m also running numpy version 1.19.4 which was suggested to be a solution for a similar problem in the above thread so that’s probably not the issue.

Some specs:

python version: 3.6.9
numpy version: 1.19.5
torch version: 1.9.0

system: Jetson Nano 2GB
os: Ubuntu 18.04.5 LTS (Bionic Beaver)

many thanks.

2

I’ve hit a similar issue in the past and were seeing that a vectorized code path was used internally starting from a specific length of the tensor. Could you check the backtrace via:

gdb --args python script.py
...
run
...
bt
3

Hi @ptrblck . Thanks for the response. Here’s the backtrace:

#0  0x0000007fb3efefc8 in exec_blas ()
   from /home/project/.local/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so
#1  0x0000007fb3ed0150 in gemm_driver ()
   from /home/project/.local/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so
#2  0x0000007fb3ed0bd0 in sgemm_thread_nn ()
   from /home/project/.local/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so
#3  0x0000007fb3ec95bc in sgemm_ ()
   from /home/project/.local/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so
#4  0x0000007fb12c9b8c in at::native::cpublas::gemm(at::native::cpublas::TransposeType, at::native::cpublas::TransposeType, long, long, long, float, float const*, long, float const*, long, float, float*, long) ()
   from /home/project/.local/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so
#5  0x0000007fb1376c48 in at::native::addmm_impl_cpu_(at::Tensor&, at::Tensor const&, at::Tensor, at::Tensor, c10::Scalar const&, c10::Scalar const&) ()
   from /home/project/.local/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so
#6  0x0000007fb1377618 in at::native::structured_addmm_out_cpu::impl(at::Tensor const&, at::Tensor const&, at::Tensor const&, c10::Scalar const&, c10::Scalar co---Type <return> to continue, or q <return> to quit---
nst&, at::Tensor const&) ()
   from /home/project/.local/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so
#7  0x0000007fb1bfe568 in at::(anonymous namespace)::wrapper_addmm_(at::Tensor&, at::Tensor const&, at::Tensor const&, c10::Scalar const&, c10::Scalar const&)
    ()
   from /home/project/.local/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so
#8  0x0000007fb1f863a4 in at::Tensor::addmm_(at::Tensor const&, at::Tensor const&, c10::Scalar const&, c10::Scalar const&) const ()
   from /home/project/.local/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so
#9  0x0000007fb12e01dc in at::native::slow_conv2d_forward_out_cpu(at::Tensor const&, at::Tensor const&, c10::ArrayRef<long>, c10::optional<at::Tensor> const&, c10::ArrayRef<long>, c10::ArrayRef<long>, at::Tensor&, at::Tensor&, at::Tensor&)::{lambda(long, long)#1}::operator()(long, long) const ()
   from /home/project/.local/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so
#10 0x0000007fb12e2b8c in at::native::slow_conv2d_forward_out_cpu(at::Tensor const&, at::Tensor const&, c10::ArrayRef<long>, c10::optional<at::Tensor> const&, c10::ArrayRef<long>, c10::ArrayRef<long>, at::Tensor&, at::Tensor&, at::Tensor&)
    ()
   from /home/project/.local/lib/python3.6/site-packages/torch/lib/libtorch_cpu.s---Type <return> to continue, or q <return> to quit---
o
#11 0x0000007fb12e372c in at::native::slow_conv2d_forward_cpu(at::Tensor const&, at::Tensor const&, c10::ArrayRef<long>, c10::optional<at::Tensor> const&, c10::ArrayRef<long>, c10::ArrayRef<long>) ()
   from /home/project/.local/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so
#12 0x0000007fb1c03a14 in c10::impl::wrap_kernel_functor_unboxed_<c10::impl::detail::WrapFunctionIntoFunctor_<c10::CompileTimeFunctionPointer<std::tuple<at::Tensor, at::Tensor, at::Tensor> (at::Tensor const&, at::Tensor const&, c10::ArrayRef<long>, c10::optional<at::Tensor> const&, c10::ArrayRef<long>, c10::ArrayRef<long>), &at::(anonymous namespace)::(anonymous namespace)::wrapper_thnn_conv2d_forward>, std::tuple<at::Tensor, at::Tensor, at::Tensor>, c10::guts::typelist::typelist<at::Tensor const&, at::Tensor const&, c10::ArrayRef<long>, c10::optional<at::Tensor> const&, c10::ArrayRef<long>, c10::ArrayRef<long> > >, std::tuple<at::Tensor, at::Tensor, at::Tensor> (at::Tensor const&, at::Tensor const&, c10::ArrayRef<long>, c10::optional<at::Tensor> const&, c10::ArrayRef<long>, c10::ArrayRef<long>)>::call(c10::OperatorKernel*, c10::DispatchKeySet, at::Tensor const&, at::Tensor const&, c10::ArrayRef<long>, c10::optional<at::Tensor> const&, c10::ArrayRef<long>, c10::ArrayRef<long>) ()
   from /home/wallbo/.local/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so
#13 0x0000007fb1b85684 in at::redispatch::thnn_conv2d_forward(c10::DispatchKeySet, at::Tensor const&, at::Tensor const&, c10::ArrayRef<long>, c10::optional<at::---Type <return> to continue, or q <return> to quit---
Tensor> const&, c10::ArrayRef<long>, c10::ArrayRef<long>) ()
   from /home/wallbo/.local/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so
#14 0x0000007fb3045f28 in torch::autograd::VariableType::(anonymous namespace)::thnn_conv2d_forward(c10::DispatchKeySet, at::Tensor const&, at::Tensor const&, c10::ArrayRef<long>, c10::optional<at::Tensor> const&, c10::ArrayRef<long>, c10::ArrayRef<long>) ()
   from /home/wallbo/.local/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so
#15 0x0000007fb30466d4 in c10::impl::wrap_kernel_functor_unboxed_<c10::impl::detail::WrapFunctionIntoFunctor_<c10::CompileTimeFunctionPointer<std::tuple<at::Tensor, at::Tensor, at::Tensor> (c10::DispatchKeySet, at::Tensor const&, at::Tensor const&, c10::ArrayRef<long>, c10::optional<at::Tensor> const&, c10::ArrayRef<long>, c10::ArrayRef<long>), &torch::autograd::VariableType::(anonymous namespace)::thnn_conv2d_forward>, std::tuple<at::Tensor, at::Tensor, at::Tensor>, c10::guts::typelist::typelist<c10::DispatchKeySet, at::Tensor const&, at::Tensor const&, c10::ArrayRef<long>, c10::optional<at::Tensor> const&, c10::ArrayRef<long>, c10::ArrayRef<long> > >, std::tuple<at::Tensor, at::Tensor, at::Tensor> (c10::DispatchKeySet, at::Tensor const&, at::Tensor const&, c10::ArrayRef<long>, c10::optional<at::Tensor> const&, c10::ArrayRef<long>, c10::ArrayRef<long>)>::call(c10::OperatorKernel*, c10::DispatchKeySet, at::Tensor const&, at::Tensor const&, c10::ArrayRef<long>, c10::optional<at::Tensor> const&, c10::ArrayRef<long>, c10::ArrayRef<long>) ()
---Type <return> to continue, or q <return> to quit---
   from /home/project/.local/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so
#16 0x0000007fb18e90f0 in at::thnn_conv2d_forward(at::Tensor const&, at::Tensor const&, c10::ArrayRef<long>, c10::optional<at::Tensor> const&, c10::ArrayRef<long>, c10::ArrayRef<long>) ()
   from /home/project/.local/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so
#17 0x0000007fb12dfdfc in at::native::thnn_conv2d(at::Tensor const&, at::Tensor const&, c10::ArrayRef<long>, c10::optional<at::Tensor> const&, c10::ArrayRef<long>, c10::ArrayRef<long>) ()
   from /home/project/.local/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so
#18 0x0000007fb1db748c in c10::impl::wrap_kernel_functor_unboxed_<c10::impl::detail::WrapFunctionIntoFunctor_<c10::CompileTimeFunctionPointer<at::Tensor (at::Tensor const&, at::Tensor const&, c10::ArrayRef<long>, c10::optional<at::Tensor> const&, c10::ArrayRef<long>, c10::ArrayRef<long>), &at::(anonymous namespace)::(anonymous namespace)::wrapper_thnn_conv2d>, at::Tensor, c10::guts::typelist::typelist<at::Tensor const&, at::Tensor const&, c10::ArrayRef<long>, c10::optional<at::Tensor> const&, c10::ArrayRef<long>, c10::ArrayRef<long> > >, at::Tensor (at::Tensor const&, at::Tensor const&, c10::ArrayRef<long>, c10::optional<at::Tensor> const&, c10::ArrayRef<long>, c10::ArrayRef<long>)>::call(c10::OperatorKernel*, c10::DispatchKeySet, at::Tensor const&, at::Tensor const&, c10::ArrayRef<long>, c10::optional<at::Tensor> const&, c10::ArrayRef<long>, c10::ArrayRef<long>) ()
---Type <return> to continue, or q <return> to quit---
   from /home/project/.local/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so
#19 0x0000007fb18e74a0 in at::thnn_conv2d(at::Tensor const&, at::Tensor const&, c10::ArrayRef<long>, c10::optional<at::Tensor> const&, c10::ArrayRef<long>, c10::ArrayRef<long>) ()
   from /home/project/.local/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so
#20 0x0000007fb12d5f30 in at::native::_convolution_nogroup(at::Tensor const&, at::Tensor const&, c10::optional<at::Tensor> const&, c10::ArrayRef<long>, c10::ArrayRef<long>, c10::ArrayRef<long>, bool, c10::ArrayRef<long>) ()
   from /home/project/.local/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so
#21 0x0000007fb1daf1f0 in c10::impl::wrap_kernel_functor_unboxed_<c10::impl::detail::WrapFunctionIntoFunctor_<c10::CompileTimeFunctionPointer<at::Tensor (at::Tensor const&, at::Tensor const&, c10::optional<at::Tensor> const&, c10::ArrayRef<long>, c10::ArrayRef<long>, c10::ArrayRef<long>, bool, c10::ArrayRef<long>), &at::(anonymous namespace)::(anonymous namespace)::wrapper__convolution_nogroup>, at::Tensor, c10::guts::typelist::typelist<at::Tensor const&, at::Tensor const&, c10::optional<at::Tensor> const&, c10::ArrayRef<long>, c10::ArrayRef<long>, c10::ArrayRef<long>, bool, c10::ArrayRef<long> > >, at::Tensor (at::Tensor const&, at::Tensor const&, c10::optional<at::Tensor> const&, c10::ArrayRef<long>, c10::ArrayRef<long>, c10::ArrayRef<long>, bool, c10::ArrayRef<long>)>::call(c10::OperatorKernel*, c10::DispatchKeySet, at::Tensor const&, at::Tensor const&, c10::option---Type <return> to continue, or q <return> to quit---
al<at::Tensor> const&, c10::ArrayRef<long>, c10::ArrayRef<long>, c10::ArrayRef<long>, bool, c10::ArrayRef<long>) ()
   from /home/project/.local/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so
#22 0x0000007fb17a62fc in at::_convolution_nogroup(at::Tensor const&, at::Tensor const&, c10::optional<at::Tensor> const&, c10::ArrayRef<long>, c10::ArrayRef<long>, c10::ArrayRef<long>, bool, c10::ArrayRef<long>) ()
   from /home/project/.local/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so
#23 0x0000007fb12dc8a8 in at::native::_convolution(at::Tensor const&, at::Tensor const&, c10::optional<at::Tensor> const&, c10::ArrayRef<long>, c10::ArrayRef<long>, c10::ArrayRef<long>, bool, c10::ArrayRef<long>, long, bool, bool, bool, bool) ()
   from /home/project/.local/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so
#24 0x0000007fb1daf0ec in c10::impl::wrap_kernel_functor_unboxed_<c10::impl::detail::WrapFunctionIntoFunctor_<c10::CompileTimeFunctionPointer<at::Tensor (at::Tensor const&, at::Tensor const&, c10::optional<at::Tensor> const&, c10::ArrayRef<long>, c10::ArrayRef<long>, c10::ArrayRef<long>, bool, c10::ArrayRef<long>, long, bool, bool, bool, bool), &at::(anonymous namespace)::(anonymous namespace)::wrapper__convolution>, at::Tensor, c10::guts::typelist::typelist<at::Tensor const&, at::Tensor const&, c10::optional<at::Tensor> const&, c10::ArrayRef<long>, c10::ArrayRef<long>, c10::ArrayRef<long>, bool, c10::ArrayRef<long>, long, bool, boo---Type <return> to continue, or q <return> to quit---
l, bool, bool> >, at::Tensor (at::Tensor const&, at::Tensor const&, c10::optional<at::Tensor> const&, c10::ArrayRef<long>, c10::ArrayRef<long>, c10::ArrayRef<long>, bool, c10::ArrayRef<long>, long, bool, bool, bool, bool)>::call(c10::OperatorKernel*, c10::DispatchKeySet, at::Tensor const&, at::Tensor const&, c10::optional<at::Tensor> const&, c10::ArrayRef<long>, c10::ArrayRef<long>, c10::ArrayRef<long>, bool, c10::ArrayRef<long>, long, bool, bool, bool, bool) ()
   from /home/wallbo/.local/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so
#25 0x0000007fb17a3eac in at::_convolution(at::Tensor const&, at::Tensor const&, c10::optional<at::Tensor> const&, c10::ArrayRef<long>, c10::ArrayRef<long>, c10::ArrayRef<long>, bool, c10::ArrayRef<long>, long, bool, bool, bool, bool) ()
   from /home/wallbo/.local/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so
#26 0x0000007fb12d5470 in at::native::convolution(at::Tensor const&, at::Tensor const&, c10::optional<at::Tensor> const&, c10::ArrayRef<long>, c10::ArrayRef<long>, c10::ArrayRef<long>, bool, c10::ArrayRef<long>, long) ()
   from /home/wallbo/.local/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so
#27 0x0000007fb1daf050 in c10::impl::wrap_kernel_functor_unboxed_<c10::impl::detail::WrapFunctionIntoFunctor_<c10::CompileTimeFunctionPointer<at::Tensor (at::Tensor const&, at::Tensor const&, c10::optional<at::Tensor> const&, c10::ArrayRef<long>, c10::ArrayRef<long>, c10::ArrayRef<long>, bool, c10::ArrayRef<long>, long), &at::(anonymous namespace)::(anonymous namespace)::wrapper_convolution>, at::---Type <return> to continue, or q <return> to quit---
Tensor, c10::guts::typelist::typelist<at::Tensor const&, at::Tensor const&, c10::optional<at::Tensor> const&, c10::ArrayRef<long>, c10::ArrayRef<long>, c10::ArrayRef<long>, bool, c10::ArrayRef<long>, long> >, at::Tensor (at::Tensor const&, at::Tensor const&, c10::optional<at::Tensor> const&, c10::ArrayRef<long>, c10::ArrayRef<long>, c10::ArrayRef<long>, bool, c10::ArrayRef<long>, long)>::call(c10::OperatorKernel*, c10::DispatchKeySet, at::Tensor const&, at::Tensor const&, c10::optional<at::Tensor> const&, c10::ArrayRef<long>, c10::ArrayRef<long>, c10::ArrayRef<long>, bool, c10::ArrayRef<long>, long) ()
   from /home/wallbo/.local/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so
#28 0x0000007fb17a2ecc in at::convolution(at::Tensor const&, at::Tensor const&, c10::optional<at::Tensor> const&, c10::ArrayRef<long>, c10::ArrayRef<long>, c10::ArrayRef<long>, bool, c10::ArrayRef<long>, long) ()
   from /home/wallbo/.local/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so
#29 0x0000007fb12d4af4 in at::native::conv1d(at::Tensor const&, at::Tensor const&, c10::optional<at::Tensor> const&, c10::ArrayRef<long>, c10::ArrayRef<long>, c10::ArrayRef<long>, long) ()
   from /home/wallbo/.local/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so
#30 0x0000007fb1daf30c in c10::impl::wrap_kernel_functor_unboxed_<c10::impl::detail::WrapFunctionIntoFunctor_<c10::CompileTimeFunctionPointer<at::Tensor (at::Tensor const&, at::Tensor const&, c10::optional<at::Tensor> const&, c10::ArrayRef<---Type <return> to continue, or q <return> to quit---
long>, c10::ArrayRef<long>, c10::ArrayRef<long>, long), &at::(anonymous namespace)::(anonymous namespace)::wrapper_conv1d>, at::Tensor, c10::guts::typelist::typelist<at::Tensor const&, at::Tensor const&, c10::optional<at::Tensor> const&, c10::ArrayRef<long>, c10::ArrayRef<long>, c10::ArrayRef<long>, long> >, at::Tensor (at::Tensor const&, at::Tensor const&, c10::optional<at::Tensor> const&, c10::ArrayRef<long>, c10::ArrayRef<long>, c10::ArrayRef<long>, long)>::call(c10::OperatorKernel*, c10::DispatchKeySet, at::Tensor const&, at::Tensor const&, c10::optional<at::Tensor> const&, c10::ArrayRef<long>, c10::ArrayRef<long>, c10::ArrayRef<long>, long) ()
   from /home/wallbo/.local/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so
#31 0x0000007fb17a6ee4 in at::conv1d(at::Tensor const&, at::Tensor const&, c10::optional<at::Tensor> const&, c10::ArrayRef<long>, c10::ArrayRef<long>, c10::ArrayRef<long>, long) ()
   from /home/wallbo/.local/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so
#32 0x0000007fb642d100 in torch::autograd::THPVariable_conv1d(_object*, _object*, _object*) ()
   from /home/wallbo/.local/lib/python3.6/site-packages/torch/lib/libtorch_python.so
#33 0x00000000005bbc8c in _PyCFunction_FastCallDict ()
#34 0x000000000052ba70 in ?? ()
#35 0x00000000005306c0 in _PyEval_EvalFrameDefault ()
---Type <return> to continue, or q <return> to quit---
#36 0x0000000000529978 in ?? ()
#37 0x000000000052b8f4 in ?? ()
#38 0x00000000005306c0 in _PyEval_EvalFrameDefault ()
#39 0x0000000000529978 in ?? ()
#40 0x0000000000607a20 in _PyObject_FastCallDict ()
#41 0x00000000005f4d34 in ?? ()
#42 0x0000000000608da8 in PyObject_Call ()
#43 0x000000000052f2d8 in _PyEval_EvalFrameDefault ()
#44 0x000000000052b108 in ?? ()
#45 0x000000000052b42c in _PyFunction_FastCallDict ()
#46 0x0000000000607a20 in _PyObject_FastCallDict ()
#47 0x00000000005f4d34 in ?? ()
#48 0x0000000000608da8 in PyObject_Call ()
#49 0x0000000000597fb4 in ?? ()
#50 0x0000000000607948 in _PyObject_FastCallDict ()
#51 0x000000000052b850 in ?? ()
#52 0x00000000005306c0 in _PyEval_EvalFrameDefault ()
#53 0x0000000000529978 in ?? ()
#54 0x0000000000607a20 in _PyObject_FastCallDict ()
#55 0x00000000005f4d34 in ?? ()
#56 0x0000000000608da8 in PyObject_Call ()
#57 0x000000000052f2d8 in _PyEval_EvalFrameDefault ()
#58 0x000000000052b108 in ?? ()
---Type <return> to continue, or q <return> to quit---
#59 0x000000000052b42c in _PyFunction_FastCallDict ()
#60 0x0000000000607a20 in _PyObject_FastCallDict ()
#61 0x00000000005f4d34 in ?? ()
#62 0x0000000000608da8 in PyObject_Call ()
#63 0x0000000000597fb4 in ?? ()
#64 0x0000000000607948 in _PyObject_FastCallDict ()
#65 0x000000000052b850 in ?? ()
#66 0x00000000005306c0 in _PyEval_EvalFrameDefault ()
#67 0x0000000000529978 in ?? ()
#68 0x0000000000607a20 in _PyObject_FastCallDict ()
#69 0x00000000005f4d34 in ?? ()
#70 0x0000000000608da8 in PyObject_Call ()
#71 0x000000000052f2d8 in _PyEval_EvalFrameDefault ()
#72 0x000000000052b108 in ?? ()
#73 0x000000000052b42c in _PyFunction_FastCallDict ()
#74 0x0000000000607a20 in _PyObject_FastCallDict ()
#75 0x00000000005f4d34 in ?? ()
#76 0x0000000000608da8 in PyObject_Call ()
#77 0x0000000000597fb4 in ?? ()
#78 0x0000000000607948 in _PyObject_FastCallDict ()
#79 0x000000000052b850 in ?? ()
#80 0x00000000005306c0 in _PyEval_EvalFrameDefault ()
#81 0x000000000052b108 in ?? ()
---Type <return> to continue, or q <return> to quit---
#82 0x000000000052b69c in ?? ()
#83 0x000000000052b8f4 in ?? ()
#84 0x0000000000530918 in _PyEval_EvalFrameDefault ()
#85 0x000000000052b108 in ?? ()
#86 0x0000000000631598 in PyRun_FileExFlags ()
#87 0x0000000000636c2c in PyRun_SimpleFileExFlags ()
#88 0x0000000000621428 in Py_Main ()
#89 0x0000000000420d3c in main ()
4

I’ve managed to fix the problem. I was directed to this set up protocol for installing pytorch specifically for the Jetson Nano. Still not sure exactly what the problem was but hopefully that link can be of help to anyone facing the same problem. Thanks for your assistance @ptrblck.

5

Illegal instruction (core dumped)

usually its a cpu problem that I have been met

This is last non-using avx pytorch version, that I have test, my cpu have no avx

pip3 install torch==1.8.1+cu111 torchvision==0.9.1+cu111 torchaudio==0.8.1 -f https://download.pytorch.org/whl/torch_stable.html