How to use torchscript C++ with nn.transformer

C++
1

Hi

I’m trying to export a transformer model with torchscript for use in C++.
I am currently passing a dummy input of all zeros for debugging. The first time I call forward in C++ I get an output from the transformer fine. The second time, even though I pass the exact same input, I get the error bellow:

terminate called after throwing an instance of 'c10::Error'
  what():  Expected a list element that subtypes 'Tensor' but got an element of type 'Optional[Tensor]'
Exception raised from createList at ../torch/csrc/jit/ir/ir.cpp:1660 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x69 (0x7f55dfefeb29 in /usr/local/include/libtorch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xd2 (0x7f55dfefbab2 in /usr/local/include/libtorch/lib/libc10.so)
frame #2: torch::jit::Graph::createList(std::shared_ptr<c10::Type> const&, c10::ArrayRef<torch::jit::Value*>) + 0x185 (0x7f5625fa25f5 in /usr/local/include/libtorch/lib/libtorch_cpu.so)
frame #3: <unknown function> + 0x372f824 (0x7f562615d824 in /usr/local/include/libtorch/lib/libtorch_cpu.so)
frame #4: <unknown function> + 0x3732108 (0x7f5626160108 in /usr/local/include/libtorch/lib/libtorch_cpu.so)
frame #5: torch::jit::differentiate(std::shared_ptr<torch::jit::Graph>&) + 0x237 (0x7f5626162497 in /usr/local/include/libtorch/lib/libtorch_cpu.so)
frame #6: <unknown function> + 0x376a3a7 (0x7f56261983a7 in /usr/local/include/libtorch/lib/libtorch_cpu.so)
frame #7: <unknown function> + 0x376b329 (0x7f5626199329 in /usr/local/include/libtorch/lib/libtorch_cpu.so)
frame #8: <unknown function> + 0x376be8e (0x7f5626199e8e in /usr/local/include/libtorch/lib/libtorch_cpu.so)
frame #9: <unknown function> + 0x373e764 (0x7f562616c764 in /usr/local/include/libtorch/lib/libtorch_cpu.so)
frame #10: torch::jit::GraphFunction::operator()(std::vector<c10::IValue, std::allocator<c10::IValue> >, std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, c10::IValue, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, c10::IValue> > > const&) + 0x3a (0x7f5625e8725a in /usr/local/include/libtorch/lib/libtorch_cpu.so)
frame #11: torch::jit::Method::operator()(std::vector<c10::IValue, std::allocator<c10::IValue> >, std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, c10::IValue, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, c10::IValue> > > const&) + 0x16a (0x7f5625e9670a in /usr/local/include/libtorch/lib/libtorch_cpu.so)
frame #12: torch::jit::Module::forward(std::vector<c10::IValue, std::allocator<c10::IValue> >) + 0x131 (0x7f5635e8ab01 in /home/russell/catkin_ws/devel/lib/libvilens.so)
frame #13: vilens::TorchModelWrapper::getBiasTransformer(Eigen::Matrix<double, -1, -1, 0, -1, -1> const&, Eigen::Matrix<double, -1, -1, 0, -1, -1> const&, Eigen::Matrix<double, -1, 1, 0, -1, 1>&) + 0x665 (0x7f5635e8e5d5 in /home/russell/catkin_ws/devel/lib/libvilens.so)
frame #14: vilens::ImuBiasObserverHandler::createFactor(vilens::State const&, gtsam::Values const&, vilens::FGNodes const&, gtsam::NonlinearFactorGraph&, gtsam::Values&, bool) + 0x4238 (0x7f5635e85808 in /home/russell/catkin_ws/devel/lib/libvilens.so)
frame #15: vilens::VilensOptimizer::addFactors(std::vector<std::unique_ptr<vilens::Handler, std::default_delete<vilens::Handler> >, std::allocator<std::unique_ptr<vilens::Handler, std::default_delete<vilens::Handler> > > >&, vilens::StateMap&) + 0xfe4 (0x7f5635d49774 in /home/russell/catkin_ws/devel/lib/libvilens.so)
frame #16: vilens::VilensManager::optimize() + 0x5e (0x7f5635f68c0e in /home/russell/catkin_ws/devel/lib/libvilens.so)
frame #17: vilens::VilensManager::optimizer() + 0xbc5 (0x7f5635f6fa85 in /home/russell/catkin_ws/devel/lib/libvilens.so)
frame #18: <unknown function> + 0xda6b4 (0x7f55dfb6b6b4 in /lib/x86_64-linux-gnu/libstdc++.so.6)
frame #19: <unknown function> + 0x8609 (0x7f56355f3609 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #20: clone + 0x43 (0x7f55df9a1133 in /lib/x86_64-linux-gnu/libc.so.6)

I am doing the tracing as below:

    def torch_script_trace(self, module, imu_window_time):
        # Tracing
        exampleUInput = torch.rand(1, 6, 100)
        exampleUInputCuda = exampleUInput.cuda()
        exampleYInput = torch.rand(1, 6, 1)
        exampleYInputCuda = exampleYInput.cuda()

        traced_script_module = torch.jit.trace(
            module, (exampleUInputCuda, exampleYInputCuda))
        return traced_script_module

I have read this tutorial on Huggingface which uses a torchscript flag but I don’t see a similar thing in nn.transformer. Is there some configuration I am missing to use torchscript with transformers?
Huggingface tutorial: Exporting transformers models — transformers 3.1.0 documentation

Thank you

2

So I eventually resolved this with two changes:

  1. Instead of tracing I used annotations Loading a TorchScript Model in C++ — PyTorch Tutorials 1.11.0+cu102 documentation
  2. I updated to the most recent libtorch. I forget what version I had before but it seems older versions did not support annotations which I think are necessary for transformers.
3

We had version 1.8. That seems to be key, ensuring that Pytorch and LibTorch have the same (or at least compatible) versions.