Creating a tuple of Tensors as input to a traced model with the C++ front end

I can’t seem to figure out how to create a tuple of tensors that are will be the input to a traced module for use with the c++ front end.

I have a model that I trace with this example code:

import torch

def forward(inputs):
    return inputs[0] + inputs[1]

example_input = tuple(torch.rand(5, 10) for _ in range(2))
traced = torch.jit.trace(forward, (example_input,))
traced.save('example.pt')
print(traced(example_input).shape)

Then I have a c++ program that tries to use it like so

#include <torch/script.h>
#include <iostream>
#include <memory>

int main(int argc, const char** argv) {
    if (argc != 2) {
        std::cerr << "usage: smoke <path-to-exported-script-module>\n";
        return -1;
    }
    std::shared_ptr<torch::jit::script::Module> module = torch::jit::load(argv[1]);
    assert(module != nullptr);

    std::vector<torch::jit::IValue> inputs;
    inputs.push_back(std::make_tuple(torch::zeros({5, 10}), torch::zeros({5, 10})));

    at::Tensor output = module->forward(inputs).toTensor();
    std::cout << output << '\n';
    return 0;
}

When I try to compile this I get an error that I can’t add a tuple to the the vector

/home/blester/torch_script/example/example.cpp:18:83: error: no matching function for call to ‘std::vector<c10::IValue, std::allocator<c10::IValue> >::push_back(std::tuple<at::Tensor, at::Tensor>)’
     inputs.push_back(std::make_tuple(torch::zeros({5, 10}), torch::zeros({5, 10})));

I also tried creating a vector of IValue to represent the tuple which was able to be compiled but at runtime threw this error

    std::vector<torch::jit::IValue> inputs;
    std::vector<torch::jit::IValue> tuple;
    tuple.push_back(torch::zeros({5, 10}));
    tuple.push_back(torch::zeros({5, 10}));
    inputs.push_back(tuple);
terminate called after throwing an instance of 'c10::Error'
  what():  Type cannot be accurately recovered from this IValue. (incompleteInferTypeFrom at /pytorch/aten/src/ATen/core/type.cpp:149)

Similarly making a vector of tensors with

    std::vector<torch::jit::IValue> inputs;
    std::vector<torch::Tensor> tuple;
    tuple.push_back(torch::zeros({5, 10}));
    tuple.push_back(torch::zeros({5, 10}));
    inputs.push_back(tuple);

compiles but running it throws another error

terminate called after throwing an instance of 'c10::Error'
  what():  Expected value of type (Tensor, Tensor) for argument '_0' in position 0, but instead got value of type Tensor[]. Declaration: forward((Tensor, Tensor) _0) -> Tensor (checkInputsAgainstSchema at /home/blester/torch_script/libtorch/include/torch/csrc/jit/script/module.h:249)

So I think I am missing something about how to create input for this case. Any guidance would be greatly appreciated.

1 Like

Hi,

I’ve created a bug report you can track here. https://github.com/pytorch/pytorch/issues/17165

Edit:
you should be able to run it with:

    std::vector<torch::jit::IValue> inputs;
    std::vector<torch::jit::IValue> tuple;
    tuple.push_back(torch::zeros({5, 10}));
    tuple.push_back(torch::zeros({5, 10}));
    inputs.push_back(torch::jit::Tuple::create(tuple));

although I’m keeping the issue open because we should consider a fix so that your examples work.