I can’t seem to figure out how to create a tuple of tensors that are will be the input to a traced module for use with the c++ front end.
I have a model that I trace with this example code:
import torch
def forward(inputs):
return inputs[0] + inputs[1]
example_input = tuple(torch.rand(5, 10) for _ in range(2))
traced = torch.jit.trace(forward, (example_input,))
traced.save('example.pt')
print(traced(example_input).shape)
Then I have a c++ program that tries to use it like so
#include <torch/script.h>
#include <iostream>
#include <memory>
int main(int argc, const char** argv) {
if (argc != 2) {
std::cerr << "usage: smoke <path-to-exported-script-module>\n";
return -1;
}
std::shared_ptr<torch::jit::script::Module> module = torch::jit::load(argv[1]);
assert(module != nullptr);
std::vector<torch::jit::IValue> inputs;
inputs.push_back(std::make_tuple(torch::zeros({5, 10}), torch::zeros({5, 10})));
at::Tensor output = module->forward(inputs).toTensor();
std::cout << output << '\n';
return 0;
}
When I try to compile this I get an error that I can’t add a tuple to the the vector
/home/blester/torch_script/example/example.cpp:18:83: error: no matching function for call to ‘std::vector<c10::IValue, std::allocator<c10::IValue> >::push_back(std::tuple<at::Tensor, at::Tensor>)’
inputs.push_back(std::make_tuple(torch::zeros({5, 10}), torch::zeros({5, 10})));
I also tried creating a vector of IValue to represent the tuple which was able to be compiled but at runtime threw this error
std::vector<torch::jit::IValue> inputs;
std::vector<torch::jit::IValue> tuple;
tuple.push_back(torch::zeros({5, 10}));
tuple.push_back(torch::zeros({5, 10}));
inputs.push_back(tuple);
terminate called after throwing an instance of 'c10::Error'
what(): Type cannot be accurately recovered from this IValue. (incompleteInferTypeFrom at /pytorch/aten/src/ATen/core/type.cpp:149)
Similarly making a vector of tensors with
std::vector<torch::jit::IValue> inputs;
std::vector<torch::Tensor> tuple;
tuple.push_back(torch::zeros({5, 10}));
tuple.push_back(torch::zeros({5, 10}));
inputs.push_back(tuple);
compiles but running it throws another error
terminate called after throwing an instance of 'c10::Error'
what(): Expected value of type (Tensor, Tensor) for argument '_0' in position 0, but instead got value of type Tensor[]. Declaration: forward((Tensor, Tensor) _0) -> Tensor (checkInputsAgainstSchema at /home/blester/torch_script/libtorch/include/torch/csrc/jit/script/module.h:249)
So I think I am missing something about how to create input for this case. Any guidance would be greatly appreciated.