I want to develop a torch-serving backend with libtorch.
as the libtorch c++ frontend example shows:
torch::jit::script::Module module;
try {
// Deserialize the ScriptModule from a file using torch::jit::load().
module = torch::jit::load(argv[1]);
}
catch (const c10::Error& e) {
std::cerr << "error loading the model\n";
return -1;
}
std::cout << "ok\n";
std::vector<torch::jit::IValue> inputs;
inputs.push_back(torch::ones({1, 3, 224, 224}));
// Execute the model and turn its output into a tensor.
at::Tensor output = module.forward(inputs).toTensor();
std::cout << output.slice(/*dim=*/1, /*start=*/0, /*end=*/5) << '\n';
now I can load traced model with libtorch , and call module.forward to do predict.
however the predict requests with inputs are coming from network by another client like python/java/c++ &.
so I need serialize the request from the client, and deserialize the request at the server backend, now I choose protobuffer to do the thing.
but I’m confused about which protodef file I can use to define the torch::jit::IValue or torch::Tensor ??
Is there anyone can help me ???