Inference in C++ pytorch

C++
1

Hi,

Is there any example for converting a detection model trained in python (weight file as .h format) to c++ and running inference using python c++?

Any response would be appreciated.
Thanks

2

The jit might be the best way to export your model to C++ and execute it.

1 Like
3

@ptrblck thank you for the response.
I checked the link but what I did not understand is if we can just trace the .pt file (trained model in python) and import in c++ or do we also need to provide the network architecture like a config file or something?
Any idea on this?
Thanks

4

You should be able to load the saved model directly in your C++ application.

1 Like
5

The traced output contains both the architecture and the weights.

1 Like
6

@ptrblck @markl Thank you for your inputs.

I tried a simple example to trace the Maskrcnn detection model using the following script

import torch
import torchvision
model = torchvision.models.detection.maskrcnn_resnet50_fpn(pretrained=True)
model.eval()

test_data = torch.rand(1, 3, 480, 640)
traced_model = torch.jit.trace(model, test_data)

However, I got this error “RuntimeError: The expanded size of the tensor (640) must match the existing size (1066) at non-singleton dimension 2. Target sizes: [3, 480, 640]. Tensor sizes: [3, 800, 1066]”

Then I changed the input to be of the size [3,800,1056], just to check and got the next error
“RuntimeError: Only tensors or tuples of tensors can be output from traced functions (getNestedOutputTrace at /pytorch/torch/csrc/jit/tracer.cpp:200)”

According to my understanding, the test_data can be any random input (can also be an image of 3 channels) needed to trace and save the model. Could someone please help address this issue?

Torch version: 1.1.0
Torchvision: 0.3.0
python: 3.5

Thank you for the responses.

7

I’m not sure why you got that error. It looks like that should work. Do you also get an error if you just call model.forward(test_data)?

8

could it be the pytorch version? as I understand there are quite some changes in this part from 1.1 to 1.2

thanks.

rgds,
CL

9

@markl

model.forward(test_data) works and outputs some tensors (boxes).
But the with torch.jit.trace I still have the issue

RuntimeError: Only tensors or tuples of tensors can be output from traced functions (getNestedOutputTrace at /pytorch/torch/csrc/jit/tracer.cpp:200)

Not sure if jit trace fully supports maskrcnn model for now.

@tancl

I updated torch and torchvision versions to 1.3 and 0.4.1 respectively. still the same error.

11

It’s possible that your model has an intermediate computation that uses a custom data structure that is not traceable.

12

looks like u hv your answer here already?

Thanks

CL