What is the recommended way to use PyTorch models in C++ for deployment?
I have read different articles on the forum but could not find any conclusion. Assuming that development effort is not an issue, will the pure C++ model be faster during training and inference compared to a Python model converted using TorchScript and then loaded in C++ application?
Are there any limitations to the Python -> TorchScript -> C++ approach compared to pure C++ models?
Also, what is the long term support plan by PyTorch team for TorchScript vs C++ frontend (to better understand the recommended approach)?
Hi @ptrblck, @yf225, @albanD, @Michael_Suo, saw your comments on other similar posts so I thought perhaps you could provide some valuable input.
I also reviewed the PyTorch C++ Deployment Story: 2019 post. But I am not clear on difference on speed between TorchScript and pure C++. Also, I could use some input on limitations of TorchScript. I read on some posts that LSTM and maybe some conditional constructs if/loops are not supported (or require extra effort that I am not completely clear on).
Writing directly in C++ is needed, for deployment, if your network topology is dynamic (depend on the inputs such as Tree) or you need to train in an environment that can’t have Python runtime. In most other cases, TorchScript (if the model has control flow or loop) or Trace is the way to go. Speed wise, it should be pretty close, script might be faster in some cases due to the optimization passes (i.e. OP fusion, constant folding…etc).