Torch 2 and libtorch c++

C++
1

Hi,

as of today (Dec. 2023): Is the recommended way to export models from PyTorch to be deployed in c++ still jit.trace / jit.script as outlined in [1]?

There seem to be many concepts related to this flying aroung:

  • torch.export [2] states “this graph can then be saved, loaded, and run in different environments and languages”.
  • [3] states “in particular aot inductor is the new recommended path for people wanting c++ deployments”
  • There is torch.compile.

I, therefore would appreciate if somewone coul clarify what is the recommended way to export pre-trained Models to c++ in the most efficient way for inference.

Appreciate any reply!

[1] Loading a TorchScript Model in C++ — PyTorch Tutorials 2.2.0+cu121 documentation
[2] torch.export — PyTorch 2.1 documentation
[3] What is the use of exportedprogram?

2 Likes
2

I would also like some guidance on this from a PyTorch dev. I have not yet come across an example of using any of the PyTorch 2 functionality (compile/export) to be able to export a simple model and load it into C++ for inference.

Is there any timeline for when torch.export will be ready to do this?/ Will it be able to do this?

I am still using torch.jit.script to make a TorchScript model I can load into C++. Despite TorchScript being in “maintenance mode”

3

[Not a Contribution]

Hi @ptrblck @tom, any pointers about this?

It seems like torch.export is the intended way for C++ (non-Python) deployment based on the first bullet above. But it’s not clear if the API to load a serialized ExportedProgram is available in libTorch.

torch.compile doesn’t seem to support serialization (as mentioned here).

Any heads up about future roadmaps to support this use-case?

Thanks!

4

[Not a Contribution]

Tagging @smth @albanD if they have any additional inputs. Thanks!

5

With 2.2, we released documentation of torch.export, along with showing exported models in C++ runtime.
Please see AOTInductor: Ahead-Of-Time Compilation for Torch.Export-ed Models — PyTorch main documentation

1 Like
6

Very interesting. Can you share if there are plans to supported inference on Windows as well? And in general, will the new export workflow have feature parity with Torchscript (support on MacOS / Windows / Ubuntu, support for calling different methods etc.)? Thank you!

7

Unfortunately I think AOTInductor won’t allow to train the model, while you can train a model imported from Torchscript.
Can someone confirm that ? It there a future for training imported model with C++ (or any language using libtorch for binding) ?