Advantages of TorchScript vs. Porting a Model to C++

Hello. I have a relatively simple model I’m interested in using in production, in particular it is a graph convolutional network CTR-GCN, used for skeletal action recognition, with not much variation in
control statements (if statements).

I’m wondering given the two choices of:

  1. Using TorchScript to scipt the model, save a checkpoint, and then load it in C++ as in this
    tutorial :
    Loading a TorchScript Model in C++ — PyTorch Tutorials 1.11.0+cu102 documentation

  2. Manually port the code to LibTorch C++ and train a new model (the training process is sufficiently fast, this is a minor concern)

At inference time, will there be a significant advantage to either of these approaches in terms of
performance? Any insights much appreciated.

In theory torchscipt can be substantially faster at inference depending on the number of fusable operations in your model (more likely if you have custom layers/activations that have many pointwise operations). If the computation time in your model skews more heavily towards compute-heavy ops like convs and matmuls then the gains may not be as dramatic, but I would still consider a quick experiment using the Python frontend of jit scripting to see if there is a substantial speedup. TorchScript — PyTorch 1.11.0 documentation

1 Like