Advantages of TorchScript vs. Porting a Model to C++

1

Hello. I have a relatively simple model I’m interested in using in production, in particular it is a graph convolutional network CTR-GCN, used for skeletal action recognition, with not much variation in
control statements (if statements).

I’m wondering given the two choices of:

  1. Using TorchScript to scipt the model, save a checkpoint, and then load it in C++ as in this
    tutorial :
    Loading a TorchScript Model in C++ — PyTorch Tutorials 1.11.0+cu102 documentation

  2. Manually port the code to LibTorch C++ and train a new model (the training process is sufficiently fast, this is a minor concern)

At inference time, will there be a significant advantage to either of these approaches in terms of
performance? Any insights much appreciated.

2

In theory torchscipt can be substantially faster at inference depending on the number of fusable operations in your model (more likely if you have custom layers/activations that have many pointwise operations). If the computation time in your model skews more heavily towards compute-heavy ops like convs and matmuls then the gains may not be as dramatic, but I would still consider a quick experiment using the Python frontend of jit scripting to see if there is a substantial speedup. TorchScript — PyTorch 1.11.0 documentation

1 Like