Hi, all. Currently I’m looking for an official PyTorch model serving solution with high performance. From the docs about PyTorch model serving, there are several options for model deployment. I’m afraid that none of the solutions can meet all the requirement at the same time.
- Deploy model using Flask: Python have GIL, there may be performance issue if the traffic is heavy.
- TorchServe: The key logic (preprocess, inference, postprocess) is in the torch_handlers. The handlers are also Python modules. There may be performance issue because of the same reason as the option above.
- Load TorchScript using LibTorch: It provides the LibTorch library instead of a model server. And the solution doesn’t contain preprocess and postprocess.
Look forward to more suggestions.