Will the real tensorrt please stand up?

torch.compile
1

I want to try a torch.compile setting the backend to ‘tensorrt’.
There are:
pip3 install tensorrt
pip3 install nvidia-tensorrt
pip3 install torch-tensorrt

I have the first two installed and I, as many others had problem with, not been able to install torch-tensorrt due to it only finding version 0.0.0 instead of the 1.4.0dev version. I’ve looked and can’t find a workaround to install it.

Here is a simple question. Given these 3 different tensorrt installation directories, when I see instructions like:
cmake ../ -DTENSORRT_ROOT=<path_to_trt>
would someone tell me how it is obvious which path I should use? And, no, it is not in the README.md file.

2

So now there is a fourth TensorRT I found. One which has NvInfer.h in it.
Will there be a fifth TensorRT if I build PyTorch with USE_TENSORRT as another subpackage to pytorch?

Will the real one please stand up and will someone please tell me when instructions refer to “the” tensorrt install what the heck does it mean?

3

Torch-TensorRT is the interface between PyTorch to TensorRT. The repository, docs, and install instructions are given in the link.
TensorRT is an SDK and used by Torch-TensorRT.

4

Thanks. I should have mentioned I wanted one with supports torch.compile. If I try to install torch-tensorrt==1.3.0 it tries to downgrade my torch to 1.13.1 instead of v2.0.0 which has compile in it.

I can find the 1.4.0, 1.4.0dev, 1.4.0dev0, … in my attempts to locate the nightly build. Unlike anything else I tried to build in the last 6 months torch-tensorrt wants me to install bazel or whatever it was called.

5

I don’t know if Torch-TensorRT is compatible with torch.compile, as the former uses TorchScript and torch.fx while the latter utilizes TorchDynamo and Inductor.
@narendasan would know more about this support and if it’s WIP or which path will be used.

6

The “tensorrt” backend has been removed in 2.0 and will be pointing to torch-tensorrt in the future. Likely that will come in PyTorch 2.1 (WIP in our repo). (Remove fx2trt/torch2trt backends by jansel · Pull Request #93822 · pytorch/pytorch · GitHub, Reserve the tensorrt backend name for torch-tensorrt by frank-wei · Pull Request #95627 · pytorch/pytorch · GitHub)

Nightlies for Torch-TensorRT can be found in the CI (e.g. https://app.circleci.com/pipelines/github/pytorch/TensorRT/2083/workflows/758ec559-da66-43dd-aeab-07233b0001ff/jobs/9609/artifacts)

7

Thanks. Given the huge perf boost others report using tensorrt for inference like voltaML, which I can confirm, shipping pytorch 2.0 with only the marginally better, for inference, “intductor” will be a disappointment.

Odd that it was removed before something to replace it was ready, unless getting it stable became too much of a hassle. Even Inductor has bugs which I have to fix in my build to get it to run to get a 5 to 15% boost in inference perf.

Yesterday I tried torch_tensorrt 1.3.1 but the following even though I’m not training anything:
RuntimeError: DiffusionWrapper is not attached to a Trainer.
using:
torch_tensorrt.compile(m.model, input=input, enabled_precisions={torch.half}, workspace_size=1<<32, require_full_compilation=True)

I’ll try with the nightly build of this.