Is "torch_tensorrt" an invalid backend?

torch.compile
1

When trying to use the "torch_tensorrt" backend from these docs: Torch-TensorRT torch.compile Backend — Torch-TensorRT v2.2.0.dev0+f617898 documentation, I get an error saying the backend doesn’t exist. Is the documentation out-dated?

2

It works for me using the current nightly binaries: torch==2.4.0.dev20240610+cu124 and torch_tensorrt==2.4.0.dev20240611+cu124:

import torch
import torch_tensorrt
import torchvision.models as models


model = models.resnet18().cuda()

optimized_model = torch.compile(model, backend="tensorrt")
print(optimized_model)
# OptimizedModule(
#   (_orig_mod): ResNet(
#     (conv1): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)
#     (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
#     (relu): ReLU(inplace=True)
#     (maxpool): MaxPool2d(kernel_size=3, stride=2, padding=1, dilation=1, ceil_mode=False)
#     (layer1): Sequential(
# ...
4

Ok, I installed torch_tensorrt with:

/tmp/train_env_v017/bin/python3 -m pip install torch_tensorrt==2.4.0.dev20240522+cu121 --extra-index-url https://download.pytorch.org/whl/nightly/cu121

Now, when I try:

        import tensorrt as trt

I get:

  File "some_path/x.py", line 157, in train
    import tensorrt as trt
  File "/tmp/train_env_v017/lib/python3.10/site-packages/tensorrt/__init__.py", line 18, in <module>
    from tensorrt_bindings import *
ModuleNotFoundError: No module named 'tensorrt_bindings'

Will debug.

Can you give a list of the package versions you’re using? I’m using

torch                         2.4.0.dev20240522+cu121
torch-fidelity                0.3.0
torch-optimizer               0.3.0
torch-summary                 1.4.5
torch_tensorrt                2.4.0.dev20240522+cu121
torchaudio                    2.2.0.dev20240522+cu121

I’m not sure if I need to be using some very-recent version of tensorrt as well.

5

These are my versions:

torch.__version__
Out[8]: '2.4.0.dev20240610+cu124'

torch_tensorrt.__version__
Out[9]: '2.4.0.dev20240611+cu124'

tensorrt.__version__
Out[10]: '10.0.1'
6

Hm, for some reason I had to manually install the bindings:

pip install tensorrt==10.0.1 tensorrt-cu12==10.0.1 tensorrt-cu12-bindings==10.0.1 tensorrt-cu12-libs==10.0.1

but after that, everything worked without issue.

7

Hi,

Does it mean that pytorch version must align with torch_tensorrt version?
If pytorch version is 2.4.1, should torch_tensorrt version be 2.4.1?

And a pytorch version of 2.4.1 does not work with torch_tensorrt 2.4.0 ?

8

I would assume patch releases are irrelevant and thus the posted combination should work, but you can simply try it and let us know.

1 Like