'Tensor' object has no attribute or method 'forward'

jit
1

Hi everyone. I am working on the CLIP project. I am trying to parse models from it, script them and save them, but I am getting this error:

$ CUDA_VISIBLE_DEVICES=0 python get_encoder.py
Using cuda
Traceback (most recent call last):
  File "/media/ssdraid/training/clip/test/get_encoder.py", line 74, in <module>
    model_scripted = torch.jit.script(model)
  File "/home/tensorflow/venvs/triton/lib/python3.10/site-packages/torch/jit/_script.py", line 1286, in script
    return torch.jit._recursive.create_script_module(
  File "/home/tensorflow/venvs/triton/lib/python3.10/site-packages/torch/jit/_recursive.py", line 458, in create_script_module
    return create_script_module_impl(nn_module, concrete_type, stubs_fn)
  File "/home/tensorflow/venvs/triton/lib/python3.10/site-packages/torch/jit/_recursive.py", line 520, in create_script_module_impl
    script_module = torch.jit.RecursiveScriptModule._construct(cpp_module, init_fn)
  File "/home/tensorflow/venvs/triton/lib/python3.10/site-packages/torch/jit/_script.py", line 615, in _construct
    init_fn(script_module)
  File "/home/tensorflow/venvs/triton/lib/python3.10/site-packages/torch/jit/_recursive.py", line 498, in init_fn
    scripted = create_script_module_impl(orig_value, sub_concrete_type, stubs_fn)
  File "/home/tensorflow/venvs/triton/lib/python3.10/site-packages/torch/jit/_recursive.py", line 520, in create_script_module_impl
    script_module = torch.jit.RecursiveScriptModule._construct(cpp_module, init_fn)
  File "/home/tensorflow/venvs/triton/lib/python3.10/site-packages/torch/jit/_script.py", line 615, in _construct
    init_fn(script_module)
  File "/home/tensorflow/venvs/triton/lib/python3.10/site-packages/torch/jit/_recursive.py", line 498, in init_fn
    scripted = create_script_module_impl(orig_value, sub_concrete_type, stubs_fn)
  File "/home/tensorflow/venvs/triton/lib/python3.10/site-packages/torch/jit/_recursive.py", line 520, in create_script_module_impl
    script_module = torch.jit.RecursiveScriptModule._construct(cpp_module, init_fn)
  File "/home/tensorflow/venvs/triton/lib/python3.10/site-packages/torch/jit/_script.py", line 615, in _construct
    init_fn(script_module)
  File "/home/tensorflow/venvs/triton/lib/python3.10/site-packages/torch/jit/_recursive.py", line 498, in init_fn
    scripted = create_script_module_impl(orig_value, sub_concrete_type, stubs_fn)
  File "/home/tensorflow/venvs/triton/lib/python3.10/site-packages/torch/jit/_recursive.py", line 520, in create_script_module_impl
    script_module = torch.jit.RecursiveScriptModule._construct(cpp_module, init_fn)
  File "/home/tensorflow/venvs/triton/lib/python3.10/site-packages/torch/jit/_script.py", line 615, in _construct
    init_fn(script_module)
  File "/home/tensorflow/venvs/triton/lib/python3.10/site-packages/torch/jit/_recursive.py", line 498, in init_fn
    scripted = create_script_module_impl(orig_value, sub_concrete_type, stubs_fn)
  File "/home/tensorflow/venvs/triton/lib/python3.10/site-packages/torch/jit/_recursive.py", line 524, in create_script_module_impl
    create_methods_and_properties_from_stubs(concrete_type, method_stubs, property_stubs)
  File "/home/tensorflow/venvs/triton/lib/python3.10/site-packages/torch/jit/_recursive.py", line 375, in create_methods_and_properties_from_stubs
    concrete_type._create_methods_and_properties(property_defs, property_rcbs, method_defs, method_rcbs, method_defaults)
RuntimeError: 
'Tensor' object has no attribute or method 'forward'.:
  File "/media/ssdraid/training/clip/test/model.py", line 156
    def forward(self, x: torch.Tensor):
        orig_type = x.dtype
        ret = super().forward(x.type(torch.float32))
              ~~~~~~~~~~~~~ <--- HERE
        return ret.type(orig_type)

Can anyone please tell me how to solve it?

The codes I am using can be found here, and the get_encoder.py script includes downloading a model from external link, so just you know.

Versions

Collecting environment information...
PyTorch version: 1.12.1+cu102
Is debug build: False
CUDA used to build PyTorch: 10.2
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.4 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: 10.0.0-4ubuntu1 
CMake version: version 3.16.3
Libc version: glibc-2.31

Python version: 3.10.4 (main, Apr  8 2022, 17:35:13) [GCC 9.4.0] (64-bit runtime)
Python platform: Linux-5.4.0-1020-ibm-x86_64-with-glibc2.31
Is CUDA available: True
CUDA runtime version: 11.2.152
GPU models and configuration: 
GPU 0: NVIDIA GeForce RTX 2080 Ti
GPU 1: NVIDIA GeForce GTX 980 Ti
GPU 2: NVIDIA GeForce RTX 2080 Ti

Nvidia driver version: 515.65.01
cuDNN version: Probably one of the following:
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn.so.8.1.1
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_adv_infer.so.8.1.1
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_adv_train.so.8.1.1
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8.1.1
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_cnn_train.so.8.1.1
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_ops_infer.so.8.1.1
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_ops_train.so.8.1.1
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.22.4
[pip3] torch==1.12.1
[pip3] torchvision==0.12.0
[conda] Could not collect

Thank you very much in advance.

2

This seems to be a known limitation and tracked here.

3

Thank you @ptrblck so much. I did the workaround mentioned there, and I didn’t get errors, except for the state_dict key error, as mentioned there as well, so I just edited their names and it worked.

I changed this:

class LayerNorm(nn.LayerNorm):
    """Subclass torch's LayerNorm to handle fp16."""

    def forward(self, x: ResidualAttentionBlock):
        orig_type = x.dtype
        ret = super().forward(x.type(torch.float32))
        return ret.type(orig_type)

to this:

class LayerNorm(nn.Module):
    def __init__(self, *args, **kwargs):
        super(LayerNorm, self).__init__()
        self.inner_layernorm = nn.LayerNorm(*args, **kwargs)

    def forward(self, x: torch.Tensor):
        orig_type = x.dtype
        ret = self.inner_layernorm(x.type(torch.float32))
        return ret.type(orig_type)

I was able to load the model and inference successfully. But, when trying to deploy it with Triton Server, I got another problem:

terminate called after throwing an instance of 'c10::Error'
  what():  isTuple() INTERNAL ASSERT FAILED at "/opt/pytorch/pytorch/aten/src/ATen/core/ivalue_inl.h":1917, please report a bug to PyTorch. Expected Tuple but got String
Exception raised from toTupleRef at /opt/pytorch/pytorch/aten/src/ATen/core/ivalue_inl.h:1917 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6c (0x7fd87054bcec in /opt/tritonserver/backends/pytorch/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xfa (0x7fd870527d60 in /opt/tritonserver/backends/pytorch/libc10.so)
frame #2: c10::detail::torchInternalAssertFail(char const*, char const*, unsigned int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0x53 (0x7fd870549a03 in /opt/tritonserver/backends/pytorch/libc10.so)
frame #3: <unknown function> + 0x396dde9 (0x7fd7e62a4de9 in /opt/tritonserver/backends/pytorch/libtorch_cpu.so)
frame #4: <unknown function> + 0x396df09 (0x7fd7e62a4f09 in /opt/tritonserver/backends/pytorch/libtorch_cpu.so)
frame #5: torch::jit::SourceRange::highlight(std::ostream&) const + 0x48 (0x7fd7e39fc158 in /opt/tritonserver/backends/pytorch/libtorch_cpu.so)
frame #6: torch::jit::ErrorReport::what() const + 0x2c3 (0x7fd7e39e2193 in /opt/tritonserver/backends/pytorch/libtorch_cpu.so)
frame #7: <unknown function> + 0x113bf (0x7fd8706023bf in /opt/tritonserver/backends/pytorch/libtriton_pytorch.so)
frame #8: <unknown function> + 0x230e2 (0x7fd8706140e2 in /opt/tritonserver/backends/pytorch/libtriton_pytorch.so)
frame #9: <unknown function> + 0x23602 (0x7fd870614602 in /opt/tritonserver/backends/pytorch/libtriton_pytorch.so)
frame #10: TRITONBACKEND_ModelInstanceInitialize + 0x3f6 (0x7fd870614a46 in /opt/tritonserver/backends/pytorch/libtriton_pytorch.so)
frame #11: <unknown function> + 0x10ce8a (0x7fd87c50be8a in /opt/tritonserver/bin/../lib/libtritonserver.so)
frame #12: <unknown function> + 0x10e971 (0x7fd87c50d971 in /opt/tritonserver/bin/../lib/libtritonserver.so)
frame #13: <unknown function> + 0x101a10 (0x7fd87c500a10 in /opt/tritonserver/bin/../lib/libtritonserver.so)
frame #14: <unknown function> + 0x1b217a (0x7fd87c5b117a in /opt/tritonserver/bin/../lib/libtritonserver.so)
frame #15: <unknown function> + 0x1c0fa1 (0x7fd87c5bffa1 in /opt/tritonserver/bin/../lib/libtritonserver.so)
frame #16: <unknown function> + 0xd6de4 (0x7fd87c04ede4 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6)
frame #17: <unknown function> + 0x8609 (0x7fd87d25e609 in /usr/lib/x86_64-linux-gnu/libpthread.so.0)
frame #18: clone + 0x43 (0x7fd87bd39163 in /usr/lib/x86_64-linux-gnu/libc.so.6)

Anyways, there is already a related discussion about it here, I will see if I can get some clues.

1 Like
4

Ok, looks like that last error is a version issue as mentioned by glaringlee. I downgraded Pytorch to the same version used by the backend in Triton then scripted the model again and it worked.