Sparse tensor not supported (Minkowski Engine)

jit
1

SparseTensor’ object is not iterable

  • I have a pytorch model trained on Minkowski Engine library and i want to deploy this model on torchserve.I am facing issues while converting ‘.pth’ model to the serialized ‘.pt’ model.
  • python enviornment -
  1. python 3.7, pytorch 1.8, Minkowski Engine (0.4.3) [GitHub - NVIDIA/MinkowskiEngine: Minkowski Engine is an auto-diff neural network library for high-dimensional sparse tensors]
  2. python 3.8, pythorch 1.10, Minkowski Engine 0.5.4
    Tried on both these enviornments, got same following issue.

  • preprocessing used in input file :
    preprocessed_data = ME.SparseTensor(features, coordinates)

  • I used torch.jit functions to convert the model into serialized form, but i am getting errors due to sparse tensor class of minkowski engine dependency in model.

Error logs

I used various methods for converting ‘.pth’ model to ‘.pt’ serialized model.

  1. torch.jit.trace, got following error :
    Traceback (most recent call last): File "model_pt_converter.py", line 65, in <module> traced_script_module = torch.jit.trace(model, preprocessed_data) File "/home/vinayak/miniconda3/envs/serve/lib/python3.7/site-packages/torch/jit/_trace.py", line 750, in trace _module_class, File "/home/vinayak/miniconda3/envs/serve/lib/python3.7/site-packages/torch/jit/_trace.py", line 941, in trace_module example_inputs = make_tuple(example_inputs) File "/home/vinayak/miniconda3/envs/serve/lib/python3.7/site-packages/torch/jit/_trace.py", line 542, in make_tuple return tuple(example_inputs) TypeError: 'SparseTensor' object is not iterable

  2. using torch.jit.script,
    torch.jit.frontend.UnsupportedNodeError: global variables aren't supported:
    SparseTensor.__init__' is being compiled since it was called from '__torch__.SparseTensor.SparseTensor
    __torch__.SparseTensor.SparseTensor' is being compiled since it was called from 'MinkowskiConvolution.forward

  3. I also tried to serve the .pth model directly, the model is not getting loaded throwing following issues -
    Load model failed: s2b-model, error: Worker died
    error: unrecognized arguments: --sock-type unix --sock-name /tmp/.ts.sock.9002
    java.lang.InterruptedException: nullI have a pytorch model trained on Minkowski Engine library and i want to deploy this model on torchserve.I am facing issues while converting ‘.pth’ model to the serialized ‘.pt’ model.
    python enviornment -

  4. python 3.7, pytorch 1.8, Minkowski Engine (0.4.3) [GitHub - NVIDIA/MinkowskiEngine: Minkowski Engine is an auto-diff neural network library for high-dimensional sparse tensors]

  5. python 3.8, pythorch 1.10, Minkowski Engine 0.5.4
    Tried on both these enviornments, got same following issue.
    preprocessing used in input file :
    preprocessed_data = ME.SparseTensor(features, coordinates)

  • I used various methods for converting ‘.pth’ model to ‘.pt’ serialized model.

  1. torch.jit.trace, got following error :
    Traceback (most recent call last): File "model_pt_converter.py", line 65, in <module> traced_script_module = torch.jit.trace(model, preprocessed_data) File "/home/vinayak/miniconda3/envs/serve/lib/python3.7/site-packages/torch/jit/_trace.py", line 750, in trace _module_class, File "/home/vinayak/miniconda3/envs/serve/lib/python3.7/site-packages/torch/jit/_trace.py", line 941, in trace_module example_inputs = make_tuple(example_inputs) File "/home/vinayak/miniconda3/envs/serve/lib/python3.7/site-packages/torch/jit/_trace.py", line 542, in make_tuple return tuple(example_inputs) TypeError: 'SparseTensor' object is not iterable

  2. using torch.jit.script,
    torch.jit.frontend.UnsupportedNodeError: global variables aren't supported:
    SparseTensor.__init__' is being compiled since it was called from '__torch__.SparseTensor.SparseTensor
    __torch__.SparseTensor.SparseTensor' is being compiled since it was called from 'MinkowskiConvolution.forward

  3. I also tried to serve the .pth model directly, the model is not getting loaded throwing following issues -
    Load model failed: s2b-model, error: Worker died
    error: unrecognized arguments: --sock-type unix --sock-name /tmp/.ts.sock.9002
    java.lang.InterruptedException: null

Installation instructions

  • Installed torchserve using pip and pytorch using conda.
  • Didn’t used docker image.

Versions

Environment headers

Torchserve branch:

torchserve==0.7.0
torch-model-archiver==0.7.0

Python version: 3.7 (64-bit runtime)
Python executable: /home/vinayak/miniconda3/envs/serve/bin/python

Versions of relevant python libraries:
future==0.18.2
numpy==1.21.5
nvgpu==0.9.0
psutil==5.9.4
requests==2.28.1
torch==1.8.2
torch-model-archiver==0.7.0
torch-workflow-archiver==0.2.6
torchaudio==0.8.2
torchserve==0.7.0
torchvision==0.13.1a0
wheel==0.37.1
torch==1.8.2
**Warning: torchtext not present …
torchvision==0.13.1a0
torchaudio==0.8.2

Java Version:

OS: Pop!_OS 20.04 LTS
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: N/A
CMake version: N/A