FasterRCNN Resnet50 JIT Trace

jit
1

Hi,

I’m trying to trace FasterRCNN to use in Pytorch Mobile on iOS.

I simply trace as shown below:

model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True)
model.eval()

input_tensor = torch.rand(1,3,224,224)
script_model = torch.jit.trace(model, input_tensor)
script_model.save("models/fRCNN_resnet50.pt")

I receive a “Only tensors or tuples of tensors can be output from traced functions (getOutput at …/torch/csrc/jit/tracer.cpp:209)” error as shown below

RuntimeError: Only tensors or tuples of tensors can be output from traced functions (getOutput at ../torch/csrc/jit/tracer.cpp:209)
frame #0: c10::Error::Error(c10::SourceLocation, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) + 135 (0x128cc29e7 in libc10.dylib)
frame #1: torch::jit::tracer::TracingState::getOutput(c10::IValue const&) + 1785 (0x120164069 in libtorch.dylib)
frame #2: torch::jit::tracer::exit(std::__1::vector<c10::IValue, std::__1::allocator<c10::IValue> > const&) + 232 (0x120167108 in libtorch.dylib)
frame #3: torch::jit::tracer::createGraphByTracing(pybind11::function const&, torch::jit::tracer::TypedStack, pybind11::function const&, bool, torch::jit::script::Module*) + 916 (0x11c957914 in libtorch_python.dylib)
frame #4: void pybind11::cpp_function::initialize<torch::jit::script::initJitScriptBindings(_object*)::$_16, void, torch::jit::script::Module&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, pybind11::function, pybind11::tuple, pybind11::function, bool, pybind11::name, pybind11::is_method, pybind11::sibling>(torch::jit::script::initJitScriptBindings(_object*)::$_16&&, void (*)(torch::jit::script::Module&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, pybind11::function, pybind11::tuple, pybind11::function, bool), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&)::'lambda'(pybind11::detail::function_call&)::__invoke(pybind11::detail::function_call&) + 197 (0x11c993185 in libtorch_python.dylib)
frame #5: pybind11::cpp_function::dispatcher(_object*, _object*, _object*) + 3324 (0x11c4cd92c in libtorch_python.dylib)
frame #6: _PyCFunction_FastCallDict + 183 (0x10e3d0167 in Python)
frame #7: call_function + 184 (0x10e452d28 in Python)
frame #8: _PyEval_EvalFrameDefault + 27511 (0x10e44f597 in Python)
frame #9: _PyEval_EvalCodeWithName + 2447 (0x10e45388f in Python)
frame #10: fast_function + 545 (0x10e454141 in Python)
frame #11: call_function + 401 (0x10e452e01 in Python)
frame #12: _PyEval_EvalFrameDefault + 27511 (0x10e44f597 in Python)
frame #13: _PyEval_EvalCodeWithName + 2447 (0x10e45388f in Python)
frame #14: fast_function + 545 (0x10e454141 in Python)
frame #15: call_function + 401 (0x10e452e01 in Python)
frame #16: _PyEval_EvalFrameDefault + 27511 (0x10e44f597 in Python)
frame #17: _PyEval_EvalCodeWithName + 2447 (0x10e45388f in Python)
frame #18: PyEval_EvalCode + 100 (0x10e448954 in Python)
frame #19: builtin_exec + 548 (0x10e445fe4 in Python)
frame #20: _PyCFunction_FastCallDict + 491 (0x10e3d029b in Python)
frame #21: call_function + 439 (0x10e452e27 in Python)
frame #22: _PyEval_EvalFrameDefault + 27511 (0x10e44f597 in Python)
frame #23: gen_send_ex + 183 (0x10e3a7fe7 in Python)
frame #24: _PyEval_EvalFrameDefault + 11552 (0x10e44b740 in Python)
frame #25: gen_send_ex + 183 (0x10e3a7fe7 in Python)
frame #26: _PyEval_EvalFrameDefault + 11552 (0x10e44b740 in Python)
frame #27: gen_send_ex + 183 (0x10e3a7fe7 in Python)
frame #28: _PyCFunction_FastCallDict + 560 (0x10e3d02e0 in Python)
frame #29: call_function + 439 (0x10e452e27 in Python)
frame #30: _PyEval_EvalFrameDefault + 27511 (0x10e44f597 in Python)
frame #31: fast_function + 381 (0x10e45409d in Python)
frame #32: call_function + 401 (0x10e452e01 in Python)
frame #33: _PyEval_EvalFrameDefault + 27511 (0x10e44f597 in Python)
frame #34: fast_function + 381 (0x10e45409d in Python)
frame #35: call_function + 401 (0x10e452e01 in Python)
frame #36: _PyEval_EvalFrameDefault + 27511 (0x10e44f597 in Python)
frame #37: _PyEval_EvalCodeWithName + 2447 (0x10e45388f in Python)
frame #38: _PyFunction_FastCallDict + 763 (0x10e45445b in Python)
frame #39: _PyObject_FastCallDict + 247 (0x10e3873e7 in Python)
frame #40: _PyObject_Call_Prepend + 149 (0x10e387505 in Python)
frame #41: PyObject_Call + 96 (0x10e387220 in Python)
frame #42: _PyEval_EvalFrameDefault + 28250 (0x10e44f87a in Python)
frame #43: _PyEval_EvalCodeWithName + 2447 (0x10e45388f in Python)
frame #44: fast_function + 545 (0x10e454141 in Python)
frame #45: call_function + 401 (0x10e452e01 in Python)
frame #46: _PyEval_EvalFrameDefault + 27670 (0x10e44f636 in Python)
frame #47: gen_send_ex + 183 (0x10e3a7fe7 in Python)
frame #48: builtin_next + 92 (0x10e446bcc in Python)
frame #49: _PyCFunction_FastCallDict + 491 (0x10e3d029b in Python)
frame #50: call_function + 439 (0x10e452e27 in Python)
frame #51: _PyEval_EvalFrameDefault + 27511 (0x10e44f597 in Python)
frame #52: _PyEval_EvalCodeWithName + 2447 (0x10e45388f in Python)
frame #53: fast_function + 545 (0x10e454141 in Python)
frame #54: call_function + 401 (0x10e452e01 in Python)
frame #55: _PyEval_EvalFrameDefault + 27511 (0x10e44f597 in Python)
frame #56: gen_send_ex + 183 (0x10e3a7fe7 in Python)
frame #57: builtin_next + 92 (0x10e446bcc in Python)
frame #58: _PyCFunction_FastCallDict + 491 (0x10e3d029b in Python)
frame #59: call_function + 439 (0x10e452e27 in Python)
frame #60: _PyEval_EvalFrameDefault + 27511 (0x10e44f597 in Python)
frame #61: _PyEval_EvalCodeWithName + 2447 (0x10e45388f in Python)
frame #62: fast_function + 545 (0x10e454141 in Python)
frame #63: call_function + 401 (0x10e452e01 in Python)

Could someone explain to me how to properly trace or script a pretrained object detection model such as this one? I don’t know which steps I might be missing if any!

1 Like
2

The MaskRCNN based models don’t support tracing due to the error you saw, but thanks to this PR you can use torch.jit.script to compile the whole network.

You’ll need to get the most current version of torchvision by building it from source, then you can do

model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True)
model.eval()
script_model = torch.jit.script(model)
1 Like
3

Hi! I know some time has passed between now and when this was response was written. Would recompiling from source still be the solution for this? Will try tonight! Thanks

4

We had a release recently, so it should work so long as you’re using the latest versions of PyTorch and torchvision.

# Clear out old versions
pip uninstall torch
pip uninstall torch
pip uninstall torchvision

# Install the most recent versions
pip install torch torchvision
5

Okay, I was able to successfuly script the model as a pt file. I proceeded to place this pt file in an iOS project via pytorch mobile bridging headers. I also changed the settings to accept an 800x800 input image since the model specifies that.

However when I try to run predict/inference on Swift/iOS I get the following error.

2020-01-29 12:43:32.086469-0500 torchMobile[11047:2729940] 
Unknown builtin op: torchvision::_new_empty_tensor_op.
Could not find any similar ops to torchvision::_new_empty_tensor_op. This op may not exist or may not be currently supported in TorchScript.
:
at /Users/hxh85ki/Desktop/Projects/thdEnv/lib/python3.6/site-packages/torchvision/ops/new_empty_tensor.py:16:11
        output (Tensor)
    """
    return torch.ops.torchvision._new_empty_tensor_op(x, shape)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE
Serialized at code/__torch__/torchvision/ops/new_empty_tensor.py:4:7
def _new_empty_tensor(x: Tensor,
    shape: List[int]) -> Tensor:
  _0 = ops.torchvision._new_empty_tensor_op(x, shape)
       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE
  return _0
'_new_empty_tensor' is being compiled since it was called from 'interpolate'
Serialized at code/__torch__/torchvision/ops/misc.py:25:2
    align_corners: Optional[bool]=None) -> Tensor:
  _1 = __torch__.torchvision.ops.misc._output_size
  _2 = __torch__.torchvision.ops.new_empty_tensor._new_empty_tensor
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE
  _3 = uninitialized(Tensor)
  if torch.gt(torch.numel(input), 0):
'interpolate' is being compiled since it was called from 'GeneralizedRCNNTransform.resize'
Serialized at code/__torch__/torchvision/models/detection/transform.py:79:4
    target: Optional[Dict[str, Tensor]]) -> Tuple[Tensor, Optional[Dict[str, Tensor]]]:
    _18 = __torch__.torchvision.models.detection.transform.resize_boxes
    _19 = __torch__.torchvision.ops.misc.interpolate
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE
    _20 = __torch__.torchvision.models.detection.transform.resize_keypoints
    _21 = uninitialized(Tuple[Tensor, Optional[Dict[str, Tensor]]])
'GeneralizedRCNNTransform.resize' is being compiled since it was called from 'GeneralizedRCNNTransform.forward'
at /Users/hxh85ki/Desktop/Projects/thdEnv/lib/python3.6/site-packages/torchvision/models/detection/transform.py:47:34
                                 "of shape [C, H, W], got {}".format(image.shape))
            image = self.normalize(image)
            image, target_index = self.resize(image, target_index)
                                  ~~~~~~~~~~~ <--- HERE
            images[i] = image
            if targets is not None and target_index is not None:
Serialized at code/__torch__/torchvision/models/detection/transform.py:29:33
        pass
      image0 = (self).normalize(image, )
      _2 = (self).resize(image0, target_index, )
                                 ~~~~~~~~~~~~ <--- HERE
      image1, target_index0, = _2
      _3 = torch._set_item(images0, i, image1)
Fatal error: Can't find the model file!: file /Users/hxh85ki/thd-visual-ai/pytorchMobile/torchMobile/torchMobile/ViewController.swift, line 110
2020-01-29 12:43:32.088293-0500 torchMobile[11047:2729940] Fatal error: Can't find the model file!: file /Users/hxh85ki/thd-visual-ai/pytorchMobile/torchMobile/torchMobile/ViewController.swift, line 110

My model file is definitely in the project and I’m able to successfully able to run any normal image classification models on iOS. But this is my first attempt with Object Detection using Faster RCNN. Where am I going wrong? Any help would be appreciated. If necessary I could also take it to the mobile thread.

From my understanding, if I can trace/script a pt file for mobile, pytorch mobile should be able to run it? This is a vanilla Faster RCNN Resnet50 fpn.

Thanks, Haris

6

We don’t currently build and ship torchvision or torchaudio for mobile. We’re looking into it.

1 Like
7

@HussainHaris David is right, the torchvision c++ APIs are not supported on mobile yet. If your local pytorch version is 1.4.0, you can use the python API below to examine the ops used by your model

torch.jit.export_opnames(traced_script_module)
8

@David_Reiss I try to load faster rcnn model with nightly libtorch on Win10. But it failed. Is it support win10?

9

Thanks, how is shipping torchvision models for object detection or segmentation such as this FasterRCNN different from say shipping a resnet50 on mobile? Isn’t resnet50 and other CV models torchvision models? Just so I can understand better!

10

Detection models use special ops that are not part of the PyTorch core. Some segmentation architectures (like U-Net) work fine without those ops. We are looking into supporting them.

Win10 is supported.

11

I’ve also run into this same issue using the pre-trained Faster RCNN model from pytorch, saved using

model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True)
model.eval()
script_module = torch.jit.script(model)
script_module.save("../models/frcnn_torchscript.pt")

Module.load using the Java wrappers in Android Studio gives:

Unknown builtin op: torchvision::_new_empty_tensor_op.
    Could not find any similar ops to torchvision::_new_empty_tensor_op. This op may not exist or may not be currently supported in TorchScript.

Is this something I could fix by compiling the latest Pytorch Android from source?

12

As David stated, F-RCNN is a detection model using special ops not part of the Pytorch Core and is therefore not directly scriptable for mobile deployment unfortunately

13

Ok thanks @HussainHaris. Do you have any insight on what this change did and what I could do to build it into the android libraries? It seems like there is a torchvision package available now (used in this example I was following) which uses at least some of the vision ops, so maybe there is a way for me to hack the missing ops in and unblock myself for a while?

14

I didn’t see this actually, maybe @David_Reiss could shed some light on these changes for MaskRCNN scriptability?

Please let me know @mrpropellers of any updates with this attempt. Currently I’m trying to add MaskRCNN Resnet50 via MLRI/Tensorflow Lite’s experimental convertor.

16

Hi @HussainHaris . It has been a while since this issue was active, so I’m wondering if you’ve had any more success with this in more recent versions of torch and torchvision. I’m trying to do the same thing as you, where I save a trace of a faster rcnn model, and the problem appears to be that those ops functions are not available for tracing.

Are there any ways to do this now that the packages have been updated.
Thank you