Slow inference with U-2-Net on iOS

Mobile
1

I’m using U-2-Net on mobile.
I converted the model to TorchScript, and used it succesfully on Android. The inference takes about 1 sec.
Now, if I use the same model on iOS, the inference is about half a minute. I disable autograd, and run the model in inference mode.
Any idea what could cause such a big performance difference using the same model?

(I use 1.6.0 nightly build on Android, and 1.5.0 production build on iOS)

2

Android 1.6.0 nightly is using XNNPACK, whereas iOS 1.5.0 is not. We’ll be enabling XNNPACK on iOS in 1.6.0 release.

3

Ok, thank you! Meanwhile we managed to speed up the iOS inference. The thing is that on Android we scripted to model by calling jit.script(), and we tried to use this exact same model on iOS, which was slow. I tried another approach for converting to TorchScript by tracing the model’s execution with jit.trace, and for some reason the “traced model” performance on iOS was on par with the Android inference. Can it be also related to XNNPACK somehow?

4

Cool, it’s nice to hear the issue has been resolved. Short answer, no. The XNNPACK is our new computation kernels that has nothing to do with scripting. Yea, tracing is supposed to be faster.

1 Like
5

hello Ahmed , I am converting U2-Net for IoS app but having issue with conversion . Could you help me by your conversion code or .ml file ?

6

Hi, sure I can help you!

in u2net_test.py write this after net.eval():

scripted = torch.jit.script(net)
torch.jit.save(scripted, "fod.p")

This is how I build the model I use on Android. On iOS however, this model didn’t work so I used the following conversion method:

In u2net_test.py, in the loop which goes through the test images, write this after inputs_test is loaded:

if traced == False:
    traced_script_module = torch.jit.trace(net, inputs_test)
    traced_script_module.save("fod_traced.p")
    traced = True

Use traced boolean to only trace a single inference, not all the inferences.
Hope it helps.
If you need help with how to use these models on Android or iOS, let me know.

8

hi dear !
my UNet model is displaying black image on IoS app , can I view your source code or model for help ?

9

Hi Ahmed,
I successfully converted the model to pytorch script, but having some trouble using it on android platform. Can you guide me how can I use this u2net model, I mean how will we give it the input and take the output in android,
Thanks in advance …