What is the most straigforward/simple/relible way to run CNN on Android device with GPU support for current time 16.02.2021?

For my best knowledge there 3 ways to do so

  1. Run your PyTorch model on Android GPU using libMACE
    Run your PyTorch model on Android GPU using libMACE | by Vitaliy Hramchenko | Medium
    Which needs to ONNX convertation

  2. Via Android NNAPI
    PyTorch Mobile Now Supports Android NNAPI | by PyTorch | PyTorch | Medium
    (Prototype) Convert MobileNetV2 to NNAPI — PyTorch Tutorials 1.7.1 documentation
    Which requires special PyTorch revision?

  3. PyTorch on Vulkan backend
    tutorials/prototype_source at master · pytorch/tutorials · GitHub
    Which allows only float32 because Vulkan itself is a Graphic engine I think.

Could you please share your thoughts, because I want to make as less effort as I possibly can. I know that all these features are prototypes but its is ok for me.

2 Likes
  1. If you use float32 model - the most straightforward will be to use our Vulkan backend. ( tutorials/vulkan_workflow.rst at master · pytorch/tutorials · GitHub )

You might found that not all the operators for your model are yet supported.
Please message them here if you find it.

You can use Vulkan backend from java, specifying Device.VULKAN during model loading

Module module = Module.load("$PATH", Device.VULKAN)

Since 1.8 version Vulkan backend is included in our main gradle artifact (org.pytorch:pytorch_android), that you can skip libtorch building process.

  1. For quantized model is better to follow Android NNAPI way.
    You do not need custom build of pytorch and can use our default maven artifact.
3 Likes

torchversion 1.11.0
RuntimeError: falseINTERNAL ASSERT FAILED at "/opt/conda/conda-bld/pytorch_1646755861072/work/torch/csrc/jit/passes/vulkan_rewrite.cpp":272, please report a bug to PyTorch. Mobile optimizaiton only available with Vulkan at the moment. Vulkan is not enabled. Please build with USE_VULKAN=1
I created bug report

I make
mModule = Module.load(MainActivity.assetFilePath(getApplicationContext(), "traced_model_vulkan.pt"), Device.VULKAN);
And get from AndroidStudio
/mnt/hugedisk/AndroidTorch/android-demo-app/ObjectDetection/app/src/main/java/org/pytorch/demo/objectdetection/MainActivity.java:190: error: no suitable method found for load(String,Device) mModule = Module.load(MainActivity.assetFilePath(getApplicationContext(), "traced_model_vulkan.pt"), Device.VULKAN); ^ method Module.load(String,Map<String,String>,Device) is not applicable (actual and formal argument lists differ in length) method Module.load(String) is not applicable (actual and formal argument lists differ in length)

This API was changed (need one more parameter for extra files):