Android GPU access, NNAPI vs Vulkan

Mobile
1

Hi,

I noticed that on Android, the NNAPI pytorch backend might decide to dispatch some operations to the GPU:

 I ExecutionPlan: Device qti-default can't do operation RESIZE_NEAREST_NEIGHBOR
 I ExecutionPlan: Device qti-dsp can't do operation RESIZE_NEAREST_NEIGHBOR
 I ExecutionPlan: Device qti-gpu can't do operation RESIZE_NEAREST_NEIGHBOR
 I ExecutionPlan: Device qti-hta can't do operation RESIZE_NEAREST_NEIGHBOR
 I ExecutionPlan: ModelBuilder::findBestDeviceForEachOperation(RESIZE_NEAREST_NEIGHBOR) = 4 (nnapi-reference)
 I ExecutionPlan: Device qti-dsp can't do operation CONCATENATION
 I ExecutionPlan: Device qti-hta can't do operation CONCATENATION
 I ExecutionPlan: ModelBuilder::findBestDeviceForEachOperation(CONCATENATION) = 0 (qti-default)

My understanding is that for the concatenation operation, the qti-gpu will be used through qti-default. What are the benefits of using Vulkan over NNAPI and vice-versa for running inference on GPU?

Thanks,

Julien

2

What are the benefits of using Vulkan over NNAPI and vice-versa for running inference on GPU?

The main difference is that with Vulkan, the code that runs on the GPU (shaders) ships as part of the PyTorch runtime. With NNAPI, we give a high-level description of the model to the OS, which then sends it to the vendor backend for execution. Right now, any updates to the backed require Android OS updates.

Benefits of Vulkan

  • Runs on Vulkan-enabled devices that don’t support NNAPI.
  • Shaders can be updated by updating PyTorch runtime, which means that bug fixes and performance improvements can roll out quickly.
  • New ops can be added in PyTorch updates, and soon custom ops will be addable by applications using PyTorch, without waiting for an OS update.
  • Soon, we’ll support an API for using Vulkan GPU tensors directly in your own shaders, so model output can be sent to rendering code without returning to CPU memory.

Benefits of NNAPI on GPU

  • Vendor backend might be more mature and/or faster than the PyTorch Vulkan backend.
  • Might be a stepping stone to using NNAPI on DSP or NPU.
3

Hi,

Thanks for the detailed answer. It was very helpful.