How can I enable GPU support for torch mobile on android?

ZackPashkin · February 25, 2020, 3:22am

Hi! Is there any way to use GPU delegate (like options.gpuDelegate for tflite models) for torch mobile on android to speed up inference time? I’ve found CPU only options both for QNNPACK and FBGEMM models . Thank you!

tom · February 25, 2020, 7:54am

Tassilo · September 22, 2020, 11:36am

Is there any update on this? We would really love to go forward using pytorch mobile, but this is a blocker.

chazo1994 · October 27, 2020, 7:26am

I’am waiting for this feature.

windmaple · November 29, 2020, 11:22am

They announced GPU support at developer day. But there is no doc. I’m curious if it’s release yet.

seungjun · November 29, 2020, 12:12pm

Mobile GPU support is yet a prototype feature available on the nightly build.
Prototype features are not documented.

You can find an example here:

github.com

pytorch/tutorials/blob/master/prototype_source/nnapi_mobilenetv2.rst

(Prototype) Convert MobileNetV2 to NNAPI
========================================

Introduction
------------

This tutorial shows how to prepare a computer vision model to use
`Android's Neural Networks API (NNAPI) <https://developer.android.com/ndk/guides/neuralnetworks>`_.
NNAPI provides access to powerful and efficient computational cores
on many modern Android devices.

PyTorch's NNAPI is currently in the "prototype" phase and only supports
a limited range of operators, but we expect to solidify the integration
and expand our operator support over time.


Environment
-----------

Install PyTorch and torchvision.

This file has been truncated. show original

windmaple · November 29, 2020, 12:40pm

Isn’t this for NNAPI? I assume on Qualcomm chip it will just use DSP/NPU?

seungjun · November 29, 2020, 1:40pm

NNAPI can use both GPUs and DSP/NPU.
For example, if you quantize your models to 8bits, DSP/NPU will be used otherwise GPU will be the main computing unit.
The quantization is optional in the above example.

More benchmarks and information could be found here.

windmaple · November 30, 2020, 8:11am

So if I pass a float graph to the benchmark binary, how does it know whether to use CPU/GPU? I mean, there must be some kind of switch, right?

I did read that blog, but it offers miserably little information about GPU.

seungjun · November 30, 2020, 10:28am

I was talking about in general NNAPI usage but didn’t run the above example myself to check the GPU acceleration. I’m sorry for the confusion.

This link might be more related to you.
It says the primary targets of the vulkan backed are mobile GPUs and pytorch should be built with a proper cmake option, USE_VULKAN.

It seems like the mobile_optimizer, torch.utils.mobile_optimizer.optimize_for_mobile already supports mobile GPU if built with vulkan enabled.

windmaple · November 30, 2020, 11:53am

I see. It seems there is no plan to support OpenCL/OpenGL then. Just Vulkan.

seungjun · December 1, 2020, 2:02am

According to this issue, OpenCL is not going to be supported.

windmaple · December 1, 2020, 6:36am

Noted. Thanks for the pointer!

Kiki_Rizki_Arpiandi · February 2, 2021, 8:59am

Will the feature available in the PyTorch next release (1.8) ?

tom · February 3, 2021, 8:27pm

You mean like this:
https://pytorch.org/tutorials/prototype/nnapi_mobilenetv2.html#introduction

dcl · February 16, 2021, 9:02pm

Seems like there a recent release of NNAPI support for PyTorch. Does that mean PyTorch model can have mobile GPU support now?

Marvin · February 15, 2023, 7:10pm

The only official document that I can find is this one: (Prototype) Use iOS GPU in PyTorch — PyTorch Tutorials 1.13.1+cu117 documentation. The document has been a while there, do we have any update since then?