How can I enable GPU support for torch mobile on android?

Hi! Is there any way to use GPU delegate (like options.gpuDelegate for tflite models) for torch mobile on android to speed up inference time? I’ve found CPU only options both for QNNPACK and FBGEMM models . Thank you!

1 Like

Is there any update on this? We would really love to go forward using pytorch mobile, but this is a blocker.

I’am waiting for this feature. :grinning: :grinning: :grinning:

They announced GPU support at developer day. But there is no doc. I’m curious if it’s release yet.

1 Like

Mobile GPU support is yet a prototype feature available on the nightly build.
Prototype features are not documented.

You can find an example here:

Isn’t this for NNAPI? I assume on Qualcomm chip it will just use DSP/NPU?

NNAPI can use both GPUs and DSP/NPU.
For example, if you quantize your models to 8bits, DSP/NPU will be used otherwise GPU will be the main computing unit.
The quantization is optional in the above example.

More benchmarks and information could be found here.

1 Like

So if I pass a float graph to the benchmark binary, how does it know whether to use CPU/GPU? I mean, there must be some kind of switch, right?

I did read that blog, but it offers miserably little information about GPU.

I was talking about in general NNAPI usage but didn’t run the above example myself to check the GPU acceleration. I’m sorry for the confusion.

This link might be more related to you.
It says the primary targets of the vulkan backed are mobile GPUs and pytorch should be built with a proper cmake option, USE_VULKAN.

It seems like the mobile_optimizer, torch.utils.mobile_optimizer.optimize_for_mobile already supports mobile GPU if built with vulkan enabled.

I see. It seems there is no plan to support OpenCL/OpenGL then. Just Vulkan.

According to this issue, OpenCL is not going to be supported.

Noted. Thanks for the pointer!

Will the feature available in the PyTorch next release (1.8) ?

You mean like this:

Seems like there a recent release of NNAPI support for PyTorch. Does that mean PyTorch model can have mobile GPU support now?