FP16 support in pytorch mobile

Mobile
1

Is there a way to run scripted half precision model in pytorch mobile? If not, will this be supported in the future? and is there way to at least convert saved fp16 scripted model to fp32 on mobile device inside pytorch mobile?

2

I’m not sure if the mobile device architecture would benefit from float16 dtypes (the x86 CPU architectures do not see a benefit from it which is also the reason float16 ops are mostly not supported on the CPU in PyTorch).

I don’t know about future plans and if some devices are planning to support float16 on mobile.

I would assume transformations are still possible on mobile, i.e. would model.to(dtype) work?

3

While arm cpus have support for fp16, we dont support that on mobile runtime to directly take advantage of fp16 features of mobile cpus.

In terms of saving fp16 and converting to fp32 on model load is something that is not quite supported natively. I have not explored this but it might be possible to do this entirely in torchscirpt without pytorch runtime needing to support this natively.

4

model.to likely wont work since on server with full pytorch runtime gets torch::jit::Module as the type returned by torch::jit::load whereas on mobile we have torch::jit::_load_for_mobile which returns torch::jit::mobile::Module.

5

@ptrblck I’m trying a FP32 and FP16 PyTorch model on Mobile. FP32 works fine, but with FP16 the app crashes.
Does Pytorch on mobile have support for FP16 now?

6

Which backend are you using? Do you see these crashes in ExecuTorch?

7

@ptrblck I’m using PyTorch Lite as the backend, and FP16 is crashing. On Executorch, both FP32 and FP16 work.

From the official PyTorch Android source code (https://github.com/pytorch/pytorch/blob/main/android/pytorch_android/src/main/java/org/pytorch/DType.java), I can see that there’s no Float16 data type defined for PyTorch.

Could you please confirm that FP16 with PyTorch Lite is supported?

8

I cannot confirm that FP16 is supported on PyTorch Lite and also assume it’s not being actively developed anymore as ExecuTorch is the actively developed and maintained mobile backend.