Compiling Pytorch Models to run on Apple ANE

Mobile
1

I benchmarked 2 different Resnet50 Models - the Apple CoreML model, available on the Apple website, and a pretrained Torchvision Resnet50 model which I converted using ONNX (Opset9) and CoreMLTools (iOS Version 13).

I tested both models on a brand new iPhone XR.

Inference Times:

  1. Apple Resnet50 : CPU Inference 100ms, GPU Inference 60ms, ANE Inference 15ms
  2. Torchvision Resnet50 : CPU Inference 100ms, GPU Inference 60ms, ANE Inference 60ms.

As you can see, the ANE Inference for the Apple version is 4x faster.

Memory Usage:

  1. Apple Resnet50 : 46 MB CPU, 92 MB GPU, 35 MB ANE
  2. Torchvision Resnet50 : 48 MB CPU, 91 MB GPU, 91 MB ANE

The Apple version uses 3x less memory than the Torchvision one.

I am concluding the Torchvision Model doesn’t run on the ANE.

I was wondering if anyone knew how the Apple Repo models were compiled in order to run on the ANE? From the model description, the Apple Models were originally written in Keras, but as far as I know the Keras implementation isn’t different from the Pytorch Torchvision one, so I am not sure where the difference comes from.

Thanks!

1 Like
2

Hello! First, I tested the results on my ANE-GPT model, considering that the ANE engine will prioritize 4-dimensional tensors, so you need to make appropriate changes to the neural network architecture. :grinning:

3

Could you clarify what you exactly mean with prioritization of 4D tensors? Should we better use batches for testing the speed or just add an extra dimension, which already happens by default
4D → , <channels/feature maps>, , , e.g. (1, 3, 224, 224)