Speed benchmarking on android?

I am interested to know how fast some of my models run on the CPUs of a Pixel 3 phone. I am a moderately experienced pytorch programmer and linux user, but I have zero experience with android. I am not looking to build an app right now; I just want to know how fast my model runs on this particular phone.

The TensorFlow repo has this barebones android test thingy for timing the latency of a neural net of your choice on an android phone in TensorFlow: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/tools/benchmark/android/README.md

Has anyone made anything similar for pytorch?

1 Like

We have a binary to do this that can run on your android phone using adb.

To build,

./scripts/build_android.sh \                                                                                                                       
-DCMAKE_PREFIX_PATH=$(python -c 'from distutils.sysconfig import get_python_lib; print(get_python_lib())') \
-DPYTHON_EXECUTABLE=$(python -c 'import sys; print(sys.executable)')

To run the binary, push it to the device using adb and run the following command
./speed_benchmark_torch --model=model.pt --input_dims="1,3,224,224" --input_type=float --warmup=10 --iter 10 --report_pep true

@supriyar do you know if FAI-PEP will integrate with pytorch mobile?

It should be possible, we already output the total network latency in a format that is acceptable by FAI-PEP.
The flow should be similar to existing caffe2 for mobile, but use the speed_benchmark_torch binary instead.

1 Like

Thanks! Is there a certain NDK version that is preferred? I know in TensorFlow, they like using old NDK versions for some reason.

Also, do we need the Android SDK to be visible to PyTorch anywhere?

The instructions for speed_benchmark_torch worked for me on the first try!

If anyone else wants try this on a Pixel 3 android phone, here is the setup that worked for me:

# in bash shell
cd pytorch #where I have my `git clone` of pytorch

export ANDROID_ABI=arm64-v8a
export ANDROID_NDK=/path/to/Android/Sdk/ndk/21.0.6113669/

./scripts/build_android.sh \
-DCMAKE_PREFIX_PATH=$(python -c 'from distutils.sysconfig import get_python_lib; print(get_python_lib())') \
-DPYTHON_EXECUTABLE=$(python -c 'import sys; print(sys.executable)') \

# speed_benchmark_torch appears in pytorch/build_android/install/bin/speed_benchmark_torch

Next, I followed these instructions to export a resnet18 torchscript model:

#in python
import torch
import torchvision

model = torchvision.models.resnet18(pretrained=True)
example = torch.rand(1, 3, 224, 224)
traced_script_module = torch.jit.trace(model, example)

Then, I put the files onto the android device

# in bash shell on linux host computer that's plugged into Pixel 3 phone
adb shell mkdir /data/local/tmp/pt
adb push build_android/install/bin/speed_benchmark_torch /data/local/tmp/pt
adb push resnet18.pt /data/local/tmp/pt

And finally I run on the android device

# in bash shell on linux host computer that's plugged into Pixel 3 phone
adb shell  /data/local/tmp/pt/speed_benchmark_torch \
--model  /data/local/tmp/pt/resnet18.pt --input_dims="1,3,224,224" \
--input_type=float --warmup=5 --iter 20

It prints:

Starting benchmark.
Running warmup runs.

Main runs.
Main run finished. Milliseconds per iter: 188.382. Iters per second: 5.30836

Pretty good! I believe resnet18 is about 4 gflop (that is, 2 gmac) per frame, so (4 gmac) / (188 ms) = 21 gflop/s. Not bad for ARM CPUs! (At least I assume it’s executing on the ARM CPUs and not any GPUs or other accelerators.)

Also, this whole process took me about 25 minutes, and everything worked on the first try. I use pytorch day-to-day, but I have very little experience with android, and this was also my first time using torchscript, so I’m surprised and impressed that it was so straightforward.


This thread is very useful and I’m trying to get this working. I can’t get past the step where build_android.sh is run without a bunch of errors. You can view my CMakeError.log here. Does anyone know what’s going on here? Alternatively, if someone could link me their speed_benchmark_torch executable that might also work.

1 Like

@nufsty2, Could I see your build command, please?

And, what operating system do you have on the computer where you are compiling the binary?

@solvingPuzzles I’m in Ubuntu 18.04. I actually just got it working. I had to run
git submodule update --init --recursive
within the pytorch clone as well as run the

./scripts/build_android.sh \
-DCMAKE_PREFIX_PATH=$(python -c 'from distutils.sysconfig import get_python_lib; print(get_python_lib())') \
-DPYTHON_EXECUTABLE=$(python -c 'import sys; print(sys.executable)')

command as sudo -E. I also had to run the python commands as sudo so it could actually write the .pt file.

@nufsty2 Way to go!

That’s a bit odd sudo was needed to write the .pt file. I haven’t had to use sudo for that. Perhaps you’re saving the .pt in a write-protected directory?

@solvingPuzzles hmm… maybe? Thanks for the help though!

If someone met this error

abort_message: assertion "terminating with uncaught exception of type c10::Error: PytorchStreamReader failed locating file bytecode.pkl: file not found ()

just save the model using _save_for_lite_interpreter like this:

It worked with me ^^