We are currently working with pytorch 1.9.1.
We built pytorch with LITE_INTERPRETER and converted our original fully quantized model that contains only conv and relu to NNAPI supported format.
Test inference code that is shown below worked successfully on Android. But forward method slows down at some point.
As shown below, in first 37 iterations, each execution takes about 78 - 80 msec. After that, however, it takes about 90 - 102 msec.
What is the cause of this feature? We found the similar issue here. Do these issues come from the same cause? I’d like to know the cause and solution of this issue.
Thanks in advance!
void testLoadAndForward(
JNIEnv *env,
jstring jModelPath)
{
c10::InferenceMode guard;
const char* modelPath = env->GetStringUTFChars(jModelPath, 0);
torch::jit::mobile::Module module = torch::jit::_load_for_mobile(modelPath);
auto get_method = module.find_method("get_all_bundled_inputs");
auto all_inputs = (*get_method)({}).toList();
auto inputs = all_inputs.get(0).toTuple()->elements();
c10::IValue t_out;
int iter = 100;
for (int i = 0; i < iter; ++i) {
std::chrono::system_clock::time_point start = std::chrono::system_clock::now();
t_out = module.forward(inputs);
std::chrono::system_clock::time_point end = std::chrono::system_clock::now();
auto duration = std::chrono::duration_cast<std::chrono::nanoseconds>(end - start).count() / 1000000.0;
LOGD("%d duration: %lf msec", i, duration);
}
}
Output is:
D/NATIVE_CODE: 0 duration: 9284.693000 msec
D/NATIVE_CODE: 1 duration: 79.760000 msec
D/NATIVE_CODE: 2 duration: 78.123000 msec
D/NATIVE_CODE: 3 duration: 78.468000 msec
D/NATIVE_CODE: 4 duration: 77.882000 msec
D/NATIVE_CODE: 5 duration: 78.987000 msec
D/NATIVE_CODE: 6 duration: 77.897000 msec
D/NATIVE_CODE: 7 duration: 78.845000 msec
D/NATIVE_CODE: 8 duration: 78.278000 msec
D/NATIVE_CODE: 9 duration: 77.879000 msec
D/NATIVE_CODE: 10 duration: 78.108000 msec
D/NATIVE_CODE: 11 duration: 78.109000 msec
D/NATIVE_CODE: 12 duration: 78.286000 msec
D/NATIVE_CODE: 13 duration: 78.376000 msec
D/NATIVE_CODE: 14 duration: 78.368000 msec
D/NATIVE_CODE: 15 duration: 78.295000 msec
D/NATIVE_CODE: 16 duration: 78.213000 msec
D/NATIVE_CODE: 17 duration: 78.345000 msec
D/NATIVE_CODE: 18 duration: 77.938000 msec
D/NATIVE_CODE: 19 duration: 78.280000 msec
D/NATIVE_CODE: 20 duration: 78.160000 msec
D/NATIVE_CODE: 21 duration: 78.574000 msec
D/NATIVE_CODE: 22 duration: 78.711000 msec
D/NATIVE_CODE: 23 duration: 78.490000 msec
D/NATIVE_CODE: 24 duration: 78.499000 msec
D/NATIVE_CODE: 25 duration: 78.521000 msec
D/NATIVE_CODE: 26 duration: 78.173000 msec
D/NATIVE_CODE: 27 duration: 78.130000 msec
D/NATIVE_CODE: 28 duration: 77.988000 msec
D/NATIVE_CODE: 29 duration: 78.555000 msec
D/NATIVE_CODE: 30 duration: 78.568000 msec
D/NATIVE_CODE: 31 duration: 78.566000 msec
D/NATIVE_CODE: 32 duration: 80.458000 msec
D/NATIVE_CODE: 33 duration: 79.139000 msec
D/NATIVE_CODE: 34 duration: 78.517000 msec
D/NATIVE_CODE: 35 duration: 78.168000 msec
D/NATIVE_CODE: 36 duration: 78.810000 msec
D/NATIVE_CODE: 37 duration: 78.541000 msec
D/NATIVE_CODE: 38 duration: 87.629000 msec
D/NATIVE_CODE: 39 duration: 89.959000 msec
D/NATIVE_CODE: 40 duration: 93.983000 msec
D/NATIVE_CODE: 41 duration: 97.401000 msec
D/NATIVE_CODE: 42 duration: 96.597000 msec
D/NATIVE_CODE: 43 duration: 90.576000 msec
D/NATIVE_CODE: 44 duration: 92.451000 msec
D/NATIVE_CODE: 45 duration: 97.267000 msec
D/NATIVE_CODE: 46 duration: 97.906000 msec
D/NATIVE_CODE: 47 duration: 97.167000 msec
D/NATIVE_CODE: 48 duration: 94.146000 msec
D/NATIVE_CODE: 49 duration: 94.663000 msec
D/NATIVE_CODE: 50 duration: 93.778000 msec
D/NATIVE_CODE: 51 duration: 99.721000 msec
D/NATIVE_CODE: 52 duration: 97.753000 msec
D/NATIVE_CODE: 53 duration: 97.071000 msec
D/NATIVE_CODE: 54 duration: 97.035000 msec
D/NATIVE_CODE: 55 duration: 101.194000 msec
D/NATIVE_CODE: 56 duration: 102.563000 msec
D/NATIVE_CODE: 57 duration: 99.331000 msec
D/NATIVE_CODE: 58 duration: 98.304000 msec
D/NATIVE_CODE: 59 duration: 91.064000 msec
D/NATIVE_CODE: 60 duration: 96.322000 msec
D/NATIVE_CODE: 61 duration: 94.546000 msec
D/NATIVE_CODE: 62 duration: 93.846000 msec
D/NATIVE_CODE: 63 duration: 98.800000 msec
D/NATIVE_CODE: 64 duration: 96.586000 msec
D/NATIVE_CODE: 65 duration: 99.268000 msec
D/NATIVE_CODE: 66 duration: 96.090000 msec
D/NATIVE_CODE: 67 duration: 96.336000 msec
D/NATIVE_CODE: 68 duration: 99.795000 msec
D/NATIVE_CODE: 69 duration: 100.935000 msec
D/NATIVE_CODE: 70 duration: 99.763000 msec
D/NATIVE_CODE: 71 duration: 98.782000 msec
D/NATIVE_CODE: 72 duration: 101.088000 msec
D/NATIVE_CODE: 73 duration: 101.943000 msec
D/NATIVE_CODE: 74 duration: 99.099000 msec
D/NATIVE_CODE: 75 duration: 99.384000 msec
D/NATIVE_CODE: 76 duration: 91.306000 msec
D/NATIVE_CODE: 77 duration: 93.050000 msec
D/NATIVE_CODE: 78 duration: 92.522000 msec
D/NATIVE_CODE: 79 duration: 91.041000 msec
D/NATIVE_CODE: 80 duration: 99.242000 msec
D/NATIVE_CODE: 81 duration: 99.460000 msec
D/NATIVE_CODE: 82 duration: 99.413000 msec
D/NATIVE_CODE: 83 duration: 99.119000 msec
D/NATIVE_CODE: 84 duration: 102.010000 msec
D/NATIVE_CODE: 85 duration: 98.925000 msec
D/NATIVE_CODE: 86 duration: 99.047000 msec
D/NATIVE_CODE: 87 duration: 95.927000 msec
D/NATIVE_CODE: 88 duration: 95.569000 msec
D/NATIVE_CODE: 89 duration: 94.219000 msec
D/NATIVE_CODE: 90 duration: 96.446000 msec
D/NATIVE_CODE: 91 duration: 96.937000 msec
D/NATIVE_CODE: 92 duration: 95.171000 msec
D/NATIVE_CODE: 93 duration: 90.502000 msec
D/NATIVE_CODE: 94 duration: 97.297000 msec
D/NATIVE_CODE: 95 duration: 100.063000 msec
D/NATIVE_CODE: 96 duration: 97.878000 msec
D/NATIVE_CODE: 97 duration: 97.199000 msec
D/NATIVE_CODE: 98 duration: 99.090000 msec
D/NATIVE_CODE: 99 duration: 92.561000 msec