For anyone coming across this with the same question, I had to specifically disable both Vulkan and XNNPACK support to build the native code for Android (-DUSE_VULKAN=OFF -DUSE_XNNPACK=OFF).
I would be interested in any tips on how to build without disabling those features, though, since I’m not sure what the consequences is of disabling them!