Quantized model is slow and gpu usage becomes high with qnnpack

[Sining_Sun]

@Vasiliy_Kuznetsov

Thanks. Apart from the layernorm problem, I found another problem. Even my network is very simple, for example, just one Linear layer without LayerNorm, the cpu usage is very high after quantization. More details can be found in in post.

Quantized model has higer cpu usage on Android

Hi all, Recently, I am deploying my model to Android. I found that my quantized model has very high cpu usage, even much higher than fp32 float model. So I tried to run a model with only one Linear layer like: Class Linear（torch.nn.Module): def _init_(self, idim, odim): self.quant= torch.quantization.QuantStub() self.dequant = torch.quantization.DequantStub() self.linear = torch.nn.Sequential(torch.nn.Linear(idim, 60), torch.nn.Linear(60, odim), torch.nn.ReLU()) def forward(self, …

This problem has been confused me for a long time.