Very Bad Bounding Boxes After Quantization of Model

[Raghav_Gurbaxani]

Hi Folks,
I have been trying to perform static quantization of a model for scene text detection and reduce the precision of the model from float32 to integer8.

I followed the steps mentioned in the official tutorial - fused the conv,batch norm and relu layers ; converted to int ; replaced concat with nn.quantized.Floattensor().
The original model works well, but the prediction from quantized model is really bad -

Prediction from FP32 model -

Prediction from INT8 model -

Not sure where I’m going wrong

Would appreciate on any suggestions where I’m going wrong during quantization.

The code is available here - https://github.com/raghavgurbaxani/experiments/blob/master/try_static_quant.py

[paganpasta]

Not sure how it might help you, but you can try using Intel’s Openvino for quantisation.

[Raghav_Gurbaxani]

yeah pytorch quantization didn’t yield very good results. Might give OpenVino or Tensortt a try.

[hx89]

Seems it’s the same topic as another thread where we are helping on debugging:

Quantization Error During Concat -- RuntimeError: Didn't find kernel to dispatch to for operator 'aten::_cat'

thanks @hx89 , if you could post that example for compare module level quantization error - It would be great In the meantime, I tried the histogram observer and the result is still pretty bad [image] any other suggestions ?