Very Bad Bounding Boxes After Quantization of Model

Hi Folks,
I have been trying to perform static quantization of a model for scene text detection and reduce the precision of the model from float32 to integer8.

I followed the steps mentioned in the official tutorial - fused the conv,batch norm and relu layers ; converted to int ; replaced concat with nn.quantized.Floattensor().
The original model works well, but the prediction from quantized model is really bad -

Prediction from FP32 model -

Prediction from INT8 model -

Not sure where I’m going wrong :confused:

Would appreciate on any suggestions where I’m going wrong during quantization.

The code is available here -

Not sure how it might help you, but you can try using Intel’s Openvino for quantisation.

yeah pytorch quantization didn’t yield very good results. Might give OpenVino or Tensortt a try.

Seems it’s the same topic as another thread where we are helping on debugging: