Have you checked the accuracy of fused_model? By checking the accuracy of fused_model before converting to int8 model we can know if the issue is in the preprocessing part or in the quantized model.

If fused_model has good accuracy, the next step we can check the quantization error of the weights. Could you try the following code:

```
def l2_error(ref_tensor, new_tensor):
"""Compute the l2 error between two tensors.
Args:
ref_tensor (numpy array): Reference tensor.
new_tensor (numpy array): New tensor to compare with.
Returns:
abs_error: l2 error
relative_error: relative l2 error
"""
assert (
ref_tensor.shape == new_tensor.shape
), "The shape between two tensors is different"
diff = new_tensor - ref_tensor
abs_error = np.linalg.norm(diff)
ref_norm = np.linalg.norm(ref_tensor)
if ref_norm == 0:
if np.allclose(ref_tensor, new_tensor):
relative_error = 0
else:
relative_error = np.inf
else:
relative_error = np.linalg.norm(diff) / ref_norm
return abs_error, relative_error
float_model_dbg = fused_model
qmodel_dbg = quantized
for key in float_model_dbg.state_dict().keys():
float_w = float_model_dbg.state_dict()[key]
qkey = key
# Get rid of extra hiearchy of the fused Conv in float model
if key.endswith('.weight'):
qkey = key[:-9] + key[-7:]
if qkey in qmodel_dbg.state_dict():
q_w = qmodel_dbg.state_dict()[qkey]
if q_w.dtype == torch.float:
abs_error, relative_error = l2_error(float_w.numpy(), q_w.detach().numpy())
else:
abs_error, relative_error = l2_error(float_w.numpy(), q_w.dequantize().numpy())
print(key, ', abs error = ', abs_error, ", relative error = ", relative_error)
```

It should print out the quantization error for each Conv weight such as:

```
features.0.0.weight , abs error = 0.21341866 , relative error = 0.01703797
features.3.squeeze.0.weight , abs error = 0.095942035 , relative error = 0.012483358
features.3.expand1x1.0.weight , abs error = 0.071949296 , relative error = 0.010309489
features.3.expand3x3.0.weight , abs error = 0.18284422 , relative error = 0.025256516
features.4.squeeze.0.weight , abs error = 0.088713735 , relative error = 0.011313644
features.4.expand1x1.0.weight , abs error = 0.0780085 , relative error = 0.0126931975
...
```