Quantization not Decreasing Model Size (Static and QAT)

quantization
#1

Hi

I am trying to quantize a text detection model based on Mobilenet (model definition here )

After inserting the quant and dequant stub, fusing all the conv+bn+relu and conv+relu, replacing cat with skip_add.cat() . I perform the static quantization (script - https://github.com/raghavgurbaxani/Quantization_Experiments/blob/master/try_quantization.py )

After performing quantization, the model size doesn’t go down (in fact it increases )

Original Size:
Size (MB): 6.623636

Fused model Size:
Size (MB): 6.638188

Quantized model Size:
Size (MB): 7.928258

I have even printed the final quantized model here

I changed the qconfig to fused_model.qconfig = torch.quantization.default_qconfig but still quantized_model size is Size (MB): 6.715115

Why doesn’t the model size reduce ?

1 Like
#3

Looking at the model def you posted, it looks like it is not yet quantized. One missing thing is calibration. You can add a calibration step after you call prepare and before you call convert:

torch.quantization.prepare(fused_model, inplace=True)

# calibrate your model by feeding it example inputs
for inputs in your_dataset:        
    fused_model(inputs)

print('Quantized model Size:')
quantized = torch.quantization.convert(fused_model, inplace=False)
print_size_of_model(quantized)
#4

Hi @Vasiliy_Kuznetsov

Thank you for your input, I have updated my script to pass in a few images into the fused model as inputs for calibration.

Please see the updated script here

But still the quantized model size is bigger than the original model -

Original Size:
Size (MB): 6.623636

Fused model Size:
Size (MB): 6.638188

Quantized model Size:
Size (MB): 6.712286

there seems to be some improvement due to the calibration, but the quantized model size is still not satisfactory compared to the original size :frowning:

Could you suggest what’s going wrong here ?

#5

@Vasiliy_Kuznetsov

I also tried a script with Quantized Aware Training -

But still the quantized model is bigger than the original model :frowning: :no_mouth:

I don’t know what’s going wrong here

Original Size:
Size (MB): 6.623636

Fused model Size:
Size (MB): 6.638188

Quantized model Size:
Size (MB): 6.712286

QAT model Size:
Size (MB): 6.712286
#6

in the paste here (https://github.com/raghavgurbaxani/Quantization_Experiments/blob/master/quantized_model.txt), the model doesn’t look quantized. One would expect to see QuantizedConv instead of Conv and QuantizedLinear instead of Linear. One thing to try could be to make sure to run the convert script and ensure that you see the quantized module equivalents afterwards.

#7

Hi @Vasiliy_Kuznetsov

Please check the updated quantized_model now -

it seems to have quantized covolutions (line 100 onwards). I don’t know why the layers before line 100 do not have quantized modules.

Do you think my quantstub and dequantstub placement is incorrect ?
Here’s the model (with quant and dequant stub)

Main script here - https://github.com/raghavgurbaxani/Quantization_Experiments/blob/master/try_qat.py

I suspect maybe my quant and dequant stub may be incorrect but apart from that I’ve followed all the steps as posted in the static quantization tutorial.

Reallly appreciate your help

#9

@Vasiliy_Kuznetsov any update on this ?

#10

Hi Raghav,

For post training quantization, we want the model to be in eval mode (see https://github.com/pytorch/pytorch/blob/530d48e93a3f04a5ec63a1b789c19a5f775bf497/torch/quantization/fuse_modules.py#L63). So, you can add a model.eval() call before you fuse modules:

model.eval()
torch.quantization.fuse_modules(...)
1 Like