Post-Training Static Quantization for Last FC layer gets rubbish results


I have this model that I have statically quantized in two manners: 1) feature extractor only and 2) feature extractor + classifier. I get adequate results with the former, but I get rubbish with the later.
I do not know where to start looking, does anyone have any idea what could be going wrong?
It seems when I quantize the classifier, there is very little variation in the classification output.
Validation error before quantization for both cases is good, but it goes down to almost 0 when I quantize the classifier.