How can I get the gradient of quantized tensor?

quantization
1

I quantized my model with post-training quantization in PyTorch. Then I want to compare the gradient of input ‘x’ before and after quantization. I first set requires_grad=True. After self.quant(x), x.requires_grad became False. When I set it to True, it reported RuntimeError: only Tensors of floating point dtype can require gradients. How can I get the gradient of quantized tensor?

x = self.preprocess(x)
x = self.quant(x)
# x.requires_grad_(True)
x = self.features(x)
2

Hi Rahul,

As the error message suggests, I don’t think it is possible to get the gradient of a quantized tensor. The only quantization mode in which we need gradients is QAT, and in this mode all weights and activations are only “fake quantized” and not actually cast to int8 values. Please see this blog post for more information.

Best,
-Andrew

3

Thanks for your reply.