Quantize activations with qinit8

quantization
1

Hello,
I was wondering if I can use observers for activations with dtype of qint8.
It seems by the documentation that it cannot be done (I want to make sure I got it correctly).

When I tried it with different observers, it failed for this kind of error when evaluating:
RuntimeError: expected scalar type QUInt8 but found QInt8 (data_ptrc10::quint8 at /pytorch/build/aten/src/ATen/core/TensorMethods.h:6322)

Does qint8 supported for activation quantization?

Thanks!

2

The default observer for activation is quint8, but you may overwrite it by creating a new activation observer such as:
new_activation_observer = MinMaxObserver.with_args(dtype=torch.qint8)

and then assign this observer to the model qconfig before quantizing the model:
myModel.qconfig = QConfig(activation=new_activation_observer,
weight=default_weight_observer)

3

Thanks.
Did you try it?
Because I just tried what you offered and got again the above-mentioned error:
RuntimeError: expected scalar type QUInt8 but found QInt8 (data_ptrc10::quint8 at /pytorch/build/aten/src/ATen/core/TensorMethods.h:6322)

I think it’s not supported, but I don’t understand why…

4

which quantized op are you calling? some ops only works for quint8

5

I tried this, but got the same error
RuntimeError: expected scalar type QUInt8 but found QInt8 (data_ptrc10::quint8 at /pytorch/build/aten/src/ATen/core/TensorMethods.h:6322)
the op is
self.conv0 = nn.Conv2d(3, 3, kernel_size=1, stride=1, padding=0, bias=False)

which quantized op are supported qint8??

6

conv only support quint8 as input, I think right now most of the ops that doing some computation on the data itself will require quint8.

for example, conv expects quint8 activations and qint8 weight, so you need to set the dtype to quint8 for activation observer and qint8 for weight observer