Quantization type error

# CODE 1 - type = uint8
model.qconfig = torch.quantization.QConfig(
activation=default_observer.with_args(dtype=torch.quint8),
weight=default_float_qparams_observer.with_args(dtype=torch.quint8))
model_prepared = torch.quantization.prepare(model, inplace=True)
torch.quantization.convert(model_prepared, inplace=True)

result : AssertionError: Weight observer must have dtype torch.qint8
(“/usr/local/lib/python3.9/dist-packages/torch/nn/quantized/modules/linear.py”, line 282, in from_float
)

# CODE 2 - type = int8
model.qconfig = torch.quantization.QConfig(
activation=default_observer.with_args(dtype=torch.qint8),
weight=default_float_qparams_observer.with_args(dtype=torch.qint8))
model_prepared = torch.quantization.prepare(model, inplace=True)
torch.quantization.convert(model_prepared, inplace=True)

result : AssertionError: The only supported weight dtype for nnq.Embedding is torch.quint8, got torch.qint8
(“/usr/local/lib/python3.9/dist-packages/torch/nn/quantized/modules/embedding_ops.py”, line 162, in from_float)

Each type make different error.
If i change weight observer’s type to int8, i can watch unsupported type error.
If i change weight observer’s type to uint8, i can watch weight observer error.

How can I solve this problems…

Hi @ELIVATOR , for embeddings the supported dtype for weight is quint8, and for other ops it’s usually qint8. You can configure this by assigning the appropriate qconfigs to the right parts of the model. For example:

qconfig_global = torch.quantization.QConfig(
    activation=default_observer,
    weight=default_weight_observer)
qconfig_emb = torch.quantization.QConfig(
    activation=default_observer,
    weight=default_float_qparams_observer)

# set global qconfig
model.qconfig = qconfig_global
# set custom qconfig for embedding layer
model.foo.bar.emb.qconfig = qconfig_emb

Sometimes there are multiple embedding layers in one model. Is there any way to set them once and for all?

If you are using the eager mode workflow, you can do something like

for _, child in model.named_modules():
    if isinstance(child, torch.nn.Embedding):
        child.qconfig = emb_qconfig

If you are using the FX graph mode workflow, you can use QConfigMapping to map a module type to a qconfig.

1 Like