Quantization accuracy debugging for custom LSTM PTQ

quantization
1

Hello,
I am trying to debug custom LSTM PTQ using torch.ao.ns._numeric_suite_fx.
This is the code after custom quantization:

# extract weight pairs
weight_comparison = ns.extract_weights('a', m, 'b', converted)

ns.extend_logger_results_with_comparison(
    weight_comparison, 'a', 'b', torch.ao.ns.fx.utils.compute_sqnr,
    'sqnr')

I got this error:

    raise GraphMatchingException(msg)
torch.ao.ns.fx.graph_matcher.GraphMatchingException: 
The subgraphs
(NSSubgraph(start_node=my_lstm, end_node=my_lstm, base_op_node=my_lstm), <class 'torch.nn.modules.rnn.LSTM'>) and
(NSSubgraph(start_node=my_lstm, end_node=my_lstm, base_op_node=my_lstm), <class 'torch.ao.nn.quantized.modules.rnn.LSTM'>)
are not related. Please ensure that the two models you pass in have the same number
of subgraphs, and each pair of subgraphs is related to each other.

Does this mean that custom LSTM quantization is not supported for those APIs?
Or I should manually match quantized.LSTM and nn.LSTM ops?

2

probably means numeric suite is not yet supporting comparing nn.LSTM and nn.quantized.LSTM. cc @Vasiliy_Kuznetsov could you comment on the plan on the support?

3

Numeric Suite for workflows such as Eager mode quantization and FX graph mode quantization is not something the core quantization team actively works on because we expect PT2.0 export quantization to be more aligned with our future plans, but we would welcome contributions.