@supriyar has tested the quantization in onnx with one of our internal models, but I’m not sure about the long term plans for that. @supriyar can you comment?
Hi, I’ve read your answer, but I am confused. You need first an onnx model which you later convert to caffe2. But if I get an error when exporting to onnx, how I can get to second step?
Traceback (most recent call last):
File "check_conv_op.py", line 92, in <module>
quantize(img)
File "check_conv_op.py", line 59, in quantize
torch.onnx.export(model, img, "8INTmodel.onnx", verbose=True)
File "/usr/local/lib/python3.7/site-packages/torch/onnx/__init__.py", line 168, in export
custom_opsets, enable_onnx_checker, use_external_data_format)
File "/usr/local/lib/python3.7/site-packages/torch/onnx/utils.py", line 69, in export
use_external_data_format=use_external_data_format)
File "/usr/local/lib/python3.7/site-packages/torch/onnx/utils.py", line 485, in _export
fixed_batch_size=fixed_batch_size)
File "/usr/local/lib/python3.7/site-packages/torch/onnx/utils.py", line 334, in _model_to_graph
graph, torch_out = _trace_and_get_graph_from_model(model, args, training)
File "/usr/local/lib/python3.7/site-packages/torch/onnx/utils.py", line 282, in _trace_and_get_graph_from_model
orig_state_dict_keys = _unique_state_dict(model).keys()
File "/usr/local/lib/python3.7/site-packages/torch/jit/__init__.py", line 302, in _unique_state_dict
filtered_dict[k] = v.detach()
AttributeError: 'torch.dtype' object has no attribute 'detach'
Hi @dassima and @jerryzh168 - did you manage to get to the bottom of this? I’m seeing exactly the same error. A simple model exports fine without quantization.
Setting a break on the point of failure, I’m seeing the object to be detached is torch.qint8
Then dumping the state_dict for both non-quantized and quantized versions, the quantized version has this as an entry - (‘fc1._packed_params.dtype’, torch.qint8). The non quantized version has only tensors.
Any thoughts as to what’s going on greatly appreciated!
Hi @jerryzh168, yes. Updated initially to 1.7.0.dev20200705+cpu and just tried torch-1.7.0.dev20200724+cpu. No luck with either.
As I hijacked an old thread, I thought best to raise a separate issue with a simple example (single fully connected layer) to replicate -
I’ve had one reply with comment explaining that exporting of quantized models is not yet supported and a link to another thread. Sounds like it’s WIP. Would be good to get your take on the example in the other thread.
How are you exporting the quantized model to ONNX? Like previously mentioned we only currently support a custom conversion flow through ONNX to Caffe2 for quantized models. The models aren’t represented in native ONNX format, but a format specific to Caffe2.
If you wish to export model to caffe2, you can follow the steps here to do so (model needs to be traced first and need to set operator_export_type to ONNX_ATEN_FALLBACK)
Export of pytorch QAT models to ONNX standard is supported now. You should be able to export the model without operator_export_type = ONNX_ATEN_FALLBACK
@neginraoof Can you post a minimal example of PyTorch QAT → ONNX and clarify which PyTorch version this became supported in? I’m on 1.8.1 and still seeing similar errors as @G4V. In all the discussion I’m able to find (forum + Github issues), @supriyar says the only support path is still involving Caffe2