# quantization **URL:** https://discuss.pytorch.org/c/quantization/17.md [Latest](https://discuss.pytorch.org/latest.md) · [Categories](https://discuss.pytorch.org/categories.md) --- ## [About the quantization category](https://discuss.pytorch.org/t/about-the-quantization-category/57360)
**Author:** [@smth](https://discuss.pytorch.org/u/smth)\ **Replies:** 0
This category is for questions, discussion and issues related to PyTorch’s quantization feature. For more information, see: https://github.com/pytorch/pytorch/issues/18318 --- ## [Pytorch FX Quantization for custom models](https://discuss.pytorch.org/t/pytorch-fx-quantization-for-custom-models/225328)
**Author:** [@sparshgarg23](https://discuss.pytorch.org/u/sparshgarg23)\ **Replies:** 0\ **Last updated:** [August 25, 2026, 3:01pm UTC](https://discuss.pytorch.org/t/pytorch-fx-quantization-for-custom-models/225328 "2026-08-25T15:01:37Z")
So below is my pytorch model Temporal Shift Model for video classification. temporal shift import torch import torch.nn as nn class TemporalShift(nn.Module): def \_\_init\_\_(self,n\_segments=8,n\_div=8): … --- ## [Torchao get RuntimeError](https://discuss.pytorch.org/t/torchao-get-runtimeerror/224810)
**Author:** [@shangxiaaabb](https://discuss.pytorch.org/u/shangxiaaabb)\ **Replies:** 0\ **Last updated:** [April 15, 2026, 7:54am UTC](https://discuss.pytorch.org/t/torchao-get-runtimeerror/224810 "2026-04-15T07:54:32Z")
I firsrt use the Trochao to quantize my model, like that: import torch import random import numpy as np import torch.nn as nn from torchao.quantization import quantize\_, Int4WeightOnlyConfig from torchao.quantization.qa… --- ## [XNNPACKQuantizer.set\_module\_name() not working as expected](https://discuss.pytorch.org/t/xnnpackquantizer-set-module-name-not-working-as-expected/224458)
**Author:** [@Pairman](https://discuss.pytorch.org/u/Pairman)\ **Replies:** 3\ **Last updated:** [March 5, 2026, 2:58am UTC](https://discuss.pytorch.org/t/xnnpackquantizer-set-module-name-not-working-as-expected/224458 "2026-03-05T02:58:52Z")
Hello. I’m experimenting with PT2E PTQ on my object detection model. Since quantizing the whole model completely zeros its precisions, I decide to only quantize its backbone like below: import argparse from copy import … --- ## [\[ROCm\]\[CI\] fp8 acceptable accuracy threshold](https://discuss.pytorch.org/t/rocm-ci-fp8-acceptable-accuracy-threshold/224499)
**Author:** [@andreasceid](https://discuss.pytorch.org/u/andreasceid)\ **Replies:** 2\ **Last updated:** [February 20, 2026, 6:35pm UTC](https://discuss.pytorch.org/t/rocm-ci-fp8-acceptable-accuracy-threshold/224499 "2026-02-20T18:35:06Z")
Hello, I observed that in: the accuracy threshold of that test is rtol=5e-2, atol=0.07. It is not clear to me why this is an acceptable accuracy threshold. Why can that threshold not be optimized for better accurac… --- ## [Extracting int8 weights and other quant params after convert\_pt2e](https://discuss.pytorch.org/t/extracting-int8-weights-and-other-quant-params-after-convert-pt2e/224471)
**Author:** [@BProvost](https://discuss.pytorch.org/u/BProvost)\ **Replies:** 1\ **Last updated:** [February 17, 2026, 11:17pm UTC](https://discuss.pytorch.org/t/extracting-int8-weights-and-other-quant-params-after-convert-pt2e/224471 "2026-02-17T23:17:06Z")
Hello, I have trained a simple linear model using QAT. I have also lowered this model to the XNNPACK backend via ExecuTorch. However, I would like to extract the actual int8 weights, weight scales, layer output scales, … --- ## [Limitations of Int8 QAT for Linear Layers](https://discuss.pytorch.org/t/limitations-of-int8-qat-for-linear-layers/224434)
**Author:** [@BProvost](https://discuss.pytorch.org/u/BProvost)\ **Replies:** 1\ **Last updated:** [February 17, 2026, 11:14pm UTC](https://discuss.pytorch.org/t/limitations-of-int8-qat-for-linear-layers/224434 "2026-02-17T23:14:54Z")
Hello, I am trying to create a simple example of a linear layer model using int8 weights and int8 activations trained with QAT. I’ve defined my own custom quantization config. I’ve noticed something odd though. If I hav… --- ## [Post training quantized model gets the error "Copying from quantized Tensor to non-quantized Tensor is not allowed" even though I'm not copying tensor](https://discuss.pytorch.org/t/post-training-quantized-model-gets-the-error-copying-from-quantized-tensor-to-non-quantized-tensor-is-not-allowed-even-though-im-not-copying-tensor/224497)
**Author:** [@Ekim\_Yukselbaba](https://discuss.pytorch.org/u/Ekim_Yukselbaba)\ **Replies:** 5\ **Last updated:** [February 12, 2026, 7:42pm UTC](https://discuss.pytorch.org/t/post-training-quantized-model-gets-the-error-copying-from-quantized-tensor-to-non-quantized-tensor-is-not-allowed-even-though-im-not-copying-tensor/224497 "2026-02-12T19:42:48Z")
I got a pretrained resnet 18 model from this lane detection repo in order to use it as an ADAS(advanced driver assistance systems) function for an electric car making competition. My current goal is to quantize this mode… --- ## [Huge accuracy drop from QAT model after convert\_pt2e](https://discuss.pytorch.org/t/huge-accuracy-drop-from-qat-model-after-convert-pt2e/224176)
**Author:** [@killamocingbird](https://discuss.pytorch.org/u/killamocingbird)\ **Replies:** 1\ **Last updated:** [January 27, 2026, 8:54pm UTC](https://discuss.pytorch.org/t/huge-accuracy-drop-from-qat-model-after-convert-pt2e/224176 "2026-01-27T20:54:10Z")
I’m trying to quantize a CNN using QAT and after running convert\_pt2e the F1 ratio drops from around 90% down to 70%. Here’s my code, not sure if doing something wrong or how to debug here. model\_for\_export = get\_model(… --- ## [Variable-bit (sub 8-bits) quantization for custom hardware deployment with power-of-two (pot) scales](https://discuss.pytorch.org/t/variable-bit-sub-8-bits-quantization-for-custom-hardware-deployment-with-power-of-two-pot-scales/175366)
**Author:** [@d0uwe](https://discuss.pytorch.org/u/d0uwe)\ **Replies:** 10\ **Last updated:** [January 6, 2026, 2:17pm UTC](https://discuss.pytorch.org/t/variable-bit-sub-8-bits-quantization-for-custom-hardware-deployment-with-power-of-two-pot-scales/175366 "2026-01-06T14:17:10Z")
Hey everyone! I am looking for a way to perform Quantization-Aware Training (QAT) using PyTorch. My usecase concerns deploying trained PyTorch models on custom hardware (silicon) and so I have a few requirements: Need… --- ## [Ost Training Quantization fails on SPAN model with type\_as](https://discuss.pytorch.org/t/ost-training-quantization-fails-on-span-model-with-type-as/223388)
**Author:** [@MarioAvolio](https://discuss.pytorch.org/u/MarioAvolio)\ **Replies:** 1\ **Last updated:** [December 4, 2025, 11:24pm UTC](https://discuss.pytorch.org/t/ost-training-quantization-fails-on-span-model-with-type-as/223388 "2025-12-04T23:24:03Z")
Hi all, I’m trying to run post training quantization (PTQ) on a super-resolution model called SPAN, but I’m running into an error when I forward a quantized input. I wrapped the model like this: @ARCH\_REGISTRY.registe… --- ## [Difference of IntxWeightOnlyConfig/UIntxWeightOnlyConfig/Int8WeightOnlyConfig/Int4WeightOnlyConfig/](https://discuss.pytorch.org/t/difference-of-intxweightonlyconfig-uintxweightonlyconfig-int8weightonlyconfig-int4weightonlyconfig/224000)
**Author:** [@FreedWu](https://discuss.pytorch.org/u/FreedWu)\ **Replies:** 2\ **Last updated:** [December 4, 2025, 11:23pm UTC](https://discuss.pytorch.org/t/difference-of-intxweightonlyconfig-uintxweightonlyconfig-int8weightonlyconfig-int4weightonlyconfig/224000 "2025-12-04T23:23:00Z")
In torchao, there are many configs with similar names. However, not any detailed document to explain their difference. Welcome to the torchao Documentation — torchao 0.13 documentation According to my understanding, all… --- ## [Quantize convolution layer](https://discuss.pytorch.org/t/quantize-convolution-layer/223999)
**Author:** [@FreedWu](https://discuss.pytorch.org/u/FreedWu)\ **Replies:** 1\ **Last updated:** [December 4, 2025, 11:21pm UTC](https://discuss.pytorch.org/t/quantize-convolution-layer/223999 "2025-12-04T23:21:19Z")
\# torchao/quantization/quant\_api.py def quantize\_( model: torch.nn.Module, config: AOBaseConfig, filter\_fn: Optional\[Callable\[\[torch.nn.Module, str\], bool\]\] = \_is\_linear, device: Optional\[torch.types.Devi… --- ## [PT2E quantization doesn't reduce the model size](https://discuss.pytorch.org/t/pt2e-quantization-doesnt-reduce-the-model-size/223271)
**Author:** [@vladimir\_risojevic](https://discuss.pytorch.org/u/vladimir_risojevic)\ **Replies:** 2\ **Last updated:** [December 4, 2025, 11:20pm UTC](https://discuss.pytorch.org/t/pt2e-quantization-doesnt-reduce-the-model-size/223271 "2025-12-04T23:20:23Z")
I’m trying to quantize a simple model using PT2E quantization workflow and reproduced the code from the Quick Start Guide. However, the sizes of the original floating point model and the quantized model are the same. Wha… --- ## [How to use quantized weights for manual implementation of the model in FPGA?](https://discuss.pytorch.org/t/how-to-use-quantized-weights-for-manual-implementation-of-the-model-in-fpga/165868)
**Author:** [@lucasmazz](https://discuss.pytorch.org/u/lucasmazz)\ **Replies:** 2\ **Last updated:** [September 28, 2025, 4:25am UTC](https://discuss.pytorch.org/t/how-to-use-quantized-weights-for-manual-implementation-of-the-model-in-fpga/165868 "2025-09-28T04:25:07Z")
Hi! I’m starting to study about the implementation of quantized models in FPGA. Although I’m still learning, I would like to know if I can use PyTorch quantization for this. I mean, I’d like to train a simple CNN model … --- ## [\[pt2e\]\[quant\] Quantization of operators with multiple outputs (RNN, LSTM)](https://discuss.pytorch.org/t/pt2e-quant-quantization-of-operators-with-multiple-outputs-rnn-lstm/218207)
**Author:** [@roman-janik-nxp](https://discuss.pytorch.org/u/roman-janik-nxp)\ **Replies:** 4\ **Last updated:** [September 15, 2025, 11:12am UTC](https://discuss.pytorch.org/t/pt2e-quant-quantization-of-operators-with-multiple-outputs-rnn-lstm/218207 "2025-09-15T11:12:40Z")
Environment: torch==2.6.0 executorch==0.5.0 Description: I’m working on quantization of LSTM module in ExecuTorch using PyTorch 2 Export Quantization mode. I perform Post Training Quantization. The nn.LSTM is exporte… --- ## [GPU MEM% allocation vs batch size and temporal dimension](https://discuss.pytorch.org/t/gpu-mem-allocation-vs-batch-size-and-temporal-dimension/223058)
**Author:** [@Ryan\_Xu](https://discuss.pytorch.org/u/Ryan_Xu)\ **Replies:** 3\ **Last updated:** [September 13, 2025, 5:46am UTC](https://discuss.pytorch.org/t/gpu-mem-allocation-vs-batch-size-and-temporal-dimension/223058 "2025-09-13T05:46:42Z")
I have a vision transformer training where the input has shape (B, T, C, H, W). I notice that when I double the batch size B, GPU MEM% also increases by ~2x. On the other hand, when I compare single timestep training (T… --- ## [TorchAO Migration](https://discuss.pytorch.org/t/torchao-migration/223016)
**Author:** [@jerryzh168](https://discuss.pytorch.org/u/jerryzh168)\ **Replies:** 0\ **Last updated:** [September 11, 2025, 4:07am UTC](https://discuss.pytorch.org/t/torchao-migration/223016 "2025-09-11T04:07:02Z")
Hi everyone, we are moving to torchao: GitHub - pytorch/ao: PyTorch native quantization and sparsity for training and inference , docs: Welcome to the torchao Documentation — torchao main documentation and please checkou… --- ## [Does export support quantized models with torchAo](https://discuss.pytorch.org/t/does-export-support-quantized-models-with-torchao/221571)
**Author:** [@Anis\_Moncef](https://discuss.pytorch.org/u/Anis_Moncef)\ **Replies:** 1\ **Last updated:** [September 11, 2025, 4:00am UTC](https://discuss.pytorch.org/t/does-export-support-quantized-models-with-torchao/221571 "2025-09-11T04:00:35Z")
Hello, I’m trying to quantize a model in FP8 using torchAO quantize\_(model, Float8WeightOnlyConfig()) My weights are quantized in FP8. However, when I use the function torch.export.export() observe dequatization layer… --- ## [Should I perform quantization after activation functions like sigmoid and SiLU?](https://discuss.pytorch.org/t/should-i-perform-quantization-after-activation-functions-like-sigmoid-and-silu/222960)
**Author:** [@IReallyWantToGoHome](https://discuss.pytorch.org/u/IReallyWantToGoHome)\ **Replies:** 0\ **Last updated:** [September 9, 2025, 5:22am UTC](https://discuss.pytorch.org/t/should-i-perform-quantization-after-activation-functions-like-sigmoid-and-silu/222960 "2025-09-09T05:22:10Z")
I’m asking because I encountered an issue. After applying a sigmoid function to a feature map, I tried to perform 16-bit asymmetric quantization based on the output’s min/max values. However, the calculated zero-point wa… --- ## [Quantization of Hybrid Pytorch Model](https://discuss.pytorch.org/t/quantization-of-hybrid-pytorch-model/222931)
**Author:** [@Aman\_230](https://discuss.pytorch.org/u/Aman_230)\ **Replies:** 0\ **Last updated:** [September 8, 2025, 4:47am UTC](https://discuss.pytorch.org/t/quantization-of-hybrid-pytorch-model/222931 "2025-09-08T04:47:22Z")
Hi, I hope you are fine and doing well. I have a pytorch model consists of Conv + Bi GRU + FC layer. I have trained it and generated a pth file. I want to quantize it, how to proceed? How to specify per channel quan… --- ## [Error while converting quantized Torch model to ONNX](https://discuss.pytorch.org/t/error-while-converting-quantized-torch-model-to-onnx/222892)
**Author:** [@Vitali-Matteo](https://discuss.pytorch.org/u/Vitali-Matteo)\ **Replies:** 0\ **Last updated:** [September 5, 2025, 11:36am UTC](https://discuss.pytorch.org/t/error-while-converting-quantized-torch-model-to-onnx/222892 "2025-09-05T11:36:43Z")
Hi everyone, I’m Matteo. I’m applying QAT to YOLOv8n model with the following configuration: QConfig( activation=FakeQuantize.with\_args( observer=MovingAverageMinMaxObserver, quant\_min=0, qu… --- ## [My model is taking too much time in calculating FFT to find top k ](https://discuss.pytorch.org/t/my-model-is-taking-too-much-time-in-calculating-fft-to-find-top-k/222783)
**Author:** [@M.A\_Shanawaz](https://discuss.pytorch.org/u/M.A_Shanawaz)\ **Replies:** 1\ **Last updated:** [September 2, 2025, 2:05pm UTC](https://discuss.pytorch.org/t/my-model-is-taking-too-much-time-in-calculating-fft-to-find-top-k/222783 "2025-09-02T14:05:19Z")
so basically my batch size is 32 d\_model is 128 d\_ff is 256 enc\_in = 5 seq\_len = 128 and pred\_len is 10 I narrow downed the bottle neck and found that my FFT step is taking too much time. i can’t use autocast to mak… --- ## [FX mode static\_quantization for YOLOv7](https://discuss.pytorch.org/t/fx-mode-static-quantization-for-yolov7/198616)
**Author:** [@RonaldYuren](https://discuss.pytorch.org/u/RonaldYuren)\ **Replies:** 16\ **Last updated:** [August 4, 2025, 9:32am UTC](https://discuss.pytorch.org/t/fx-mode-static-quantization-for-yolov7/198616 "2025-08-04T09:32:53Z")
I use the FX\_mode post-training-static quantization on YOLOv7， but I always see the error about the control-flow (which is happend at yolov7/models/yolo.py , that function is fuseforward()， and I don’t know how to solv… --- ## [Could not run 'aten::quantize\_per\_tensor' with arguments from the 'QuantizedCPU' backend](https://discuss.pytorch.org/t/could-not-run-aten-quantize-per-tensor-with-arguments-from-the-quantizedcpu-backend/151382)
**Author:** [@sarramrg](https://discuss.pytorch.org/u/sarramrg)\ **Replies:** 7\ **Last updated:** [July 17, 2025, 1:03pm UTC](https://discuss.pytorch.org/t/could-not-run-aten-quantize-per-tensor-with-arguments-from-the-quantizedcpu-backend/151382 "2025-07-17T13:03:20Z")
Hello everyone :smile: Currently, I have a model trained on Pytorch. Its size is around 42 Mb. The expected inputs of this model are (1, 3, 512, 512) images. This model should be deployed on an iOS mobile app but first … --- ## [RuntimeError: quantized::conv2d\_prepack() is missing value for argument 'stride'](https://discuss.pytorch.org/t/runtimeerror-quantized-conv2d-prepack-is-missing-value-for-argument-stride/220771)
**Author:** [@FreedWu](https://discuss.pytorch.org/u/FreedWu)\ **Replies:** 1\ **Last updated:** [July 1, 2025, 11:30pm UTC](https://discuss.pytorch.org/t/runtimeerror-quantized-conv2d-prepack-is-missing-value-for-argument-stride/220771 "2025-07-01T23:30:34Z")
I try to use torch.ao.quantization to quantize a model. However, it will give me an error. If I remove the lines about quantization, it can work. Traceback (most recent call last): File "/dev/shm/test/compress.py", l… --- ## [Why is there such a significant difference between floating-point convolution and quantized integer convolution results?](https://discuss.pytorch.org/t/why-is-there-such-a-significant-difference-between-floating-point-convolution-and-quantized-integer-convolution-results/220920)
**Author:** [@bitzs](https://discuss.pytorch.org/u/bitzs)\ **Replies:** 2\ **Last updated:** [June 30, 2025, 11:28am UTC](https://discuss.pytorch.org/t/why-is-there-such-a-significant-difference-between-floating-point-convolution-and-quantized-integer-convolution-results/220920 "2025-06-30T11:28:31Z")
Hello everyone on the forum In my process of learning PyTorch quantization, I implemented convolution in two ways. Given quantized input and weight tensors, along with the scale and zero\_point for the input, weight, and… --- ## [\[MPS\] When device='mps', aten.linear.default op is not decomposed](https://discuss.pytorch.org/t/mps-when-device-mps-aten-linear-default-op-is-not-decomposed/220573)
**Author:** [@saeonnuri](https://discuss.pytorch.org/u/saeonnuri)\ **Replies:** 1\ **Last updated:** [June 5, 2025, 1:55am UTC](https://discuss.pytorch.org/t/mps-when-device-mps-aten-linear-default-op-is-not-decomposed/220573 "2025-06-05T01:55:02Z")
Summary I was following the “Writing your own quantized tensor” tutorial with apple silicon chip(M4 pro), but unlike tutorial example, the aten.linear.default is not decomposed into aten.mm.default, and aten.t.default. C… --- ## [Logits mismatch between PyTorch inference and manual implementation](https://discuss.pytorch.org/t/logits-mismatch-between-pytorch-inference-and-manual-implementation/219568)
**Author:** [@greifswald](https://discuss.pytorch.org/u/greifswald)\ **Replies:** 1\ **Last updated:** [April 29, 2025, 12:42am UTC](https://discuss.pytorch.org/t/logits-mismatch-between-pytorch-inference-and-manual-implementation/219568 "2025-04-29T00:42:05Z")
I’m trying to manually reproduce the inference forward-pass to understand exactly how quantized inference works. To do so, I trained and quantized a model in PyTorch using QAT, manually simulate the forward pass, and com… --- ## [QAT model drops accuracy after converting with torch.ao.quantization.convert](https://discuss.pytorch.org/t/qat-model-drops-accuracy-after-converting-with-torch-ao-quantization-convert/219533)
**Author:** [@du\_tran\_ngoc](https://discuss.pytorch.org/u/du_tran_ngoc)\ **Replies:** 1\ **Last updated:** [April 29, 2025, 12:36am UTC](https://discuss.pytorch.org/t/qat-model-drops-accuracy-after-converting-with-torch-ao-quantization-convert/219533 "2025-04-29T00:36:54Z")
Hello everyone. I am implementing QAT model yolov8 in 4bit mode for weight and 8bit for activation by setting quant\_min, quant\_max in config. The model when training and eval gives quite good results, however when I con… [Next page](https://discuss.pytorch.org/c/quantization/17.md?page=1)