It is necessary to use CPU to do QAT

Hi,

I was wondering if it is possible to do QAT with GPU. According to this tutorial ( (beta) Static Quantization with Eager Mode in PyTorch — PyTorch Tutorials 1.12.1+cu102 documentation), we need to use CPU.

However, according to this blog( Introduction to Quantization on PyTorch | PyTorch), we can use either CPU or GPU.

Thanks for your attention

Yes, QAT training works on GPU.

1 Like

If you scroll all the way down to the QAT cell in this notebook there is an example of QAT with GPU

Thanks for all replies here. It seem QAT is supported by GPU. Maybe this tutorial ((beta) Static Quantization with Eager Mode in PyTorch — PyTorch Tutorials 1.12.1+cu102 documentation) has to be changed a little?

Since at the beginning, it says “Note that quantization is currently only supported for CPUs, so we will not be utilizing GPUs / CUDA in this tutorial.”

Thanks @jerryzh168

QAT supports GPU in the beginning, I think that line means it’s not supported in GPU for inference. We had been working on supporting GPU inference through TensorRT and cudnn, but haven’t tried officially releasing them

1 Like