It is necessary to use CPU to do QAT

quantization
1

Hi,

I was wondering if it is possible to do QAT with GPU. According to this tutorial ( (beta) Static Quantization with Eager Mode in PyTorch — PyTorch Tutorials 1.12.1+cu102 documentation), we need to use CPU.

However, according to this blog( Introduction to Quantization on PyTorch | PyTorch), we can use either CPU or GPU.

Thanks for your attention

2

Yes, QAT training works on GPU.

1 Like
3

If you scroll all the way down to the QAT cell in this notebook there is an example of QAT with GPU

4

Thanks for all replies here. It seem QAT is supported by GPU. Maybe this tutorial ((beta) Static Quantization with Eager Mode in PyTorch — PyTorch Tutorials 1.12.1+cu102 documentation) has to be changed a little?

Since at the beginning, it says “Note that quantization is currently only supported for CPUs, so we will not be utilizing GPUs / CUDA in this tutorial.”

Thanks @jerryzh168

5

QAT supports GPU in the beginning, I think that line means it’s not supported in GPU for inference. We had been working on supporting GPU inference through TensorRT and cudnn, but haven’t tried officially releasing them

2 Likes