Is it a good idea to use float16/bfloat16 for inference?

mixed-precision
1

Hi

I am currently using bfloat16 for inference and the model seems to be performing well (with torch.amp.autocast).
Is it a common practice to use reduce precision for inference as well as for training? I am not confident enough because most articles about reduced precision are focusing on training.

2

Yes, lower dtypes can be used for inference, too.

3

Thank you for the clear answer!