# What does PyTorch do when calling tensor.half() **URL:** \ **Category:** Uncategorized\ **Created:** [October 18, 2019, 7:13pm UTC](https://discuss.pytorch.org/t/what-does-pytorch-do-when-calling-tensor-half/58597 "2019-10-18T19:13:42Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Beinan\_Wang](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/beinan_wang/32/14160_2.png) [@Beinan\_Wang](https://discuss.pytorch.org/u/Beinan_Wang)\ **Post date:** [October 18, 2019, 7:13pm UTC](https://discuss.pytorch.org/t/what-does-pytorch-do-when-calling-tensor-half/58597/1 "2019-10-18T19:13:42Z")
I want to understand how pytorch does fp16 inference. Say I have a pretrained fp32 model and I run fp16 inference by calling model.half(). Then when I run y = model(x), does pytorch simply calculate in fp16 format or are there some hidden optimizations applied to make sure there is not significant drop in accuracy? ---
**Author:** ![albanD](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/32/215_2.png) [@albanD](https://discuss.pytorch.org/u/albanD)\ **Post date:** [October 18, 2019, 7:48pm UTC](https://discuss.pytorch.org/t/what-does-pytorch-do-when-calling-tensor-half/58597/2 "2019-10-18T19:48:29Z")
If you only do `model.half()` and then forward. Pytorch will only convert all the model weights to half precision and then forward with that. If you want something smarter (that keeps single precision buffers for some ops for stability), you can check out [nvidia’s amp package](https://github.com/NVIDIA/apex/tree/master/apex/amp). ---
**Author:** ![Beinan\_Wang](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/beinan_wang/32/14160_2.png) [@Beinan\_Wang](https://discuss.pytorch.org/u/Beinan_Wang)\ **Post date:** [October 19, 2019, 3:21am UTC](https://discuss.pytorch.org/t/what-does-pytorch-do-when-calling-tensor-half/58597/3 "2019-10-19T03:21:17Z")
Thank you for the clarification. ---
**Author:** ![jiacheng1gujiaxin](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/f17d59/32.png) [@jiacheng1gujiaxin](https://discuss.pytorch.org/u/jiacheng1gujiaxin)\ **Post date:** [November 12, 2019, 4:02am UTC](https://discuss.pytorch.org/t/what-does-pytorch-do-when-calling-tensor-half/58597/4 "2019-11-12T04:02:10Z")
Can nvidia-apex only be used for inference? I want to deploy my model with mix precision, but I find that when I run it, my memory is not reduced. futhermore, when i run fp16 inference by calling model.half(), the memory is not reduced either ---
**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\ **Post date:** [November 12, 2019, 4:20am UTC](https://discuss.pytorch.org/t/what-does-pytorch-do-when-calling-tensor-half/58597/5 "2019-11-12T04:20:00Z")
> [@jiacheng1gujiaxin](#): > > Can nvidia-apex only be used for inference? No, you can use `apex` for training and inference. > [@jiacheng1gujiaxin](#): > > I want to deploy my model with mix precision, but I find that when I run it, my memory is not reduced. > > futhermore, when i run fp16 inference by calling model.half(), the memory is not reduced either If you are checking the used memory via `nvidia-smi`, note that you might see the cached memory as well. `torch.cuda.memory_allocated()` might give you a better number. ---
**Author:** ![jiacheng1gujiaxin](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/f17d59/32.png) [@jiacheng1gujiaxin](https://discuss.pytorch.org/u/jiacheng1gujiaxin)\ **Post date:** [November 14, 2019, 8:39am UTC](https://discuss.pytorch.org/t/what-does-pytorch-do-when-calling-tensor-half/58597/6 "2019-11-14T08:39:19Z")
May i ask you how can i deploy my model with fp16 would get significant performance promote. is tensorrt helpful? ---
**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\ **Post date:** [November 14, 2019, 7:00pm UTC](https://discuss.pytorch.org/t/what-does-pytorch-do-when-calling-tensor-half/58597/7 "2019-11-14T19:00:02Z")
Depending on your model, you might just call `model.half()` on it. However, IIRC you could export the vanilla FP32 model and pass it to TensorRT, which should use the specified precision. ---
**Author:** ![Rohith\_Rajesh](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rohith_rajesh/32/17647_2.png) [@Rohith\_Rajesh](https://discuss.pytorch.org/u/Rohith_Rajesh)\ **Post date:** [July 17, 2021, 11:46am UTC](https://discuss.pytorch.org/t/what-does-pytorch-do-when-calling-tensor-half/58597/8 "2021-07-17T11:46:01Z")
Hi! Does that mean the activations will also be computed in FP16?