Yes, it’s possible to use just parts of your model in FP16. Autograd will take care of transforming all gradients to the necessary dtype.
Also, if you are using cuDNN, some layers also take mixed precision input, so that you don’t have to specifically cast them to FP16 or FP32.
Have a look at NVIDIA’s apex for some mixed precision utilities as well as this post for more information regarding apex.