I’m thinking of using autocast for speeding up processing of frozen backbone. If I use autocast context manager in forward, will it every time spend time on conversion of model weights?
How do I ensure it does conversion of model parameters fp32->fp16 exactly once?
My alternative is to manually convert frozen backbone’s parameters to fp16, and maybe not use any autocast at all, but I assume autocast is a better bet because it has special cases for ops that still must be run in fp32.