Does amp autocast cache fp16 copies of model parameters?

I’m thinking of using autocast for speeding up processing of frozen backbone. If I use autocast context manager in forward, will it every time spend time on conversion of model weights?

How do I ensure it does conversion of model parameters fp32->fp16 exactly once?

My alternative is to manually convert frozen backbone’s parameters to fp16, and maybe not use any autocast at all, but I assume autocast is a better bet because it has special cases for ops that still must be run in fp32.

Yes, autocast uses caching internally unless you manually disable it and the checks can be found here.

It’ll be done automatically for you if the parameter is found in the cache (which won’t be the case if e.g. you’ve modified it).

1 Like