Increased memory usage with AMP

mixed-precision
4

As you’ve pointed out, the memory savings depend on the model architecture and e.g. the ratio f parameters to activations.
This post breaks it down for some common use cases.