To train my model, I must use AMP, but AMP causes Div/0 in Adam.
To fix this, I have to increase the default value of eps from 1e-8 to 1e-4. However, I noticed that by doing so, my model progress much slower.
My default LR rate is 0.001.
What is the effect of eps have on Adam??