ZeroDivisionError: float division by zero when applying "O2"

I’m trying to apply gradient compression with mixed-precision training (amp, mode = “O2” with dynamic loss scale), however I encounter with an error in the middle of training:

  File "/home/anaconda3/lib/python3.7/site-packages/ray/tune/", line 248, in run
  File "/home/anaconda3/lib/python3.7/site-packages/ray/tune/", line 316, in entrypoint
  File "/home/anaconda3/lib/python3.7/site-packages/ray/tune/", line 575, in _trainable_func
    output = fn()
  File "/home/", line 634, in train
    scaled_loss.backward() # calculate the gradients
  File "/home/anaconda3/lib/python3.7/", line 119, in __exit__
  File "/home/anaconda3/lib/python3.7/site-packages/apex/amp/", line 123, in scale_loss
  File "/home/anaconda3/lib/python3.7/site-packages/apex/amp/", line 190, in post_backward_with_master_weights
  File "/home/anaconda3/lib/python3.7/site-packages/apex/amp/", line 119, in unscale
    self.unscale_python(model_grads, master_grads, scale)
  File "/home/anaconda3/lib/python3.7/site-packages/apex/amp/", line 88, in unscale_python
ZeroDivisionError: float division by zero

I’m applying this to ResNet model (ResNet 50, for example), and all the hyperparameter values are reasonable. I’m not entirely sure how to fix this, will changing to static loss scale works? I was also wondering what’s the difference between dynamic loss scaling and static loss scaling, and whether that would lead to difference in run time or accuracy

We recommend to use the native mixed-precision training utility via torch.cuda.amp instead of apex/amp.
You can find some examples here.