What could cause loss jumping up for CNN + RPN for 1 epoch?

I am training a CNN with RPN at the end, rather deep, but straightforward model. I am training with focal loss and saw such a peak on my training loss. How could I explain it? I mean, it happened around 80th epoch when the model was already rather stable. I am just conceptually wondering what could possible push the loss up for 1 epoch and then back to normal again? Haven’t seen such behaviour before, only much smaller jumps. The loss is an average for thousands of training examples, not just for one minibatch.

Opera Snapshot_2020-02-04_220933_localhost

The peak is actually very high (see below), but capped by 2 on the plot:

Epoch 84 mean loss 0.05488062464491334 LR: 0.001
Epoch 85 mean loss 0.05492022068288196 LR: 0.001
Epoch 86 mean loss 198953985.74916962 LR: 0.001
Epoch 87 mean loss 0.05357671019970312 LR: 0.001
Epoch 88 mean loss 0.05381619437844289 LR: 0.001

Could it be that one of the minibatches went to INF (buffer overflow?) and it is the result of averaging this over batch_count? Any thoughts or ideas appreciated.