How to deal with sudden loss spike?

Model is tacotron2 gst.
Almost same to this https://github.com/mozilla/TTS/blob/master/models/tacotrongst.py
I left train at night and fond this
https://i.imgur.com/59vuPyf.png

I have 2 thoughts on this:
1 try to track what data is it, may be it is corrupted.
2 skip step and doesnt apply changes to model if loss spike.

Is it good ideas? If so, what is best way to do it? May be examples or someone already make lib about it?
Or should i do something else?