How to monitor the "dead" neuron during training process?


I’m seeking suggestions w.r.t the best practice to monitor the “dead” neuron ratio during training process in pytorch. The goal is to probably set this as one of the early-stop signals so that I could abandon a certain model when I see that number goes up too high say 30% etc.


If you do weight decay, you can do this: to get the variance of the output units’ weight.

If you don’t do weight decay, and have to check the unit activation, look up Module.register_forward_hook. This allows you to check the output of intermediate layers quite conveniently.