I’m seeking suggestions w.r.t the best practice to monitor the “dead” neuron ratio during training process in pytorch. The goal is to probably set this as one of the early-stop signals so that I could abandon a certain model when I see that number goes up too high say 30% etc.
If you do weight decay, you can do this:
model.target_layer.weight.data.var(1) to get the variance of the output units’ weight.
If you don’t do weight decay, and have to check the unit activation, look up Module.register_forward_hook. This allows you to check the output of intermediate layers quite conveniently.