How to monitor the "dead" neuron during training process?

1

Hello,

I’m seeking suggestions w.r.t the best practice to monitor the “dead” neuron ratio during training process in pytorch. The goal is to probably set this as one of the early-stop signals so that I could abandon a certain model when I see that number goes up too high say 30% etc.

Thanks,

2

If you do weight decay, you can do this: model.target_layer.weight.data.var(1) to get the variance of the output units’ weight.

If you don’t do weight decay, and have to check the unit activation, look up Module.register_forward_hook. This allows you to check the output of intermediate layers quite conveniently.

3 Likes