Hey,
I’m new to Pytorch. I’m trying to implement a compressed network where I switch off a neuron after every few epochs throughout training based on some threshold criteria(something similar to dropout). I’m not sure how to implement this in forward if that’s where masking of a neuron should be done. Thanks in advance.
I think this should be done during forward pass. Switching off means setting it’s value to zero?
If so, you would need to have a mask matrix which contains zeros and ones and you would have to multiply it to the outputs.
The only thing is, that you would have to update this mask by hand, as you cannot compute suitable gradients.
1 Like
Thank you so much 