Left out gradient for some operations

autograd
1

Hi, I am a new user to pytorch. I used group lasso and wanted to zero out the small values of weight matrix after the backprop. But when I tried to do so, the gradient for the zero out operation kind of cancel out itself. I checked some posts in the pytorch forum and found out people would manually set the require gradient option to false.

So now the problem is, I want to zero out small values in the weight matrix, and I also want the weight matrix to update normally through backprop, so that means I can’t set the require gradient option to false, right? I just want to left out the gradient for the zero out operation. Is there a way to do that? Or if you have a better solution?

Here is the code for the zero out operation:

for param in model.parameters():
    cond = torch.abs(param.data) < threshold
    weight_ = param.data + 0
    weight_[cond] = 0
    param.data = weight_

Thanks in advance. Help is greatly appreciated.

2

What you need is to register_hook http://pytorch.org/docs/master/autograd.html#torch.autograd.Variable.register_hook. In your hook function, you can dynamically modify the gradients. :slight_smile:

3

Thanks a lot Simon, later I found out that the zeroout operation actually doesn’t compute gradient at all, so I just set both the data and gradient data of some component in the weight matrix to zero, and that seems to be working.

And now the question is, the code I am using might be too slow, is there like some built-in method I could use to increase the speed?

code:

(final is the loss)

Thanks again!!