Calculate the grad when weight is not a parameter

In my model, the weight of conv is determined by another tensor(let’s call it weight_real) on the fly while running model, the actual parameter to be optimized is weight_real but not the weight itself.

So I create a new module that calls F.conv2d in forward. What I expected is the grad propagates first from the output to the weight, then from the weight to weight_real.

But I met a trouble that, if I set the weight tensor as a parameter, then the weight will be a leaf node, weight_real can’t get any grad, if I set the weight tensor as a normal tensor, the weight tensor itself can’t get any grad. It seems that only if weight is a parameter but not a normal tensor then its grad will be calculated.

Is there any way to solve this problem? :slight_smile: Thanks in advance!

Hi,

The only gradients you actually care about are for weight_real no?
In that case you can just make weight_real a Parameter of your module and create weight in a differentiable manner during the forward.

1 Like

Hi,

Many thanks. Surprisingly it works. However I’m still confused, how can weight_real get the correct grad when weight has no grad? (I set a breakpoint on optimizer.step() and the grad of weight is None)

Hi,

the .grad field is only populated for leaf tensors. That is a tensor created by the user where requires_grad=True is specified.
If you want the .grad value for an intermediate result to inspect it, you can call .retain_grad() on it during the forward pass.

1 Like

Thanks for your great help!