Specify parameter sharing pattern in nn.Parameter

[KFrank]

Hi Cai!

@Chen-Cai-OSU

What I want is to fix the lower part to be 0 all the time. I think I have to unregister those numbers but I didn’t find a clean way to do so.

An entire tensor is a single Parameter. You can’t unregister just some
elements of a tensor.

Your best bet will be to “freeze” the lower triangle at zero by zeroing out
that part of the tensor’s gradient. Here’s a post from @ptrblck that shows
how to do this:

Creating a Triangular weight matrix with a hidden Layer instead of a fully connected one

You could initialize the weight matrices in your desired shape, e.g. by using torch.triu, and then zeroing out the gradients at the same locations. Here is a small code snippet for this use case, which could be a good starter: # Init def triu_init(m): if isinstance(m, nn.Linear): with torch.no_grad(): m.weight.copy_(torch.triu(m.weight)) # Zero out gradients def get_zero_grad_hook(mask): def hook(grad): return grad * mask return hook # Setup model = nn…

Best.

K. Frank