Multiply Parameter by a parameter in computation graph

1

I have a fully connected layer, I want to multiply first half of weights by a number K that has itself to be learnt.
As in suppose the weight of fully connected network is W [mxn] and I have a another parameter K that is initialized to say 0.5.
I want to perform W[:m//2,:] = W[:m//2,:] * K and I need K to change as well
Is there some way to do this, will backpropogation work if I multiply two parameters???

2

Modifying parameters this way doesn’t work.
In my opinion the best way to do this is to create your own layer using W and K as parameters and using torch.nn.functional.linear in the forward.
for the multiplication, (provided m is even), I’d probably suggest

   mul = torch.ones(2, 1, 1, device=K.device)
   mul[0, 0, 0] = K
   weight = (W.view(2, m//2, -1) * mul).view(m, -1)

and then using weight. Backpropagation should work this way.
There is a somewhat elaborate way to replace parameters with calculated quantities, see e.g. the spectral norm implementation for inspiration. Unfortunately, there doesn’t seem general interest for a more generally applicable interface (I’ve tried to pitch my ideas at https://github.com/pytorch/pytorch/issues/7313 ).

Best regards

Thomas

1 Like
3

Oh thanks!
Functional approach seems cool I guess, bit not neat but good enough

Thanks

4

Hi, in your example use of mul is to convert parameter into tensor before multiplication right??

EDIT : seems it doesnt make a difference with or without it

5

Oops, the code example missed the top line because I left it on the triple backtick line :/.
The mul is target the first half only.