After randomly initializing the weight matrices of a GRU (let’s call them W) I need to transform them by means of some function. For the sake of this discussion, let’s simplify and say I want to multiply W by a scalar:
W <- alpha * W
where alpha is a scalar parameter that I want my optimization algorithm to optimize.
Is there a way for me to accomplish this without rewriting the whole GRU code from scratch?
What I have tried so far:
def __init__(self):
self.gru = ...
alpha = torch.Tensor([0.5])
self.alpha = nn.Parameter(alpha, requires_grad=True)
def forward(self, x):
self.gru.weight_hh_l0.data = self.alpha * self.gru.weight_hh_l0
... normal forward code ...
If I optimize all parameters, this doesn’t work. If I optimize all parameters except gru.weight_hh_l0, it still doesn’t work.