Just to confirm, for sharing weights, can directly re-use the same Variable?
Seems like this is the case empirically:
W_init = torch.rand(3, 2)
W = autograd.Variable(W_init.clone(), requires_grad=True)
x = autograd.Variable(torch.rand(1, 3))
h1 = x @ W
h2 = h1 @ W.transpose(0, 1)
grad_out = torch.rand(1, 3)
h2.backward(grad_out)
print('W.grad', W.grad)
W1 = autograd.Variable(W_init.clone(), requires_grad=True)
W2 = autograd.Variable(W_init.clone(), requires_grad=True)
h1 = x @ W1
h2 = h1 @ W2.transpose(0, 1)
h2.backward(grad_out)
print('W1.grad + W2.grad', W1.grad + W2.grad)
=> output shows that W1.grad + W2.grad is identical to W.grad, even though W was re-used directly, without doing anything special etc => seems like empierically supports the idea that we can just directly re-use Variables if we want, for sharing the weights?