I am implementing custom RNN. It sounds weird but it should have separate hidden states for all inputs. I will use it inside of graph convolution. As an input, it takes edge weights one by one. I am sharing the restricted case where I have only 3 inputs as an illustration.
class RNN(nn.Module):
def __init__(self):
super(RNN,self).__init__()
self.weight = Parameter(torch.rand(3,1, requires_grad=True))
self.weight_h = Parameter(torch.rand(3,1, requires_grad=True))
self.hidden = Parameter(torch.rand(3,1,requires_grad=True))
self.bias = Parameter(torch.rand(3,1, requires_grad=True))
self.tanh = Tanh()
self.iteration = 0
def forward(self, x):
self.hidden[self.iteration] = self.tanh(x*self.weight[self.iteration] + self.hidden[self.iteration].clone()*self.weight_h[self.iteration] + self.bias[self.iteration])
self.iteration = self.iteration + 1
return self.hidden[self.iteration-1]
model = RNN()
mael = nn.L1Loss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
x = torch.rand(3,1)
for i in range(2):
optimizer.zero_grad()
out = model(x[i])
loss = mael(out, x[i+1])
loss.backward()
optimizer.step()
print(loss.item())
However, the code gives these error messages. Probably, the hidden state is changed in-place operation. I could not find a way to update a hidden state.
RuntimeError Traceback (most recent call last)
<ipython-input-11-24c2beaf081c> in <module>()
7 out = model(x[i])
8 loss = mael(out, x[i+1])
----> 9 loss.backward()
10 optimizer.step()
11 print(loss.item())
/usr/local/lib/python3.6/dist-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)
130 Variable._execution_engine.run_backward(
131 tensors, grad_tensors_, retain_graph, create_graph,
--> 132 allow_unreachable=True) # allow_unreachable flag
133
134
RuntimeError: leaf variable has been moved into the graph interior
Tanks in advance.