Hi. I am trying to define a module that accepts a tensor of parameters but allowing to control the slice of this tensor that will actually be learned, for example:
class MyModule(nn.Module): def __init__(self, m, learn_slice): super().__init__() self.m = m self.m_params = nn.Parameter(m[learn_slice]) self.m[learn_slice] = self.m_params def forward(self, x): return torch.dot(self.m, x)
When I evaluate the gradient for the first time it works well, but after the second
backward() call I get the following error:
m = torch.rand(100) x = torch.rand(100) model = MyModule(m, slice(10, 20)) y = model(x) y.backward() y = model(x) y.backward()
RuntimeError: Trying to backward through the graph a second time, but the saved intermediate results have already been freed. Specify retain_graph=True when calling backward the first time.
If I always add
backward() it does work, but I am not sure whether it is the right way of doing this.
What am I missing here? Thanks!