Why should paramters for modules be a 'leaf tensor'?

ildoonet · January 2, 2019, 12:31pm

pytorch/pytorch/blob/aec9fdf0a41fc304e1bd5424b9baf83101834083/torch/nn/modules/module.py#L153


    elif hasattr(self, name) and name not in self._parameters:
        raise KeyError("attribute '{}' already exists".format(name))


    if param is None:
        self._parameters[name] = None
    elif not isinstance(param, Parameter):
        raise TypeError("cannot assign '{}' object to parameter '{}' "
                        "(torch.nn.Parameter or None required)"
                        .format(torch.typename(param), name))
    elif param.grad_fn:
        raise ValueError(
            "Cannot assign non-leaf Tensor to parameter '{0}'. Model "
            "parameters must be created explicitly. To express '{0}' "
            "as a function of another Tensor, compute the value in "
            "the forward() method.".format(name))
    else:
        self._parameters[name] = param


def add_module(self, name, module):
    r"""Adds a child module to the current module.

I encountered this error message in the above line, while developing MAML-like architecture which requires to calculate high order derivatives of accumulated gradient of parameters.

I wanted to set new parameters with grad_fn, but as shown in the above line of codes, pytorch requires parameters of modules should be leaf tensors.

Is there any reason for this? Or can I ignore this error message(eg. delete the line)?

As far as I can find, high-order autograd works for every Functions in pytorch(eg. convNd, linear, …) so I guess it is not a problem when I change network parameters with the non-leaf ones.

tom · January 2, 2019, 11:55pm

You can’t do that currently in PyTorch. It’s a bit of an administrative problem more than a fundamental one (albeit not an entirely trivial one, I once implemented a PoC for this).
Currently, the functional interfaces and computing things in forward is what you have to get along with.

Best regards

Thomas