# How does one have the parameters of a model NOT BE LEAFS?

**URL:** <https://discuss.pytorch.org/t/how-does-one-have-the-parameters-of-a-model-not-be-leafs/70076>\
**Category:** Uncategorized\
**Created:** [February 17, 2020, 5:24pm UTC](https://discuss.pytorch.org/t/how-does-one-have-the-parameters-of-a-model-not-be-leafs/70076 "2020-02-17T17:24:42Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![pinocchio](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pinocchio/32/22093_2.png) [@pinocchio](https://discuss.pytorch.org/u/pinocchio)\
**Post date:** [February 17, 2020, 5:24pm UTC](https://discuss.pytorch.org/t/how-does-one-have-the-parameters-of-a-model-not-be-leafs/70076/1 "2020-02-17T17:24:42Z")

</div>

I want the parameters of the model to NOT be leafs. I also want the parameters to be part of the computation graph and I am dynamically changing the parameters with code. So I want:

- Parameters to NOT be leafs (but still be held inside a `nn.Module` so that I can use the dynamically changing parameters as part of the computation graph).
- I want the “updating model” to be part of the computation graph at any time but it seems setting the parameters to leaf is stopping the flow of gradients

How can I do that?

(plan to post later a working self-contained simplified jupyter demo of what more or less is going to make it easier to help me)

* * *

SO: [https://stackoverflow.com/questions/60271131/how-does-one-have-parameters-in-a-pytorch-model-not-be-leafs-and-be-in-the-compu](https://stackoverflow.com/questions/60271131/how-does-one-have-parameters-in-a-pytorch-model-not-be-leafs-and-be-in-the-compu)

---

<div class="post-metadata">

**Author:** ![albanD](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/32/215_2.png) [@albanD](https://discuss.pytorch.org/u/albanD)\
**Post date:** [February 17, 2020, 9:40pm UTC](https://discuss.pytorch.org/t/how-does-one-have-the-parameters-of-a-model-not-be-leafs/70076/2 "2020-02-17T21:40:14Z")

</div>

Hi,

The general trick to make sure they are not leafs anymore is to first delete de field containing an nn.Parameter. Then you can set it with a Tensor that is not a leaf.

---

<div class="post-metadata">

**Author:** ![pinocchio](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pinocchio/32/22093_2.png) [@pinocchio](https://discuss.pytorch.org/u/pinocchio)\
**Post date:** [February 17, 2020, 9:50pm UTC](https://discuss.pytorch.org/t/how-does-one-have-the-parameters-of-a-model-not-be-leafs/70076/3 "2020-02-17T21:50:23Z")

</div>

tried what you suggested but it didn’t work…

```auto
        del loss_net.fc0.weight
        #setattr(loss_net.fc0,'weight', wt)
        loss_net.fc0.weight = wt

```

what did you have in mind?

* * *

Error msg:

```auto
TypeError: cannot assign 'torch.FloatTensor' as parameter 'weight' (torch.nn.Parameter or None expected)

```

---

<div class="post-metadata">

**Author:** ![pinocchio](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pinocchio/32/22093_2.png) [@pinocchio](https://discuss.pytorch.org/u/pinocchio)\
**Post date:** [February 17, 2020, 9:54pm UTC](https://discuss.pytorch.org/t/how-does-one-have-the-parameters-of-a-model-not-be-leafs/70076/4 "2020-02-17T21:54:05Z")

</div>

variations that don’t work:

```auto
        #del loss_net.fc0.weight
        #setattr(loss_net.fc0, 'weight', nn.Parameter( wt ))
        #setattr(loss_net.fc0, 'weight', wt)
        #loss_net.fc0.weight = wt
        #loss_net.fc0.weight = nn.Parameter( wt )

```

---

<div class="post-metadata">

**Author:** ![albanD](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/32/215_2.png) [@albanD](https://discuss.pytorch.org/u/albanD)\
**Post date:** [February 17, 2020, 10:06pm UTC](https://discuss.pytorch.org/t/how-does-one-have-the-parameters-of-a-model-not-be-leafs/70076/5 "2020-02-17T22:06:58Z")

</div>

Hi,

This does work:

```python
m = nn.Linear(10, 10)
del m.weight
m.weight = torch.rand(10)

```

If you try to print the weight after the del, you should get that you module has no attribute ‘weight’, is that what you get?

---

<div class="post-metadata">

**Author:** ![pinocchio](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pinocchio/32/22093_2.png) [@pinocchio](https://discuss.pytorch.org/u/pinocchio)\
**Post date:** [February 17, 2020, 10:10pm UTC](https://discuss.pytorch.org/t/how-does-one-have-the-parameters-of-a-model-not-be-leafs/70076/6 "2020-02-17T22:10:16Z")

</div>

> [@albanD](#):
>
> If you try to print the weight after the del, you should get that you module has no attribute ‘weight’, is that what you get?

yes, I get that error msg:

```auto
AttributeError: 'Linear' object has no attribute 'weight'

```

---

<div class="post-metadata">

**Author:** ![pinocchio](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pinocchio/32/22093_2.png) [@pinocchio](https://discuss.pytorch.org/u/pinocchio)\
**Post date:** [February 17, 2020, 10:11pm UTC](https://discuss.pytorch.org/t/how-does-one-have-the-parameters-of-a-model-not-be-leafs/70076/7 "2020-02-17T22:11:08Z")

</div>

But I get another error now:

```auto
RuntimeError: OrderedDict mutated during iteration

```

I am looping over all the parameters of my (potentially arbitrary) model. So, I guess python gets unhappy that I’ve mutated it…

---

<div class="post-metadata">

**Author:** ![pinocchio](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pinocchio/32/22093_2.png) [@pinocchio](https://discuss.pytorch.org/u/pinocchio)\
**Post date:** [February 17, 2020, 10:12pm UTC](https://discuss.pytorch.org/t/how-does-one-have-the-parameters-of-a-model-not-be-leafs/70076/8 "2020-02-17T22:12:02Z")

</div>

@albanD

This is the self contained dummy example/prototype I built to debug this issue:

```auto
import torch
import torch.nn as nn

import copy

from collections import OrderedDict

# img = torch.randn([8,3,32,32])
# targets = torch.LongTensor([1, 2, 0, 6, 2, 9, 4, 9])
# img = torch.randn([1,3,32,32])
# targets = torch.LongTensor([1])
x = torch.randn(1)
target = 12.0*x**2

criterion = nn.CrossEntropyLoss()

#loss_net = nn.Sequential(OrderedDict([('conv0',nn.Conv2d(in_channels=3,out_channels=10,kernel_size=32))]))
loss_net = nn.Sequential(OrderedDict([('fc0', nn.Linear(in_features=1,out_features=1))]))

hidden = torch.randn(size=(1,1),requires_grad=True)
updater_net = nn.Sequential(OrderedDict([('fc0',nn.Linear(in_features=1,out_features=1))]))
print(f'updater_net.fc0.weight.is_leaf = {updater_net.fc0.weight.is_leaf}')
#
nb_updates = 2
for i in range(nb_updates):
    print(f'i = {i}')
    new_params = copy.deepcopy( loss_net.state_dict() )
    ## w^<t> := f(w^<t-1>,delta^<t-1>)
    for (name, w) in loss_net.named_parameters():
        print(f'name = {name}')
        print(w.size())
        hidden = updater_net(hidden).view(1)
        print(hidden.size())
        #delta = ((hidden**2)*w/2)
        delta = w + hidden
        wt = w + delta
        print(wt.size())
        new_params[name] = wt
        #del loss_net.fc0.weight
        #print(loss_net.fc0.weight)
        #setattr(loss_net.fc0, 'weight', nn.Parameter( wt ))
        #setattr(loss_net.fc0, 'weight', wt)
        #loss_net.fc0.weight = torch.randn(1)
        #loss_net.fc0.weight = nn.Parameter( wt )
    ##
    loss_net.load_state_dict(new_params)
#
print()
print(f'updater_net.fc0.weight.is_leaf = {updater_net.fc0.weight.is_leaf}')
outputs = loss_net(x)
loss_val = 0.5*(target - outputs)**2
loss_val.backward()
print()
print(f'-- params that dont matter if they have gradients --')
print(f'loss_net.grad = {loss_net.fc0.weight.grad}')
print('-- params we want to have gradients --')
print(f'hidden.grad = {hidden.grad}')
print(f'updater_net.fc0.weight.grad = {updater_net.fc0.weight.grad}')
print(f'updater_net.fc0.bias.grad = {updater_net.fc0.bias.grad}')

```

Hope it helps us…

its current (incorrect) output:

```auto
updater_net.fc0.weight.is_leaf = True
i = 0
name = fc0.weight
torch.Size([1, 1])
torch.Size([1])
torch.Size([1, 1])
name = fc0.bias
torch.Size([1])
torch.Size([1])
torch.Size([1])
i = 1
name = fc0.weight
torch.Size([1, 1])
torch.Size([1])
torch.Size([1, 1])
name = fc0.bias
torch.Size([1])
torch.Size([1])
torch.Size([1])

updater_net.fc0.weight.is_leaf = True

-- params that dont matter if they have gradients --
loss_net.grad = tensor([[9.6197]])
-- params we want to have gradients --
hidden.grad = None
updater_net.fc0.weight.grad = None
updater_net.fc0.bias.grad = None

```

---

<div class="post-metadata">

**Author:** ![albanD](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/32/215_2.png) [@albanD](https://discuss.pytorch.org/u/albanD)\
**Post date:** [February 18, 2020, 4:24pm UTC](https://discuss.pytorch.org/t/how-does-one-have-the-parameters-of-a-model-not-be-leafs/70076/9 "2020-02-18T16:24:00Z")

</div>

Hi,

`load_state_dict()` load the weights in a non-differentiable manner. So you end up with leafs as expected.

Here is a version that works as expected:

```python

def del_attr(obj, names):
    if len(names) == 1:
        delattr(obj, names[0])
    else:
        del_attr(getattr(obj, names[0]), names[1:])
def set_attr(obj, names, val):
    if len(names) == 1:
        setattr(obj, names[0], val)
    else:
        set_attr(getattr(obj, names[0]), names[1:], val)

nb_updates = 2
for i in range(nb_updates):
    print(f'i = {i}')
    new_params = copy.deepcopy( loss_net.state_dict() )
    ## w^<t> := f(w^<t-1>,delta^<t-1>)
    for (name, w) in list(loss_net.named_parameters()):
        hidden = updater_net(hidden).view(1)
        #delta = ((hidden**2)*w/2)
        delta = w + hidden
        wt = w + delta
        del_attr(loss_net, name.split("."))
        set_attr(loss_net, name.split("."), wt)
    ##
#
print()
print(f'updater_net.fc0.weight.is_leaf = {updater_net.fc0.weight.is_leaf}')
print(f'loss_net.fc0.weight.is_leaf = {loss_net.fc0.weight.is_leaf}')
outputs = loss_net(x)
loss_val = 0.5*(target - outputs)**2
loss_val.backward()
print()
print(f'-- params that dont matter if they have gradients --')
print(f'loss_net.grad = {loss_net.fc0.weight.grad}')
print('-- params we want to have gradients --')
print(f'hidden.grad = {hidden.grad}') # None because this is not a leaf, it is overriden in the for loop above.
print(f'updater_net.fc0.weight.grad = {updater_net.fc0.weight.grad}')
print(f'updater_net.fc0.bias.grad = {updater_net.fc0.bias.grad}')

```

---

<div class="post-metadata">

**Author:** ![pinocchio](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pinocchio/32/22093_2.png) [@pinocchio](https://discuss.pytorch.org/u/pinocchio)\
**Post date:** [February 18, 2020, 4:30pm UTC](https://discuss.pytorch.org/t/how-does-one-have-the-parameters-of-a-model-not-be-leafs/70076/10 "2020-02-18T16:30:42Z")

</div>

> [@albanD](#):
>
> def del\_attr(obj, names): if len(names) == 1: delattr(obj, names[0]) else: del\_attr(getattr(obj, names[0]), names[1:]) def set\_attr(obj, names, val): if len(names) == 1: setattr(obj, names[0], val) else: set\_attr(getattr(obj, names[0]), names[1:], val)

@albanD oh wow! The grads are being printed for sure now! Who also but the albanD boss to come to rescue…

Sorry for being demanding, but is it ok if you comment or explain what you did that made things work? Especially this code:

```auto
def del_attr(obj, names):
    if len(names) == 1:
        delattr(obj, names[0])
    else:
        del_attr(getattr(obj, names[0]), names[1:])
def set_attr(obj, names, val):
    if len(names) == 1:
        setattr(obj, names[0], val)
    else:
        set_attr(getattr(obj, names[0]), names[1:], val)

```

Thanks so much again…your a saviour… 💪 🙂

---

<div class="post-metadata">

**Author:** ![pinocchio](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pinocchio/32/22093_2.png) [@pinocchio](https://discuss.pytorch.org/u/pinocchio)\
**Post date:** [February 18, 2020, 4:31pm UTC](https://discuss.pytorch.org/t/how-does-one-have-the-parameters-of-a-model-not-be-leafs/70076/11 "2020-02-18T16:31:19Z")

</div>

self contained script that seems to work:

```auto
import torch
import torch.nn as nn

from torchviz import make_dot

import copy

from collections import OrderedDict

# img = torch.randn([8,3,32,32])
# targets = torch.LongTensor([1, 2, 0, 6, 2, 9, 4, 9])
# img = torch.randn([1,3,32,32])
# targets = torch.LongTensor([1])
x = torch.randn(1)
target = 12.0*x**2

criterion = nn.CrossEntropyLoss()

#loss_net = nn.Sequential(OrderedDict([('conv0',nn.Conv2d(in_channels=3,out_channels=10,kernel_size=32))]))
loss_net = nn.Sequential(OrderedDict([('fc0', nn.Linear(in_features=1,out_features=1))]))

hidden = torch.randn(size=(1,1),requires_grad=True)
updater_net = nn.Sequential(OrderedDict([('fc0',nn.Linear(in_features=1,out_features=1))]))
print(f'updater_net.fc0.weight.is_leaf = {updater_net.fc0.weight.is_leaf}')
#
def del_attr(obj, names):
    if len(names) == 1:
        delattr(obj, names[0])
    else:
        del_attr(getattr(obj, names[0]), names[1:])
def set_attr(obj, names, val):
    if len(names) == 1:
        setattr(obj, names[0], val)
    else:
        set_attr(getattr(obj, names[0]), names[1:], val)

nb_updates = 2
for i in range(nb_updates):
    print(f'i = {i}')
    new_params = copy.deepcopy( loss_net.state_dict() )
    ## w^<t> := f(w^<t-1>,delta^<t-1>)
    for (name, w) in list(loss_net.named_parameters()):
        hidden = updater_net(hidden).view(1)
        #delta = ((hidden**2)*w/2)
        delta = w + hidden
        wt = w + delta
        del_attr(loss_net, name.split("."))
        set_attr(loss_net, name.split("."), wt)
    ##
#
print()
print(f'updater_net.fc0.weight.is_leaf = {updater_net.fc0.weight.is_leaf}')
print(f'loss_net.fc0.weight.is_leaf = {loss_net.fc0.weight.is_leaf}')
outputs = loss_net(x)
loss_val = 0.5*(target - outputs)**2
loss_val.backward()
print()
print(f'-- params that dont matter if they have gradients --')
print(f'loss_net.grad = {loss_net.fc0.weight.grad}')
print('-- params we want to have gradients --')
print(f'hidden.grad = {hidden.grad}') # None because this is not a leaf, it is overriden in the for loop above.
print(f'updater_net.fc0.weight.grad = {updater_net.fc0.weight.grad}')
print(f'updater_net.fc0.bias.grad = {updater_net.fc0.bias.grad}')
make_dot(loss_val)

```

output:

```auto
updater_net.fc0.weight.is_leaf = True
i = 0
i = 1

updater_net.fc0.weight.is_leaf = True
loss_net.fc0.weight.is_leaf = False

-- params that dont matter if they have gradients --
loss_net.grad = None
-- params we want to have gradients --
hidden.grad = None
updater_net.fc0.weight.grad = tensor([[0.7152]])
updater_net.fc0.bias.grad = tensor([-7.4249])

```

---

<div class="post-metadata">

**Author:** ![albanD](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/32/215_2.png) [@albanD](https://discuss.pytorch.org/u/albanD)\
**Post date:** [February 18, 2020, 4:43pm UTC](https://discuss.pytorch.org/t/how-does-one-have-the-parameters-of-a-model-not-be-leafs/70076/12 "2020-02-18T16:43:17Z")

</div>

Since you have arbitrary name, but you need to be able to delete and set attributes (to delete the nn.Parameter and set a Tensor instead). So this is a recursive function that given a list of all the names like [“foo”, “bar”, “weight”] will either set or delete obj.foo.bar.weight.

---

<div class="post-metadata">

**Author:** ![pinocchio](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pinocchio/32/22093_2.png) [@pinocchio](https://discuss.pytorch.org/u/pinocchio)\
**Post date:** [February 18, 2020, 5:32pm UTC](https://discuss.pytorch.org/t/how-does-one-have-the-parameters-of-a-model-not-be-leafs/70076/13 "2020-02-18T17:32:03Z")

</div>

AlbanD, this does quite work. I wanted to make sure it worked by printing the computation graph and increasing the `nb_updates` should increase the computation graph but it doesn’t. I believe it has something to do with the fact we are changing/mutating the OrderDict/params **and** looping through it at the same time. I tried to fix it by collecting the parameters before and after running the loop but the somehow it thinks its empty for the second loop (i.e. `t=1` doesn’t print anything when it should):

```auto
import torch
import torch.nn as nn

from torchviz import make_dot

import copy

from collections import OrderedDict

def del_attr(obj, names):
    if len(names) == 1:
        delattr(obj, names[0])
    else:
        del_attr(getattr(obj, names[0]), names[1:])
        
def set_attr(obj, names, val):
    if len(names) == 1:
        setattr(obj, names[0], val)
    else:
        set_attr(getattr(obj, names[0]), names[1:], val)

        
x = torch.randn(1, requires_grad=True)
y = torch.randn(1, requires_grad=True)

criterion = nn.CrossEntropyLoss()

loss_net = nn.Sequential(OrderedDict([('l_fc0', nn.Linear(in_features=1,out_features=1, bias=True))]))
loss_net.l_fc0.weight.requires_grad=False
loss_net.l_fc0.bias.requires_grad=False

hidden = torch.randn(size=(1,1),requires_grad=True)
updater_net = nn.Sequential(OrderedDict([('u_fc0',nn.Linear(in_features=1,out_features=1))]))
updater_net.u_fc0.bias.requires_grad = False
print(f'updater_net.u_fc0.weight.is_leaf = {updater_net.u_fc0.weight.is_leaf}')
#
outputs_virgin = loss_net(x)
params = dict(dict(loss_net.named_parameters()),**{'x':x})
make_dot(outputs_virgin, params=params).render('loss_net_x', format='png')
#
nb_updates = 2
params = list(loss_net.named_parameters())
for t in range(nb_updates):
    print(f't = {t}')
    ## w^<t> := f(w^<t-1>,delta^<t-1>)
    for (name, w) in params:
        delta = updater_net(hidden).view(1)
        wt = w + delta
        print(f'w^<{t}> = {wt}')
        del_attr(loss_net, name.split("."))
        set_attr(loss_net, name.split("."), wt)
    params = list(loss_net.named_parameters())

print()
print(f'updater_net.u_fc0.weight.is_leaf = {updater_net.u_fc0.weight.is_leaf}')
print(f'loss_net.l_fc0.weight.is_leaf = {loss_net.l_fc0.weight.is_leaf}')

outputs = loss_net(x)
loss_val = (outputs - y)**2
loss_val.backward()
print()
print(f'-- params that dont matter if they have gradients --')
print(f'loss_net.grad = {loss_net.l_fc0.weight.grad}')
print('-- params we want to have gradients --')
print(f'hidden.grad = {hidden.grad}') # None because this is not a leaf, it is overriden in the for loop above.
print(f'updater_net.u_fc0.weight.grad = {updater_net.u_fc0.weight.grad}')
#print(f'updater_net.u_fc0.bias.grad = {updater_net.u_fc0.bias.grad}')
params = dict(dict(updater_net.named_parameters()), **{'x':x,'y':y,'hidden':hidden},** dict(loss_net.named_parameters()))
make_dot(loss_val, params=params).render('loss_val', format='png')

```

---

<div class="post-metadata">

**Author:** ![albanD](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/32/215_2.png) [@albanD](https://discuss.pytorch.org/u/albanD)\
**Post date:** [February 18, 2020, 6:06pm UTC](https://discuss.pytorch.org/t/how-does-one-have-the-parameters-of-a-model-not-be-leafs/70076/14 "2020-02-18T18:06:15Z")

</div>

Well you should not use `loss_net.named_parameters()` anymore. we override them. So you should “save” them before doing the first override. Like `params = list(loss_net.named_parameters())` before the for-loop.  
Then you can set them back (as leafs) when you want to make a new iteration. So maybe extract them as a state\_dict and restore them with load\_state\_dict.

---

<div class="post-metadata">

**Author:** ![pinocchio](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pinocchio/32/22093_2.png) [@pinocchio](https://discuss.pytorch.org/u/pinocchio)\
**Post date:** [February 18, 2020, 7:32pm UTC](https://discuss.pytorch.org/t/how-does-one-have-the-parameters-of-a-model-not-be-leafs/70076/15 "2020-02-18T19:32:09Z")

</div>

> [@albanD](#):
>
> Then you can set them back (as leafs) when you want to make a new iteration. So maybe extract them as a state\_dict and restore them with load\_state\_dict.

I don’t think we should do that, if we do they would be used as non-leafs in the next iteration and won’t appear in the computation graph properly. It should be a chain of caused by the iterative use of `wt`.

Let me think about it…

I also thought we could have some special string like the word `param` in the field when we set it and thus we could loop through all the fields of the object that contain that special string.

---

<div class="post-metadata">

**Author:** ![albanD](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/32/215_2.png) [@albanD](https://discuss.pytorch.org/u/albanD)\
**Post date:** [February 18, 2020, 7:37pm UTC](https://discuss.pytorch.org/t/how-does-one-have-the-parameters-of-a-model-not-be-leafs/70076/16 "2020-02-18T19:37:31Z")

</div>

Note that if the goal is to do learning through your optimizer step, the [higher](https://github.com/facebookresearch/higher) library already implements this. And has a nice API for for all that 🙂

---

<div class="post-metadata">

**Author:** ![pinocchio](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pinocchio/32/22093_2.png) [@pinocchio](https://discuss.pytorch.org/u/pinocchio)\
**Post date:** [February 20, 2020, 12:57am UTC](https://discuss.pytorch.org/t/how-does-one-have-the-parameters-of-a-model-not-be-leafs/70076/17 "2020-02-20T00:57:53Z")

</div>

thanks for that!

I’ve been playing around with the library but was wondering if it was possible to have a trainable step-size with that library. Is it possible?

I tried:

```auto
#
child_model = nn.Sequential(OrderedDict([
        ('conv1', nn.Conv2d(in_channels=3,out_channels=2,kernel_size=5)),
        ('relu1', nn.ReLU()),
        ('Flatten', Flatten()),
        ('fc', nn.Linear(in_features=28*28*2,out_features=10) )
    ]))
eta = nn.Sequential(OrderedDict([
    ('fc', nn.Linear(1,1)),
    ('sigmoid', nn.Sigmoid())
]))
inner_opt = torch.optim.Adam(child_model.parameters(), lr=eta)
meta_params = itertools.chain(child_model.parameters(),eta.parameters())
meta_opt = torch.optim.Adam(meta_params, lr=1e-3)

```

but it failed with error:

```auto
Exception has occurred: TypeError
'<=' not supported between instances of 'float' and 'Sequential'

```

---

<div class="post-metadata">

**Author:** ![pinocchio](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pinocchio/32/22093_2.png) [@pinocchio](https://discuss.pytorch.org/u/pinocchio)\
**Post date:** [February 20, 2020, 1:09am UTC](https://discuss.pytorch.org/t/how-does-one-have-the-parameters-of-a-model-not-be-leafs/70076/18 "2020-02-20T01:09:16Z")

</div>

Or even better the update rule be some sort of NN…

---

<div class="post-metadata">

**Author:** ![pinocchio](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pinocchio/32/22093_2.png) [@pinocchio](https://discuss.pytorch.org/u/pinocchio)\
**Post date:** [February 20, 2020, 1:21am UTC](https://discuss.pytorch.org/t/how-does-one-have-the-parameters-of-a-model-not-be-leafs/70076/19 "2020-02-20T01:21:17Z")

</div>

starting to think that going back to your settatrr approach might be better X’D…

> [@Implement a meta-trainable step size](https://discuss.pytorch.org/t/implement-a-meta-trainable-step-size/70396):
>
> I want to implement a (meta) trainable step size. I tried it with this post: [https://discuss.pytorch.org/t/how-does-one-have-the-parameters-of-a-model-not-be-leafs/70076/17](https://discuss.pytorch.org/t/how-does-one-have-the-parameters-of-a-model-not-be-leafs/70076/17) and with the higher library ([https://github.com/facebookresearch/higher](https://github.com/facebookresearch/higher)) with no luck… I tried: eta = torch.tensor([0.5], requires\_grad=True).view(1) inner\_opt = torch.optim.Adam(child\_model.parameters(), lr=eta) #meta\_params = itertools.chain(child\_model.parameters(),eta.parameters()) meta\_params = itertools.chain(child\_…

---

<div class="post-metadata">

**Author:** ![albanD](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/32/215_2.png) [@albanD](https://discuss.pytorch.org/u/albanD)\
**Post date:** [February 20, 2020, 3:15pm UTC](https://discuss.pytorch.org/t/how-does-one-have-the-parameters-of-a-model-not-be-leafs/70076/20 "2020-02-20T15:15:34Z")

</div>

You can simply replace the optimizer by a nn ?  
Or fill-in the `.grad` fiels with your nn’s result and then use a simple SGD to do the step.

[Next page](https://discuss.pytorch.org/t/how-does-one-have-the-parameters-of-a-model-not-be-leafs/70076.md?page=2)