# Implement a meta-trainable step size

**URL:** <https://discuss.pytorch.org/t/implement-a-meta-trainable-step-size/70396>\
**Category:** Uncategorized\
**Created:** [February 20, 2020, 1:16am UTC](https://discuss.pytorch.org/t/implement-a-meta-trainable-step-size/70396 "2020-02-20T01:16:25Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![pinocchio](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pinocchio/32/22093_2.png) [@pinocchio](https://discuss.pytorch.org/u/pinocchio)\
**Post date:** [February 20, 2020, 1:16am UTC](https://discuss.pytorch.org/t/implement-a-meta-trainable-step-size/70396/1 "2020-02-20T01:16:25Z")

</div>

I want to implement a (meta) trainable step size. I tried it with this post:

> [@How does one have the parameters of a model NOT BE LEAFS?](https://discuss.pytorch.org/t/how-does-one-have-the-parameters-of-a-model-not-be-leafs/70076/17):
>
> thanks for that! I’ve been playing around with the library but was wondering if it was possible to have a trainable step-size with that library. Is it possible? I tried: # child\_model = nn.Sequential(OrderedDict([('conv1', nn.Conv2d(in\_channels=3,out\_channels=2,kernel\_size=5)), ('relu1', nn.ReLU()), ('Flatten', Flatten()), ('fc', nn.Linear(in\_features=28\*28\*2,out\_features=10) )])) eta = nn.Sequential(OrderedDict([ ('fc', nn.Linear(1,1)), ('sigmoid…

and with the higher library ([GitHub - facebookresearch/higher: higher is a pytorch library allowing users to obtain higher order gradients over losses spanning training loops rather than individual training steps.](https://github.com/facebookresearch/higher)) with no luck…

I tried:

```auto
eta = torch.tensor([0.5], requires_grad=True).view(1)
inner_opt = torch.optim.Adam(child_model.parameters(), lr=eta)
#meta_params = itertools.chain(child_model.parameters(),eta.parameters())
meta_params = itertools.chain(child_model.parameters())
meta_opt = torch.optim.Adam(meta_params, lr=1e-3)
# do meta-training/outer training minimize outerloop: min_{theta} sum_t L^val( theta^{T} - eta* Grad L^train(theta^{T}) ) 
nb_outer_steps = 10 # note, in this case it's the same as number of meta-train steps (but it's could not be the same depending how you loop through the val set)
for outer_i, (outer_inputs, outer_targets) in enumerate(testloader, 0):
    meta_opt.zero_grad()
    if outer_i >= nb_outer_steps:
        break
    # do inner-training/MAML; minimize innerloop: theta^{T} - eta* Grad L^train(theta^{T}) ~ argmin L^train(theta)
    nb_inner_steps = 3
    with higher.innerloop_ctx(child_model, inner_opt) as (fmodel, diffopt):

```

with error:

```auto
Exception has occurred: RuntimeError
Only Tensors created explicitly by the user (graph leaves) support the deepcopy protocol at the moment

```

which wouldn’t work anyway cuz eta might become negative suddenly so I really want to cap it with a sigmoid function but had to try something…

* * *

related:

SO: [machine learning - How does one implement a meta-trainable step size in Pytorch? - Stack Overflow](https://stackoverflow.com/questions/60311609/how-does-one-implement-a-meta-trainable-step-size-in-pytorch)