Is there any way the optimizer can figure out he is dealing with head layers?

1

Hi all,

I want a custom optimizer to react to the last layers of a graph in a specific way. Usually, the last group in a param_group is the last, but many networks have multiple output layers. Is there any way to know at the optimization step of the optimizer which ones are output layers?

Thanks in advance.

2

If you have named the output layers, maybe you can segregate the layers based on that while doing
the back-propagation yourself.

for name, param in model.named_parameters():
    print name, param.data
3

Hi

All optimizers accept param_groups as argument to construct them.
So you can require the user to provide parameters with two groups called body and ouput in your optimizer init.
That way you know exactly which is which.

4

I was actually looking if there was any way to do that generically, but @albanD approach is a reasonable requirement.

5

Unfortunately, in pytorch, a user can define its network in any way it wants. So I don’t think there is a generic way to do this :confused: