How to set different learning rates for different part of the same tensor?

1

Is it possible to set different learning rates for different part of a tensor?

2

That’s not possible as you would be hitting the same limitations explained in your other topic.

3

So, the only way to have a different learning rates is to manually scale the gradients on the backward pass. Right?

4

No, you could also use different (sub-)tensors and stack/concatenate them in the actual forward pass as explained e.g. here.

1 Like