# Different samples use different learning rates

**URL:** <https://discuss.pytorch.org/t/different-samples-use-different-learning-rates/58234>\
**Category:** autograd\
**Created:** [October 15, 2019, 2:01am UTC](https://discuss.pytorch.org/t/different-samples-use-different-learning-rates/58234 "2019-10-15T02:01:23Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![shirui-japina](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shirui-japina/32/15285_2.png) [@shirui-japina](https://discuss.pytorch.org/u/shirui-japina)\
**Post date:** [October 15, 2019, 2:01am UTC](https://discuss.pytorch.org/t/different-samples-use-different-learning-rates/58234/1 "2019-10-15T02:01:23Z")

</div>

I want to use different learning rates for different input samples to carry out model training. Any good suggestions?🤔

---

<div class="post-metadata">

**Author:** ![shirui-japina](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shirui-japina/32/15285_2.png) [@shirui-japina](https://discuss.pytorch.org/u/shirui-japina)\
**Post date:** [October 15, 2019, 3:00am UTC](https://discuss.pytorch.org/t/different-samples-use-different-learning-rates/58234/2 "2019-10-15T03:00:58Z")

</div>

The way I think of is:

_Assuming using `nn.BCELoss()` as loss function._  
As [here](https://pytorch.org/docs/master/nn.functional.html#loss-functions) said,

> `'none'` : no reduction will be applied

So I can get the loss value of each sample by:  
`loss_reduction_none = nn.BCELoss(reduction='none')`

Then I can adjust the loss value of each sample like:

```auto
loss_reduction_none[1] = criterion_reduction_none[1] * scaling_factor_1,
loss_reduction_none[0] = criterion_reduction_none[2] * scaling_factor_2,
...
loss_reduction_none[n] = criterion_reduction_none[n] * scaling_factor_n,

```

Because the _ **learning rate** _ works in this way.

> w\_new = w\_old - _ **learning\_rate** _ \* (∂ loss / ∂ weight)

As I mentioned above, I adjust the loss function value of each sample.  
That indirectly solves the problem of learning rate adjustment for each sample.

Then I can use backward and step function:  
`loss_reduction_none.backward()`  
`optimizer.step()`

**But I am not sure if PyTorch will use the loss function value which I have modified.**  
**Actually, I don’t know the parameter _reduction_ is designed for what kind of demand.**

Is this the right answer?🤔

Maybe this question is something like below:

> [@How are optimizer.step() and loss.backward() related?](https://discuss.pytorch.org/t/how-are-optimizer-step-and-loss-backward-related/7350):
>
> Hi. I am pretty new to Pytorch and keep surprised with the performance of Pytorch slight_smile I have followed tutorials and there’s one thing that is not clear. How the optimizer.step() and loss.backward() related? Does optimzer.step() function optimize based on the closest loss.backward() function? When I check the loss calculated by the loss function, it is just a Tensor and seems it isn’t related with the optimizer. Here’s my questions: (1) Does optimzer.step() function optimize bas…

---

<div class="post-metadata">

**Author:** ![Oli](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/oli/32/9773_2.png) [@Oli](https://discuss.pytorch.org/u/Oli)\
**Post date:** [October 15, 2019, 5:56am UTC](https://discuss.pytorch.org/t/different-samples-use-different-learning-rates/58234/3 "2019-10-15T05:56:58Z")

</div>

I second @shirui-japina’s suggestions. Do a `nn.LossFn(reduction='none')`, weigh the different samples to your liking and then do your own reduction (e.g. mean).

---

<div class="post-metadata">

**Author:** ![shirui-japina](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shirui-japina/32/15285_2.png) [@shirui-japina](https://discuss.pytorch.org/u/shirui-japina)\
**Post date:** [October 15, 2019, 6:36am UTC](https://discuss.pytorch.org/t/different-samples-use-different-learning-rates/58234/4 "2019-10-15T06:36:46Z")

</div>

Thanks for your reply.

> Do a `nn.LossFn(reduction='none')` , weigh the different samples to your liking

If the method above is correct (In fact, I think that’s correct),  
but why I need to **do reduction (e.g. mean)**?  
If I don’t do that, what will happen?

From mathematical theory

- _ **Batch Gradient Descent** _  
When the number of samples is large, the training process will be slow.

- _ **Stochastic Gradient Descent** _  
Decreased accuracy, not globally optimal.

- _ **Mini-batch Gradient Descen** _  
Use a subset of samples to update each parameter.

Usually we use _ **Mini-batch Gradient Descen-like** _ to update the parameters. Which means we get the average value of the loss function to update parameters.  
**Is the process of averaging (do reduction (e.g. mean)) equivalent to this process?**  
Or **the process of averaging (do reduction (e.g. mean))** is just for PyTorch operating mechanism?

---

<div class="post-metadata">

**Author:** ![Oli](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/oli/32/9773_2.png) [@Oli](https://discuss.pytorch.org/u/Oli)\
**Post date:** [October 15, 2019, 8:48am UTC](https://discuss.pytorch.org/t/different-samples-use-different-learning-rates/58234/5 "2019-10-15T08:48:30Z")

</div>

> [@shirui-japina](#):
>
> Usually we use _ **Mini-batch Gradient Descen-like** _ to update the parameters. Which means we get the average value of the loss function to update parameters.  
> **Is the process of averaging (do reduction (e.g. mean)) equivalent to this process?**

Yup 🙂 We have a loss for each sample in your batch, then mean/average that to get **one** loss-float which we can use to backpropagate. (edit: doesn’t need to be mean, but other reduction techniques such a sum)

---

<div class="post-metadata">

**Author:** ![shirui-japina](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shirui-japina/32/15285_2.png) [@shirui-japina](https://discuss.pytorch.org/u/shirui-japina)\
**Post date:** [October 15, 2019, 9:22am UTC](https://discuss.pytorch.org/t/different-samples-use-different-learning-rates/58234/6 "2019-10-15T09:22:36Z")

</div>

Thanks for your reply.😃  
I think I thoroughly understood this problem.😃😃

---

<div class="post-metadata">

**Author:** ![shirui-japina](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shirui-japina/32/15285_2.png) [@shirui-japina](https://discuss.pytorch.org/u/shirui-japina)\
**Post date:** [October 15, 2019, 2:07pm UTC](https://discuss.pytorch.org/t/different-samples-use-different-learning-rates/58234/7 "2019-10-15T14:07:47Z")

</div>

I have reconsidered it. And found some problems.

> [@How are optimizer.step() and loss.backward() related?](https://discuss.pytorch.org/t/how-are-optimizer-step-and-loss-backward-related/7350/2):
>
> optimizer.step is performs a parameter update based on the current gradient (stored in .grad attribute of a parameter) and the update rule. As an example, the update rule for SGD is defined here: [https://github.com/pytorch/pytorch/blob/cd9b27231b51633e76e28b6a34002ab83b0660fc/torch/optim/sgd.py#L63](https://github.com/pytorch/pytorch/blob/cd9b27231b51633e76e28b6a34002ab83b0660fc/torch/optim/sgd.py#L63). Calling .backward() mutiple times accumulates the gradient (by addition) for each parameter. This is why you should call optimizer.zero\_grad() after each .step() call. Note that following the f…

Topic above describes the operating mechanism of .backward() and optimizer.step() in PyTorch.  
Simple understanding is:

- _**.backward()**_  
Get the gradient for the parameters in model.

- _**optimizer.step()**_  
performs a parameter update based on the _current_ gradient (stored in `.grad` attribute of a parameter) and the update rule.

I mean, the function _**optimizer.step()**_ considers the update method ,and actually update the parameters, but not PyTorch users to consider the update method. (**We don’t have to do reduction (e.g. mean) or something by manual**)

What we should do is,

1. Before the function .backward(), adjust the loss function value of each sample.

2. Then get gradient for the parameters in model by .backward().

3. At the end, update the parameters based on the _current_ gradient by optimizer.step().

The parameter in LossFn() _ **reduction** _ can set as: `'none'` | `'mean'` | `'sum'`.  
But not here to get the average of loss function value for the parameters updating.  
(Although I don’t know what these `'mean'` | `'sum'` are for.)

---

<div class="post-metadata">

**Author:** ![Oli](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/oli/32/9773_2.png) [@Oli](https://discuss.pytorch.org/u/Oli)\
**Post date:** [October 15, 2019, 2:34pm UTC](https://discuss.pytorch.org/t/different-samples-use-different-learning-rates/58234/8 "2019-10-15T14:34:49Z")

</div>

If you set the reduction flag to ‘none’ and try to do the backward() on the un-reduced loss, you get this error message

`RuntimeError: grad can be implicitly created only for scalar outputs`

---

<div class="post-metadata">

**Author:** ![shirui-japina](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shirui-japina/32/15285_2.png) [@shirui-japina](https://discuss.pytorch.org/u/shirui-japina)\
**Post date:** [October 15, 2019, 2:37pm UTC](https://discuss.pytorch.org/t/different-samples-use-different-learning-rates/58234/9 "2019-10-15T14:37:43Z")

</div>

😱😱😱
