Different samples use different learning rates

[shirui-japina (Shirui Zhang)]

I have reconsidered it. And found some problems.

How are optimizer.step() and loss.backward() related?

optimizer.step is performs a parameter update based on the current gradient (stored in .grad attribute of a parameter) and the update rule. As an example, the update rule for SGD is defined here: https://github.com/pytorch/pytorch/blob/cd9b27231b51633e76e28b6a34002ab83b0660fc/torch/optim/sgd.py#L63. Calling .backward() mutiple times accumulates the gradient (by addition) for each parameter. This is why you should call optimizer.zero_grad() after each .step() call. Note that following the f…

Topic above describes the operating mechanism of .backward() and optimizer.step() in PyTorch.
Simple understanding is:

• .backward()
  Get the gradient for the parameters in model.

• optimizer.step()
  performs a parameter update based on the current gradient (stored in .grad attribute of a parameter) and the update rule.

I mean, the function optimizer.step() considers the update method ,and actually update the parameters, but not PyTorch users to consider the update method. (We don’t have to do reduction (e.g. mean) or something by manual)

What we should do is,

1. Before the function .backward(), adjust the loss function value of each sample.

2. Then get gradient for the parameters in model by .backward().

3. At the end, update the parameters based on the current gradient by optimizer.step().

The parameter in LossFn() reduction can set as: 'none' | 'mean' | 'sum'.
But not here to get the average of loss function value for the parameters updating.
(Although I don’t know what these 'mean' | 'sum' are for.)