# Proper way to do gradient clipping? **URL:** \ **Category:** Uncategorized\ **Created:** [January 25, 2017, 4:49pm UTC](https://discuss.pytorch.org/t/proper-way-to-do-gradient-clipping/191 "2017-01-25T16:49:21Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![kim.seonghyeon](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kim.seonghyeon/32/3045_2.png) [@kim.seonghyeon](https://discuss.pytorch.org/u/kim.seonghyeon)\ **Post date:** [January 25, 2017, 4:49pm UTC](https://discuss.pytorch.org/t/proper-way-to-do-gradient-clipping/191/1 "2017-01-25T16:49:21Z")
Is there a proper way to do gradient clipping, for example, with Adam? It seems like that the value of Variable.data.grad should be manipulated (clipped) before calling optimizer.step() method. I think the value of Variable.data.grad can be modified in-place to do gradient clipping. Is it safe to do? Also, Is there a reason that Autograd RNN cells have separated biases for input-to-hidden and hidden-to-hidden? I think this is redundant and has a some overhead. ---
**Author:** ![apaszke](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/apaszke/32/21_2.png) [@apaszke](https://discuss.pytorch.org/u/apaszke)\ **Post date:** [January 25, 2017, 4:52pm UTC](https://discuss.pytorch.org/t/proper-way-to-do-gradient-clipping/191/2 "2017-01-25T16:52:34Z")
You can safely modify `Variable.grad.data` in-place after the backward pass finishes. For example see how it’s done in the [language modelling example](https://github.com/pytorch/examples/blob/master/word_language_model/main.py#L84-L91). The reason for that is that it has a nice user facing API where you have both weight tensors exposed. Also, it opens up a possibility of doing batched matrix multiply on the inputs for all steps, and then only applying the hidden-to-hidden weights (it’s not yet added there). If you measure the overhead and prove us that it can be implemented in a clean and fast way, we’ll happily accept a PR or change it. ---
**Author:** ![kim.seonghyeon](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kim.seonghyeon/32/3045_2.png) [@kim.seonghyeon](https://discuss.pytorch.org/u/kim.seonghyeon)\ **Post date:** [January 25, 2017, 7:30pm UTC](https://discuss.pytorch.org/t/proper-way-to-do-gradient-clipping/191/3 "2017-01-25T19:30:15Z")
I have tested nn.LSTM against simple LSTM implementation and found almost no difference in the performance. Maybe I overestimated the overhead of the additional addition with simple guess. Thank you! ---
**Author:** ![apaszke](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/apaszke/32/21_2.png) [@apaszke](https://discuss.pytorch.org/u/apaszke)\ **Post date:** [January 25, 2017, 7:57pm UTC](https://discuss.pytorch.org/t/proper-way-to-do-gradient-clipping/191/4 "2017-01-25T19:57:41Z")
If you’re running on GPU you’ll also likely see great speedups from using cuDNN LSTM implementation. ---
**Author:** ![kim.seonghyeon](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kim.seonghyeon/32/3045_2.png) [@kim.seonghyeon](https://discuss.pytorch.org/u/kim.seonghyeon)\ **Post date:** [January 25, 2017, 11:34pm UTC](https://discuss.pytorch.org/t/proper-way-to-do-gradient-clipping/191/5 "2017-01-25T23:34:25Z")
I have tested in CPU and got no better results than just few milliseconds. (for someone who may try to implement LSTM for benchmarking 🙂 ) I think some more addition is insignificant than another expensive computations, like multiplication of weight matrices, nonlinear activation functions, or even python loop itself. ---
**Author:** ![dmadeka1](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/dmadeka1/32/18387_2.png) [@dmadeka1](https://discuss.pytorch.org/u/dmadeka1)\ **Post date:** [April 2, 2017, 12:02am UTC](https://discuss.pytorch.org/t/proper-way-to-do-gradient-clipping/191/6 "2017-04-02T00:02:04Z")
Quick question about this @apaszke , are the `Variable.grad.data` that we should pass to our clip function a part of the model object or (if we use a different optimizer) - the optimizer object? In the sense, does optimizer itself call backward? In which case, the code below should pass optimizer to the clip function right? ```python optimizer.zero_grad() output, hidden = model(data, hidden) loss = criterion(output.view(-1, ntokens), targets) loss.backward() clipped_lr = lr * clip_gradient(model, clip) for p in model.parameters(): p.data.add_(-clipped_lr, p.grad.data) optimizer.step() ``` ---
**Author:** ![apaszke](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/apaszke/32/21_2.png) [@apaszke](https://discuss.pytorch.org/u/apaszke)\ **Post date:** [April 3, 2017, 10:39pm UTC](https://discuss.pytorch.org/t/proper-way-to-do-gradient-clipping/191/8 "2017-04-03T22:39:16Z")
I’m sorry but I don’t understand the question. Optimizer never calls `backward()` itself, unless you give it a callable argument (see `torch.optim` docs for more details on that). BTW you might want to use `torch.nn.utils.clip_grad_norm` now. ---
**Author:** ![mcleonard](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/898d66/32.png) [@mcleonard](https://discuss.pytorch.org/u/mcleonard)\ **Post date:** [May 2, 2017, 5:55pm UTC](https://discuss.pytorch.org/t/proper-way-to-do-gradient-clipping/191/9 "2017-05-02T17:55:41Z")
Maybe I’m doing something wrong here, but using gradient clipping like ```auto nn.utils.clip_grad_norm(model.parameters(), clip) for p in model.parameters(): p.data.add_(-lr, p.grad.data) ``` makes my network train much slower than with `optimizer.step()`. Here’s what it looks like with gradient clipping, with `clip=5`: ```auto Epoch: 1/10... Step: 10... Loss: 4.4288 Epoch: 1/10... Step: 20... Loss: 4.4274 Epoch: 1/10... Step: 30... Loss: 4.4259 Epoch: 1/10... Step: 40... Loss: 4.4250 Epoch: 1/10... Step: 50... Loss: 4.4237 Epoch: 1/10... Step: 60... Loss: 4.4223 Epoch: 1/10... Step: 70... Loss: 4.4209 Epoch: 1/10... Step: 80... Loss: 4.4193 Epoch: 1/10... Step: 90... Loss: 4.4188 Epoch: 1/10... Step: 100... Loss: 4.4174 ``` And without gradient clipping, everything else equal: ```auto Epoch: 1/10... Step: 10... Loss: 3.2837 Epoch: 1/10... Step: 20... Loss: 3.1901 Epoch: 1/10... Step: 30... Loss: 3.1512 Epoch: 1/10... Step: 40... Loss: 3.1296 Epoch: 1/10... Step: 50... Loss: 3.1170 Epoch: 1/10... Step: 60... Loss: 3.0758 Epoch: 1/10... Step: 70... Loss: 2.9787 Epoch: 1/10... Step: 80... Loss: 2.9104 Epoch: 1/10... Step: 90... Loss: 2.8271 Epoch: 1/10... Step: 100... Loss: 2.6813 ``` There is probably something I don’t understand, but I’m just switching out those two bits of code. ---
**Author:** ![apaszke](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/apaszke/32/21_2.png) [@apaszke](https://discuss.pytorch.org/u/apaszke)\ **Post date:** [May 2, 2017, 10:55pm UTC](https://discuss.pytorch.org/t/proper-way-to-do-gradient-clipping/191/10 "2017-05-02T22:55:53Z")
Maybe you’re clipping them to very small values. It’s a possible effect ---
**Author:** ![WendyShang](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/wendyshang/32/1617_2.png) [@WendyShang](https://discuss.pytorch.org/u/WendyShang)\ **Post date:** [July 10, 2017, 9:27pm UTC](https://discuss.pytorch.org/t/proper-way-to-do-gradient-clipping/191/11 "2017-07-10T21:27:38Z")
The one comes with nn.util clips in proportional to the magnitude of the gradients. Thus you’d like to make sure it is not too small for your particular model as Adam said (I think :p). The old-fashioned way of clipping/clampping is ``` def gradClamp(parameters, clip=5): for p in parameters: p.grad.data.clamp_(max=clip) ``` ---
**Author:** ![Brando\_Miranda](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/brando_miranda/32/14355_2.png) [@Brando\_Miranda](https://discuss.pytorch.org/u/Brando_Miranda)\ **Post date:** [August 23, 2017, 4:15pm UTC](https://discuss.pytorch.org/t/proper-way-to-do-gradient-clipping/191/13 "2017-08-23T16:15:21Z")
for people trying to just get an answer quickly: ``` torch.nn.utils.clip_grad_norm(mdl_sgd.parameters(),clip) ``` or with in-place clamp: ``` W.grad.data.clamp_(-clip,clip) ``` * * * also similar Q: > [@Gradient clipping](https://discuss.pytorch.org/t/gradient-clipping/2836): > > Hi everyone, I am working on implementing Alex Graves model for handwriting synthesis (this is is the [link](https://arxiv.org/pdf/1308.0850.pdf)) In page 23, he mentions the output derivatives and LSTM derivatives How can I do this part in PyTorch? Thank you, Omar ---
**Author:** ![ntubertchen](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ntubertchen/32/6677_2.png) [@ntubertchen](https://discuss.pytorch.org/u/ntubertchen)\ **Post date:** [June 4, 2018, 8:00am UTC](https://discuss.pytorch.org/t/proper-way-to-do-gradient-clipping/191/14 "2018-06-04T08:00:58Z")
I thought nn.utils.clip\_grad\_norm(model.parameters(), clip) is supposed to finish the job. What is: for p in model.parameters(): p.data.add\_(-lr, p.grad.data) for? Can someone give a more explicit explain? Is it because after I use gradient clipping, I may not use adam optimizer? ---
**Author:** ![Neta\_Zmora](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/neta_zmora/32/23393_2.png) [@Neta\_Zmora](https://discuss.pytorch.org/u/Neta_Zmora)\ **Post date:** [June 4, 2018, 10:33pm UTC](https://discuss.pytorch.org/t/proper-way-to-do-gradient-clipping/191/15 "2018-06-04T22:33:15Z")
@ntubertchen Hi, Use `torch.nn.utils.clip_grad_norm` to keep the gradients within a specific range (clip). In RNNs the gradients tend to grow very large (this is called ‘the exploding gradient problem’), and clipping them helps to prevent this from happening . It is probably helpful to look at the [implementation](https://pytorch.org/docs/master/_modules/torch/nn/utils/clip_grad.html) because it teaches us that: 1. “The norm is computed over all gradients together, as if they were concatenated into a single vector.” 2. You can control the norm type ([lp-norm](https://en.wikipedia.org/wiki/Lp_space#The_p-norm_in_finite_dimensions), with p defaulting to 2; or the L-inf norm). 3. All of the gradient coefficients are multiplied by the same `clip_coef`. 4. clip\_grad\_norm is invoked after all of the gradients have been updated. I.e. between loss.backward() and optimizer.step(). So during loss.backward(), the gradients that are propagated backwards are not clipped, until the backward pass completes and clip\_grad\_norm() is invoked. optimizer.step() will then use the updated gradients. Regarding the code you ask about: ```auto for p in model.parameters(): p.data.add_(-lr, p.grad.data) ``` This iterates across all of the model.parameters() and performs an in-place multiply-add on each of the parameter tensors §. p.data.add\_ is functionally equal to: ```auto p.data = p.data + (-lr * p.grad.data) ``` In other words, this performs a similar function as optimizer.step(), using the gradients to updates the model parameters, but without the extra sophistication of a torch.optim.Optimizer. If you use the above code, then you should not use an optimizer (and vice-versa). Cheers, Neta ---
**Author:** ![tom](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tom/32/3162_2.png) [@tom](https://discuss.pytorch.org/u/tom)\ **Post date:** [June 5, 2018, 6:40am UTC](https://discuss.pytorch.org/t/proper-way-to-do-gradient-clipping/191/16 "2018-06-05T06:40:09Z")
Note that `clip_grad_norm_` modifies the gradient _after the entire backpropagation has taken place_. In the RNN context it is common to restrict the gradient _that is being backpropagated_ during the calculation. This is described e.g. in [Alex Graves’ famous RNN paper](https://arxiv.org/abs/1308.0850). To do the latter, you typically use `register_hook` on the inputs or outputs of certain operations, e.g. with `lambda x: x.clamp(-10,10)` to do element-wise clipping. For a practical example, you could search for `register_hook` in [my Graves handwriting generation notebook](https://github.com/t-vi/pytorch-tvmisc/blob/master/misc/graves_handwriting_generation.ipynb). Best regards Thomas ---
**Author:** ![hughperkins](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hughperkins/32/915_2.png) [@hughperkins](https://discuss.pytorch.org/u/hughperkins)\ **Post date:** [June 5, 2018, 9:31am UTC](https://discuss.pytorch.org/t/proper-way-to-do-gradient-clipping/191/17 "2018-06-05T09:31:48Z")
@tom Awesome info. Thanks! ---
**Author:** ![Jared\_77](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jared_77/32/5293_2.png) [@Jared\_77](https://discuss.pytorch.org/u/Jared_77)\ **Post date:** [June 22, 2018, 8:40pm UTC](https://discuss.pytorch.org/t/proper-way-to-do-gradient-clipping/191/18 "2018-06-22T20:40:34Z")
> [@Brando\_Miranda](#): > > torch.nn.utils.clip\_grad\_norm(mdl\_sgd.parameters(),clip) Thanks. Where does this go in relation to forward and backward propagation? ---
**Author:** ![jukiewiczm](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jukiewiczm/32/12934_2.png) [@jukiewiczm](https://discuss.pytorch.org/u/jukiewiczm)\ **Post date:** [September 23, 2018, 7:50pm UTC](https://discuss.pytorch.org/t/proper-way-to-do-gradient-clipping/191/20 "2018-09-23T19:50:45Z")
> [@Neta\_Zmora](#): > > In other words, this performs a similar function as optimizer.step(), using the gradients to updates the model parameters, but without the extra sophistication of a torch.optim.Optimizer. If you use the above code, then you should not use an optimizer (and vice-versa). What’s the reason for not using the `optimizer.step` after clipping the gradients? ---
**Author:** ![Neta\_Zmora](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/neta_zmora/32/23393_2.png) [@Neta\_Zmora](https://discuss.pytorch.org/u/Neta_Zmora)\ **Post date:** [September 25, 2018, 7:03pm UTC](https://discuss.pytorch.org/t/proper-way-to-do-gradient-clipping/191/21 "2018-09-25T19:03:53Z")
No reason: you can certainly use `optimizer.step()` and it will most likely lead to a better solution since the optimizer will update the parameters in a more sophisticated way (e.g. using momentum). ---
**Author:** ![jukiewiczm](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jukiewiczm/32/12934_2.png) [@jukiewiczm](https://discuss.pytorch.org/u/jukiewiczm)\ **Post date:** [September 25, 2018, 10:42pm UTC](https://discuss.pytorch.org/t/proper-way-to-do-gradient-clipping/191/22 "2018-09-25T22:42:48Z")
My bad, I thought what you suggest is that if you do gradient clipping, then you should (for some reason) use custom updates instead of `optimizer.step()`. Now I got it, you meant that if you use custom updates, then you should not use `optimizer.step()` (to avoid mixing custom and auto updates). Makes sense! ---
**Author:** ![aaronlelevier](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/7cd45c/32.png) [@aaronlelevier](https://discuss.pytorch.org/u/aaronlelevier)\ **Post date:** [November 11, 2018, 2:46am UTC](https://discuss.pytorch.org/t/proper-way-to-do-gradient-clipping/191/23 "2018-11-11T02:46:01Z")
You need to use both `optimizer.step` and clip right? Because `optimizer.step` calcultes the gradient and then you want to clip those gradients to prevent vanishing on the next training step? [Next page](https://discuss.pytorch.org/t/proper-way-to-do-gradient-clipping/191.md?page=2)