Gradient Clipping is not work as expected

1

Hi all, I have an exploding gradient problem when train the minibatch for 150-200 epochs with batch size = 256 and there’s about 30-60 minibatch (This depends on my specific config). But I have an exploding gradient issues even if I add the code below.

image
image842×88 13.4 KB

As you can see this below images, notice that in step about 40k there’s the swing of gradients between ± 20k, 40k and 60k respectively. I don’t know why this happens because i use the clip_grad_value_ above. Also Using the learning rate decay from 0.01 to about 0.008 at step 40k.

image
image1763×378 26.6 KB

Or do I need to update the weight parameters by myself something like this
image
But i think optimizer.step() should do the job and the clip_grad_value_ is an inplace operation so i don’t need to take the return value from function. Please correct if i did anything wrong. Thank you very much

Best regards,
Mint

2

Hi it seems correct to me however, perhaps your norm bound is too high ? What happens when you lower it from 100 to 10 or so ?

3

Hi, I’ve decrease it to ±1 and it’s still have these problem. But there’s 2 different kind of computing the bound by norm or by a value. Maybe using value is a cause of the problem. I’m not sure about this

4

any solution? I still got this problem
Thank you

5

It seems correct to me. Can you, for example, try printing the L2 norm of gradients over iterations to see if the code is really working as expected?