'WeightNormInterfaceBackward0' returned nan values in its 0th output

I’m training a TCN model for time series prediction, the structure is as follows:


But as the training progressed, I suddenly got the following error report:

‘WeightNormInterfaceBackward0’ returned nan values in its 0th output.

I carefully checked the parameters of the model and found that some of them were particularly strange, the values of the parameters were particularly small (1e-16, 1e-17),and the corresponding gradients were almost 0. After backward, they became nan

I have tried to lowe the learning rate, but this time it just took a little longer to train and I still get the same reported error! And I dont’t know why this happen…

Can anyone who also met similar problems help me?
What’s the problem and How can I correct it?