Lstm.weight_hh_l0 doesn't update after gradient descent (but the other parameters do update!)

autograd
3

I have a similar LSTM model for imdb sentiment classification. Noticed that one of the LSTM parameters doesn’t change at all.

# trainable parameters
['embedding.weight', 'lstm.weight_ih_l0', 'lstm.weight_hh_l0', 'lstm.bias_ih_l0', 'lstm.bias_hh_l0', 'linear.weight', 'linear.bias']
# unchanged parameter
lstm.weight_ih_l0

Edit : I’ve checked the gradients. They are very small for lstm.weight_ih_l0 but not zero, in the range of e^-42.

1 Like