Hi,
I’m implementing a model for a binary NLP classification task with a bi-RNN and an Attention mechanism on top and I would like to get the gradients of the embeddings with respect to the predominant predicted class. My issue is I found various approaches to obtain the gradient and they yield various results. The approaches I tried are:
-
torch.autograd.grad( loss, model.embedding.weight ) and retrieve element 0.
-
To add retain_grad() on the embeddings and retrieve the gradient after calling loss.backward() by doing embedding.grad
-
To backward through the output by y_predicted.backward( (y_predicted > 0.5 ).float() ) where the (y_predicted > 0.5) will return a mask of equal size to y_predicted with 1’s occupying the predominant class. Following this backward I then retrieve the gradient with embedding.grad.
Can anyone advise on which of these three (if any) is the correct approach? If not is there a more robust way to obtain the embedding gradients starting from the predominant class?
Thanks!
George