LSTM gradient shape

1

Hi,

I’m trying to understand LSTMs and backpropagation with LSTMs. I have a simple architecture.

def __init__(self, input_dim=100, hidden_dim=100, output_dim=216, num_layers=2):
    super(LSTMGenerator, self).__init__()
    self.input_dim = input_dim
    self.hidden_dim = hidden_dim
    self.output_dim = output_dim
    self.num_layers = num_layers
    self.layer1 = nn.LSTM(self.input_dim, self.hidden_dim, self.num_layers, bidirectional=False, dropout=0.5)
    self.out = nn.Linear(self.hidden_dim, self.output_dim)

I’m using the pytorch register_backward_hook function. I’m trying to modify the gradient in the backward pass. Should I use grad_input or grad_output? Which gradient is being used to update the previous weights?

I printed out the gradient shapes. What do the values in grad_input and grad_output mean?

length of grad input 3
grad input 0 shape torch.Size([216])
grad input 1 shape torch.Size([128, 100])
grad input 2 shape torch.Size([100, 216])

length of grad output 1
grad ouput 0 shape torch.Size([128, 216])

Any help would be appreciated.