LSTMCell implementation

jjery2243542 · November 1, 2018, 8:56am

I want to use high-order gradient with LSTMCell. However, this function is not supported. So I want to implement the cell with linear layer. Also, I want to initialize my lstm cell to a well-trained lstm cell.
But I cannot find the order of the gates. The following is my implementation. I am wondering what is the order of gates in pytorch LSTMCell implementation. Thank you.

gist.github.com

https://gist.github.com/jjery2243542/a26ee50b97ad8f7f7fae486ea798fae3

MyLSTMCell.py

class MyLSTMCell(torch.nn.Module):
    def __init__(self, input_size, hidden_size):
        super().__init__()
        self.input_layer = torch.nn.Linear(input_size, hidden_size * 4)
        self.hidden_layer = torch.nn.Linear(hidden_size, hidden_size * 4)
        self.h = hidden_size

    def forward(self, inp, hidden_tuple):
        z_tm1, c_tm1 = hidden_tuple
        h = self.h

This file has been truncated. show original

tom · November 1, 2018, 10:33am

The order within the parameters is the same as for LSTM, so you can consult the Variables section in the LSTM documentation.

Best regards

Thomas