Sequential LSTM

1

Hi,

I am new to PYTORCH. I am trying to use ‘nn.Sequential’ to build a single layer LSTM (just for the sake of trial)

rnn = nn.Sequential(
    nn.LSTM(10, 20, 2)
 )
input = Variable(torch.randn(100, 3, 10))
h0 = Variable(torch.randn(2, 3, 20))
output, hn = rnn(input, (h0))

However, this gives an error

forward() takes exactly 2 arguments (3 given)

although the same example works if I don’t use Sequential

rnn = nn.LSTM(10, 20, 2)
input = Variable(torch.randn(100, 3, 10))
h0 = Variable(torch.randn(2, 3, 20))
output, hn = rnn(input, (h0))

Can you please advice on this?
Thank you

1 Like
2

An LSTM has two internal states: h and c. I don’t know the why of that behaviour with Sequential but try either passing both h0 and c0 values or not passing them at all in order to let the model do the default initialisation.

Let me know how it goes…

EDIT: not passing any state value works for me…

3

You’re right, I just tried it, and it works without passing h and c.
However, I don’t understand if this is correct or no. In the examples in the documentation, you have to pass h and c.
How can I do this in case of a Sequential model?

4

nn.Sequential is not meant for building a model that operates on time sequences; nn.LSTM will do that out of the box. nn.Sequential is for stringing together several layers that don’t use time sequences into one model that runs the layers one after another.

3 Likes
5

@jekbradbury I see. I thought it is similar to Sequential Model in Keras.
What if I want to stack layers of Convolution, LSTM and Dense in one model? Is there an example on how to do this in PyTorch?
Also, what if I want to stack LSTM layers with different number of hidden units? How can I do that?

6

I think the easiest way to get a feel for these things would be to play around with a batch of data at the interactive prompt, seeing the sizes that come out of calls to Linear, Conv1D, and LSTM modules; you’ll want to write a forward method for your model that passes the data around between those modules and uses .view to reshape tensors.

1 Like
7

Thank you @jekbradbury . I will spend more time playing with the different modules

8

@jekbradbury

Is it ever possible to do this:

model = nn.Sequential(
                     nn.LSTM(...),
                     # since LSTM will return a tuple, anything here in the middle to make this whole thing work?
                     nn.Linear(...)
              )
3 Likes
9

This is how you use with nn.Sequential. I haven’t figure out how to supply initial h0 yet, so don’t pass h0 in, it will init as default

rnn = nn.Sequential(
    nn.LSTM(10, 20, 2)
 )
input = Variable(torch.randn(100, 3, 10))
h0 = Variable(torch.randn(2, 3, 20))
c0 = Variable(torch.randn(2, 3, 20))
output, hn = rnn(input)

If you want to pass h0 (also, you must pass c0 with h0), perhaps you should let lstm outside nn.Sequential

lstm =  nn.LSTM(10, 20, 2)
input = Variable(torch.randn(100, 3, 10))
h0 = Variable(torch.randn(2, 3, 20))
c0 = Variable(torch.randn(2, 3, 20))
output. hn = lstm(input, (h0,c0)) # omit both h0, c0, or must pass a tuple of both (h0, c0).
2 Likes