Problem migrating code from Torch to PyTorch

1

I have a code I wrote in Torch which I am migrating to PyTorch. Its a regression-based learning problem. In Torch, after defining my network, I define concatenate my inputs along the 2nd axis as follows

    inputs = torch.cat({inputs[1], inputs[2], inputs[3], inputs[4], 
                        inputs[5], inputs[6], inputs[7], inputs[8],
                        inputs[9]}, 2)

And the network is as defined:

  (lstm1): LSTM(9, 9)
  (lstm2): LSTM(9, 6)
  (lstm3): LSTM(6, 6)
  (drop): Dropout (p = 0.3)
  (fc1): Linear (6 -> 3)
  (fc2): Linear (3 -> 3)

In torch, I can forward call and backward call okay as the inputs tensor id of batch size [100, 9] torch.*Tensor.

I construct the network the same way in PyTorch and concatenate my nine inputs along the 1st axis as

	inputs = torch.cat((base_in, base_out, left_in, 
						left_out, right_in, right_out, 
						z, pitch, yaw), 1)

I define my model in PyTorch as

class LSTMModel(nn.Module):
    def __init__(self, nFeatures, nCls, nHidden, nineq=12, neq=0, eps=1e-4, 
                noutputs=3,numLayers=1):
        super(LSTMModel, self).__init__()
        
        self.nFeatures = nFeatures
        self.nHidden = nHidden
        self.nCls = nCls
        self.nineq = nineq
        self.neq = neq
        self.eps = eps

        self.cost = nn.MSELoss(size_average=False)
        self.noutputs = noutputs
        # self.neunet = nn.Sequential()
        self.lstm1  = nn.LSTM(nHidden[0],nHidden[0],num_layers=numLayers)
        self.lstm2  = nn.LSTM(nHidden[0],nHidden[1],num_layers=numLayers)
        self.lstm3  = nn.LSTM(nHidden[1],nHidden[2],num_layers=numLayers)
        self.drop   = nn.Dropout(0.3)
        self.fc1 = nn.Linear(nHidden[2], noutputs)
        self.fc2 = nn.Linear(noutputs, noutputs)

        self.M = Variable(torch.tril(torch.ones(nCls, nCls)))
        self.L = Parameter(torch.tril(torch.rand(nCls, nCls)))
        self.G = Parameter(torch.Tensor(nineq/2, nCls).uniform_(-1,1))

        """
        define constraints, z_i, and slack variables, s_i,
        for six valves. z_i and c_i are learnable parameters
        """
        self.z0 = Parameter(torch.zeros(nCls))
        self.s0 = Parameter(torch.ones(nineq/2))
        self.z0p = Parameter(torch.zeros(nCls))
        self.s0p = Parameter(torch.ones(nineq/2))

    def forward(self, x):
        nBatch = x.size(0)
        # LSTM-dropout-LSTM-dropout-lstm-dropout-FC-QP-FC
        x = x.view(nBatch, -1)
        x = self.lstm1(x)
        x = self.drop(x)
        x = self.lstm2(x)
        x = self.drop(x)
        x = self.lstm3(x)
        x = self.drop(x)
        x = self.fc1(x)

But my calls to forward gives me runtime errors like so

RuntimeError: matrices expected, got 1D, 2D tensors at /home/robotec/Documents/NNs/locuclab-pytorch/torch/lib/TH/generic/THTensorMath.c:1224

I wonder what I must be doing wrong.

2

The input to an nn.LSTM module should be a time series, num_timesteps x batch_size x num_features. You’re passing it a single timestep – you most likely want to use nn.LSTMCell.

3

Also, note that {...} is a set in Python, so it doesn’t preserve ordering! You should use [...] in the torch.cat call!

4

Thank you, guys. There’s still; something I am not too clear of. When I am defining a model in torch, usually, I can use nn.Sequential to place my layers in a container form such as

        neunet = nn.Sequential()
       local rnn        
        rnn = nn.FastLSTM(9, 9, 5)
        neunet:add(rnn) 
        rnn = nn.FastLSTM(9, 6, 5)
        neunet:add(rnn) 
        rnn = nn.FastLSTM(6, 3, 5)
        neunet:add(rnn) 
        neunet:add(nn.Dropout(0.3))
        neunet:add(nn.Linear(3, 3, bias))      
       neunet = nn.Sequencer(neunet, 3)

Now in PyTorch, I defined the same model (thanks to James’ pointing out to me that nn.LSTM is different from nn.LSTMCell):

class LSTMModel(nn.Module):
    def __init__(self, noutputs=3,numLayers=3):
        super(LSTMModel, self).__init__()
        
        self.nHidden = [9, 6, 6]
        self.cost = nn.MSELoss(size_average=False)
        self.noutputs = noutputs
        self.num_layers =  numLayers
        
        self.lstm0     = nn.LSTM(self.nHidden[0], self.nHidden[0], self.num_layers, 
                                batch_first= True, dropout=0.3)
        self.lstm1     = nn.LSTM(self.nHidden[1], self.nHidden[1], self.num_layers, 
                                batch_first= True, dropout=0.3)
        self.lstm2     = nn.LSTM(self.nHidden[2], self.nHidden[2], self.num_layers, 
                                batch_first= True, dropout=0.3)
        self.fc = nn.Linear(self.nHidden[2], noutputs)

    def forward(self, x):

        h0 = Variable(torch.zeros(self.num_layers, x.size(0), self.nHidden[0])) 
        c0 = Variable(torch.zeros(self.num_layers, x.size(0), self.nHidden[0]))
        
        #Now propagate the rnn 
        #rnn layer 1
        out0, _ = self.lstm0(x, (h0, c0))  
        
        #second rnn layer
        h1 = Variable(torch.zeros(self.num_layers, x.size(0), self.nHidden[1])) 
        c1 = Variable(torch.zeros(self.num_layers, x.size(0), self.nHidden[1]))
        out1, _ = self.lstm1(out0, (h1, c1)) 
        
        #third rnn layer
        h2 = Variable(torch.zeros(self.num_layers, x.size(0), self.nHidden[2])) 
        c2 = Variable(torch.zeros(self.num_layers, x.size(0), self.nHidden[2]))
        out2, _ = self.lstm2(out1, (h2, c2)) 
        
        #hidden layer of last time step
        out = self.fc(out2[:, -1, :])        
        return out

So I forward a DoubleTensor defined as

x = Variable(torch.randn(50000, 9), requires_grad=False) through this model and I get weird errors like

RuntimeError: matrices expected, got 1D, 2D tensors at /home/robotec/Documents/NNs/locuclab-pytorch/torch/lib/TH/generic/THTensorMath.c:1224
5

@apaszke, that was defined in the torch code that I earlier defined, haha!

6

You never need to create zero tensors as the initial hidden state of an nn.LSTM; that’s done automatically. But I’m still confused as to what you’re trying to do with the LSTMs; are you passing in a time series? If so, it needs to be three-dimensional (time by batch by features).

7

Yes, it is a time series. Could you give a minimal working example of how I can correctly form a PyTorch model and a time series input tensor based off the snippet I wrote above? Sorry to be a pain.

8

If x is 50000 by 9, then what’s your time dimension, what’s your batch dimension, and what’s your feature dimension?

9

mini-batch dimension is 100, feature dimension is 3, time dimension is 5.

10

Then you need to pass in a tensor of dimension 5 x 100 x 3

11

@jekbradbury, how do you suggest the initial states of the hidden state be initialized as in the example I give?

Like this?
h0 = Variable((self.num_layers, x.size(0), self.nHidden[0]))

12

Hi there
i was having similar question in this topic.
i have a time sequence data (accelerations sample in time) and i want to figure out the step length with a Bi-LSTM network, but i have problem making my input data. can you help me how to make dataset with my own data ? i dont know how to put them in order if i want to make BiLSTM input and what should my features be
i think i might use a window over my data and chose with a width of 50 accelerations i would have 50 features, but since there must be a time dimension it does’nt make sense to have a window over data as a feature
I would appreciate if you could help me with this.