Using inputs of very different lengths to LSTM

Hi all,

I am new to PyTorch. I have the following setting:

  • inputs time series of length: N
  • for each datapoint in the time series I have a target vector of length N where y_i is 0 (no event) or 1 (event)

I have many of these signals. Each signal has a different length which depends on the recording time. For example one recording can be N = 1000 datapoints and another N = 1 Million datapoints.

I have understood that one way to manage this is by using sequence padding. However, given the very very different lengths of the recordings this will mean that I will end up with a tensor which is very large (of length of my longest recording) and mostly constituted of zeros.

Is zero padding, in this context, the best way to go? or is there an alternative?

Thanks,
Jo

Anyone :slight_smile: ?

Assuming you are using a stateful network, you should be able to split your data up into smaller sequences and feed them in to your network one after the other, meaning you will not have to pad each recording to the length of your longest recording.

1 Like

Thanks al3x!

Does it makes sense to you the way I am doing it?

####################
#defining the neural network - LSTM
####################
class Sequence(nn.Module):

def __init__(self,hidden_dim):
    super(Sequence, self).__init__()
    self.hidden_dim = hidden_dim
    self.lstm1 = nn.LSTMCell(1, hidden_dim)
    self.lstm2 = nn.LSTMCell(hidden_dim, hidden_dim)
    self.linear = nn.Linear(hidden_dim, 1)

def forward(self, input, future = 0):
    outputs = []
    h_t = torch.zeros(input.size(0), self.hidden_dim, dtype=torch.double)
    c_t = torch.zeros(input.size(0), self.hidden_dim, dtype=torch.double)
    h_t2 = torch.zeros(input.size(0), self.hidden_dim, dtype=torch.double)
    c_t2 = torch.zeros(input.size(0), self.hidden_dim, dtype=torch.double)

    for i, input_t in enumerate(input.chunk(input.size(1), dim=1)):
        h_t, c_t = self.lstm1(input_t, (h_t, c_t))
        h_t2, c_t2 = self.lstm2(h_t, (h_t2, c_t2))
        output = self.linear(h_t2)
        outputs += [output]

    outputs = torch.stack(outputs, 1).squeeze(2)
    return outputs

####################
#begin to train
####################

for i in range(5):
print('STEP: ', i)

def closure(): # this is needed because the LBFGS optimizer need to compute the error over a coupple of steps
    optimizer.zero_grad() # optimizer
    out_tab = torch.empty(1, 1, dtype=torch.float64)
    target_tab = torch.empty(1, 1, dtype=torch.float64)

    for j in range(NB_REC_TRAIN):
        out = seq(torch.from_numpy(signal_arr[j]))
        out_tab = torch.cat((out_tab, out), 0)
        target_tab = torch.cat((target_tab, torch.from_numpy(anns_arr[j])), 0)
    
    loss = criterion(out_tab, target_tab)
    print('loss:', loss.item())
    loss.backward()
    return loss
optimizer.step(closure)

In the inner for loop, I go through each time series one by one. I included this loop within the closure() function.

Does this makes sense?

Yes, that is an alternative to the method I suggested and looks like a reasonable solution.

Thanks Alex! I will move forward with that then.