Hey all
I am currently experimenting with different LSTM models for time-series prediction and am noticing a strange behaviour difference between two implementations that should be (from my understanding) fundamentally the same.
For my first implementation I used two LSTMCells and avoided pytorch’s built in batching. I initially did this because my training data has variable sequence lengths and I was averse to dealing with the variable input length when it comes to batching, which is why I kind of cheated with the batch dimension in the forward function. Thus to handle batches, (as shown below) I run each sample in the batch through the network individually, calculate the loss and the associated gradients, caching the gradients and updating the network using the accumulated gradients.
For my second model I tried to utilize pytorch optimizations and instead of LSTMCell I used actual LSTM layers and the built-in batching, which should result in a considerable speed up. Note that for this model, I always use num_layers=2.
I then tried to compare how these two models train and found that for implementation 1, the model exhibits two phases, one where it quickly decreases, and another where it has noisy oscillations around a slowly decreasing average (much like an L shape). For model 2 I expected the same behaviour but found that while it also has the same initial phase where it quickly decreases and leads into the same second phase seen in implementation 1, once in this phase the MSELoss occasionally spikes up really high, and then the model enters another phase where it quickly decreases until it levels out again.
Is there any obvious reason why I may be seeing this difference? For both of these models I am using the Adam optimizer with its default parameters, and torch.nn.MSELoss() for the loss function. Ultimately I’d like to use Implementation 2 because it does lead to a considerable speed up in the training process, but I’m concerned that I am doing something wrong.
Update: I’ve realized that since I am not dividing the loss function by the batch size for implementation 1, the gradients may differ significantly in magnitude but this would mean that I’d have larger gradients for the first implementation, would it not? Meaning that if I was to see spiking behaviour, it should be for Imp 1 instead of Imp 2.
Implementation 1: using naive batching
Model
class PyLSTMCells(nn.Module):
# init layers
def __init__(self, input_size, hidden_size, output_size):
super(PyLSTMCells, self).__init__()
# attach hidden layer size to class
self.hidden_size = hidden_size
# define layers
self.l1 = nn.LSTMCell(input_size, hidden_size)
self.l2 = nn.LSTMCell(hidden_size, hidden_size)
self.o = nn.Linear(hidden_size, output_size)
# define forward pass over single time-series
def forward(self, input):
# get tensor type of input
Ttype = input.type()
# add an additional "first" dimension to work
# with the LSTMCell interface
pad_input = input.unsqueeze(0)
# define hidden/cell states
h1 = torch.zeros(1, self.hidden_size).type(Ttype)
c1 = torch.zeros(1, self.hidden_size).type(Ttype)
h2 = torch.zeros(1, self.hidden_size).type(Ttype)
c2 = torch.zeros(1, self.hidden_size).type(Ttype)
# loop over time series and get final output
TS_len = pad_input.shape[1]
for i in range(TS_len):
h1, c1 = self.l1(pad_input[:,:,i], (h1,c1))
h2, c2 = self.l2(h1, (h2,c2))
output = self.o(h2)
# return output, ignoring the false "first" dim
# that was added
return output[0,:]
Training Method
opt.zero_grad() # Adam optimizer with default parameters
batch_loss = 0
for i in range(batch_size):
X, lbl = get_example()
output = model.forward(X) # where model is the network defined above
l = loss(output, lbl) # MSELoss
l.backward()
batch_loss += l
avg_batch_loss = batch_loss/batch_size
opt.step()
Implementation 2: LSTM layers and Pytorch batching
Model
class BatchPyLSTM(nn.Module):
# init network
def __init__(self, input_size, hidden_size, output_size, num_layers=2):
super(BatchPyLSTM, self).__init__()
# attach important vars to class
self.input_size = input_size
self.hidden_size = hidden_size
self.num_layers = num_layers
# define layers
self.lstm = nn.LSTM(input_size, hidden_size, num_layers, batch_first=True)
self.lin = nn.Linear(hidden_size, output_size)
# init hidden and cell states
def init_hidden(self, Ttype, batch_size):
h0 = torch.zeros(self.num_layers, batch_size, self.hidden_size).type(Ttype) # hidden states
c0 = torch.zeros(self.num_layers, batch_size, self.hidden_size).type(Ttype) # cell states
return h0,c0
# forward pass over entire batch
def forward(self, input, seq_lens):
# get tensor type of input and batch size
Ttype = input.type()
batch_size = input.shape[0]
# init hidden and cell states
h0, c0 = self.init_hidden(Ttype, batch_size)
# reshape before packing to match pytorch interface
pckd_input = input.permute(0,2,1)
# pack input so the LSTM knows now to consider the padded values
pckd_input = nn.utils.rnn.pack_padded_sequence(pckd_input, seq_lens,
batch_first=True)
# feed through LSTM
lstm_otp,_ = self.lstm(pckd_input,(h0,c0))
# repad lstm_otp into normal tensor object so we can actually work with it
lstm_otp,_ = nn.utils.rnn.pad_packed_sequence(lstm_otp, batch_first=True,
padding_value=float('nan'))
# extract lstm output for final time-step of each series
fin_lstm_otp = torch.stack([ o[l-1,:] for o,l in zip(lstm_otp, seq_lens) ])
# feed lstm output tensors
otp = self.lin(fin_lstm_otp)
return otp
Training Method
opt.zero_grad() # Adam optimizer with default parameters
X_batch, lbl_batch, seq_lens = get_batch(batch_size)
output = model.forward() # implementation 2 network
avg_batch_loss = loss(output, lbl_batch) # MSELoss
avg_batch_loss.backward()
opt.step()