Hello guys! I’m having a bit of a problem trying to implement a small LSTM network.
I’m using a sequence of 20 values as input and the network has to predict certain output. The data is scaled between 0 and 1, and the dataset looks like this:
Input:
array([[0.3616897 , 0.50186179, 0.46220047, 0.48337192],
[0.38939199, 0.5308964 , 0.47071214, 0.48807264],
[0.43114892, 0.55613415, 0.47903991, 0.49106299],
...,
[0.48847856, 0.55368452, 0.48759646, 0.48916795],
[0.49450675, 0.57330196, 0.48922357, 0.49509893],
[0.49728463, 0.58997826, 0.49048734, 0.50129733]])
Output:
array([[0.33857308, 0.50931249],
[0.3834156 , 0.53883397],
[0.42320043, 0.5688971 ],
...,
[0.48872479, 0.56165727],
[0.50050588, 0.58804321],
[0.49346006, 0.59605735]])
So I would use the 20 first values of Input as the first sequence and the desired output would be the 20th element of Output.
This is how my network class looks like:
class neuralNet(nn.Module):
def __init__(self):
super(neuralNet, self).__init__()
self.lstm = nn.LSTM(4, 4)
self.fc1 = nn.Linear(4, 16)
self.out_real = nn.Linear(16, 1)
self.out_im = nn.Linear(16, 1)
def forward(self, X):
x, _ = self.lstm(X)
x = F.leaky_relu(self.fc1(x[-1].view(X.shape[1], -1)))
sal_real = F.leay_relu(self.out_real(x))
sal_im = F.leaky_relu(self.out_im(x))
return sal_real, sal_im
This is how I define the Dataloader:
class _data_(Dataset):
def __init__(self, X, y, window):
self.X = X
self.y = y
self.window = window
def __len__(self):
return self.X.shape[0]
def __getitem__(self, idx):
return self.X[idx:idx+(self.window)], self.y[idx+self.window-1]
I use the Adam optimizer, a batch size of 1024, however I’ve tried with several different batch sizes and learning rates, and this is how my training loop looks like:
network.train()
train_loss = []
for epochs in range(100):
for x, y_true in loader:
optimizer.zero_grad()
out_real, out_im = network(x.view(20, 1024, 4).type(torch.FloatTensor))
loss_1 = torch.sqrt(((out_real-y_true[:, 0].view(-1, 1).type(torch.FloatTensor))**2).mean())
loss_2 = torch.sqrt(((out_im-y_true[:, 1].view(-1, 1).type(torch.FloatTensor))**2).mean())
loss = loss_1+loss_2
loss.backward()
optimizer.step()
train_loss.append(loss.detach().numpy().reshape(-1))
print(epochs)
However, the loss quickly drops to around 0.23 and then just stays there, predicting very similar values no matter the input. I checked the gradients of the LSTM layer and the gradients are very small:
print(network.lstm.weight_ih_l0.grad)
tensor([[ 2.4166e-05, 1.1411e-04, 6.1285e-05, 8.6557e-05],
[ 5.0125e-05, -1.9328e-05, 1.7648e-05, 2.4593e-06],
[ 6.6281e-05, 6.8287e-05, 6.3350e-05, 6.7243e-05],
[ 2.7249e-05, 5.8169e-05, 4.2136e-05, 5.0256e-05],
[ 2.1470e-05, 1.0980e-04, 5.8128e-05, 8.3109e-05],
[ 7.5999e-05, -7.7817e-06, 3.6380e-05, 1.7472e-05],
[ 1.1556e-04, 1.3376e-04, 1.1724e-04, 1.2893e-04],
[ 3.5581e-05, 6.9475e-05, 4.9838e-05, 5.9006e-05],
[-1.0905e-04, -4.3272e-04, -2.4602e-04, -3.3566e-04],
[ 4.4844e-04, 2.4696e-06, 2.3953e-04, 1.3694e-04],
[ 1.7014e-04, 1.8785e-04, 1.6552e-04, 1.8307e-04],
[-2.9961e-04, -1.2150e-03, -6.4513e-04, -9.0194e-04],
[ 2.6096e-05, 2.2836e-04, 1.0026e-04, 1.6018e-04],
[ 1.4060e-04, -6.1505e-05, 3.9611e-05, -1.6231e-05],
[ 2.9458e-04, 1.2405e-04, 1.8125e-04, 1.3770e-04],
[ 4.3766e-05, 1.5784e-04, 8.4850e-05, 1.1989e-04]])
The loss function:
This is my first time working with LSTMs, and I’ve tried with different widths/depths and different activation functions on the neural network, but I’m still stuck with the same problem, so I’m assuming the problem is with my code.
Thanks a lot in advance!