I am implementing in PyTorch an LSTM model to predict if the closing value of a stock will go up or down in the next 5 and 10 minutes.
Specifically, I am using 24 years of 5 minute data with 19 features, divided in chunks of one week per forecast (using 7 different stocks)
The problem I’m facing is the fact that, no matter what, the LSTM model seems to predict values around one specific value to always minimize the loss, which it does not go down very much.
I pre-prepare the inputs and targets in torch.tensors with [batch_size, sequence_len, features] (which in my case is [32, 2016, 19]), normalize them between 0 and 1 and feed them to my LSTM model which is structured like this:
class MultiInputOutputLSTM(nn.Module):
def __init__(self, input_size, hidden_size, num_layers, output_size, dropout, lr, batch_size):
super(MultiInputOutputLSTM, self).__init__()
self.input_size = input_size
self.hidden_size = hidden_size
self.num_layers = num_layers
self.dropout = dropout
self.batch_size = batch_size
self.loss_list = []
self.accuracy = 0
self.predictions_list = [0]
self.lstm = nn.LSTM(input_size = self.input_size, hidden_size = self.hidden_size, num_layers = self.num_layers, dropout = self.dropout, batch_first=True)
self.fc = nn.Linear(hidden_size, output_size, bias=True)
self.sigmoid = nn.Sigmoid()
self.criterion = nn.BCEWithLogitsLoss()
self.optimizer = torch.optim.RMSprop(self.parameters(), lr = lr, alpha=0.9, weight_decay=1e-4, momentum=0.5)
self.scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(self.optimizer, 'min')
def forward(self, x):
h0 = torch.zeros(self.num_layers, self.batch_size, self.hidden_size)
c0 = torch.zeros(self.num_layers, self.batch_size, self.hidden_size)
lstm_out, _ = self.lstm(x, (h0, c0))
output = self.fc(lstm_out[:, -1, :])
return output
def train_step(self ,x, y):
self.train()
predictions_1 = torch.round(self.forward(x))
predictions = self.forward(x)
if (predictions_1.detach().cpu().numpy()[0] == y.detach().cpu().numpy()[0]).all():
self.accuracy += 1
self.predictions_list.append(predictions.detach().cpu().numpy()[0][0])
penalty = torch.mean((predictions-0.5)**2)
loss = self.criterion(predictions, y) + penalty
self.scheduler.step(loss)
self.optimizer.zero_grad()
self.loss_list.append(loss.item())
mean_loss = sum(self.loss_list)/len(self.loss_list)
loss.backward()
torch.nn.utils.clip_grad_norm_(self.parameters(), max_norm=6)
self.optimizer.step()
return loss.item(), mean_loss, self.accuracy, predictions.detach().cpu().numpy()[0], y.detach().cpu().numpy()[0]
def test_step(self, inputs, targets):
self.eval()
with torch.no_grad():
outputs = torch.round(self.forward(inputs))
outputs = outputs.detach().cpu().numpy()[0]
targets = targets.detach().cpu().numpy()[0]
if (outputs == targets).all():
accuracy = 1
else:
accuracy = 0
return accuracy, outputs, targets
The targets are 1 if the price goes up, 0 else wise.
The hyper parameters are:
input_size = 19
hidden_size = 3
num_layers = 5
output_size = 2
lr = 0.001
num_epochs = 5
batch_size = 32
dropout = 0.3
The model creates predictions at the start and then they will change ± 1e-3, 1e-4, so basically the same through the training.
The train-test split is 85-15%.
Here is a list of what I have tried:
lowering or increasing learning rate (from 0.00001 to 0.1), output size (with more and less forecasts), batch_size (from 1 to 256), num_layers (1-5), input_size (with 1, 2, 3… 19 features), dropout (0 to 0.5) and num_epochs (from 1 to 100).
I have tried Adam optimiser, then SDG, then RMSProp optimiser changing alpha, momentum and weight_decay.
Also tried BCELoss (with the sigmoid activation layer in self.forward()), L1Loss, MSELoss, CrossEntropyLoss before settling to a loss that penalises values too close to 0.5 (that does not help very much, but at this point I’ll try anything.
The mean loss goes down from 0.924 to 0.889.
The accuracy is, as you can imagine, very low (from 5.56% to 11.67%): the only reason that it is not zero is because of pure luck.
I am also trying to forecast the difference between last item of closing column of the data frame and the future next two values, but… with the same problems.
Ticker - Df - Loss - Mean Loss - Predictions - Targets - Epoch
AAPL 1/120 0.808 0.808 [0.5642182 0.11684255] [1. 1.] 1/50
AAPL 2/120 0.828 0.818 [0.5629723 0.12899561] [1. 1.] 1/50
AAPL 3/120 0.834 0.823 [0.5472506 0.13749745] [1. 1.] 1/50
AAPL 4/120 0.839 0.827 [0.52973044 0.13656825] [0. 0.] 1/50
AAPL 5/120 0.821 0.826 [0.52082217 0.13681749] [0. 0.] 1/50
AAPL 6/120 0.804 0.822 [0.5117941 0.14727135] [1. 1.] 1/50
AAPL 7/120 0.779 0.816 [0.49826628 0.1547919 ] [0. 0.] 1/50
AAPL 8/120 0.845 0.820 [0.49666688 0.1573844 ] [0. 0.] 1/50
…
AAPL 1/120 0.762 0.751 [0.4200741 0.24348469] [1. 1.] 50/50
AAPL 2/120 0.787 0.751 [0.43581426 0.23458275] [1. 1.] 50/50
AAPL 3/120 0.797 0.751 [0.43012726 0.23912352] [1. 1.] 50/50
AAPL 4/120 0.805 0.751 [0.43174705 0.23598525] [0. 0.] 50/50
AAPL 5/120 0.794 0.751 [0.41290796 0.24462458] [0. 0.] 50/50
AAPL 6/120 0.777 0.751 [0.42214283 0.2398287 ] [1. 1.] 50/50
AAPL 7/120 0.747 0.751 [0.42418063 0.24077193] [0. 0.] 50/50
AAPL 8/120 0.822 0.751 [0.44875318 0.23495609] [0. 0.] 50/50
...
I got the data from finam.ru: if you want to download it be careful as the more you go back in time the more the data gets corrupted and requires some processing.
With very little changes I saw that the predictions seem always to shift towards 0.5: a recurrent “anomaly” (i do not know if this is supposed to happen or not) is that the predictions are correct or very close, but the loss doesn’t go down or it just goes up.
How the loss happens seems random and the results do not have any pattern.
Any help is appreciated.
Thank you