Dropout in LSTMCell

PyQuantQ · October 1, 2018, 10:53am

How to implement dropout if I’m using LSTMCell instead of LSTM?
Let’s stick to the sine-wave example because my architecture is similar:

github.com

pytorch/examples/blob/master/time_sequence_prediction/train.py

from __future__ import print_function
import torch
import torch.nn as nn
import torch.optim as optim
import numpy as np
import matplotlib
matplotlib.use('Agg')
import matplotlib.pyplot as plt

class Sequence(nn.Module):
    def __init__(self):
        super(Sequence, self).__init__()
        self.lstm1 = nn.LSTMCell(1, 51)
        self.lstm2 = nn.LSTMCell(51, 51)
        self.linear = nn.Linear(51, 1)

    def forward(self, input, future = 0):
        outputs = []
        h_t = torch.zeros(input.size(0), 51, dtype=torch.double)
        c_t = torch.zeros(input.size(0), 51, dtype=torch.double)

This file has been truncated. show original

If I try to update weights by accessing them directly

self.lstmCell_1 = nn.LSTMCell(self.input_features, self.hidden_features)
self.dropout = nn.Dropout(p=0.1, inplace=True)
...
self.dropout(self.self.lstmCell_1.weights_ih)

it results in an error.

I don’t want to implement my own LSTMCell, neither do I want to use LSTM, because I need predictions to be made further in time, not just the single next value, therefore I need to control the flow of data between LSTMCell units like in the sine-wave example.

Any ideas?

PyQuantQ · October 2, 2018, 1:29pm

Or maybe it is possible to realize the same sine-wave predictor with just LSTM without going into data flow control of LSTMCell?