# RNN predicting a constant output **URL:** \ **Category:** Uncategorized\ **Created:** [March 20, 2019, 12:37pm UTC](https://discuss.pytorch.org/t/rnn-predicting-a-constant-output/40397 "2019-03-20T12:37:21Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![abn](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/5e9695/32.png) [@abn](https://discuss.pytorch.org/u/abn)\ **Post date:** [March 20, 2019, 12:37pm UTC](https://discuss.pytorch.org/t/rnn-predicting-a-constant-output/40397/1 "2019-03-20T12:37:21Z")
Hi, PyTorch community! I am new to PyTorch and deep learning in general. I am using Elman RNN ([Ref](https://pytorch.org/docs/stable/nn.html?highlight=rnn#torch.nn.RNN)) in a regression analysis problem. However, the RNN is always predicting a constant output. I have tried - 1. Changing batch size 2. Scaling the input and output by a constant factor but still, the issue persists. ```python class RNN(nn.Module): def __init__ (self, input_size, output_size, hidden_dim, n_layers): super(RNN, self). __init__ () self.hidden_dim=hidden_dim self.rnn = nn.RNN(input_size, hidden_dim, n_layers, batch_first=True) self.fc = nn.Linear(hidden_dim, output_size) def forward(self, x, hidden): # x (batch_size, seq_length, input_size) # hidden (n_layers, batch_size, hidden_dim) # r_out (batch_size, time_step, hidden_size) batch_size = x.size(0) r_out, hidden = self.rnn(x, hidden) output = self.fc(r_out) return output, hidden ``` ```python # hyperparameters input_size=3 output_size=3 hidden_dim=14 n_layers=1 lr=0.1 # instantiate an RNN rnn = RNN(input_size, output_size, hidden_dim, n_layers) print(rnn) # MSE loss and Adam optimizer with a learning rate of 0.01 criterion = nn.MSELoss() optimizer = torch.optim.Adam(rnn.parameters(), lr=lr) ``` ```python final_hidden = None # train the RNN def train(rnn, n_steps, print_every): hidden = None for batch_i, step in enumerate(range(n_steps)): prediction, hidden = rnn(x_tensor, hidden) final_hidden = hidden ## Representing Memory ## # make a new variable for hidden and detach the hidden state from its history # this way, we don't backpropagate through the entire history hidden = hidden.data loss = criterion(prediction, y_tensor) loss_val.append(loss.item()) optimizer.zero_grad() loss.backward() optimizer.step() if batch_i%print_every == 0 or batch_i == n_steps-1: print("Epoch: {0}/{1}".format(batch_i+1, n_steps)) print('Loss: ', loss.item()) print(prediction.data) return rnn ``` Thanks for any advice in advance! ---
**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\ **Post date:** [March 20, 2019, 4:18pm UTC](https://discuss.pytorch.org/t/rnn-predicting-a-constant-output/40397/2 "2019-03-20T16:18:44Z")
Could you print the shapes of `prediction` and `y_tensor` before passing them to your criterion? Your model seems to train well using random inputs and targets. ---
**Author:** ![abn](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/5e9695/32.png) [@abn](https://discuss.pytorch.org/u/abn)\ **Post date:** [March 20, 2019, 5:20pm UTC](https://discuss.pytorch.org/t/rnn-predicting-a-constant-output/40397/3 "2019-03-20T17:20:24Z")
@ptrblck , I am getting ```python prediction shape torch.Size([25, 20, 3]) y_tensor shape torch.Size([25, 20, 3]) ``` ---
**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\ **Post date:** [March 20, 2019, 8:38pm UTC](https://discuss.pytorch.org/t/rnn-predicting-a-constant-output/40397/4 "2019-03-20T20:38:55Z")
Thanks for the information! In that case, could you try to lower the learning rate and see if the outputs stays constant? If that doesn’t help, you should try to overfit a small data sample (e.g. just 10 samples) and see, if your model is able to learn this sample at all. ---
**Author:** ![abn](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/5e9695/32.png) [@abn](https://discuss.pytorch.org/u/abn)\ **Post date:** [March 21, 2019, 1:59pm UTC](https://discuss.pytorch.org/t/rnn-predicting-a-constant-output/40397/5 "2019-03-21T13:59:59Z")
@ptrblck Thanks for replying! It is still giving constant output. I tried `lr=0.001` and 20000 epochs with the following x\_tensor ```auto [[0.5 3. 9.5] [0.6 3. 9.5] [0.7 3. 9.5] [0.8 3. 9.5] [0.5 3.5 9.5] [0.6 3.5 9.5] [0.7 3.5 9.5] [0.8 3.5 9.5] [0.5 4. 9.5] [0.6 4. 9.5]] ``` and the following y\_tensor ```auto [[1.18310e+02 1.00000e-01 5.35000e-01] [1.18290e+02 7.00000e-02 5.85000e-01] [1.18253e+02 3.00000e-02 6.35000e-01] [1.18148e+02 3.00000e-02 6.35000e-01] [1.18228e+02 3.00000e-02 5.60000e-01] [1.18148e+02 3.00000e-02 5.78000e-01] [1.18083e+02 8.00000e-02 6.68000e-01] [1.18060e+02 7.00000e-02 6.88000e-01] [1.18033e+02 9.00000e-02 5.58000e-01] [1.18048e+02 6.00000e-02 6.38000e-01]] ``` and final predicted output was: ```auto tensor([[[1.1816e+02, 5.9165e-02, 6.0802e-01], [1.1816e+02, 5.9165e-02, 6.0802e-01], [1.1816e+02, 5.9165e-02, 6.0802e-01], [1.1816e+02, 5.9165e-02, 6.0802e-01], [1.1816e+02, 5.9165e-02, 6.0802e-01], [1.1816e+02, 5.9165e-02, 6.0802e-01], [1.1816e+02, 5.9165e-02, 6.0802e-01], [1.1816e+02, 5.9165e-02, 6.0802e-01], [1.1816e+02, 5.9165e-02, 6.0802e-01], [1.1816e+02, 5.9165e-02, 6.0802e-01]]]) ``` ```auto ``` ---
**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\ **Post date:** [March 23, 2019, 11:01am UTC](https://discuss.pytorch.org/t/rnn-predicting-a-constant-output/40397/6 "2019-03-23T11:01:14Z")
I think the value ranges might be a problem in your use case. Since your model trains fine using random (normalized) data, I would suggest to normalize the input data and run the training again. Also, if that doesn’t help, you could normalize the output additionally and de-normalize it for prediction. The actual target signal seems to be a small signal on top of a mean value. ---
**Author:** ![abn](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/5e9695/32.png) [@abn](https://discuss.pytorch.org/u/abn)\ **Post date:** [March 23, 2019, 12:10pm UTC](https://discuss.pytorch.org/t/rnn-predicting-a-constant-output/40397/7 "2019-03-23T12:10:21Z")
Thanks! Will try the same. ---
**Author:** ![robd2](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/779978/32.png) [@robd2](https://discuss.pytorch.org/u/robd2)\ **Post date:** [March 23, 2019, 3:59pm UTC](https://discuss.pytorch.org/t/rnn-predicting-a-constant-output/40397/8 "2019-03-23T15:59:48Z")
Yes these true number (vs classification) prediction models can be tricky. Since your new to Pytorch I would further reduce the complexity of this data set and try a univariate input/output test vs the multivariate you currently have. Try to get the model to memorize a sign wave shape. Feel free to post more code for us to review.