I have a very simple model that has a torch.nn.LSTM that takes in a sequence of length 30 and produces a sequence of length 30. The LSTM is followed by two Linear layers with ReLU activation to produce the final predictions of length 30. The number of features in both input and output of LSTM is 1. The predictions have intermittent zero values, while the target at the time of training is never zero (I use [0, 1] normalization, and then replace the zeroes by 1e-4).
I have tried changing:
- optimizer
- depth
- number of features in LSTM cells
and regardless, the output has zeroes at some places in the predictions. Why is this happening and how do I ensure that the output of the model is never zero (because the physical quantity I am predicting cannot be zero)?