Feeding None as hidden state to LSTM/GRU

leckofunny · May 18, 2021, 2:08pm

Hello everybody,

I just came across a behavior that I would expect to throw an exception.
If you feed None as hidden state to nn.GRU or nn.LSTM, it won’t throw an exception.
Instead it will use a hidden state made of zeros.

I’m wondering whether this behavior is intentionally or not.

torch version 1.8.1+cu111

pascal_notsawo · May 18, 2021, 2:16pm

This is intentional.
If you look at the source code you will see :

github.com

pytorch/pytorch/blob/master/torch/nn/modules/rnn.py#L659


if isinstance(orig_input, PackedSequence):
    input, batch_sizes, sorted_indices, unsorted_indices = input
    max_batch_size = batch_sizes[0]
    max_batch_size = int(max_batch_size)
else:
    batch_sizes = None
    max_batch_size = input.size(0) if self.batch_first else input.size(1)
    sorted_indices = None
    unsorted_indices = None


if hx is None:
    num_directions = 2 if self.bidirectional else 1
    real_hidden_size = self.proj_size if self.proj_size > 0 else self.hidden_size
    h_zeros = torch.zeros(self.num_layers * num_directions,
                          max_batch_size, real_hidden_size,
                          dtype=input.dtype, device=input.device)
    c_zeros = torch.zeros(self.num_layers * num_directions,
                          max_batch_size, self.hidden_size,
                          dtype=input.dtype, device=input.device)
    hx = (h_zeros, c_zeros)
else: