I found that under some circumstances, an RNN gives slightly different results for instances of the same sequence in a batch of many different sequences.
It is difficult to isolate the problem, and changing small details like the random seed or the constants in the input may make it disappear. With this rather cabalistic set of inputs, though, I get unexpectedly different outputs for the sequence [5]
import torch
from torch import nn
from torch.nn.utils.rnn import pack_padded_sequence, pad_packed_sequence
indices = torch.tensor([[64, 35, 35],
[13, 24, 6],
[37, 5, 7],
[11, 44, 10],
[ 5, 11, 23],
[19, 4, 36],
[45, 33, 42],
[ 5, 11, 23],
[15, 10, 5],
[ 7, 4, 8],
[19, 9, 8],
[46, 10, 0],
[12, 4, 0],
[15, 10, 0],
[ 6, 7, 0],
[12, 4, 0],
[ 6, 7, 0],
[ 4, 10, 0],
[ 7, 4, 0],
[12, 4, 0],
[25, 26, 0],
[ 6, 7, 0],
[12, 4, 0],
[ 4, 10, 0],
[ 3, 0, 0],
[ 3, 0, 0],
[28, 0, 0],
[ 5, 0, 0],
[23, 0, 0],
[ 3, 0, 0],
[62, 0, 0],
[16, 0, 0],
[16, 0, 0],
[16, 0, 0],
[ 3, 0, 0],
[28, 0, 0],
[ 5, 0, 0],
[ 4, 0, 0],
[16, 0, 0],
[ 6, 0, 0],
[16, 0, 0],
[ 3, 0, 0],
[28, 0, 0],
[ 5, 0, 0],
[16, 0, 0],
[ 6, 0, 0],
[16, 0, 0],
[16, 0, 0],
[ 3, 0, 0],
[ 5, 0, 0]])
lengths = torch.sum(indices != 0, 1)
torch.set_printoptions(precision=10)
torch.manual_seed(42)
embeddings = nn.Embedding(66, 5, padding_idx=0)
lstm = nn.LSTM(5, 50, batch_first=True)
embedded = embeddings(indices)
packed = pack_padded_sequence(embedded, lengths, batch_first=True)
hidden, _ = lstm(packed)
padded, _ = pad_packed_sequence(hidden, batch_first=True)
only_5 = (indices[:, 0] == 5) & (lengths == 1)
print('Output for sequences consisting only of [5]')
print(padded[only_5, 0, :3])
Output:
Output for sequences consisting only of [5]
tensor([[-0.0135216247, 0.0015346913, 0.0631567389],
[-0.0135216247, 0.0015346913, 0.0631567389],
[-0.0135216247, 0.0015346913, 0.0631567389],
[-0.0135216229, 0.0015346938, 0.0631567389]],
grad_fn=<IndexBackward>)
Is there anyway to fix that?