Transformer Embedding - IndexError: index out of range in self

nlp
2

Hello again,

In error trace of yours (error in decoder stage)

  File "~/transformer.py", line 20, in forward
    x = self.embedding(x)

can you add print(torch.max(x)) before the line x = self.embedding(x)

I guess the error is because of x contains id that is >=3194. If the value is greater than 3194, then pytorch will raise the error mentioned in the stack trace.

It means that while you are converting tokens to ids, you are assigning the value greater than 3194.

One way to debug this is checking the max value for the batch before sending to model. Once the value is greater than or equal to 3194, then in that batch you find out for which sample via

# x-> (batch_size, seq_len)
torch.max(x, dim=0)

and finding the index where 3194 comes. With this you know the index of the sample, now you can traceback to ids and then tokens and then to the original sentence as well.

Hope it helps.

1 Like