torch.nn.CTCLoss returns infinity for some inputs

1

Hi,
I tried to replace CrossEntropyLoss with CTCLoss but I often get infinity outputs,
setting zero_infinity=True just makes it output 0 forever.
Any idea why?

import torch
import torch.nn as nn

probs = torch.tensor(
[[[-3.6852, -3.8380, -3.8917, -3.8420, -4.1217, -3.5202, -3.8134,
    -3.3443, -4.5487, -3.8015, -3.9781, -3.0194, -4.0216, -3.5204,
    -4.2357, -4.2557, -4.6121, -3.9420, -3.7618, -4.0152, -3.7686,
    -3.2913, -3.7141, -3.9715, -3.3414, -3.7910, -3.5413, -3.7887,
    -3.9627, -3.7912, -4.6102, -3.8574, -4.7015, -4.1991, -3.4495,
    -3.6784, -3.7860, -4.4842, -3.9561, -4.3670, -4.5784, -4.1805,
    -4.8021, -4.1785, -3.1910, -4.4310, -4.5877, -4.8743, -4.7679]],

[[-3.6433, -3.8820, -3.8138, -3.8630, -4.0570, -3.4049, -3.8250,
    -3.4900, -4.6103, -3.7115, -4.0392, -3.1994, -4.0751, -3.4773,
    -4.1737, -4.2480, -4.7001, -3.9189, -3.7473, -3.9203, -3.8479,
    -3.3608, -3.5682, -3.9201, -3.3088, -3.8354, -3.6256, -3.6313,
    -3.9220, -3.7211, -4.6523, -3.8543, -4.6962, -4.3603, -3.3139,
    -3.5954, -3.7370, -4.4784, -3.9313, -4.4535, -4.6581, -4.1376,
    -4.8446, -4.2231, -3.2970, -4.3959, -4.6230, -4.8661, -4.8886]],

[[-3.6278, -3.9100, -3.7469, -3.8844, -3.9681, -3.4005, -3.8425,
    -3.6862, -4.4955, -3.6545, -4.0343, -3.4717, -4.1044, -3.5075,
    -4.0795, -4.1596, -4.6185, -3.8613, -3.7557, -3.8153, -3.9310,
    -3.5221, -3.4801, -3.8535, -3.3727, -3.8728, -3.7805, -3.5169,
    -3.8751, -3.6887, -4.5521, -3.8429, -4.5521, -4.4218, -3.2836,
    -3.5790, -3.6708, -4.3754, -3.9049, -4.3983, -4.6034, -4.0614,
    -4.7089, -4.1932, -3.4841, -4.2479, -4.5258, -4.6913, -4.8192]]])
labels = torch.tensor([[11, 24, 24]])
lengths = torch.tensor([3])

criterion = nn.CTCLoss(blank=48, reduction='none')
lossv = criterion(probs, labels, lengths, lengths)
print(lossv)

probs were aquired from GRU output using:
probs = F.log_softmax(outputs, dim=2).transpose(0, 1)
the model should classify 48 classes and outputs 49 classes

Thanks in advance!

2

Hi,

CTCLoss returns inf if the input length is too short. It appears that for the PyTorch CTCLoss, the input lengths are too short if input_len = target_len, actually that’s also not the primary use case for CTCLoss. I don’t know whether that’s intentional or a bug of PyTorch’s implementation, as this limitation does not make sense to me.

Regards,
Unity05

3

Thank you very much, that’s it!
It seems like the PyTorch Documentation is wrong as it states “…which limits the length of the target sequence such that it must be ≤ the input length”
https://pytorch.org/docs/stable/generated/torch.nn.CTCLoss.html

4

Indeed, however I still believe that the implementation is wrong as what the PyTorch Documentation states makes perfect sense to me.
Do you know any user here who is to be pinged in such cases?

5

Hi again,

I’ve found out that that is not a bug or a documentation issue.
It’s due to your special example:
your labels are [11, 24, 24]. Therefore especially the last two are the same. So why does the problem occur, when the lengths are equal? Well, for the input to represent the same label twice in a row, there must be a blank in between those two, as that’s how CTC works, what is impossible to achieve with equal input lengths and target lengths.

Regards,
Unity05

6

Ah I see, so I need to insert a blank between them (and double the inputs).
Thank you again! :smile:

7

I’m not sure if I’ve understood you right, but your labels are fine, they just don’t work with an input of same length. :grinning_face_with_smiling_eyes:

8

Yeah you’re right again, blanks must be in the outputs of the GRU not the labels.
First time CTC user :sweat_smile:

9

Nice, always something to learn there is. :grinning_face_with_smiling_eyes: