Does anybody know how to implement label smoothing (LS) with CTCLoss? I found a lot of articles about CrossEntropyLoss with Label smoothing, but nothing about CTCLoss.
I’ve found a paper (https://arxiv.org/pdf/1711.02212.pdf) mentioning about the LS with CTCLoss, with an equivalent approach with KLDiv, but have no idea how to implement it with PyTorch. TIA!
PS:
I implemented the CTCLoss with Label smoothing like:
class SmoothCTCLoss(_Loss):
def __init__(self, num_classes, blank=0, weight=0.01):
super().__init__(reduction='mean')
self.weight = weight
self.num_classes = num_classes
self.ctc = nn.CTCLoss(reduction='mean', blank=blank, zero_infinity=True)
self.kldiv = nn.KLDivLoss(reduction='batchmean')
def forward(self, log_probs, targets, input_lengths, target_lengths):
ctc_loss = self.ctc(log_probs, targets, input_lengths, target_lengths)
kl_inp = log_probs.transpose(0, 1)
kl_tar = torch.full_like(kl_inp, 1. / self.num_classes)
kldiv_loss = self.kldiv(kl_inp, kl_tar)
#print(ctc_loss, kldiv_loss)
loss = (1. - self.weight) * ctc_loss + self.weight * kldiv_loss
return loss
Can anybody confirm if this is correct or not? I also referred to this: