Negative indexing of a 1D tensor: is this something to do with ignore_index in NLLLoss?

(Seonil Son) #1

Days ago I tried to use LabelSmoothingLoss from the openNMT and something I have question about its implementation.

in its implementation it has one_hot[self.ignore_index] = 0 and the index is default self.ignore_index = -100 . I don’t get what does it mean nor whether it is related with nn.NCELoss 's ignore_index.

Could somebody kindly explain what it does? I tried to follow source codes and still not sure what it’s doing.

class LabelSmoothingLoss(nn.Module):
    """
    With label smoothing,
    KL-divergence between q_{smoothed ground truth prob.}(w)
    and p_{prob. computed by model}(w) is minimized.
    """
    def __init__(self, label_smoothing, tgt_vocab_size, ignore_index=-100):
        assert 0.0 < label_smoothing <= 1.0
        self.ignore_index = ignore_index
        super(LabelSmoothingLoss, self).__init__()

        smoothing_value = label_smoothing / (tgt_vocab_size - 2)
        one_hot = torch.full((tgt_vocab_size,), smoothing_value)
        one_hot[self.ignore_index] = 0 # here is the thing. 
        self.register_buffer('one_hot', one_hot.unsqueeze(0))

        self.confidence = 1.0 - label_smoothing

  

you can see openNMT_LabelSmoothingLoss source code here