Different weighting of labels using CrossEntropyLoss

1

Hello there, i am using CrossEntropyLoss for training on sequence data. On my sequence there are sparse single events of which i have labels and their position as indices for the output sequence. I would like the neighbouring samples of a single event to be labeld in the same manner as the event it self but with reduced weighting. This is what i have in mind in pseudo code:

weight = 0.5
for inputs, labels, label_indices in dataloader:
    outputs = model(inputs)
    normal_loss = criterion(outputs[label_indices], labels)
    left_loss = criterion(outputs[label_indices - 1], labels)
    right_loss = criterion(outputs[label_indices + 1], labels)
    loss = normal_loss + left_loss * weight + right_loss * weight
    
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

I this approach suitable?

2

Your approach should work, if you also slice the labels tensor.
However, if might be easier to avoid the loss reduction via reduction='none' and multiply your loss tensor with a weight tensor.
However, I doubt you will see a huge performance difference. :wink:

1 Like