I’m working on a problem that requires cross entropy loss in the form of a reconstruction loss. I would like to weight the loss for each sample in the mini-batch differently. The current API for cross entropy loss only allows weights of shape C. I would like to pass in a weight matrix of shape batch_size , C so that each sample is weighted differently.
Is there a way to do this? The only workaround I can think of is to evaluate the loss for each sample in the mini-batch and pass in a new set of weights each time.
Sorry I don’t really understand your question.
Are you saying that for a batch N, you want to get the loss between each data point in N and it’s corresponding prediction?
I’m using a denoising autoencoder and wanted to use a weighted loss function to emphasise the loss for the corrupted components of the data. A batch size of 1 could work but the model would perform poorly.
The only solution I can think of is to do something like:
loss = []
for i, element in enumerate(batch):
# calculate weights
l = F.cross_entropy(y_pred[i], y_true[i], weight=weights[i])
loss.append(l)
If each sample had its own weight, then ur model won’t be able to generalize properly on data that wasn’t part of the training data.
It’s like having a data points in a graph that make up a straight line. You would want to get the slope (m) of the straight line after making the plot of the entire data points, rather than finding m for each and every single point.
Hi. You can probably do with reduction ‘none’. Something like this should probably work. criterion = CrossEntropyLoss(reduction='none') loss = criterion(network_out, label) --> shape batch_size x 1 loss = loss * batch_weights loss = torcn.mean(loss)