Hi, all
Recently I implement a customized loss function, within a batch, there are positive sample loss and negative sample loss, my overall loss will sum these positive and negative loss over a batch, I want minimize the positive sample loss and the negative loss, in other words, the absolute value of negative loss will go higher as optimization proceeds.
An obvious solution would be to set the batch size 1, decreasing speed dramatically, so, is there any better solution or a hint ?
Let me know if this is what you want:
You have a loss function that returns positive and negative losses. You want to minimize positive losses as well as the negative losses.
I think you can define an overall loss that sums together all of those losses. You can call backward() on that, and then send the gradients into an optimizer.
Exactly what I did, it turns out, that, the overall loss is decreasing, while there is always one of them increasing, positive loss or negative loss, whichever. Iām doubting the gradient calculated in this way, e.g., sums positive and negative loss all together, will be opposed to the minimization direction, as one of the positive and negative loss will go higher.