How to use Focal Loss for an imbalanced data for binary classification problem?

[Mona_Jalal]

I have been searching in GitHub, Google, and PyTorch forum but it doesn’t seem there is a training for using PyTorch-based focal loss for an imbalanced dataset for binary classification. Further, there has been so many variation of the said loss. Is there any standardized version of this loss given its effectiveness and popularity inside the newer PyTorch library itself? If not, the experts in the field, which open-source implementation of the “focal loss” for “binary classification” in PyTorch do you suggest?

Further, if 14% of my dataset is the positive class, how do I assign these weights to the said focal loss when using it?

Implementing Focal Loss for a binary classification problem

So I have been trying to implement Focal Loss recently (for binary classification), and have found some useful posts here and there, however, each solution differs a little from the other. Here, it’s less of an issue, rather a consultation. My question comes in two parts: I would like advice on the correctness of my -inspired by others- implementation (i.e., is it correct? if not, please advise.). Is there any improvements to consider (e.g., efficiency or generalizability)? Here is the snippe…

and

How to get alpha values of focal loss for multiclass classification?

I am sure the gamma value of focal loss is fixed(=2) but I am confused about how to get alpha values of focal loss for multiclass classification? Is the alpha value fixed or not? I do not understand the following line: It may be set by inverse class frequency

^ The questions posed above are left without an answer unfortunately.

For example, here’s another implementation, that I found on GitHub

def focal_loss_lgb_eval_error(y_pred, dtrain, alpha, gamma):
    """
    Adapation of the Focal Loss for lightgbm to be used as evaluation loss

    Parameters:
    -----------
    y_pred: numpy.ndarray
        array with the predictions
    dtrain: lightgbm.Dataset
    alpha, gamma: float
        See original paper https://arxiv.org/pdf/1708.02002.pdf
    """
    a,g = alpha, gamma
    y_true = dtrain.label
    p = 1/(1+np.exp(-y_pred))
    loss = -( a*y_true + (1-a)*(1-y_true) ) * (( 1 - ( y_true*p + (1-y_true)*(1-p)) )**g) * ( y_true*np.log(p)+(1-y_true)*np.log(1-p) )
    return 'focal_loss', np.mean(loss), False

jrzaurin/LightGBM-with-Focal-Loss/blob/master/examples/Lightgbm_with_Focal_Loss.ipynb

Here’s another answer from PyTorch Forum:

BCE_loss = F.binary_cross_entropy_with_logits(inputs, targets, reduction='none')
pt = torch.exp(-BCE_loss) # prevents nans when probability 0
F_loss = self.alpha * (1-pt)**self.gamma * BCE_loss
return focal_loss.mean()

Is this a correct implementation for focal loss in pytorch?

Try this: BCE_loss = F.binary_cross_entropy_with_logits(inputs, targets, reduction='none') pt = torch.exp(-BCE_loss) # prevents nans when probability 0 F_loss = self.alpha * (1-pt)**self.gamma * BCE_loss return focal_loss.mean() Remember the alpha to address class imbalance and keep in mind that this will only work for binary classification. I think this is very similar to your implementation it just uses the BCE function which does the sigmoid and those .mul() for you and also prevents the n…

I think as a suggestion would be best that this “focal loss” be in the PyTorch itself.

[hamedB (Hamed Behzadi)]

Hi @Mona_Jalal

Thanks for sharing your experience. Kindly I have a question about alpha value in this loss. The common value is 0.25. Is this actually the weight of classes with lower amount of data (i.e., positive class)? In the Pytorch implementation of the weighted BCE loss (i.e., F.binary_cross_entropy_with_logits) the positive class (class with lower sample) has higher weights, close to one for example 0.7. And 1-0,7 as the weight for negative class (the class with the higher number of samples).