I have tried using the following custom loss class:
# class BCEWithLogitsLoss(): class CustomWBCE(): def __init__(self, class_weights=None, **kwargs): self.class_weights = class_weights def __call__(self, output:Tensor, target:Tensor, **kwargs)->Rank0Tensor: output = torch.sigmoid(output) if output.min() <= 0 or output.max() >= 1: # print('Wrong value in output, will give loss nan') output = torch.clamp(torch.sigmoid(output),min=1e-8,max=1 - 1e-8) if self.class_weights is not None: assert len(self.class_weights) == 2 loss = self.class_weights * (target * torch.log(output)) + \ self.class_weights * ((1 - target) * torch.log(1 - output)) else: loss = target * torch.log(output) + (1 - target) * torch.log(1 - output) loss = torch.neg(torch.mean(loss)) return loss
The reason I apply the sigmoid is because I am using this in a fast.ai Learner, which does not apply an activation function at the end of the network by default. Sometimes my model outputed a very small value (-136 for example) and torch.sigmoid’s result was 0., which led to a -inf in the torch.log; that’s why I added the torch.clamp.
I have tried it both with weights computed with the formula:
total = negative + positive w0 = positive / total w1 = negative / total
And with weights (0.5, 0.5) to test it out.
Both times I got very poor results:
AUC scores (this ranges from 0 to 1, 0.5 being random) after training with different weights:
- Calculated weights (0.20, 0.80): 0.51 AUC
- Equal weights (0.5, 0.5): 0.45 AUC (worse than random somehow)
Now, if I try to train the model in the same way, but using
loss_bce = fastai.layers.BCEWithLogitsFlat() (flattens the tensors before applying torch.nn.modules.loss.BCEWithLogitsFlat) it gets to 0.83 AUC.
The model is a DenseNet121 based binary classifier; I trained it on X-Rays from the CheXPert14 dataset.
I am on PyTorch version 1.0.1 and fast.ai version 1.0.52.
Do you have any idea what I’m doing wrong?