Custom loss functions

[ptrblck]

Depending on your loss function, you could just multiply the positive and negative losses with your weights.
Maybe nn.BCEWithLogitsLoss might fit your use case providing pos_weight.

[YingLaiLin (Leon Lin)]

Yes, Thanks for you advise. I can split it into two kinds of loss function and just sum it up weightly. Much thanks!

[MariosOreo]

Dear @ptrblck,

Could you further explain the weight in nn.CrossentropyLoss() and nn.BCELoss, pos_weight in nn.BCEWithLogitsLoss()?

• weight in CrossentropyLoss is a Tensor of size C, but why does it should have the size of nbatch in nn.BCELoss()? And it seems that weight in BCELoss does not work for unbalanced data, right? ( because the weight is related to nbatch )
• Does pos_weight has the same effect with weight in nn.CrossentropyLoss?

Thanks in advance.

[ptrblck]

The weight argument in nn.BCE(WithLogits)Loss has the shape of the input batch, since the loss functions take floating point targets, which does not correspond to a class weighting schema. pos_weight on the other side is closer to a class weighting, as it only weights the positive examples. Furthermore, you can balance the recall and precision changing the pos_weight argument.

[msrdinesh (Mandava Sai Raghavendra Dinesh)]

Hey @ptrblck can you share, a similar dummy function to cross entropy loss. It would be helpful to me. Thanks in Advance!

[ptrblck]

Sure, here is the simple version without weighting, different reduction types etc:

def my_cross_entropy(x, y):
    log_prob = -1.0 * F.log_softmax(x, 1)
    loss = log_prob.gather(1, y.unsqueeze(1))
    loss = loss.mean()
    return loss


criterion = nn.CrossEntropyLoss()

batch_size = 5
nb_classes = 10
x = torch.randn(batch_size, nb_classes, requires_grad=True)
y = torch.randint(0, nb_classes, (batch_size,))

loss_reference = criterion(x, y)
loss = my_cross_entropy(x, y)

print(loss_reference - loss)
> tensor(0., grad_fn=<SubBackward0>)

[netaglazer (neta)]

but the loss inputs are tensors, how does it work?

[ptrblck]

Could you add some more details to your question please?
What is confusing about input tensors to a loss function?

[netaglazer (neta)]

i mean, for each batch the input of the loss function is a list of all predictions and labels in the current batch, and the loss is built for input of only one prediction and label

so how it should be implement?

and another thing - how the backward() of costume function should be implemented?

[netaglazer (neta)]

i mean, for each batch the input of the loss function is a list of all predictions and labels in the current batch, and the loss is built for input of only one prediction and label

so how it should be implement?

and another thing - how the backward() of costume function should be implemented?

[ptrblck]

I think you could index your output and target at the desired location and pass it to your criterion:

model = models.resnet18()
output = model(torch.randn(10, 3, 224, 224))
target = torch.randint(0, 1000, (10,))

criterion = nn.CrossEntropyLoss()

loss = criterion(output[0:1], target[0:1])
loss.backward()

[ashj (Ashish Kubade)]

@netaglazer
I believe if you are worried about the first dimension being the Batch index, pytorch automatically extracts the individual predictions and accumulated the loss as batch loss. So, you can write your loss function assuming your batch has only one sample.
@ptrblck could you please correct me if my understanding about loss function above is wrong?

[Rachel_Copperman]

Hi, Could you please help with some code I have?

[ptrblck]

Feel free to create a new topic with your problem description as well as your code, so that we can have a look.

[Rachel_Copperman]

opened

Loss function with small amount of positives

Hello, I have a binary classification model. My data is having 1 million samples, and only 100 of the are tagged with 1 (rest tagged with 0). My model now is a simple fully-connected. I’m having some problems with finding a loss function that will be able to overcome the small amount of true positives. I saw BCEWithLogitsLoss, but had some problems with it, so I used the next loss function I run into (which actually makes the same, I think) : def weighted_binary_cross_entropy(output, target…

[lycaenidae]

@ptrblck

loss = my_cross_entropy(x, y)

Do we need to implement forward pass for my_cross_entropy function?

[ptrblck]

my_cross_entropy is implemented as a simple function so you can just call it.
You could of course wrap it in an nn.Module and put the operations in the forward method, if that’s more convenient or if you need to store some internal states.

[abhinavtripathi95 (Abhinav Tripathi)]

Hello @ptrblck,
I am using a custom contrastive loss function as

def loss_contrastive(euclidean_distance, label_batch):
    margin = 100
    loss = torch.mean( (label_batch) * torch.pow(euclidean_distance, 2) +
                    (1-label_batch) * torch.pow(torch.clamp(margin - euclidean_distance, min=0.0), 2))

However, I get this error

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-95-9478fc9e762e> in <module>
----> 1 interp = Interpretation.from_learner(learn)

~/anaconda3/envs/pytorch/lib/python3.6/site-packages/fastai/train.py in from_learner(cls, learn, ds_type, activ)
    158     def from_learner(cls, learn: Learner,  ds_type:DatasetType=DatasetType.Valid, activ:nn.Module=None):
    159         "Gets preds, y_true, losses to construct base class from a learner"
--> 160         preds_res = learn.get_preds(ds_type=ds_type, activ=activ, with_loss=True)
    161         return cls(learn, *preds_res)
    162 

~/anaconda3/envs/pytorch/lib/python3.6/site-packages/fastai/basic_train.py in get_preds(self, ds_type, activ, with_loss, n_batch, pbar)
    339         callbacks = [cb(self) for cb in self.callback_fns + listify(defaults.extra_callback_fns)] + listify(self.callbacks)
    340         return get_preds(self.model, self.dl(ds_type), cb_handler=CallbackHandler(callbacks),
--> 341                          activ=activ, loss_func=lf, n_batch=n_batch, pbar=pbar)
    342 
    343     def pred_batch(self, ds_type:DatasetType=DatasetType.Valid, batch:Tuple=None, reconstruct:bool=False,

~/anaconda3/envs/pytorch/lib/python3.6/site-packages/fastai/basic_train.py in get_preds(model, dl, pbar, cb_handler, activ, loss_func, n_batch)
     44            zip(*validate(model, dl, cb_handler=cb_handler, pbar=pbar, average=False, n_batch=n_batch))]
     45     if loss_func is not None:
---> 46         with NoneReduceOnCPU(loss_func) as lf: res.append(lf(res[0], res[1]))
     47     if activ is not None: res[0] = activ(res[0])
     48     return res

TypeError: loss_contrastive() got an unexpected keyword argument 'reduction'

How to make the loss function compatible?

[ptrblck]

It seems you are using your custom loss function in FastAI, which apparently expects the reduction keyword for all loss functions.
A potential workaround would be to add the reduction argument,only accept 'mean' as a valid input type, and raise a NotImplementedError for other values.

[marziehoghbaie]

@ptrblck

def my_loss(output, target):
    loss = torch.mean((output - target)**2)
    return loss

Hi,
I use the same loss function, but I get this error: RuntimeError: Can only calculate the mean of floating types. Got Long instead.
Can somebody help me out?