I’m working in Binary Classification with 1d Convolutional Network. For research purposes I’ve to implement my own Loss Function which is similar to BCELoss. For that reason I’ve started trying to implement my own BCE loss function with autograd:
class DiscriminatorLoss(torch.autograd.Function): @staticmethod def forward(ctx,d_out,labels): loss = labels*torch.log(d_out)+(1-labels)*torch.log(1-d_out) ctx.d_out,ctx.labels = input,labels return loss @staticmethod def backward(ctx, grad_output): d_out,labels = ctx.d_out,ctx.labels grad_input = -labels/d_out + ((1-labels)/(1-d_out)) return grad_input,None
labels are tensor like:
d_out=tensor([[0.5412, 0.5225], | labels=tensor([[0, 1], [0.5486, 0.5167], | [0, 1], [0.5391, 0.5061],...])| [0, 1],...])
However, this doesn’t work properly. The problem is that in the middle of the training process, the outputs of the net (
d_out) turn to strange values like:
tensor([[9.9000e-08, 9.9000e-01], [9.9000e-08, 9.9000e-01], [9.9000e-08, 9.9000e-01],....])
And it get stuck there for the rest of the training.
I’ve also implemented the BCELoss function from Pytorch
nn.BCELoss() (https://pytorch.org/docs/stable/nn.html#loss-functions). And with that function, the net works so I believe that the problem is in my loss function. To be more exact, the forward() works well as it returns the same loss as nn.BCELoss. So the problem is in backward().
Can anyone help me? What I’m doing wrong in backward() function?
PS.: The outputs of the net are treat to not be exactly 0 or 1 in order to not generate
-inf values in Cross Entropy Loss.