Custom loss(weighted angular distance) in pytorch 0.4.1

autograd
1

Hi,

I am creating my own custom loss in pytorch, this loss is based on angular distance using the cosine similarity. The difference is that in my case I added a weight factor in order to magnify the error in some cases. The thing is, the network I am trying to train is not working as expected. For this reason I would like to know if the implementation that I made in pytorch is correct. I am using pytorch 0.4.1, I read some topics on this forum related on the custom losses with old versions of pytorch. Far as I saw one of the common error is the unpack of the Variables during the calculation , but in pytorch 0.4.1 this does not matter, right? I think my implementation is correct but maybe I miss some details of how this implementation needs to be done in pytorch. I would like to be sure that the error is not something related to my understanding of the implementation of losses in pytorch. Thanks in advance.

def weighted_angular_MSE(input,target,weight)
	input = torch.mul(F.normalize(input,p=2,dim=1),0.99999)
	target = torch.mul(F.normalize(target,p=2,dim=1),0.99999)
	
	error =torch.div(torch.acos(torch.sum(torch.mul(input,target),dim=1,keepdim=True)),math.pi)

	return torch.div(torch.sum(torch.mul(torch.pow(error,2)[0,:],weight)), torch.nonzero(weight.data).size(0))

class WeightedAngularMSE(nn.Module):
      def __init__(self):
      	super(WeightedAngularMSE,self).__init__()
      def forward(self, input, target, weight):
      	return weighted_angular_MSE(input,target,weight)
2

As long as you are using PyTorch methods, the computation should be fine.
I can’t see any obvious error in your code.

As long as weights has some non-zero values, the division should also be fine.
What kind of strange behavior are you experiencing?

3

Thanks for your comments @ptrblck! What happens is that my network is not learning, as you can see in the following image:

But if the loss function is correct, I think the problem may be something else, like the network is not good enough for my problem or the dataset. I just wanted to make sure that I was not making any silly mistakes by implementing my loss function in pytorch.

4

Well, the code looks alright.
Regarding the validity of the code, it seems the PyTorch implementation uses the dot product and vector norms to calculate the cosine while your code seems a bit more complicated.

Are you sure this line is correct:

torch.acos(torch.sum(torch.mul(input,target),dim=1,keepdim=True))

I have my doubts about the torch.acos as currently input and target are just normal vectors, i.e. both should be normalized with the product of both norms.
Could you check it and see if I’m mistaken?

1 Like
5

@ptrblck, you are completly right. It was a big mistake from my side. Thanks for see it! At this moment I am running the experiment again. To avoid future issues I am using cosine_similarity of pytorch. Thanks again!

1 Like