# CosineEmbeddingLoss() strange behaviour

I have been trying for a while to use CosineEmbeddingLoss() as the loss function for a network that I am working on. However, when I tried a simple example to check some of it’s features, I am running into some strange behaviour, as below.

If, for example, I do the following, then the behaviour is as expected…

``````>>> loss = torch.nn.CosineEmbeddingLoss()
>>> input1 = torch.rand(1,2,10,10)
>>> y3 = torch.ones(input1.size())
>>> loss(input1,input1,y3)
tensor(0.)
``````

However, if I define a random tensor and a tensor mask, and try again, I am geetting several weird results, as follows:

``````>>> loss = torch.nn.CosineEmbeddingLoss()
>>> a = torch.rand(3,3)
>>> a
tensor([[0.3415, 0.3722, 0.7526],
[0.4574, 0.1331, 0.1791],
[0.0287, 0.1917, 0.7043]])
>>> b = torch.tensor([[0.,0.,0.],[0.,1.,0.],[0.,0.,0.]])
>>> a * b
tensor([[0.0000, 0.0000, 0.0000],
[0.0000, 0.1331, 0.0000],
[0.0000, 0.0000, 0.0000]])
>>> y = torch.ones(a.size())
>>> loss(a,a,y)
tensor(0.)
>>> loss(a*b, a*b, y)
tensor(0.6667)
``````

When comparing a x b with a x b, we should be looking at exactly the same tensors in this case,hence the loss should be 0. However, it is not… Similarly, I get an identical answer if I do the following:

``````>>> a = torch.tensor([[0,0,0],[0,1,0],[0,0,0]])
>>> b = torch.tensor([[0,0,0],[0,5,0],[0,0,0]])
>>> y = torch.ones(a.size())
>>> loss(a,a,y)
tensor(0.6667)
>>> loss(a,b,y)
tensor(0.6667)
``````

The problem arises, as the cosine similarity between zero vectors is undefined.
In PyTorch `nn.CosineEmbeddingLoss` returns apparently a loss of 1 for zero tensors:

``````loss = torch.nn.CosineEmbeddingLoss(reduction='none')
b = torch.tensor([[0.,0.,0.],[0.,1.,0.],[0.,0.,0.]])
print(loss(b, b, torch.ones_like(b)))
> tensor([[1., 0., 1.],
[1., 0., 1.],
[1., 0., 1.]])
``````

which would return `0.6667` if you calculate its mean.

Thank you @ptrblck! You are a legend!

1 Like