CosineEmbeddingLoss() strange behaviour

1

I have been trying for a while to use CosineEmbeddingLoss() as the loss function for a network that I am working on. However, when I tried a simple example to check some of it’s features, I am running into some strange behaviour, as below.

If, for example, I do the following, then the behaviour is as expected…

>>> loss = torch.nn.CosineEmbeddingLoss()
>>> input1 = torch.rand(1,2,10,10)
>>> y3 = torch.ones(input1.size())
>>> loss(input1,input1,y3)
tensor(0.)

However, if I define a random tensor and a tensor mask, and try again, I am geetting several weird results, as follows:

>>> loss = torch.nn.CosineEmbeddingLoss()
>>> a = torch.rand(3,3)
>>> a
tensor([[0.3415, 0.3722, 0.7526],
        [0.4574, 0.1331, 0.1791],
        [0.0287, 0.1917, 0.7043]])
>>> b = torch.tensor([[0.,0.,0.],[0.,1.,0.],[0.,0.,0.]])
>>> a * b
tensor([[0.0000, 0.0000, 0.0000],
        [0.0000, 0.1331, 0.0000],
        [0.0000, 0.0000, 0.0000]])
>>> y = torch.ones(a.size())
>>> loss(a,a,y)
tensor(0.)
>>> loss(a*b, a*b, y)
tensor(0.6667)

When comparing a x b with a x b, we should be looking at exactly the same tensors in this case,hence the loss should be 0. However, it is not… Similarly, I get an identical answer if I do the following:

>>> a = torch.tensor([[0,0,0],[0,1,0],[0,0,0]])
>>> b = torch.tensor([[0,0,0],[0,5,0],[0,0,0]])
>>> y = torch.ones(a.size())
>>> loss(a,a,y)
tensor(0.6667)
>>> loss(a,b,y)
tensor(0.6667)

Can someone please help me understand why this behaviour is occuring? Thank you!

2

The problem arises, as the cosine similarity between zero vectors is undefined.
In PyTorch nn.CosineEmbeddingLoss returns apparently a loss of 1 for zero tensors:

loss = torch.nn.CosineEmbeddingLoss(reduction='none')
b = torch.tensor([[0.,0.,0.],[0.,1.,0.],[0.,0.,0.]])
print(loss(b, b, torch.ones_like(b)))
> tensor([[1., 0., 1.],
        [1., 0., 1.],
        [1., 0., 1.]])

which would return 0.6667 if you calculate its mean.

3

Thank you @ptrblck! You are a legend! :slight_smile:

1 Like