Does CosineEmbeddingLoss support multidimensional input tensor?

1

Suppose I have a feature map of size 10x10, where each element is a 2D embedding, and I want to calculate the cosine embedding loss of two such tensors (2x10x10).

I tried the following approach,

loss_func = nn.CosineEmbeddingLoss()
a = Variable(torch.randn([1,2,10,10]), requires_grad=True)
b = Variable(torch.randn([1,2,10,10]), requires_grad=True)
c = Variable(torch.from_numpy(np.ones([1,10,10])), requires_grad=False)
loss = loss_func(a, b, c)
loss.backward()

however, an error popped out,

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ryli/.local/lib/python2.7/site-packages/torch/autograd/variable.py", line 167, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, retain_variables)
  File "/home/ryli/.local/lib/python2.7/site-packages/torch/autograd/__init__.py", line 99, in backward
    variables, grad_variables, retain_graph)
  File "/home/ryli/.local/lib/python2.7/site-packages/torch/autograd/function.py", line 91, in apply
    return self._forward_cls.backward(self, *args)
  File "/home/ryli/.local/lib/python2.7/site-packages/torch/autograd/function.py", line 205, in wrapper
    outputs = fn(ctx, *tensor_args)
  File "/home/ryli/.local/lib/python2.7/site-packages/torch/nn/_functions/loss.py", line 74, in backward
    _idx = _idx.view(-1, 1).expand(gw1.size())
RuntimeError: The expanded size of the tensor (10) must match the existing size (100) at non-singleton dimension 2. at /pytorch/torch/lib/TH/generic/THTensor.c:309

If I did everything correctly, does this mean that CosineEmbeddingLoss currently does not support multidimensional embedding? If so, is there a better work-around than explicitly calculating the loss for every location and summing them up?

Thanks in advance!

System information:

  • OS: Linux
  • PyTorch version: 0.3.0.post4
  • Python version: 2.7.12
2

Are you trying to treat the (10x10) tensors as (100) tensors for the purpose of cosine similarity? In that case you can use view() to create a view that is of the correct shape.
http://pytorch.org/docs/master/tensors.html#torch.Tensor.view

However, the shape of your label tensor c suggests that you might be trying to take cosine similarity along the 1st dimension (0 indexed). In that case you could use permute() or transpose().
http://pytorch.org/docs/master/tensors.html#torch.Tensor.transpose

3

I have the same question, but in an even simpler case. How to get the loss comparing 2 batches of strings…

This works fine…

    def test_cosine_embedding_loss(self):
        seq = Variable(torch.randn(1, 5).double())
        model = nn.Linear(5, 5).double()
        target = Variable(seq.data.clone())
        output = model(seq)
        criterion = torch.nn.CosineEmbeddingLoss()
        mask = Variable(torch.ones(1), requires_grad=False)
        loss = criterion(output, target, mask)
        print(loss)
        loss.backward()

But this fails…

    def test_cosine_embedding_loss(self):
        seq = Variable(torch.randn(1, 1, 5).double())
        model = nn.Linear(5, 5).double()
        target = Variable(seq.data.clone())
        output = model(seq)
        criterion = torch.nn.CosineEmbeddingLoss()
        mask = Variable(torch.ones(1, 1), requires_grad=False)
        loss = criterion(output, target, mask)
        print(loss)
        loss.backward()

Error

RuntimeError: inconsistent tensor size, expected src [1 x 5] and mask [1 x 1] to have the same number of elements, but got 5 and 1 elements respectively at d:\pytorch\pytorch\torch\lib\th\generic/THTensorMath.c:197

Ok… so then. like the error says do…

    def test_cosine_embedding_loss(self):
        seq = Variable(torch.randn(1, 1, 5).double())
        model = nn.Linear(5, 5).double()
        target = Variable(seq.data.clone())
        output = model(seq)
        criterion = torch.nn.CosineEmbeddingLoss()
        mask = Variable(torch.ones(1, 5), requires_grad=False)
        loss = criterion(output, target, mask)
        print(loss)
        loss.backward()

And we get…

RuntimeError: The expanded size of the tensor (1) must match the existing size (5) at non-singleton dimension 1. at d:\pytorch\pytorch\torch\lib\th\generic/THTensor.c:309

There’s not a lot in the documenation that says what the expected dims of the 3rd parameter is supposed to be… Anyone got this working?

4

Cosine embedding loss only supports 2-D tensors. The tensors you’re trying to pass in are 3-D. (I’ll take a look to see if we should make this clear in docs, error messages, and if we should support higher dim)

5

Thanks Roy,

3D would make sense in the case of batches of strings

for example…

(batch, sequence, character_embeddings)

I guess to do this in 2D, you would split the batch dimension and compute the loss one at a time?

6

Yeah I think it makes sense to support higher dimensions. I’ll add it in at some point. In the meantime, you can use .view() to create a view of your tensor that is of the correct shape.