# Pairwise cosine distance

**URL:** <https://discuss.pytorch.org/t/pairwise-cosine-distance/30961>\
**Category:** vision\
**Created:** [November 30, 2018, 1:12pm UTC](https://discuss.pytorch.org/t/pairwise-cosine-distance/30961 "2018-11-30T13:12:09Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![learnpytorch](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/learnpytorch/32/10072_2.png) [@learnpytorch](https://discuss.pytorch.org/u/learnpytorch)\
**Post date:** [November 30, 2018, 1:12pm UTC](https://discuss.pytorch.org/t/pairwise-cosine-distance/30961/1 "2018-11-30T13:12:09Z")

</div>

I want to find cosine distance between each pair of 2 tensors.  
That is given [a,b] and [p,q], I want a 2x2 matrix which finds  
[ cosDist(a,p), cosDist(a,q)  
cosDist(b,p), cosDist(b,q) ]  
I want to be able to use this matrix for triplet loss with hard mining.  
What is the best way to do this?

Thanks

---

<div class="post-metadata">

**Author:** ![InnovArul](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/innovarul/32/5282_2.png) [@InnovArul](https://discuss.pytorch.org/u/InnovArul)\
**Post date:** [November 30, 2018, 3:45pm UTC](https://discuss.pytorch.org/t/pairwise-cosine-distance/30961/2 "2018-11-30T15:45:45Z")

</div>

I am not sure about cosine distance. You might do something similar to below code.

For L2 loss, you might use this:

> <https://github.com/jiyanggao/Video-Person-ReID/blob/master/losses.py#L47-L89>

---

<div class="post-metadata">

**Author:** ![learnpytorch](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/learnpytorch/32/10072_2.png) [@learnpytorch](https://discuss.pytorch.org/u/learnpytorch)\
**Post date:** [November 30, 2018, 4:32pm UTC](https://discuss.pytorch.org/t/pairwise-cosine-distance/30961/3 "2018-11-30T16:32:24Z")

</div>

Thanks @InnovArul, I had been referring to this code. I was wondering if it is possible to use nn.CosineSimilarity() instead of computing the cosine similarity manually, just to be sure that there are no errors by me.

---

<div class="post-metadata">

**Author:** ![John1231983](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/john1231983/32/4040_2.png) [@John1231983](https://discuss.pytorch.org/u/John1231983)\
**Post date:** [October 18, 2019, 2:41am UTC](https://discuss.pytorch.org/t/pairwise-cosine-distance/30961/4 "2019-10-18T02:41:33Z")

</div>

@learnpytorch: Have you find the solution?

---

<div class="post-metadata">

**Author:** ![Deeply](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/deeply/32/3840_2.png) [@Deeply](https://discuss.pytorch.org/u/Deeply)\
**Post date:** [October 18, 2019, 1:58pm UTC](https://discuss.pytorch.org/t/pairwise-cosine-distance/30961/5 "2019-10-18T13:58:23Z")

</div>

> [@learnpytorch](#):
>
> Cosine distance between each pair of 2 tensors.

Something like:

```auto
import torch

def cosine_distance_torch(x1, x2=None, eps=1e-8):
    x2 = x1 if x2 is None else x2
    w1 = x1.norm(p=2, dim=1, keepdim=True)
    w2 = w1 if x2 is x1 else x2.norm(p=2, dim=1, keepdim=True)
    return 1 - torch.mm(x1, x2.t()) / (w1 * w2.t()).clamp(min=eps)

def cosine_similarity_n_space(m1=None, m2=None, dist_batch_size=100):
    NoneType = type(None)
    device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
    if type(m1) != torch.Tensor: # only numpy conversion supported
        m1 = torch.from_numpy(m1).float()
    if type(m2) != torch.Tensor and type(m2)!=NoneType:
        m2 = torch.from_numpy(m2).float() # m2 could be None
        
    m2 = m1 if m2 is None else m2
    assert m1.shape[1] == m2.shape[1]
    
    result = torch.zeros([1, m2.shape[0]])
    
    for row_i in range(0, int(m1.shape[0] / dist_batch_size) + 1):
        start = row_i * dist_batch_size
        end = min([(row_i + 1) * dist_batch_size, m1.shape[0]])
        if end <= start:
            break # cause I'm too lazy to elegantly handle edge cases
        rows = m1[start: end] 
        # sim = cosine_similarity(rows, m2) # rows is O(1) size        
        sim = cosine_distance_torch(rows.to(device), m2.to(device))
        
        result = torch.cat( (result, sim.cpu()), 0)
        
               
    result = result[1:, :] # deleting the first row, as it was used for setting the size only
    del sim
    return result.numpy() # return 1 - ret # should be used with sklearn cosine_similarity

```

---

<div class="post-metadata">

**Author:** ![John1231983](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/john1231983/32/4040_2.png) [@John1231983](https://discuss.pytorch.org/u/John1231983)\
**Post date:** [October 18, 2019, 2:53pm UTC](https://discuss.pytorch.org/t/pairwise-cosine-distance/30961/6 "2019-10-18T14:53:29Z")

</div>

Thanks but it too complex and time consuming

---

<div class="post-metadata">

**Author:** ![Oli](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/oli/32/9773_2.png) [@Oli](https://discuss.pytorch.org/u/Oli)\
**Post date:** [October 18, 2019, 2:55pm UTC](https://discuss.pytorch.org/t/pairwise-cosine-distance/30961/7 "2019-10-18T14:55:17Z")

</div>

I worked on this during the summer. Cosine worked well for me 🙂

Have a look at [this file](https://github.com/OlofHarrysson/imagecopyfinder/blob/master/triplet.py)

---

<div class="post-metadata">

**Author:** ![John1231983](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/john1231983/32/4040_2.png) [@John1231983](https://discuss.pytorch.org/u/John1231983)\
**Post date:** [October 18, 2019, 3:32pm UTC](https://discuss.pytorch.org/t/pairwise-cosine-distance/30961/8 "2019-10-18T15:32:01Z")

</div>

@Oli: You just used original cosine sim function. It only returns the triangle value of matrix. I want to return a paired sim, so it will be full matrix

---

<div class="post-metadata">

**Author:** ![Deeply](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/deeply/32/3840_2.png) [@Deeply](https://discuss.pytorch.org/u/Deeply)\
**Post date:** [October 18, 2019, 11:25pm UTC](https://discuss.pytorch.org/t/pairwise-cosine-distance/30961/9 "2019-10-18T23:25:24Z")

</div>

@John1231983; it isn’t time consuming if the GPU is enabled.

**Usage example:**

```auto
input1 = torch.randn(5, 7)
input2 = torch.randn(5, 7)

dist_mat = cosine_distance_torch(input1, input2, 2.4)
print(dist_mat )

```

---

<div class="post-metadata">

**Author:** ![John1231983](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/john1231983/32/4040_2.png) [@John1231983](https://discuss.pytorch.org/u/John1231983)\
**Post date:** [October 18, 2019, 11:36pm UTC](https://discuss.pytorch.org/t/pairwise-cosine-distance/30961/10 "2019-10-18T23:36:49Z")

</div>

Actually, I have input of 5D. so I sure that your way is time-comsuming

---

<div class="post-metadata">

**Author:** ![Deeply](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/deeply/32/3840_2.png) [@Deeply](https://discuss.pytorch.org/u/Deeply)\
**Post date:** [October 18, 2019, 11:39pm UTC](https://discuss.pytorch.org/t/pairwise-cosine-distance/30961/11 "2019-10-18T23:39:03Z")

</div>

In fact, it does not work for a 5D input.

---

<div class="post-metadata">

**Author:** ![John1231983](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/john1231983/32/4040_2.png) [@John1231983](https://discuss.pytorch.org/u/John1231983)\
**Post date:** [October 18, 2019, 11:40pm UTC](https://discuss.pytorch.org/t/pairwise-cosine-distance/30961/12 "2019-10-18T23:40:02Z")

</div>

@Deeply: I think torch.bmm is more simple and faster your way. But in my case. bmm is memory error

---

<div class="post-metadata">

**Author:** ![uniquefine](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/uniquefine/32/40651_2.png) [@uniquefine](https://discuss.pytorch.org/u/uniquefine)\
**Post date:** [July 26, 2021, 3:33pm UTC](https://discuss.pytorch.org/t/pairwise-cosine-distance/30961/13 "2021-07-26T15:33:55Z")

</div>

You can use this snippet. Cosine similarity is the same as the scalar product of the normalized inputs and you can get the pw scalar product through matrix multiplication.  
Cosine distance in turn is just 1-cosine\_similarity.

```auto
def pw_cosine_distance(input_a, input_b):
   normalized_input_a = torch.nn.functional.normalize(input_a)  
   normalized_input_b = torch.nn.functional.normalize(input_b)
   res = torch.mm(normalized_input_a, normalized_input_b.T)
   res *= -1 # 1-res without copy
   res += 1
   return res

```

---

<div class="post-metadata">

**Author:** ![ebee](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ebee/32/49275_2.png) [@ebee](https://discuss.pytorch.org/u/ebee)\
**Post date:** [May 5, 2022, 8:34am UTC](https://discuss.pytorch.org/t/pairwise-cosine-distance/30961/14 "2022-05-05T08:34:23Z")

</div>

[This github issue](https://github.com/pytorch/pytorch/issues/48306) gives a nice way to achieve this using the built in `cosine_similarity` function.  
Essentially:

```auto
th.nn.functional.cosine_similarity(x[:,:,None], x.t()[None,:,:])  

```

---

<div class="post-metadata">

**Author:** ![111470](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/111470/32/35079_2.png) [@111470](https://discuss.pytorch.org/u/111470)\
**Post date:** [May 24, 2023, 8:53am UTC](https://discuss.pytorch.org/t/pairwise-cosine-distance/30961/15 "2023-05-24T08:53:07Z")

</div>

I assume this solution is sample and clean:

since pairwise\_cosine\_similarity already achieved pairwise cosine distance compute, but do not support batch input. [Cosine Similarity — PyTorch-Metrics 0.11.4 documentation (torchmetrics.readthedocs.io)](https://torchmetrics.readthedocs.io/en/stable/pairwise/cosine_similarity.html)  
We can vmap this pairwise\_cosine\_similarity to make it aviliable for batch data.

```auto
# torch 1.13.0

import functorch
from torchmetrics.functional import pairwise_cosine_similarity
batched_pairwise_cosine_similarity= functorch.vmap(pairwise_cosine_similarity)

a=torch.randn(64,150,10)
b=torch.randn(64,100,10)

batched_pairwise_cosine_similarity(a,b) # [64,150,100]

```