Torch.nn.functional.cosine_similarity got unexpected behavier with torch.int8 dtype

projects
1

PyTorch version: 1.9.0

torch.nn.functional.cosine_similarity got unexpected behavier with torch.int8 dtype.

>>> a = torch.randn(10)*100
>>> a
tensor([ 47.8993, -27.2694,  20.9548, -13.5573, -54.0388, -76.8524,  -7.4037,
        -95.7477, 108.9276, -44.5625])
>>> b = a
>>> F.cosine_similarity(a, b, dim=-1)
tensor(1.)
>>> F.cosine_similarity(a.to(torch.int8), b, dim=-1)
tensor(3.5215e+12)
>>> F.cosine_similarity(a.to(torch.int16), b, dim=-1)
tensor(1.0000)

As we can see, F.cosine_similarity(a.to(torch.int8), b, dim=-1) got result of tensor(3.5215e+12), while the expected result is tensor(1.).

2

This looks like an overflow using int8 internally. Would you mind creating an issue on GitHub so that we could track and fix it?

3

ok, I have created an issue: torch.nn.functional.cosine_similarity got unexpected behavier with torch.int8 dtype · Issue #61454 · pytorch/pytorch · GitHub

1 Like