evgenizer
(ESorkin)
December 28, 2018, 7:25pm
1
I am confused: can you help me understand why does torch.norm(x) depend on the size of x? Thanks!
N=10000
x=torch.ones(N)
n1=torch.norm(x)
n2=numpy.sqrt(N)
[output]: n1=tensor(100.), n2=100.
N=40000
x=torch.ones(N)
n1=torch.norm(x)
n2=numpy.sqrt(N)
[output]: n1=tensor(266.0605) n2=200.
smth
December 29, 2018, 1:37am
2
We calculate the Vector 2-norm by default. And that is a function of the total sum, by definition:
http://mathworld.wolfram.com/L2-Norm.html
However, once you hit floating point precision limits, you get “inaccurate” results…
smth
December 29, 2018, 1:40am
3
I’m taking a look at what these accuracy limits are, wrt floating point, and if that might be affecting things.
evgenizer
(ESorkin)
December 29, 2018, 1:40am
4
I know that, but for x=torch.ones(40000), the answer should be 200, not 266.0605 as I get with torch.norm(x)
smth
December 29, 2018, 1:43am
5
I’m sorry, I posted my answer as I was typing it, by mistake.
I looked into this further, it looks like a bug that we introduced in 1.0.0, I’m taking a further look and filing an issue.
It is only affected on the CPU, and produces the correct result on GPU.
evgenizer
(ESorkin)
December 29, 2018, 1:45am
6
Got it, I think gpu is affected as well, though.
smth
December 29, 2018, 1:47am
7
I checked the GPU implementation via:
>>> import torch
>>> x = torch.ones(40000)
>>> torch.norm(x)
tensor(266.0605)
>>> torch.norm(x.to(dtype=torch.float32, device='cuda'))
tensor(200., device='cuda:0')
Seems to work fine.
Are you seeing incorrect result on GPU as well?
evgenizer
(ESorkin)
December 29, 2018, 1:50am
8
Double checked it – no, you are right, GPU result is correct. Thanks!
smth
December 29, 2018, 1:57am
9
I filed an issue at https://github.com/pytorch/pytorch/issues/15602
It will for sure be fixed in our next minor release on Jan 15th, and will be fixed in our nightlies much sooner than that, I’m having it looked at with high priority.
Really sorry for the bug!
1 Like
smth
January 16, 2019, 11:23pm
10
just fyi, this is fixed now via https://github.com/pytorch/pytorch/pull/15885 and will go into the 1.0.1 release in the time window of < 1 week from now.
1 Like