How does torch.normal work with a negative standard deviation?

1

I saw some code in a repo that was using torch.normal() with a possibly negative standard deviation. I thought the code was wrong and that torch.normal() would surely give an error with a negative standard deviation, but when I tried it, it just worked with the resulting sample being positive…

I had trouble finding the exact implementation in the pytorch library so I was unable to see the actual code to see what it is doing, it seems like it just takes the absolute value, but I am not sure. How does this work?

2

Hi

As far as I can remember you cannot use negative value for std and you should not be able to use as it does not make any sense mathematically.

To just verify, I ran it and it gives a proper error:

torch.manual_seed(8)
torch.normal(0, -1, (10, ))

# output
# RuntimeError                              Traceback (most recent call last)
# <ipython-input-6-f3bb1e6c1119> in <module>()
    #   1 torch.manual_seed(8)
# ----> 2 torch.normal(0, -1, (10, ))

# RuntimeError: normal_ expects std > 0.0, but found std=-1

Bests

3

hmm… I see. That is what I would expect to see. Your method does give an error, but I was using a slightly different argument.

I was doing it this way without error…

mu = torch.tensor(0.0)
sigma = torch.tensor(-3.0)
torch.normal(mu, sigma)

# ...output
>>> torch.normal(mu, sigma)
tensor(-3.6977)
>>> torch.normal(mu, sigma)
tensor(-3.0448)
>>> torch.normal(mu, sigma)
tensor(-0.2327)

Any idea why the arguments set like this are valid? If I add a size argument to the above code it fails with an error. But without the size argument, it seems to work.

4

Hi Jeff (and Doosti)!

For what it’s worth, I can reproduce your and Doosti’s results on both
pytorch 1.6.0 and on 1.8.0.dev20201203. That is, I get the “normal_
expects std > 0.0” error when I call torch.normal() with the third (size)
argument, but not when I don’t.

I suppose you could say that it’s not a bug when software fails to issue
an error when the user does something wrong. However, pytorch is
generally pretty liberal about issuing invalid-argument errors, and here
we have the inconsistent reporting of the error based on exactly how
the function is called. So I would call it a bug.

Not that is matters, but as to what torch.normal() actually does:

It appears that torch.normal(), if effect, generates a normal deviate
(a sample from a normal distribution with mean zero and standard
deviation one), multiplies it by the passed-in sigma, and then shifts
it by the passed in mu. This is statistically the same as just using
abs (sigma), but gives specific values that differ. Here is a quick
example:

>>> torch.__version__
'1.6.0'
>>> torch.random.manual_seed (2020)
<torch._C.Generator object at 0x7f0d2dc95910>
>>> torch.normal (torch.tensor([0.0]), torch.tensor ([10.0]))
tensor([12.3723])
>>> torch.random.manual_seed (2020)
<torch._C.Generator object at 0x7f0d2dc95910>
>>> torch.normal (torch.tensor([0.0]), torch.tensor ([-10.0]))
tensor([-12.3723])

Best.

K. Frank

5

Thanks for the detailed explanation @KFrank and checking the code @Nikronic!
@jwillette could you create a GitHub issue so that we could make sure the methods raise the same error?

6

I already started one here torch.normal only issues error for negative sigma when shape argument is given · Issue #49998 · pytorch/pytorch · GitHub

1 Like