What is the meaning of function name grad_fn returns

autograd
1

Hi all,

I’m kind of new to PyTorch. I found it very interesting in 1.0 version that grad_fn attribute returns a function name with a number following it. like

>>> b
tensor([[2., 2.],
        [2., 2.]], grad_fn=<AddBackward0>)

I want to know the meaning of that 0 so I tried some more

The first different number I found is to calculate the mean of the tensor. It gives

>>> c=torch.mean(b)
>>> c
tensor(2., grad_fn=<MeanBackward1>)

The number after the function name becomes 1. When I explicitly give the dimension I want the mean, it gives

>>> d = torch.mean(b, dim=1)
>>> e = torch.mean(b, dim=0)
>>> d
tensor([2., 2.], grad_fn=<MeanBackward0>)
>>> e
tensor([2., 2.], grad_fn=<MeanBackward0>)

The number became 0 again!

So I tried something similar, the standard deviation! However this time it runs slightly different

>>> f = torch.std(b)
>>> f
tensor(0., grad_fn=<StdBackward0>)

>>> g = torch.std(b, dim=1)
>>> h = torch.std(b, dim=0)
>>> g
tensor([0., 0.], grad_fn=<StdBackward1>)
>>> h
tensor([0., 0.], grad_fn=<StdBackward1>)

It’s just the opposite to what happened in torch.mean. When I specified the dimension the nubmer becomes 1 and when I didn’t the nubmer became 0.

These number, in basic algebra functions like addition and subtraction, only shows up in version 1.0.1. More complicated functions like mean and std perform exactly like the previous version (0.4.1). Following examples are calculated in PyTorch 0.4.1

>>> a = torch.zeros(2,2,requires_grad=True)
>>> b = a+2
>>> b
tensor([[2., 2.],
        [2., 2.]], grad_fn=<AddBackward>)

>>> torch.mean(b)
tensor(2., grad_fn=<MeanBackward1>)
>>> torch.std(b)
tensor(0., grad_fn=<StdBackward0>)
>>> torch.std(b,dim=1)
tensor([0., 0.], grad_fn=<StdBackward1>)
>>> torch.mean(b,dim=1)
tensor([2., 2.], grad_fn=<MeanBackward0>)

So could someone who is familiar with PyTorch autograd mechanics explain the meaning of these gradient function names? Thanks in advance!

5 Likes