Difference between .grad and ._grad

autograd
1

I notice that the Variable class has two similar members .grad and ._grad. What is the difference between them? In the pytorch a3c implementation pytorch-a3c, there is a piece of code:

def ensure_shared_grads(model, shared_model):
  for param, shared_param in zip(model.parameters(), shared_model.parameters()):
      if shared_param.grad is not None:
          return
      shared_param._grad = param.grad

Why update member ._grad of the shared_model?

2 Likes
2

Variable.grad is not writable, while Variable._grad is writable. If you want to force the value of the gradient to equal another value, you must modify ._grad and not .grad otherwise, you will have the exception:

AttributeError: attribute 'grad' of 'torch._C._VariableBase' objects is not writable
3 Likes
3

I just checked this using a simple script

import torch
import torch.nn as nn
from torch.autograd import Variable
from torch.optim import SGD

x = Variable(torch.zeros(3, 4), requires_grad=True)

a = (3 * x).sum()

oa = SGD([x], momentum=0.0, lr=1.0)

for i in range(100):
    oa.zero_grad()
    a.backward()
    print('-' * 40)
    print(x.grad)
    x._grad = Variable(torch.ones(x.grad.size()))
    print(x.grad)
    oa.step()
    print(x)

However this behavior seems undocumented…

1 Like
4

However this solution seems undocumented

I sense a solution for this :slight_smile: .

Screen Shot 2017-07-14 at 1.03.09 AM.png2106×366 56 KB

As a bonus, you then get the snazzy ‘contributor’ annotation after your name, in anything you post into the Issues page:

Screen Shot 2017-07-14 at 1.04.34 AM.png1636×296 56.5 KB

1 Like
5

it’s undocumented because it’s not meant to be used by the user. Generally all stuff starting with _ is treated as internal stuff.

2 Likes
6

If we want to manually set the gradient of a variable, would you recommend using ._grad ?

1 Like
7

I think soumith is pretty clear about this:

It is not meant to be used by the user.

8

So if you wanted to manually set the gradient of a variable, what would be the correct command?

9

A gradient is defined wrt some operations, it has a precise mathematical signification.

There is no point to manually modify it. You may want to modify the usage of the gradient while, for example, setting the gradient descent (in that case you can do something inheriting from torch.optim.Optimizer)

10

what about the case in my example on the top?

11

if you want to modify the gradient, use hooks. this tutorial in introduces hooks: http://pytorch.org/tutorials/beginner/former_torchies/nn_tutorial.html#forward-and-backward-function-hooks

12

so, our user should forget the existence of ‘._grad’, just use ‘.grad’ ?
thanks!

13

what do you mean?
if you do not specify the parameter “retain_graph” to True, isn’t the loss only can be backward one time?