Is there any essential difference between * and + for Pytorch autograd?

1

I was trying to understand the autograd mechanism in more depth. To test my understanding, I tried to write the following code which I expected to produce an error (i.e., Trying to backward through the graph a second time).

b = torch.Tensor([0.5])
for i in range(5):
    b.data.zero_().add_(0.5)
    b = b + a
    c = b*a
    c.backward()

Apparently, it should report an error when c.backward() is called for the second time in the for loop, because the history of b has been freed， however, nothing happens.

But when I tried to change b + a to b * a as follows,

b = torch.Tensor([0.5])
for i in range(5):
    b.data.zero_().add_(0.5)
    b = b * a
    c = b*a
    c.backward()

It did report the error I was expecting.
This looks pretty weird to me. I don’t understand why there is no error evoked for the former case, and why it makes a difference to change from + to *.

2

Don’t use Tensor(...) or .data any more! They have been deprecated since PyTorch 0.4.

the reason you are not getting an error for the first is that the + backward doesn’t need any saved tensors while the multiplication needs the other argument to compute the input gradient from the output gradient. These are the ones getting freed in the backpropagation and so you get the error.

Best regards

Thomas

1 Like