Call .backward() to clear graph

[Yoo]

I was reading this article in which it says the graph will be cleaned in the step loss.backward():

Why do we need to set the gradients manually to zero in pytorch?

Here are three equivalent code, with different runtime/memory comsumption. Assume that you want to run sgd with a batch size of 100. (I didn’t run the code below there might be some typos, sorry in advance) 1: single batch of 100 (least runtime, more memory) # some code # Initialize dataset with batch size 100 for input, target in dataset: pred = net(input) loss = crit(pred, target) # one graph is created here opt.zero_grad() loss.backward() # graph is cleared here …

Is there any way that I could get access to the underlying graph to make sure it is freed?
I tried to check loss.grad_fn and found it stayed the same after backward(), although the id changed

x = torch.tensor([0.5, 0.75], requires_grad=True)
y = torch.tensor([0.1, 0.90], requires_grad=True)
z = torch.exp(x * y).sum()
print(z.grad_fn)
z.backward() 
print(z.grad_fn)

[srishti-git1110 (Srishti Gureja)]

@Yoo

the graph will be cleaned in the step loss.backward()

What this strictly means is the the references to the saved tensors are lost but the underlying graphs still hangs on in the memory. Are you asking something along the lines?
This might be the reason why the grad_fn remains intact as you checked.

[Yoo]

@srishti-git1110

. Are you asking something along the lines?

Hi thank you. I was asking if I could check the references were indeed lost after I call backward(), like I detach a tensor and I could clearly see the difference

And how this step will save the memory if the graph still hangs there?

[srishti-git1110 (Srishti Gureja)]

@Yoo

I was asking if I could check the references were indeed lost after I call backward()

Yes. The references to the saved tensors are definitely lost after a backward call unless you specify retain_graph=True as an argument to the backward method which you shouldn’t unless necessary.

To see this, try a second backward call right after the first, and the error message shall convey the same.

@Yoo

how this step will save the memory

As the references to the saved tensors are lost, memory is aggressively freed.

[Yoo]

What does “references” explicitly mean? Does it mean saved intermediate values of the graph?

[srishti-git1110 (Srishti Gureja)]

How things work internally (what data structures are used etc.) is something I’m not completely sure about.
But yes, “references” to the tensors that’ll be required for gradient computation on a backward call are saved as the graph builds in the forward pass.

You might also like to read more on how PyTorch uses dynamic computational graphs.