How Computation Graph in PyTorch is created and freed?

autograd
4

@smth
Thanks for your explanation, I have one more question, if I add retain_graph=True in the following code, will all intermediary results in 100 graph exist in memory, or previous 99 graphs are freed and only the last graph exist in memory?
import torch

for i in range(100):
    a = torch.autograd.Variable(torch.randn(2, 3).cuda(), requires_grad=True)
    y = torch.sum(a)
    y.backward(retain_graph=True)