What's the difference between retain_graph and create_graph?

[onlytailei (lei.tai)]

So retrain_graph is used for the second time backward?
and create_graph is used for higher order derivative of graph parameters?

[bpetruzzo (Bernhard Petersch)]

I’m also interested to know!

It seems that calling torch.autograd.grad with BOTH set to “True” uses (much) more memory than only setting retain_graph=True. In the master docs retain_graph mentions the “graph used to compute the grad”, and create_graph mentions that the “graph of the derivative will be constructed, allowing to compute higher order derivative products”.
I found the choice of name for those flags a bit confusing, since it seemed logical on first sight that in order to retain the graph it also need to be created. On second thought and rereading the wording in the docs it seems though that those flags concern two different graphs, as you also asked/suggested @onlytailei.

I think this is the confirmation:

@albanD

Hi,

For this to work you need to pass the create_graph=True option to the first .backward() to let it know that you need to be able to call .backward() on the grad itself.
Also if you need the gradient wrt a specific variable, you can use the autograd.grad(outputs, inputs) function to get the derivative of the output(s) wrt the input(s). For example out.backward() is equivalent to autograd.grad(out, model.parameters()) .