Cannot calculate second order gradients even though `create_graph=True`

The graph was created and loss is attached to it.
Using torchviz you would get this graph for the first forward pass with alpha included:

graph_with_alpha

and this graph for the second part:

graph2_wo_alpha

As you can see, alpha is in the graph in the last multiplication.
However, these graphs are not depending on each other, as they were created in two separate forward passes and new input.

1 Like