Pytorch only keeps the leaf gradients.
You could save them yourself using hook, as discussed here How do I calculate the gradients of a non-leaf variable w.r.t to a loss function?
Best regards
Thomas
1 Like