Computing Jacobian and Hessian of Loss Biases

what is the simplest way to compute full Jacobian and Hessian of Loss neural network biases only.
I want to do this layer wise but considering one layer at a time and I am not worried about speed for now.



You can check the implementation here: where you give only the bias as inputs.

This gives me error:
one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor []] is at version 2; expected version 1 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).

If I do recursive grad(grad), I don’t get the full hessian as we all already know.

Does replacing this:

grad_x, = torch.autograd.grad(flat_y, x, grad_y, retain_graph=True, create_graph=create_graph)


grad_x, = torch.autograd.grad(flat_y[i], x, retain_graph=True, create_graph=create_graph)

solves the error?

1 Like

No, now I get:
grad can be implicitly created only for scalar outputs

The y should be a single Tensor right?
So flat_y here should be a 1D Tensor. And flat_y[i] will be a single number. So it is a scalar output.
Is the error coming from this function?

Thanks, got it…
It works now.