First and second derivates of the output with respect to the input inside a loss function

Everything looks fine to me.
Maybe computation of derivatives is causing the issue.

In this post I have references to a notebook that solves possion equation which contains solving second order PDEs. I hope it helps.