Thanks for the response. At this point, it might be clarifying to point you to a thread that I started after this one that has a few more details on my problem: Higher-order gradients w.r.t. different functions.
I am currently doing two backward passes, but in principle it seems like I should be able to do just one.