# Gradient descent with nested functions

Edit: I spent a few minutes replacing the LaTeX with images but it was rejected because I am a new user. Sorry if it is difficult to read.

This is a code question where I am curious how to expand a simple example I am using to a more complex one using the autograd system in PyTorch. I am able to perform gradient descent with the following simple function I have written:

import torch

print(f"Initializing at {w}.")

weights = [w.tolist()]
costs = [nll(w).tolist()[0]]

for i in range(iterations):
nll_w = nll(w)
nll_w.backward()

weights.append(w.tolist())
costs.append(nll(w).tolist()[0])

return weights, costs


You just define a cost function (in this case, negative log-likelihood) with an unknown quantity you wish to estimate and with the appropriate learning rate you can find a weight estimate given some sample data.

I am trying to do something similar for a cost function with nested variable and I am not sure how to do it. I have the following cost function (fictitious):

$C(\theta, a, b) = \ln(5) - a^2 - \sin(5) - b^3$

where $a = \frac{\theta^2-5}{3}$ and $b = \frac{\theta^3-6}{7}$. Thus my cost function actually has variables I wish to find minimum values for that themselves can be expanded into a fraction.

I want to find perform gradient descent where autograd determines the partial derivatives for $\frac{\partial C}{\partial \theta}$, $\frac{\partial C}{\partial a}$, and $\frac{\partial C}{\partial b}$ and performs gradient descent to determine the minima.

The issue is that if I define my cost function in terms of the three variables as shown above, then the cost function has no knowledge that $a$ and $b$ are actually fractions that contain $\theta$. But if I expand out my cost function, replacing $a$ and $b$ with their actual values, then the cost function is no longer using the inputs $a$ and $b$ anywhere and the gradients cannot be calculated.

So my question is about how to modify my code to achieve this.