Using previous output values as part of a loss function

I am trying to implement TRPO, and I need the gradient of the network parameters w.r.t. the KL divergence between the current action distribution and the action distribution after the parameters are changed.

Put simply, we have:
action_dstribution0 <-- Network(state)

We need to calculate something to the effect of:
autograd.backward(KL(Network(state), action_distribution0), [torch.ones(1)])

What is the best way to implement this? How do I make sure action_distribution0 doesn’t get backpropagated through?

It sounds very similar to Q-learning - just use .detach() to prevent the gradients from getting propagated:

action_distribution0 = model(state).detach()
# detach() blocks the gradient
# also action_distribution0 doesn't require grad now, so it can be a loss target

# =========================================
# change the parameters of the network here
# =========================================

KL(Network(state), action_distribution0).backward()

This worked great, thanks!

In the dcgan example, the G(z) output is copied over before pushing through D (link).
Is this done in order to avoid backpropagation through G?
Would this be an equivalent way of doing things?

fake = netG(noise)
fake = fake.detach() # this is not an inplace op, so need to re-assign?
output = netD(fake)

yes, detaching sounds better. When I wrote that example, .detach() was not available. Please send a PR to fix that :slight_smile: