How to combine 2 networks, but back-prop only on one of them


If I have 2 networks, the output of the first network is passed into the 2nd network.
I would run back-prop on an objective function using output of the 2nd network. However, I only want to update the first network and leave the 2nd network alone (by minimization). Later on, I want to update only the 2nd network and leave the 1st alone (by maximization).

Right now, I am thinking of just reconstructing the network each time and copying the weights, but I was wondering if there’s a better way to do this?


I think the simplest way is to have two different optimizers.
Each one work with the parameters of one of the two networks. Then you do your forward/backward as usual and just call .step() for the optimizer of the network you currently want to update.

1 Like

Hello, I think that’s a great idea!! Thanks.