Backpropagation through two concatenated networks

Hello Everyone !
I am doing one problem which aims in improving the semantic segmentation pixelwise results. The approach requires two networks (1. CNN 2. MLP), which are concatenated. The output of net1 i.e. CNN is 3 numbers , which are then serves as input to net2 i.e. MLP. The output of net2 (MLP) is a single number (not an array or higher dimensional tensor), at the output of net2, MSE Loss is calculated.
Now I have to backpropagate and update the gradients of both networks using the calculated MSE loss. The backpropagation of net2 is straight forward, i.e. loss.backward(), but how to do backpropagation of net1 using the gradients of net2 ?

1 Like

When you do loss.backward(), the gradients are calculated for both the networks and stored along with the parameters, assuming you are doing something like net2(net1(input)). Then according to how your optimizer is defined, when you do optimizer.step() the weights will get updated.

@Megh_Bhalerao
Thank you for answering. Please see the overview of the process
x=net1(input)
x1 = f(x) {some processing, means doing some summation operation, to calculate the input of net2}
y=net2(x1)
loss = |y-gt| {gt is ground truth}
loss.backward()
so, does this loss.backward(), can take care of calculation of gradients of both the networks. In optimizer, the parameters of both the networks will be passed.

Yes, to the best of my knowledge, because there is no reason the computational graph should break by doing the f(x). Yes, if you pass both the model’s parameters to the optimizer, the the loss.backward() will calculate the gradients of both the networks.