Clarification on computing loss and backprop for multiple input

how should i compute the loss in the following situation

In one forward pass i do:

Predicted_Values_1 = Model(input1)
Predicted_Values_2 = Model(input2)

in the above the same Model is used for both inputs.
now i do:

Loss_1 = loss(Predicted_Values_1,Expected_Values_1)
Loss_2 = loss(Predicted_Values_2,Expected_Values_2)

Can i consider my total loss as sum of Loss_1 and Loss_2 and then do back-propagation for it or i should do something else?

Also, is there any cases that this way of back-propagation wont work this easily and i gotta be careful?


Sum of loss and then do back-propagation would work.

But, It depends how you want to model your loss function and depending on the loss function, your model will perform. e.g. you can model loss as follows

total_loss = alpha * Loss_1 + (1-alpha) * Loss_2
total_loss = alpha * Loss_1 + beta * Loss_2

Where, alpha and beta are hyper-parameters.
When you have more than one loss, then usually we combine then using some function (Which will determine performance of your model) and then backprop.

Another example for loss function is following:
E.g. here, input1 and input2 were expected to give same output. In that case, you might want to penalize network more if Predicted_Values_1 and Predicted_Values_2 are different, Such a loss function will ensure network is trained to produce same output for both the inputs. (this loss function is on top of normal loss functions to achieve better results as similar output is expected and known)

Summary: Loss function should be modeled depending on what is the expected output of both the predicted values and how can we use them to make network learn faster.

1 Like

Thank you for your answer.
I just provided 2 losses for simplicity, in my code I have more than 2 losses.
I think i got the gist of it :slight_smile:

Just one more thing, can I have alpha and beta to be trainable as well? :question:

You need to tune alpha and beta with your cross validation set.

AutoML is framework which does this automatically for you. But just like other hyper-parameters, You will have to do it manually for now.

I am working on a project to update such hyperparameters as well from cross-validation set. Will update once I complete it. Till then, happy tuning!!

1 Like