If I have a single network that is outputting two sets of outputs, one that performs a classification operation and another that performs a regression operation, how would I go about computing the loss and backpropagating? I understand I could use two different loss functions add them and then call backward, however regression and classification losses have different units, so would this skew my results in an unpredictable way? How can I go about training a single network to output these two types of outputs?
One way is to take a weighted sum of the regression loss and classification loss.