Hello,
I am working on domain adaptation for regression. So my predictive loss function is mean square loss and domain loss is binary cross-entropy loss. When I am training the network by jointly optimizing both the loss functions with the same optimizer and learning rate, my mean square loss converges soon and domain loss gets saturated in a few epochs and does not get converged latter. Therefore, I am not getting an improvement in performance using domain adaptation. COuld you please help me on how to fix this problem. Should I use different learning rates for the loss functions? If so how should I choose the learning rate decays of bothe loss functions and how can I use different learning rates. Should I initialize different optimizers? You help will be greatly appreciated.