You could apply a manual weighting as seen in this post and optimize the weights during the training.
However, I would assume the model could try to move the weight values to zeros or large negative values to decrease the weighted loss, so you might need to e.g. normalize them or add a penalty.