How to train an ML to predict improvement in patient recovery?

This is not specific programming or API-related question but more of modeling and framing of the problem statement in to PyTorch models.

I am interested in training an ML model that would predict improvement in a patient’s injury.

Assume doctors have collected two sets of MRI images from hundreds of patients who have suspected brain injury. The first set was taken at 2 weeks after the injury and the second set was taken 6 months after the injury. Additionally, doctors have their own assessment of the injury (say a score indicating the strength of the injury). So the patient’s recovery from the injury can be better (increase in the score and hence positive) or worse (decrease in the score and hence negative) from 2 weeks to 6 months.

How to frame this problem so that I can train an ML model that takes the 2 weeks image and 6 months image as its inputs and predicts if the recovery is positive or negative? I can use the injury score provided by the doctors as a ground truth. But what sort of loss function to use to train this model in an end-to-end fashion?

I know I need an ML model that takes in two inputs X and X' and outputs a score s and the score should be any positive number if the recovery is good or any negative number if the recovery is bad. I am not sure what loss function to use so that I can train the model. Note that X represents the MRI scan taken at 2 weeks and X' represents the MRI scan taken at 6 months.

I think the easy way is to approach this as [binary] classification problem, with added probability calibration after training.

If you want to also utilize the magnitude of change of doctor scores, perhaps KLDivLoss could be used (map value to “soft” targets like (0.9, 0.1) by some ad-hoc formula, calibrating doctors’ scores).