I would use the validation loss and would not use the test loss at all (neither for the scheduler, nor early stopping etc., as I would consider it a data leak).
Once your training is finished using the training and validation datasets, you would use the test dataset once and deploy the model.
1 Like