Loss goes up after loading checkpoint

Can you load the checkpoint, freeze all parameters, and perform inference-like training to see the resulting loss?