I find it better to just
del loss, model, optimizer
torch.cuda.empty_cache()
And then recreate the model and optimizer how you did in the first place.
Note that not deleting the loss resulted in cuda memory errors as it is connected to the model’s gradients graphs and thus keeps the whole thing in memory.