Is this usage of MSELoss correct?

One is enough. The loss here has no (learnable) parameter (in case you define it as a module), it is just a function (F.mse_loss).

By the way, when you define it as a pytorch module, it is always F.mse_loss that is called in forward : pytorch/loss.py at master · pytorch/pytorch · GitHub
One of the purposes of the module is that you can do loss.to("device_name") as you did.