Always the same gradient at a warmstart model

I followed the pytorch tutorial for warmstarting a model (Saving and Loading Models — PyTorch Tutorials 2.2.0+cu121 documentation). But when I want to train the model and check the gradients, I always get the same gradient every iteration and the model gets worse every iteration. This is the code where I first save the model and the reload the model. Then I set up the optimizer for training

image