Accuracy decreases after saving and loading

Hmm i see. I have no clue. Are you using same batch size?
You can try to compare if loaded state dict equals the one which provides good acc.

You may have a bug choosing when to save / from where to load.

Have a look at this:

Check batchnorm is ok