So, what I understand is that if you pass input x to model in case of model.train() everything is ok but for the same input x, model.eval() is giving worse result. model.eval() only affect layers such as batchnorm and dropout, so I guess you have such layers in your model definition. I think this post in that case can help alot as this is a tricky challenge sometimes:
There are few tricks to solve issue in the aforementioned thread. I hope it helps.