Did you try to use different seeds without the validation loop and check how the training behaves?
I guess your training might not be very stable so that calling into the pseudo-random number generator during validation would change the next random values and your model would diverge during the training.
Here is an explanation of the RNG behavior during the validation loop, if a DataLoader is used with a workatround. Also here is a similar issue to yours.