CrossEntropy value changes with batch size at initial validation - is this a bug?

i have two models that differ only by the batch size, meaning they have identical weight initialization and all other parameters.

i perform initial validation, which means before any training to the models i simply run inference.

the cross_entropy loss for batch size = 3 is 0.9632

the cross_entropy loss for batch size = 1 is 0.8576

i’m not using any batch norm layers in my model, so this has no effect.

what else can cause such behavior?

Are you using other normalization layers, which might be size-dependent or e.g. dropout without calling model.eval()?
If not, could you post a minimal, executable code snippet showing this issue, please?

  • i’m using GroupNorm.

  • using dropout but calling model.eval() so shouldn’t take any effect

unfortunately i can’t share executable code at the moment, was hoping to get a general idea what might cause this behavior