CrossEntropy value changes with batch size at initial validation - is this a bug?

1

i have two models that differ only by the batch size, meaning they have identical weight initialization and all other parameters.

i perform initial validation, which means before any training to the models i simply run inference.

the cross_entropy loss for batch size = 3 is 0.9632

the cross_entropy loss for batch size = 1 is 0.8576

i’m not using any batch norm layers in my model, so this has no effect.

what else can cause such behavior?

2

Are you using other normalization layers, which might be size-dependent or e.g. dropout without calling model.eval()?
If not, could you post a minimal, executable code snippet showing this issue, please?

3

  • i’m using GroupNorm.

  • using dropout but calling model.eval() so shouldn’t take any effect

unfortunately i can’t share executable code at the moment, was hoping to get a general idea what might cause this behavior