Activation are slightly alterd even in .eval() depending on the batch

1

Hey Guys,

I am using multiple nn.BatchNorm1d() layers. I noticed that the activation of the same inputs change slightly when I include the input in different batches. Even when I am using model.eval().

So I have input A and I feed it to the model in the same batch as the the inputs B and C. The activation of A will differ to the Activation of A when I fed it through the network together with B C, D, E.

Is this due to the BatchNorm I feel like model.eval(), should prevent this behavior. But maybe I am messed up somewhere.

Would there be a option to prevent this behavior

2

How large are the relative and absolute errors?
Note that you might run into the limited floating point precision e.g. due to different batch sizes, different algorithms etc. You could try to use float64 and see if the error would be lower (and if you would really need it).