BatchNorm epsilon factor tuning

Dear all,

I am using some model with BatchNorm2d with default eps=1e-5. But dumping the model trained parameters, it turns out that the running variances as a mean value of the order of 1e-6.

So the denominator of the BN sqrt(var + eps) is dominated by the eps factor.

What do you think ? should I put eps at 1e-7 level for instance ?