How does one use the mean and std from training in Batch Norm?

5

after reading the BN code in detail and posts around here + the original paper the conclusion is here Inconsistent Batchnorm behavior in eval and training modes - #4 by Brando_Miranda in summary:

BN intended behaviour:

  • Importantly, during inference (eval/testing) running_mean, running_std is used (because they want a deterministic output and to use estimates of the population statistics).
  • During training the batch statistics is used but a population statistic is estimated with running averages. I assume the reason batch_stats is used during training is to introduce noise that regularizes training (noise robustness)

ref: [1502.03167] Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

So the main mystery is to figure out why my models were saved this way and their running averages from training removed.

meta-leanring context: Is there data leakage in the maml-omniglot example? · Issue #107 · facebookresearch/higher · GitHub