I went through the code of batchnorm.py and found this statement in batchnorm.py:
line 78 in pytorch 1.3.1:
return F.batch_norm( input, self.running_mean, self.running_var, self.weight, self.bias, self.training or not self.track_running_stats, exponential_average_factor, self.eps)
Maybe this statement should be
self.training and self.track_running_stats according to the documentation
If track_running_stats is set to False, this layer then does not keep running estimates, and batch statistics are instead used during evaluation time as well.
F.batch_norm should only receive training=True if self.training=True and self.track_running_stats=True