What is the running mean of BatchNorm if gradients are accumulated?

[crcrpar (Masaki Kozuki)]

Hi, @Zhang_Chi

Batch Normalization updates its running mean and variance every call of forward method.
Also, by default, BatchNorm updates its running mean by running_mean = alpha * mean + (1 - alpha) * running_mean (the details are here).

As to accumulating gradients, this thread “How to implement accumulated gradient？ - #8 by Gopal_Sharma” might help you.

@Zhang_Chi

or example,now the batch size is 2, and i backward after five iterations, which is the same with a batch of 10.

As a side note, I don’t think the accumulated gradients and the gradient will be the same in the example.