What is the running mean of BatchNorm if gradients are accumulated?

[crcrpar (Masaki Kozuki)]

1. Yes.
2. Accumulated gradients will be the same if you divide them by the number of iterations. I referred below.

@apaszke

Not that loss functions average over the batch size, so if you do multiple backprops you might need to average over the number of the for loop iterations.