Problem encountered when tested with batchnorm layers

vision
1

Hi guys, when tested with batchnorm layers now, i was confused by something:

input = torch.rand(10,5)
model = nn.BatchNorm1d(5)
out = model(input)
print(out.mean(0), out.var(0))

Here I would expect the mean be zero while var be 1 if I didn’t misunderstand the original batchnorm operation cause we havent update the gamma and beta value of the batchnorm layer. However, i always get a strange value such as 1.1102 for var and ~0 for mean. Any suggestions?

Besides, I will get 1 and 0 for weight and bias(gamma and beta) when:

print(model.weight, model.bias)
2

anyone has similar questions?

3

The number of elements might be too small and you might get values closer to the zero mean and unit variance by using a larger input, such as [100, 5].

4

Thanks, I tried and it does have some changes. I just wonder why pytorch cannot normalize it correctly. A bit weird.

5

I don’t think PyTorch cannot “normalize it correctly”, but batchnorm layers don’t use the unbiased variance calculation, which is probably why you might think it’s incorrect. If you want to use the biased one, you could create a custom layer by e.g. following this implementation.

6

Thanks so much. You are right, pytorch used 1/n not 1/(n-1). I can now match the output with the manual calculated one.