BatchNorm1d with one value per channel

Hi,

I got an old code using BatchNorm1d in a MLP network (probably a modification from CNN code). And when I update the pytorch version to 0.3.0, it gives me the following error:

ValueError: Expected more than 1 value per channel when training, got input size [1L, 100L]

which I guess is because in MLP, there is only one channel for each layer and the new BatchNorm1d does not allow such case.

To reproduce this error, you can just generate a 1d vector and feed it to the BatchNorm1d function, for example

m = nn.BatchNorm1d(100)
input = autograd.Variable(torch.randn(1, 100))
output = m(input)

Training the original model took several weeks and there’s a lot of work to do if I have to reconstruct the network.

Can anyone enlighten me with a workaround for this issue?

Thanks!

I’m a little confused – what happens when you do a batchnorm with one value per channel? The formula in the docs suggests that it would go to + inf or - inf (because Var[x] = 0 due to there being only one value?), yes?

I’m not sure if this is what you’re looking for, but you can turn training mode off for BatchNorm1d and then it’ll take inputs with minibatch size 1:

m = nn.BatchNorm1d(100)
m.train(False)
input = autograd.Variable(torch.randn(1, 100))
output = m(input)

Thanks for your answer! Your solution worked. I don’t think it will go to inf in this case. As long as the input x isn’t a scalar or something like [1,1…1,1], var(x) will be nonzero. Also as stated in the document, there is the epsilon there which will prevent the inf case from happening. I tried using a scalar, and it didn’t explode.