I got an old code using BatchNorm1d in a MLP network (probably a modification from CNN code). And when I update the pytorch version to 0.3.0, it gives me the following error:
ValueError: Expected more than 1 value per channel when training, got input size [1L, 100L]
which I guess is because in MLP, there is only one channel for each layer and the new BatchNorm1d does not allow such case.
To reproduce this error, you can just generate a 1d vector and feed it to the BatchNorm1d function, for example
m = nn.BatchNorm1d(100)
input = autograd.Variable(torch.randn(1, 100))
output = m(input)
Training the original model took several weeks and there’s a lot of work to do if I have to reconstruct the network.
Can anyone enlighten me with a workaround for this issue?
I’m a little confused – what happens when you do a batchnorm with one value per channel? The formula in the docs suggests that it would go to + inf or - inf (because Var[x] = 0 due to there being only one value?), yes?
Thanks for your answer! Your solution worked. I don’t think it will go to inf in this case. As long as the input x isn’t a scalar or something like [1,1…1,1], var(x) will be nonzero. Also as stated in the document, there is the epsilon there which will prevent the inf case from happening. I tried using a scalar, and it didn’t explode.