Use global statistics of BatchNorm in training mode

[deepak242424 (Deepak Mittal)]

I ran the code with model.eval(). After the last batch norm layer all the values are becoming Nan.However when I set model.train() the code is working fine.

Further I saw, the weights and bias parameters of the Batch Norm layers are going to Nan. This seems quite a weird behaviour.

At the following threads also people are trying to do the exact same thing. I followed all the solutions mentioned there but still no luck.

Proper way of fixing batchnorm layers during training

@maunz I had same problem. It seems I setting eval mode wrong. This thread helped me Freeze BatchNorm layer lead to NaN def set_bn_to_eval(m): classname = m.class.name if classname.find(‘BatchNorm’) != -1: m.eval() net.apply(set_bn_to_eval) if you want to set_bn_to_eval of some subnet or some basenetwork then just use. net..apply(set_bn_to_eval) hope this helps.

Freeze BatchNorm layer lead to NaN

Hi, everyone I want to freeze BatchNorm while fine-tuning my resnet (I mean, use global mean/std and freeze weight and bias in BN), but the loss is so large and become nan at last: iter = 0 of 20000 completed, loss = [ 15156.56640625] iter = 1 of 20000 completed, loss = [ nan] iter = 2 of 20000 completed, loss = [ nan] the code I used to freeze BatchNorm is: def freeze_bn(model): for name, module in model.named_children(): if isinstance(module, nn.BatchNorm2d): …