Weight Initialization for BatchNorm in DCGAN Tutorial

https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html#weight-initialization

def weights_init(m):
    classname = m.__class__.__name__
    if classname.find('Conv') != -1:
        nn.init.normal_(m.weight.data, 0.0, 0.02)
    elif classname.find('BatchNorm') != -1:
        nn.init.normal_(m.weight.data, 1.0, 0.02)
        nn.init.constant_(m.bias.data, 0)

My question is why BatchNorm2d init with mean 1 and add bias 0 at the end
Whats the reason behind it ?

A weight of ~1 and bias of ~0 in nn.BatchNorm will pass the normalized activations to the next layer.
In your example the weight is sampled from a normal distribution with a small stddev which is approx. 1.
It’s a valid strategy to init BatchNorm layers and is also discussed here.

1 Like