PyTorch self.module?

1

Hi, I have came across something like this, can someone explain what is the purpose of this chunk of code?

Thank you very much :slight_smile:

for m in self.modules():
            if isinstance(m, nn.Conv2d):
                n = m.kernel_size[0] * m.kernel_size[1] * m.out_channels
                m.weight.data.normal_(0, math.sqrt(2. / n))
            elif isinstance(m, nn.Conv3d):
                n = m.kernel_size[0] * m.kernel_size[1]*m.kernel_size[2] * m.out_channels
                m.weight.data.normal_(0, math.sqrt(2. / n))
            elif isinstance(m, nn.BatchNorm2d):
                m.weight.data.fill_(1)
                m.bias.data.zero_()
            elif isinstance(m, nn.BatchNorm3d):
                m.weight.data.fill_(1)
                m.bias.data.zero_()
            elif isinstance(m, nn.Linear):
                m.bias.data.zero_()
5 Likes
2

I’m not sure what you mean by self.module in your title…

The self.modules() method returns an iterable to the many layers or “modules” defined in the model class. This particular piece of code is using that self.modules() iterable to initialize the weights of the different layers present in the model. isinstance() checks if the particular layer “m” is an instance of a conv2d or linear or conv3d layer etc. and initializes the weights accordingly.

Why don’t you check the docs for all these functions to get a better understanding?

model.modules()
model.named_modules()
model.parameters()
model.named_parameters()
model.children()
model.named_children()
5 Likes
3

Thank you very much for your answer.

P.S. I am new to PyTorch and not very aware of the available resources. :sweat_smile:

3 Likes
4

@shubhvachher I have another question regarding this. Why initialize the weights manually?
Correct me if I am wrong, the weights are initialized when you create your layers when using the torch.nn modules and these weights are random. Why re-initializing them?

Thank you :smile:

5

Weights are initialized to small random numbers on creation (it actually depends on the type of layer you’re using but for argument’s sake we’ll just say small random numbers), usually this isn’t a huge deal as your model will eventually learn the correct weights through gradient descent. However, depending on the model complexity this may take a long time or get stuck in a local minimum. You can set the weights manually to “guide” your model to faster convergence if you have an idea on what would work better.

1 Like
6

I’m guessing you don’t mean initializing weights with those of a pretrained model but the different types of initializations in literature.

I’m not an expert on this but I think for learning from scratch, the initializations done by pytorch internally are good enough for most tasks. You could do these initializations yourself if subject matter experts in your learning task or information you gathered from your paper review suggests some initialization techniques for that task fair better than others.

7

You are not alone, there are no stupid questions, learning requires asking questions that are known to others , but not yourself. There was a starting point for all when we asked what a PC was. Personal Caretaker was popular prior to 90’s LOL

2 Likes
8

I also have to say the brilliance and help on here is incredible. I browse to see if people are being helped and treated fairly, and I must say, out of hundreds of sites this is in the top 5 for active help. Kudos people! A+

2 Likes
9

Gentle reminder: when someone says they are new to the PyTorch community or NN architectures in general - be constructive not condescending with answers. I am on my 3rd year of building probabilistic NN models and I too haven’t memorized the Pytorch + Pyro handbooks. Remember, it takes courage to ask questions!

-Sarah Amiri