Initialise weights of SOME layers

1

Hello!

I’m working on an architecture that involves initialising some of the weights. I need to initialise most linear and conv2d layers (and there’s so many of them!) but not all of them.

I know we can say this to initialise the weights for all layers of a given type:

       for m in self.modules():
            if (isinstance(m, nn.Linear) or isinstance(m, nn.Conv2d)):
                torch.nn.init.xavier_uniform_(m.weights)
            elif isinstance(m, nn.BatchNorm2d):
                (do something else)

How do I add “exceptions” - eg “initialise all conv layers with xavier except for the first 3”? Is it possible to say something like

if isinstance(m, nn.Linear) and m!=fc1:
   torch.nn.init ....

? The linear layers appear on their own (self.fc1 = nn.Linear()) and the conv layers are part of nn.Sequential

Thanks!
NK

2

You could probably use self.named_modules() and use the names like fc1 to filter out the desired layers.

3

Hi Peter,

Thanks for getting back to me! I’ve tried using self.named_modules() but I’m not sure how to specify the name.

When I say for m in self.named_modules() and print m’s in that loop, I get this

layers2
layers2783×491 138 KB

I’ve tried saying if m=="fc1a" or if m==fc1a and a bunch of other things in the named_modules loop but it never works. I’ve looked at the documentation page but it didn’t help. I’ve also tried to refer to the sequential fragment but, once again, I don’t really know how to refer to “encoder.0” etc

How exactly do I specify the names in the named_modules loop?

4

You could use m[0] to access the name or just use this loop instead:

for name, module in self.named_modules():
    if name == '...