Module.children() vs Module.modules()

I was trying to remove the last layer (fc) of Resnet18 to create something like this by using the following

pretrained_model = models.resnet18(pretrained=True)
for param in pretrained_model.parameters():
    param.requires_grad = False
my_model = nn.Sequential(*list(pretrained_model.modules())[:-1])
model = MyModel(my_model)

As it turns out this did not work (the layer is still there in the new model). I then found this post and used the below line which works.

my_model = nn.Sequential(*list(pretrained_model.children())[:-1])

What is the difference between the two and why does one work but not the other?

22 Likes

.modules() also returns pretrained_model as one of the elements.

For example see:

list(nn.Sequential(nn.Linear(10, 20), nn.ReLU()).modules())
Out[9]:
[Sequential (
   (0): Linear (10 -> 20)
   (1): ReLU ()
 ), Linear (10 -> 20), ReLU ()]

In [10]: list(nn.Sequential(nn.Linear(10, 20), nn.ReLU()).children())
Out[10]: [Linear (10 -> 20), ReLU ()]
23 Likes

Ah, Makes sense.
Sorry for such a basic question but what is the intended use for .modules()?

4 Likes

If you want to recursively iterate over modules, then you want to use .modules()
For example, in the following network

m = nn.Sequential(nn.Linear(2,2), 
                  nn.ReLU(),
                  nn.Sequential(nn.Sigmoid(), nn.ReLU()))

calling m.children() will return

[Linear (2 -> 2), ReLU (), Sequential (
   (0): Sigmoid ()
   (1): ReLU ()
 )]

which means that it does not go inside the second Sequential, and thus does not print individually Sigmoid. On the other hand, m.modules() recursively walks into all modules in the network, and yields

[Sequential (
   (0): Linear (2 -> 2)
   (1): ReLU ()
   (2): Sequential (
     (0): Sigmoid ()
     (1): ReLU ()
   )
 ), Linear (2 -> 2), ReLU (), Sequential (
   (0): Sigmoid ()
   (1): ReLU ()
 ), Sigmoid (), ReLU ()]
75 Likes

So, when i iterate, is there a way to get these indiviual single components and get rid of those Sequential components?

13 Likes

@ post above me
You could iterate through the whole network. Think of it as a tree.

all_layers = []
def remove_sequential(network):
    for layer in network.children():
        if type(layer) == nn.Sequential: # if sequential layer, apply recursively to layers in sequential layer
            remove_sequential(layer)
        if list(layer.children()) == []: # if leaf node, add it to list
            all_layers.append(layer)
13 Likes

Hi fmassa

In this operation, the names of all the nodes are different from the original name. For example, the original name of a convolution layer may be ‘conv_3_3’. However, the name becomes some numeric number, e.g., ‘10’, under this operation. Is there way to remain each layer’s original name after this kind of operation?

2 Likes

I bumped to that issue while trying to find memory leakage in a custom architecture. To get the number of the children that are not parents to any other module, thus the real number of modules inside the provided one, I am using this recursive function:

def dim(module):
    total_num = 0
    f = False
    for child in module.children():
        f = True
        total_num += dim(child)
    if not f:
        return 1
    return total_num

I hope it helps

or

isinstance(layer, nn.Sequential)

instead of

type(layer) == nn.Sequential

as it is considered more pythonic?

4 Likes

my modules are mapped to a name. I want to get the module and the name (but not the parameters). Is there a way to do that?

e.g.

mport torch
import torch.nn as nn

from collections import OrderedDict

params = OrderedDict([
    ('fc0', nn.Linear(in_features=4,out_features=4)),
    ('ReLU0', nn.ReLU()),
    ('fc1:final', nn.Linear(in_features=4,out_features=1))
])
mdl = nn.Sequential(params)

answering my own question: How to get the module names of nn.Sequential

You could print all names and sub-modules using:

for name, module in model.named_modules():
    print(name)

If you want to directly access these modules, you can just use:

print(model.conv0)
1 Like

this might useful for you if you are trying to modify a pre-trained net (like I was): How to modify a pretrained model

1 Like

Try this simple thing

for k, m in model._modules.items():
   print(m) # m is the indiviual single components