When should I use nn.ModuleList and when should I use nn.Sequential?

vision
21

Thanks for your great explanation

23

I wonder if it work if I put multi nn.Sequential blocks into nn.ModueleList?

24

It would assume it should work. Are you seeing any issues with this approach?

25

I guess nn.Sequential will be registered parameters to my optimizer like ModuleList?

26

Yes, both nn.Sequential and nn.ModuleList which are assigned to an nn.Module instance as an attribute will register the parameters.
Also, both methods provide the .parameters() method, which can be used to pass the parameters to an optimizer.

Let me know, if you encounter any issues.

4 Likes
27

Great explanation and easy to understand!

28

I came from TF for a few days and i can be wrong on this as i am still getting started on Pytorch. I had the same doubt on this too and realized that if you need a real fine tune in your model than you probably need the nn.ModuleList. Many Yolo implementations (in Pytorch) runs appending layer by layer with nn.ModuleList. I would just take care as most of the times will be cleaner to just use nn.Sequential.

With nn.ModuleList:

class MyModule(nn.Module):
    def __init__(self, layers_count=10):
    	super(MyModule, self).__init__()

		self.layers = nn.ModuleList([nn.Linear(10, 10) for _ in range(layers_count)])

    def forward(self, x):
    	for layer in self.layers:
    		x = layer(x)

    	return x

Now with nn.Sequential:

class MyModule(nn.Module):
    def __init__(self, layers_count=10):
    	super(MyModule, self).__init__()

		layers = [nn.Linear(10, 10) for _ in range(layers_count)]
		self.layers = nn.Sequential(*layers)

    def forward(self, x):
    	return self.layers(x)

I think both codes do the same but the 2nd is less verbose as you need to iterate on ModuleList…

1 Like
29

nn.Flatten is available in Pytorch already, no need to write it for yourself.

30

I love you! I love you! Best explanation of nn.ModuleList anywhere on the interwebs