The difference in usage between nn.ModuleList and python list

1

I can’t figure out what the purpose of nn.ModuleList is because as far as I know, list can be used in most scenarios I come up with.
For example, for the newtork structure below,

	def __init__(self):
		super(MyNN, self).__init__()
		fcs = [nn.Sequential(
				 nn.Linear(fc_input_size, fc_hidden_size),
				 nn.ReLU(),
				 nn.Linear(fc_hidden_size,num_classes)
		) for fc_hidden_size in fc_hidden_sizes]
		#self.network = nn.ModuleList(fcs)
		self.network = fcs

I don’t see the difference between list and nn.ModuleList.
Can anyone please give an example of where nn.ModuleList has advantage and explain it’s designed purpose?

1 Like
2

When you ask for model.parameters(), the parameters of layers inside nn.ModuleList will be returned.
But if it is regular list, we dont look inside the list for nn.Module layers.

In your example, if you do:

list(MyNN().parameters()) then you will see that the weight of the nn.Linear in self.network will be missing.

15 Likes
3

I tried what you suggested, and it’s indeed the case. Thanks a lot!

4

I’m late to the party, but does this mean using list instead of ModuleList will be wrong? i.e. will the weights of my networks inside the list still be updated during training?

1 Like
5

Yes the weights of the modules inside the python list will not be updated in training, unless you manually add them to the list of parameters passed to the optimizer. Moreover, even if you do that, when you want to save the model parameters using model.state_dict(), the parameters of modules inside the python list won’t be saved.

7 Likes
6

In Training a classifier example of pytorch 60 min blitz tutorial the modules are not added in Modulelist still the optimizer is able to access the parameters. Why is that?

7

Double post from here with a potential answer.