How can I give a name to each module in ModuleList?

1

I have the following component in my model:

feedfnn = []
for task_name, num_class in self.tasks:
    if self.config.nonlinear_fc:
        ffnn = nn.Sequential(OrderedDict([
            ('dropout1', nn.Dropout(self.config.dropout_fc)),
            ('dense1', nn.Linear(self.config.nhid * self.num_directions * 8, self.config.fc_dim)),
            ('tanh', nn.Tanh()),
            ('dropout2', nn.Dropout(self.config.dropout_fc)),
            ('dense2', nn.Linear(self.config.fc_dim, self.config.fc_dim)),
            ('tanh', nn.Tanh()),
            ('dropout3', nn.Dropout(self.config.dropout_fc)),
            ('dense3', nn.Linear(self.config.fc_dim, num_class))
        ]))
    else:
        ffnn = nn.Sequential(OrderedDict([
            ('dropout1', nn.Dropout(self.config.dropout_fc)),
            ('dense1', nn.Linear(self.config.nhid * self.num_directions * 8, self.config.fc_dim)),
            ('dropout2', nn.Dropout(self.config.dropout_fc)),
            ('dense2', nn.Linear(self.config.fc_dim, self.config.fc_dim)),
            ('dropout3', nn.Dropout(self.config.dropout_fc)),
            ('dense3', nn.Linear(self.config.fc_dim, num_class))
        ]))
    feedfnn.append(ffnn)
self.ffnn = nn.ModuleList(feedfnn)

When I print my model, I get the description of the above component as:

(ffnn): ModuleList (
(0): Sequential (
  (dropout1): Dropout (p = 0)
  (dense1): Linear (4096 -> 512)
  (dropout2): Dropout (p = 0)
  (dense2): Linear (512 -> 512)
  (dropout3): Dropout (p = 0)
  (dense3): Linear (512 -> 2)
)
(1): Sequential (
  (dropout1): Dropout (p = 0)
  (dense1): Linear (4096 -> 512)
  (dropout2): Dropout (p = 0)
  (dense2): Linear (512 -> 512)
  (dropout3): Dropout (p = 0)
  (dense3): Linear (512 -> 3)
)
(2): Sequential (
  (dropout1): Dropout (p = 0)
  (dense1): Linear (4096 -> 512)
  (dropout2): Dropout (p = 0)
  (dense2): Linear (512 -> 512)
  (dropout3): Dropout (p = 0)
  (dense3): Linear (512 -> 3)
)
)

Can I put a specific name like (task1): Sequential, (task2): Sequential instead of (0): Sequential, (1): Sequential?

1 Like
2

It doesn’t look like it

You could probably modify ModuleList to do so (ie, make your own class that extends Module or ModuleList). You can achieve something like what you want by modifying its setattr attribute.

3

ModuleList is specifically designed to act like a list and thus can’t do that. Currently the ModuleList assumes the key names are sequential numbers to access the items in the list. You could probably create an empty Module and add items using the add_module method.

4

If the OrderedDict constructor of nn.Sequential takes a list of tuples ((name1, mod1), (name1, mod2), ..., (nameN, modN)), then would it be possible to do the following?

    ...
    feedfnn.append((task_name, ffnn))
self.ffnn = nn.Sequential(OrderedDict(feedfnn))

After all, nn.Sequential is a subclass of nn.Module.

5

Specifically (and simplified to focus on the key point):

In [1]: cat net.py
import torch.nn as nn
from collections import OrderedDict

class Example(nn.Module):
    def __init__(self, tasks):
        super(Example, self).__init__()
        self.tasks = tasks
    
        feedfnn = []
        for task_name, num_class in self.tasks:
            ffnn = nn.Sequential(OrderedDict([
                ('dense1', nn.Linear(10, 10)),
                ('tanh', nn.Tanh()),
                ('dense2', nn.Linear(10, 10)),
                ('tanh', nn.Tanh()),
                ('dense3', nn.Linear(10, 10))
            ]))
            feedfnn.append((task_name, ffnn))
        self.ffnn = nn.Sequential(OrderedDict(feedfnn))
    
if __name__ == '__main__':
    tasks = (('task1', None), ('task2', None), ('task3', None))
    example = Example(tasks)
    print(example)

In [2]: run net.py
Example (
  (ffnn): Sequential (
    (task1): Sequential (
      (dense1): Linear (10 -> 10)
      (tanh): Tanh ()
      (dense2): Linear (10 -> 10)
      (dense3): Linear (10 -> 10)
    )
    (task2): Sequential (
      (dense1): Linear (10 -> 10)
      (tanh): Tanh ()
      (dense2): Linear (10 -> 10)
      (dense3): Linear (10 -> 10)
    )
    (task3): Sequential (
      (dense1): Linear (10 -> 10)
      (tanh): Tanh ()
      (dense2): Linear (10 -> 10)
      (dense3): Linear (10 -> 10)
    )
  )
)
2 Likes
6

Thanks for the suggestion. But the use of Sequential would probably make all the individual feed-forward neural network sequential. So, for example, if I have 3 tasks, I need 3 individual FFNN which will be used for the respective task. Currently, I am using the self.ffnn for n tasks as self.ffnn[i] stands for ith task’s FFNN. So, I think I can’t use Sequential in this purpose.

7

This is what I was suggesting in code. But you’d have to make sure task_names are not the same so they don’t overwrite each other.

...
    feedfnn.append((task_name, ffnn))
self.ffnn = nn.Module()
for k, v in feedfnn:
    self.ffnn.add_module(k, v)
2 Likes