How to use nn.DataParallel while accessing the attributes of module?

1

I do not understand the technique described in PyTorch documentation (Multi-GPU examples: https://pytorch.org/tutorials/beginner/former_torchies/parallelism_tutorial.html) for creating a subclass of nn.DataParallel to access the attributes of a module.

class MyDataParallel(nn.DataParallel):
    def __getattr__(self, name):
        return getattr(self.module, name)

Can someone provide an example to illustrate this?

2

When you use your model in DataParallel mode, you cannot directly access the attributes of your model.
For example suppose you had an fc attribute for the fully connected layer and you could access it as net.fc.

Suppose you used DataParallel as net = nn.DataParallel(net). To access the same attribute now you need to do net.module.fc. This might make parts of your code dependent on weather DataParallel was used or not. This is because the decision to use DataParallel depends on the number of GPUs available, etc.

Hence to write code which is agnostic of these situations, the custom __getattr__ function is defined, which is basically retrieving the relevant attribute.

1 Like