Function/Class for Generating Variable-Depth Networks

2

  1. The for loop is not a problem and you shouldn’t see any Python overhead, as the main workload is in the layer execution, not the for loop evaluation in Python. nn.ModuleList uses a for loop as the example use case.

  2. If you store the modules in a Python list, they won’t be registered properly, and model.parameters() won’t return all parameters. Use nn.ModuleList instead.

  3. It depends on your coding style. Stateless modules, e.g. ReLU can be used in their functional API (F.relu) without the necessity to store some parameters manually. If you register these modules in your __init__, you could e.g. swap them with another activation function without manipulating the forward method, but I would still say it’s more a coding style question.