Is there performance difference when using different ways to create models?

1

Will there be any performance/memory/other difference if I do the following to create a model:

  1. using nested modules (for example, if I make Conv-BN-ReLU a module, which is nested in a Residual Module, which is nested in a stage module, which is nested in a body module, which is nested in the module of the whole model) vs. writing all layers inside a single module without nesting
  2. using Sequential vs Functional api (assuming no branching)
  3. putting simple operations like add, scaling, etc. into a custom module and using the custom module as a layer vs directly adding the tensor in forward

Any help would be great. Thanks in advance

2

All your points seem to be concerned about the Python call overhead by nesting objects.
While I think you might see more overhead in the actual function calls, I would claim they might be tiny for “real” workloads where you are able to keep the GPU busy and would thus probably stick to the “cleanest” coding approach.
If you are really concerned you could profile a “flat” vs. “nested” example model and check if you would be able to see any difference.

1 Like