My model has several sub-modules, stored in a
nn.ModuleList member, in
forward() if I calling each sub-module like this
def forward(self, x): out = [m(x) for m in self.sub_module_list] # where self.sub_module_list is nn.ModuleList object out = torch.cat(out)
forward() of each submodule run parallel or sequential?
I suppose if module is jit traced the answer is clear: it is parallel since that’s exactly what static computation graph want to do. But if not jit traced, will this line of code run sequentially?