I’m wondering if you use data parallel, will the sub module also be data parallel? For example, I write one model as A and in B I directly used A, when I use data parallel to wrap B, will A also be data parallel?
nn.DataParallel replicates the passed module on your GPUs, so if
A is a member of
B, both should run in parallel.