DataParallel on large network that take up more than one GPU

5

I agree. DataParallel assume that each GPU can take at least one data sample on its own.

I did apply dataparallel to the parent module, without specifying any gpu to any module… I found out module2 is assigned to cuda:3 after encourter the error…