DataParallel does not split data memory, instead seems to copy it

[JuanFMontesinos (Juan Montesinos)]

DataParallel imbalanced memory usage

Hi there, I’m going to re-edit the whole thread to introduce a unlikely behavior with DataParallel Right now there are several recent posts about this topic and I would like to summarize the problem. Right now it seems there is an imbalaced usage of GPUs when calling DataParallel. From my experience and other users’ explanations I will explain why this happens: Using DataParallel there are 4 types of data to consider: Input Output Ground…