CUDA: Out of memory error when using multi-gpu

[Usman_Mahmood]

@ptrblck more memory compared to other gpus or more memory compared to if you were only using 1 gpu?
When I run my code with 1 gpu and batch size 16, it works. But when I run same code with same batch size using 2 gpus (with equal memory) I get out of memory error, and on GPU 1 not on GPU 0, which is strange because my default device is GPU 0.
My issue looks similar to one discussed here:

DataParallel memory consumption in PyTorch 0.4

I believe I’m seeing a certain loss of functionality after upgrading from PyTorch 0.3.1 to 0.4.0. Specifically I’m trying to use nn.DataParallel to train, on two GPU’s, a model with a parameter that takes up over half the memory of either GPU. When the DataParallel library code attempts to replicate the model over both GPU’s it broadcasts the parameters to both, and runs out of GPU memory during the broadcast operation. In 0.3.1 the broadcast operation was implemented in Python, and contained…