# Gradient mismatch for complex networks using nn.DataParallel **URL:** \ **Category:** complex\ **Created:** [March 13, 2023, 6:05pm UTC](https://discuss.pytorch.org/t/gradient-mismatch-for-complex-networks-using-nn-dataparallel/174728 "2023-03-13T18:05:05Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Minennick](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/minennick/32/57060_2.png) [@Minennick](https://discuss.pytorch.org/u/Minennick)\ **Post date:** [March 13, 2023, 6:05pm UTC](https://discuss.pytorch.org/t/gradient-mismatch-for-complex-networks-using-nn-dataparallel/174728/1 "2023-03-13T18:05:05Z")
Dear all, I am currently building a network that gets complex numbers as input, processes them using complex layers, and gives back again complex numbers. I already recognized, that passing the input to the model, causes the input to automatically be viewed as 2-channeled real values, as is the case for applying torch.view\_as\_real(). Since I handle data in the complex view the first thing I do in the forward() method of my network is transforming it back to complex view. When I use only one GPU to train, everything works fine and I have the expected behavior. Given that I want to train on multiple GPUs I get the following error message in the backward pass. ```auto Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass RuntimeError: Function GatherBackward returned an invalid gradient at index 0 - got [2, 3, 128, 128, 2] but expected shape compatible with [2, 3, 128, 128] ``` Now for me, it looks like that as soon as I train on multiple GPUs the gradients are in the shape of the 2-channeled real view, while my inputs and weights are in complex view. I am currently using DataParallel as: ```auto if not 'CUDA_VISIBLE_DEVICES' in os.environ: os.environ['CUDA_VISIBLE_DEVICES'] = f'{args.gpus}' model = torch.nn.DataParallel(model) ``` and pytorch 1.12.1 py3.9\_cuda10.2\_cudnn7.6.5\_0 Maybe someone can shed some light on why this happens, and how to fix. Best regards, Niklas ---
**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\ **Post date:** [March 14, 2023, 6:14am UTC](https://discuss.pytorch.org/t/gradient-mismatch-for-complex-networks-using-nn-dataparallel/174728/2 "2023-03-14T06:14:22Z")
I’m not sure why the error is raised, but note that `nn.DataParallel` is in “maintenance mode” as of [this RFC](https://github.com/pytorch/pytorch/issues/65936) and should eventually be removed. We recommend using `DistributedDataParallel` for a better performance and support. Could you try to use your code using `DDP` and see if the same error is raised? ---
**Author:** ![Minennick](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/minennick/32/57060_2.png) [@Minennick](https://discuss.pytorch.org/u/Minennick)\ **Post date:** [March 14, 2023, 1:25pm UTC](https://discuss.pytorch.org/t/gradient-mismatch-for-complex-networks-using-nn-dataparallel/174728/3 "2023-03-14T13:25:21Z")
Dear Piotr, thank you very much for the advice. Using DDP instead of DP actually solved the issue. From my point of view, I sadly did not find out what the actual problem is with using DP. But in case someone else has this problem I want to refer to this [medium article](https://medium.com/codex/a-comprehensive-tutorial-to-pytorch-distributeddataparallel-1f4b42bb1b51), which helped in understanding and setting up the DDP along with the official Pytorch documentation. Best regards, Niklas ---
**Author:** ![Stanislaw\_Raczynski](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/stanislaw_raczynski/32/59624_2.png) [@Stanislaw\_Raczynski](https://discuss.pytorch.org/u/Stanislaw_Raczynski)\ **Post date:** [May 15, 2023, 1:19pm UTC](https://discuss.pytorch.org/t/gradient-mismatch-for-complex-networks-using-nn-dataparallel/174728/4 "2023-05-15T13:19:59Z")
Hi, I’m running Pytorch 2.0.1 with CUDA 12.1 and am getting the same problem. I get the following exception on GPU:1, on GPU:0 it runs without issues. ```auto RuntimeError: Function GatherBackward returned an invalid gradient at index 0 - got [5, 512, 901, 2] but expected shape compatible with [5, 512, 901] > /work/miniconda3/envs/sayso_dev/lib/python3.9/site-packages/torch/autograd/ __init__.py(200)backward() 198 # some Python versions print out the first line of a multi-line function 199 # calls in the traceback and some print out the last line --> 200 Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 201 tensors, grad_tensors_, retain_graph, create_graph, inputs, 202 allow_unreachable=True, accumulate_grad=True) # Calls into the C++ engine to run the backward pass ``` ---
**Author:** ![Minennick](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/minennick/32/57060_2.png) [@Minennick](https://discuss.pytorch.org/u/Minennick)\ **Post date:** [May 15, 2023, 1:23pm UTC](https://discuss.pytorch.org/t/gradient-mismatch-for-complex-networks-using-nn-dataparallel/174728/5 "2023-05-15T13:23:28Z")
@Stanislaw_Raczynski Do you use DataParallel or Distributed Data Parallel? ---
**Author:** ![Stanislaw\_Raczynski](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/stanislaw_raczynski/32/59624_2.png) [@Stanislaw\_Raczynski](https://discuss.pytorch.org/u/Stanislaw_Raczynski)\ **Post date:** [May 15, 2023, 1:33pm UTC](https://discuss.pytorch.org/t/gradient-mismatch-for-complex-networks-using-nn-dataparallel/174728/6 "2023-05-15T13:33:48Z")
Sorry I wasn’t clear. I was using `DataParallel`, as in the initial post. I am trying to move to `DistributedDataParallel` as per suggestions from ptrblck. The error above points to a complex tensor (result of `torch.stft()`) within my model.