# Dataparallel issue with flatten parameter **URL:** \ **Category:** Uncategorized\ **Created:** [October 3, 2017, 9:59pm UTC](https://discuss.pytorch.org/t/dataparallel-issue-with-flatten-parameter/8282 "2017-10-03T21:59:21Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![kishwarshafin](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/a88e57/32.png) [@kishwarshafin](https://discuss.pytorch.org/u/kishwarshafin)\ **Post date:** [October 3, 2017, 9:59pm UTC](https://discuss.pytorch.org/t/dataparallel-issue-with-flatten-parameter/8282/1 "2017-10-03T21:59:21Z")
``` def lstmLayer(self, input): hidden = self.init_hidden(input.size(2)) for i in range(input.size(0)): out, hidden = self.lstm1(input[i], hidden) hidden = self.repackage_hidden(hidden) return out ``` Here’s my code, but when I wrap my code with dataparallel, I get this message: `UserWarning: RNN module weights are not part of single contiguous chunk of memory. This means they need to be compacted at every call, possibly greately increasing memory usage. To compact weights again call flatten_parameters(). out, hidden = self.lstm1(input[i], hidden)` I am not sure where to call the flatten parameter, or is it something else I need to do? ---
**Author:** ![smth](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/smth/32/13_2.png) [@smth](https://discuss.pytorch.org/u/smth)\ **Post date:** [October 11, 2017, 5:59am UTC](https://discuss.pytorch.org/t/dataparallel-issue-with-flatten-parameter/8282/2 "2017-10-11T05:59:12Z")
i think it’s this way: `self.lstm1.flatten_parameters()` ---
**Author:** ![blackyang](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/blackyang/32/1777_2.png) [@blackyang](https://discuss.pytorch.org/u/blackyang)\ **Post date:** [October 17, 2017, 11:57pm UTC](https://discuss.pytorch.org/t/dataparallel-issue-with-flatten-parameter/8282/3 "2017-10-17T23:57:45Z")
Thanks, but _where_ to call this function? If self is a `nn.Module`, should we call it in ` __init__ ()` function or `forward()` function? Should we call it before or after `self.lstm1(input, hidden)`? I always get illegal memory access errors ---
**Author:** ![wgharbieh](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/w/5daacb/32.png) [@wgharbieh](https://discuss.pytorch.org/u/wgharbieh)\ **Post date:** [October 20, 2017, 3:08pm UTC](https://discuss.pytorch.org/t/dataparallel-issue-with-flatten-parameter/8282/4 "2017-10-20T15:08:58Z")
@blackyang I think you have to call `self.lstm1.flatten_parameters()` above `out, hidden = self.lstm1(input[i], hidden)` that fixed this issue for me. ---
**Author:** ![wangke](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/wangke/32/13653_2.png) [@wangke](https://discuss.pytorch.org/u/wangke)\ **Post date:** [May 28, 2018, 3:03pm UTC](https://discuss.pytorch.org/t/dataparallel-issue-with-flatten-parameter/8282/5 "2018-05-28T15:03:43Z")
@wgharbieh @smth I have added `self.lstm1.flatten_parameters()` above `out, hidden = self.lstm1(input[i], hidden)`, but I faced another error and got the following message: ```auto Traceback (most recent call last): File "steps/run_dpcl.py", line 600, in main(device) File "steps/run_dpcl.py", line 417, in main train(model, mix_mean, mix_var, clean_mean, clean_var, device, ema) File "steps/run_dpcl.py", line 149, in train output, hidden = model(mix_v, hidden, lengths) File "/mnt/tools/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in __call__ result = self.forward(*input, **kwargs) File "/mnt/tools/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 114, in forward outputs = self.parallel_apply(replicas, inputs, kwargs) File "/mnt/tools/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 124, in parallel_apply return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)]) File "/mnt/tools/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 65, in parallel_apply raise output File "/mnt/tools/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 41, in _worker output = module(*input, **kwargs) File "/mnt/tools/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in __call__ result = self.forward(*input, **kwargs) File "/mnt/workspace/pytorch/deep_clustering/model/blstm_dpcl.py", line 58, in forward output, hidden = self.blstm(inputs, hidden) File "/mnt/tools/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in __call__ result = self.forward(*input, **kwargs) File "/mnt/tools/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/nn/modules/rnn.py", line 192, in forward output, hidden = func(input, self.all_weights, hx, batch_sizes) File "/mnt/tools/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/nn/_functions/rnn.py", line 323, in forward return func(input, *fargs, **fkwargs) File "/mnt/tools/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/nn/_functions/rnn.py", line 287, in forward dropout_ts) RuntimeError: torch/csrc/autograd/variable.cpp:115: get_grad_fn: Assertion `output_nr == 0` failed. ``` ---
**Author:** ![wgharbieh](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/w/5daacb/32.png) [@wgharbieh](https://discuss.pytorch.org/u/wgharbieh)\ **Post date:** [May 28, 2018, 3:43pm UTC](https://discuss.pytorch.org/t/dataparallel-issue-with-flatten-parameter/8282/6 "2018-05-28T15:43:35Z")
Does your code run if you do not use `self.lstm1.flatten_parameters()`? If so, it could be because you are using `DataParallel`, can you try using `DistributedDataParallel` instead? It also depends on the pytorch version that you are using, the fix above seems to work up to pytorch 0.3.0. If you are using pytorch 0.3.1 or 0.4, there is an issue about it on github here: [https://github.com/pytorch/pytorch/issues/7092](https://github.com/pytorch/pytorch/issues/7092) ---
**Author:** ![kishwarshafin](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/a88e57/32.png) [@kishwarshafin](https://discuss.pytorch.org/u/kishwarshafin)\ **Post date:** [May 28, 2018, 4:00pm UTC](https://discuss.pytorch.org/t/dataparallel-issue-with-flatten-parameter/8282/7 "2018-05-28T16:00:31Z")
Try switching from `DataParallel` to `DistributedDataParallel`. I kind of had a hard time making this work with multiple gpus using `DataParallel` if I remember correctly.