# Why do we need "flatten\_parameters" when using RNN with DataParallel

**URL:** <https://discuss.pytorch.org/t/why-do-we-need-flatten-parameters-when-using-rnn-with-dataparallel/46506>\
**Category:** distributed\
**Created:** [May 29, 2019, 6:50am UTC](https://discuss.pytorch.org/t/why-do-we-need-flatten-parameters-when-using-rnn-with-dataparallel/46506 "2019-05-29T06:50:34Z")\
**Posts on this page:** 1\
**Showing post:** 2

<div class="post-metadata">

**Author:** ![wwiiiii](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/wwiiiii/32/6613_2.png) [@wwiiiii](https://discuss.pytorch.org/u/wwiiiii)\
**Post date:** [May 30, 2019, 6:57am UTC](https://discuss.pytorch.org/t/why-do-we-need-flatten-parameters-when-using-rnn-with-dataparallel/46506/2 "2019-05-30T06:57:20Z")

</div>

After reading some related codes, I think I almost get it but still have few questions.  
So what I understand is

* * *

Everytime when we make new RNN module instance, it [allocates](https://github.com/pytorch/pytorch/blob/c611630b9df4a25007dd299676b2bb88bf692784/torch/nn/modules/rnn.py#L72) new `w_ih, w_hh, b_ih, b_hh` tensors and register them as `Parameter` for each layer, direction.

But it’s not guranteed that new tensors are contiguous on GPU memory, performance can be dropped due to the fragmentation. So we call `flatten_parameters` function at the end of constructor to [aggregate all the weight tensors into continuous space of GPU memory](https://github.com/pytorch/pytorch/blob/c611630b9df4a25007dd299676b2bb88bf692784/torch/nn/modules/rnn.py#L90).

This task is done as

1. [Allocate](https://github.com/pytorch/pytorch/blob/d1623f4cc9334ade7376b3685bb91d3071e3b418/aten/src/ATen/native/cudnn/RNN.cpp#L647) one big buffer tensor called `weight_buf`
2. Copy values of weight tensor into `weight_buf`
3. Make each weight tensor’s internal data pointer [indicating](https://github.com/pytorch/pytorch/blob/d1623f4cc9334ade7376b3685bb91d3071e3b418/aten/src/ATen/native/cudnn/RNN.cpp#L469) `weight_buf + offset`

(The real execution steps are 1-\>3-\>2 in real code)

But when we use nn.DataParallel, it replicates original module(which is allocated only on certain GPU device) to every GPU it uses, then weight tensors are fragmented again since there’s no gurantee that replicated tensors are still contiguous on memory space.

Therefore we should `flatten_parameters` again everytime the module is replicated to another GPU, and the best place to put function call would be the head of `forward` function (of nn.Module), because `forward` function of `nn.Module` on each GPU is called only one time when `forward` of `nn.DataParallel` is called.

Although I never used `nn.DistributedDataParallel`, I guess that the reason why it doesn’t need the `flatten_parameters` call is because when it allocates new instance of RNN module, `flatten_parameters` are automatically called, then it doesn’t move internal data position on memory unlike `nn.DataParallel`, but it only copies some values into it.

* * *

And questions are

1. Do I understand right? Is there any misunderstood point?

2. When we do the step 3 of aggregation(=Make each weight tensor’s internal data pointer indicating `weight_buf + offset`), we call the get\_parameters function and it

---

_[View the full topic](https://discuss.pytorch.org/t/why-do-we-need-flatten-parameters-when-using-rnn-with-dataparallel/46506)._