# The problem for RNN training with DataParallel

**URL:** <https://discuss.pytorch.org/t/the-problem-for-rnn-training-with-dataparallel/23221>\
**Category:** vision\
**Created:** [August 16, 2018, 2:20am UTC](https://discuss.pytorch.org/t/the-problem-for-rnn-training-with-dataparallel/23221 "2018-08-16T02:20:01Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![wqn628](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/wqn628/32/13783_2.png) [@wqn628](https://discuss.pytorch.org/u/wqn628)\
**Post date:** [August 16, 2018, 2:20am UTC](https://discuss.pytorch.org/t/the-problem-for-rnn-training-with-dataparallel/23221/1 "2018-08-16T02:20:01Z")

</div>

hello, today I train the nn.LSTM with multi-gpu by nn.DataParallel. I set the batch\_size=12 in data.DataLoader, so the shape of input is (12, T, \*), T represents the max frame in the 12 sentences(these 12 sentences has been padded into equal length), and \* represents the dimension of the feature for each frame. the number of gpu is 3.  
the model is as follows:  
############################################################  
train\_rnn\_pack\_sort.py[115] 2018-55-16 09:55:20 INFO: Let’s use 3 GPUs!  
train\_rnn\_pack\_sort.py[126] 2018-55-16 09:55:20 INFO: DataParallel(  
(module): LSTM(  
(lstm): Sequential(  
(0): LSTM(840, 256, batch\_first=True, bidirectional=True)  
(1): LSTM(512, 256, batch\_first=True, bidirectional=True)  
(2): LSTM(512, 256, batch\_first=True, bidirectional=True)  
(3): LSTM(512, 256, batch\_first=True, bidirectional=True)  
(conn): LSTM(512, 4800, batch\_first=True, bidirectional=True)  
)  
)  
)  
###########################################################  
the code for using the nn.DataParallel  
########################################  
if th.cuda.is\_available():  
if args.multi\_gpu == 1:  
if th.cuda.device\_count() \> 1:  
logger.info(“Let’s use {} GPUs!”.format(th.cuda.device\_count()))  
nnet = nn.DataParallel(nnet, device\_ids=[0,1,2]) # dim = 0 [30, xxx] → [10, …], [10, …], [10, …] on 3 GPUs  
elif th.cuda.device\_count() == 1:  
logger.info(“!!!WARNINGS:Only 1 GPU detected and Let’s use the only one GPUs!!!”)  
else:  
logger.info(“!!!ERROR!!!”)  
else:  
logger.info(“!!!Training on single GPU!!!”)

```
else:
    logger.info("!!!WARNINGS:NO GPUs detected! AND The Net Will Be Trained On CPU!!!")
logger.info(nnet)

```

###########################  
the propagation of the model is as follows：  
###########################  
class LSTM(nn.Module):  
def **init** (self, input\_size, hidden\_layer, hidden\_size, num\_classes,  
rnn\_type=‘lstm’, dropout=0.0, bidirect=True, residual=False):  
super(LSTM, self). **init** ()  
if bidirect:  
layer = [nn.LSTM( input\_size=input\_size, hidden\_size=hidden\_size, batch\_first=True, bidirectional=bidirect, dropout=dropout)]  
for i in range(hidden\_layer):  
layer.append(nn.LSTM(hidden\_size_2, hidden\_size, batch\_first=True, dropout=0.0, bidirectional=bidirect))  
self.lstm = nn.Sequential(layer)  
self.lstm.add\_module(“conn”,nn.LSTM(hidden\_size2, num\_classes, batch\_first=True, dropout=0.0, bidirectional=bidirect))  
def forward(self, x, input\_len):  
max\_length = x.shape[1]  
x = rnn\_pack.pack\_padded\_sequence(x, input\_len, batch\_first=True)  
x = self.lstm(x)  
out , \_ = rnn\_pack.pad\_packed\_sequence(x, total\_length=max\_length, batch\_first=True)  
return out  
##########################  
def train\_one\_epoch(nnet, criterion, optimizer, train\_loader, num\_parallel, is\_rnn=False):  
nnet.train()  
pos\_frames = 0.0  
train\_frames = 0.0  
for index, (key, feats, labels, len\_list) in enumerate(train\_loader):  
print “=============”  
print key  
print feats.shape ##(12,T,_)  
print labels.shape ##(12,T)  
#print type(len\_list)  
#print len\_list.shape  
print “=============”  
labels = labels.view(labels.shape[0], labels.shape[1])  
input\_len = np.array(len\_list)  
input\_sort\_id = np.argsort(-input\_len)  
input\_len = input\_len[input\_sort\_id]  
feat\_mat = feats[th.LongTensor(input\_sort\_id)]  
feat\_mat = Variable(feat\_mat.cuda())  
input\_unsort\_id = th.LongTensor(np.argsort(input\_sort\_id))  
optimizer.zero\_grad()  
if is\_rnn:  
label\_mat = labels.view(labels.size(0) \* labels.size(1))  
targets = Variable(label\_mat.cuda())  
out = nnet(feat\_mat, input\_len)  
out = out[input\_unsort\_id]  
out\_num = out.shape[2]  
max\_frame = int(max(len\_list))  
.  
.  
.  
but the error occurs:

 ![image](https://discuss.pytorch.org/uploads/default/original/2X/a/a7e68ebdae793c3290801f642dda0d945d8cad6f.png)  
Any help would be appreciated…

---

<div class="post-metadata">

**Author:** ![PTA](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/c5a1d2/32.png) [@PTA](https://discuss.pytorch.org/u/PTA)\
**Post date:** [August 16, 2018, 3:24am UTC](https://discuss.pytorch.org/t/the-problem-for-rnn-training-with-dataparallel/23221/2 "2018-08-16T03:24:07Z")

</div>

In DataParallel, you need to set your batch size in `forward` method. Also, please format your code when pasting since it is not readable now.

---

<div class="post-metadata">

**Author:** ![wqn628](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/wqn628/32/13783_2.png) [@wqn628](https://discuss.pytorch.org/u/wqn628)\
**Post date:** [August 16, 2018, 3:42am UTC](https://discuss.pytorch.org/t/the-problem-for-rnn-training-with-dataparallel/23221/3 "2018-08-16T03:42:34Z")

</div>

Thanks for your application,but I wonder how to set batch size in `forward` method ?

---

<div class="post-metadata">

**Author:** ![PTA](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/c5a1d2/32.png) [@PTA](https://discuss.pytorch.org/u/PTA)\
**Post date:** [August 16, 2018, 3:47am UTC](https://discuss.pytorch.org/t/the-problem-for-rnn-training-with-dataparallel/23221/4 "2018-08-16T03:47:32Z")

</div>

It should be straightforward by taking the length of the input. If not, you may need to format your code first to make sure people could read that.

---

<div class="post-metadata">

**Author:** ![wqn628](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/wqn628/32/13783_2.png) [@wqn628](https://discuss.pytorch.org/u/wqn628)\
**Post date:** [August 16, 2018, 3:52am UTC](https://discuss.pytorch.org/t/the-problem-for-rnn-training-with-dataparallel/23221/5 "2018-08-16T03:52:40Z")

</div>

I 'm very sorry…here are the formated code.

```auto
class LSTM(nn.Module):
def init(self, input_size, hidden_layer, hidden_size, num_classes,
rnn_type=‘lstm’, dropout=0.0, bidirect=True, residual=False):
super(LSTM, self).init()
if bidirect:
layer = [nn.LSTM( input_size=input_size, hidden_size=hidden_size, batch_first=True, bidirectional=bidirect, dropout=dropout)]
for i in range(hidden_layer):
layer.append(nn.LSTM(hidden_size2, hidden_size, batch_first=True, dropout=0.0, bidirectional=bidirect))
self.lstm = nn.Sequential(layer)
self.lstm.add_module(“conn”,nn.LSTM(hidden_size2, num_classes, batch_first=True, dropout=0.0, bidirectional=bidirect))
def forward(self, x, input_len):
max_length = x.shape[1]
x = rnn_pack.pack_padded_sequence(x, input_len, batch_first=True)
x = self.lstm(x)
out , _ = rnn_pack.pad_packed_sequence(x, total_length=max_length, batch_first=True)
return out

```

```auto
def train_one_epoch(nnet, criterion, optimizer, train_loader, num_parallel, is_rnn=False):
nnet.train()
pos_frames = 0.0
train_frames = 0.0
for index, (key, feats, labels, len_list) in enumerate(train_loader):
print “=============”
print key
print feats.shape ##(12,T,)
print labels.shape ##(12,T)
#print type(len_list)
#print len_list.shape
print “=============”
labels = labels.view(labels.shape[0], labels.shape[1])
input_len = np.array(len_list)
input_sort_id = np.argsort(-input_len)
input_len = input_len[input_sort_id]
feat_mat = feats[th.LongTensor(input_sort_id)]
feat_mat = Variable(feat_mat.cuda())
input_unsort_id = th.LongTensor(np.argsort(input_sort_id))
optimizer.zero_grad()
if is_rnn:
label_mat = labels.view(labels.size(0) * labels.size(1))
targets = Variable(label_mat.cuda())
out = nnet(feat_mat, input_len)
out = out[input_unsort_id]
out_num = out.shape[2]
max_frame = int(max(len_list))
.
.
.

```

```auto
if th.cuda.is_available():
if args.multi_gpu == 1:
if th.cuda.device_count() > 1:
logger.info(“Let’s use {} GPUs!”.format(th.cuda.device_count()))
nnet = nn.DataParallel(nnet, device_ids=[0,1,2]) # dim = 0 [30, xxx] -> [10, …], [10, …], [10, …] on 3 GPUs
elif th.cuda.device_count() == 1:
logger.info("!!!WARNINGS:Only 1 GPU detected and Let’s use the only one GPUs!!!")
else:
logger.info("!!!ERROR!!!")
else:
logger.info("!!!Training on single GPU!!!")
else:
    logger.info("!!!WARNINGS:NO GPUs detected! AND The Net Will Be Trained On CPU!!!")
logger.info(nnet)

```

---

<div class="post-metadata">

**Author:** ![PTA](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/c5a1d2/32.png) [@PTA](https://discuss.pytorch.org/u/PTA)\
**Post date:** [August 16, 2018, 4:00am UTC](https://discuss.pytorch.org/t/the-problem-for-rnn-training-with-dataparallel/23221/6 "2018-08-16T04:00:06Z")

</div>

> [@wqn628](#):
>
> input\_len

It seems you are doing the sort/unsort stuff for sequence packing out of the forward method, which you should do inside based on your input (param #1 for forward method). What happened here is, each of your GPU got 4 sequences, with a sorted length of 12 sequences.

---

<div class="post-metadata">

**Author:** ![wqn628](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/wqn628/32/13783_2.png) [@wqn628](https://discuss.pytorch.org/u/wqn628)\
**Post date:** [August 16, 2018, 4:05am UTC](https://discuss.pytorch.org/t/the-problem-for-rnn-training-with-dataparallel/23221/7 "2018-08-16T04:05:09Z")

</div>

> [@PTA](#):
>
> seems you are doing the sort/unsort stuff for sequence packing out of the forward method, which you should do inside based on your input

do you mean that i should sort the sequence in the forward method rather than in the train\_one\_epoch method?

---

<div class="post-metadata">

**Author:** ![PTA](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/c5a1d2/32.png) [@PTA](https://discuss.pytorch.org/u/PTA)\
**Post date:** [August 16, 2018, 4:40am UTC](https://discuss.pytorch.org/t/the-problem-for-rnn-training-with-dataparallel/23221/8 "2018-08-16T04:40:51Z")

</div>

yes you should try that

---

<div class="post-metadata">

**Author:** ![wqn628](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/wqn628/32/13783_2.png) [@wqn628](https://discuss.pytorch.org/u/wqn628)\
**Post date:** [August 16, 2018, 5:25am UTC](https://discuss.pytorch.org/t/the-problem-for-rnn-training-with-dataparallel/23221/9 "2018-08-16T05:25:34Z")

</div>

ok，thank you very much，I‘ll try it。

---

<div class="post-metadata">

**Author:** ![wqn628](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/wqn628/32/13783_2.png) [@wqn628](https://discuss.pytorch.org/u/wqn628)\
**Post date:** [August 16, 2018, 5:40am UTC](https://discuss.pytorch.org/t/the-problem-for-rnn-training-with-dataparallel/23221/10 "2018-08-16T05:40:33Z")

</div>

here is the revised code. x is the input tensor (12, T, \*), len\_list is a tuple with 12 elements which represent the length of sequence(the number of frames).

```auto
    def forward (self, x, len_list):
        input_len = np.array(len_list)
        input_sort_id = np.argsort(-input_len)
        input_len = input_len[input_sort_id]
        x = x[th.LongTensor(input_sort_id)]
        max_length = x.shape[1]
        x = rnn_pack.pack_padded_sequence(x, input_len, batch_first=True)
        x = self.lstm(x)
        out , _ = rnn_pack.pad_packed_sequence(x, total_length=max_length, batch_first=True)
        return out

```

after i run this script again,the error occurs as follows:

```auto
Traceback (most recent call last):
  File "./train/train_rnn_pack_sort.py", line 195, in <module>
    train(args)
  File "./train/train_rnn_pack_sort.py", line 150, in train
    tr_frame_acc = train_epoch(nnet, criterion, optimizer, train_loader, num_parallel, train_dataset.num_frames, is_rnn=True)
  File "./train/train_rnn_pack_sort.py", line 60, in train_epoch
    train_frame, pos_frames = common_pack_sort.train_one_epoch(nnet, criterion, optimizer, train_loader, num_parallel, is_rnn)
  File "/search/speech/wangqingnan/asr_tools/pytorch/asr_egs/common/common_pack_sort.py", line 103, in train_one_epoch
    out = nnet(feats, len_list)
  File "/search/speech/wangqingnan/Anaconda/anaconda2/lib/python2.7/site-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/search/speech/wangqingnan/Anaconda/anaconda2/lib/python2.7/site-packages/torch/nn/parallel/data_parallel.py", line 114, in forward
    outputs = self.parallel_apply(replicas, inputs, kwargs)
  File "/search/speech/wangqingnan/Anaconda/anaconda2/lib/python2.7/site-packages/torch/nn/parallel/data_parallel.py", line 124, in parallel_apply
    return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
  File "/search/speech/wangqingnan/Anaconda/anaconda2/lib/python2.7/site-packages/torch/nn/parallel/parallel_apply.py", line 65, in parallel_apply
    raise output
AttributeError: 'tuple' object has no attribute 'size'

```

any help would be appreciated