I tried to develop mult-gpu CRNN with CTCLoss, when modified code with DataParallel and execute train code and got this error:
Traceback (most recent call last):
File “train.py”, line 373, in
main()
File “train.py”, line 85, in main
train(model, train_dataloader, criterion, optimizer, epoch, scaler, writer)
File “train.py”, line 230, in train
loss = criterion(output_log_probs, target, images_lengths, target_length)
File “/home/gpudev/miniconda3/envs/gpu38/lib/python3.8/site-packages/torch/nn/modules/module.py”, line 1110, in _call_impl
return forward_call(*input, **kwargs)
File “/home/gpudev/miniconda3/envs/gpu38/lib/python3.8/site-packages/torch/nn/modules/loss.py”, line 1743, in forward
return F.ctc_loss(log_probs, targets, input_lengths, target_lengths, self.blank, self.reduction,
File “/home/gpudev/miniconda3/envs/gpu38/lib/python3.8/site-packages/torch/nn/functional.py”, line 2599, in ctc_loss
return torch.ctc_loss(
RuntimeError: input_lengths must be of size batch_size
But it was not occured when do with single gpu…I cannot understand about it well
I hope it was just my fault but other projects are getting same issue (and it does not work for me, too) : https://github.com/Holmeyoung/crnn-pytorch/issues/31
So I want to know that CTCLoss can use with DataParallel or other way with 2 or more gpus and how to set data with