# Error in use multiple gpu in my source

**URL:** <https://discuss.pytorch.org/t/error-in-use-multiple-gpu-in-my-source/165622>\
**Category:** distributed\
**Created:** [November 10, 2022, 7:01am UTC](https://discuss.pytorch.org/t/error-in-use-multiple-gpu-in-my-source/165622 "2022-11-10T07:01:47Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![setare](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/b782af/32.png) [@setare](https://discuss.pytorch.org/u/setare)\
**Post date:** [November 10, 2022, 7:01am UTC](https://discuss.pytorch.org/t/error-in-use-multiple-gpu-in-my-source/165622/1 "2022-11-10T07:01:47Z")

</div>

Hi I want to run my project on two gpu parallel so after I write my code, I write this command  
in this commands I write my model in paralell and then in pl.trainer in add parameter of gpus=2  
’  
def main():  
datamodule = DataModule(train\_ds, val\_ds)  
mymodel = mymodel(config)  
trainer = pl.Trainer(  
accelerator=“gpu”,  
gpus=2,  
)  
trainer.fit(model, datamodule)

if **name** == “ **main** ”:  
main()

but when I run it I see this error how to edit my source code?( I’d appreciate it if you could help me with it)  
‘TypeError: `Trainer.fit()` requires a `LightningModule`, got: DataParallel’

---

<div class="post-metadata">

**Author:** ![fduwjj](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/fduwjj/32/47855_2.png) [@fduwjj](https://discuss.pytorch.org/u/fduwjj)\
**Post date:** [November 14, 2022, 7:04pm UTC](https://discuss.pytorch.org/t/error-in-use-multiple-gpu-in-my-source/165622/2 "2022-11-14T19:04:38Z")

</div>

> [@setare](#):
>
> parallel so after I write my code, I write this command  
> in this commands I write my model in

Looks like this is something more related to lightning. Maybe can you post the all of your code here so that we can better help you? Thanks!

---

<div class="post-metadata">

**Author:** ![setare](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/b782af/32.png) [@setare](https://discuss.pytorch.org/u/setare)\
**Post date:** [November 14, 2022, 7:26pm UTC](https://discuss.pytorch.org/t/error-in-use-multiple-gpu-in-my-source/165622/3 "2022-11-14T19:26:45Z")

</div>

Hi, yes sure. Thank you for your responce.  
my source code is…

I send my source if it could be better show my error

This is data module

```auto
class DataModule(pl.LightningDataModule):

  def __init__ (self, train_dataset, val_dataset, batch_size = 1):
    super(DataModule, self). __init__ ()
    self.train_dataset = train_dataset
    self.val_dataset = val_dataset
    self.batch_size = batch_size
  def train_dataloader(self):
    return DataLoader(self.train_dataset, batch_size = self.batch_size, collate_fn = collate_fn, shuffle = True, num_workers = 0, pin_memory = True)
  
  def val_dataloader(self):
    return DataLoader(self.val_dataset, batch_size = self.batch_size,collate_fn = collate_fn, shuffle = False, num_workers = 0, pin_memory = True)

```

and this is part of model

```auto
class LaTrForVQA(pl.LightningModule):
  def __init__ (self, config , learning_rate = 1e-4, max_steps = 100000//2):
    super(LaTrForVQA, self). __init__ ()   
    self.config = config
    self.save_hyperparameters()
    self.latr = LaTr_for_finetuning(config)
    self.training_losses = []
    self.validation_losses = []
    self.max_steps = max_steps

  def configure_optimizers(self):
    return torch.optim.AdamW(self.parameters(), lr = self.hparams['learning_rate'])

  def forward(self, batch_dict):
    boxes = batch_dict['boxes']
    img = batch_dict['img']
    question = batch_dict['question']
    words = batch_dict['tokenized_words']
    answer_vector = self.latr(lang_vect = words, 
                               spatial_vect = boxes, 
                               img_vect = img, 
                               quest_vect = question
                               )
    return answer_vector

  def calculate_metrics(self, prediction, labels):

      ## Calculate the accuracy score between the prediction and ground label for a batch, with considering the pad sequence
      batch_size = len(prediction)
      ac_score = 0

      for (pred, gt) in zip(prediction, labels):
        ac_score+= calculate_acc_score(pred.detach().cpu(), gt.detach().cpu())
      ac_score = ac_score/batch_size
      return ac_score

  def training_step(self, batch, batch_idx):
    answer_vector = self.forward(batch)

    ## https://discuss.huggingface.co/t/bertformaskedlm-s-loss-and-scores-how-the-loss-is-computed/607/2
    loss = nn.CrossEntropyLoss()(answer_vector.reshape(-1,self.config['classes']), batch['answer'].reshape(-1))
    _, preds = torch.max(answer_vector, dim = -1)

    ## Calculating the accuracy score
    train_acc = self.calculate_metrics(preds, batch['answer'])
    train_acc = torch.tensor(train_acc)

    ## Logging
    self.log('train_ce_loss', loss,prog_bar = True)
    self.log('train_acc', train_acc, prog_bar = True)
    self.training_losses.append(loss.item())

    return loss

  def validation_step(self, batch, batch_idx):
    logits = self.forward(batch)
    loss = nn.CrossEntropyLoss()(logits.reshape(-1,self.config['classes']), batch['answer'].reshape(-1))
    _, preds = torch.max(logits, dim = -1)

    ## Validation Accuracy
    val_acc = self.calculate_metrics(preds.cpu(), batch['answer'].cpu())
    val_acc = torch.tensor(val_acc)

    ## Logging
    self.log('val_ce_loss', loss, prog_bar = True)
    self.log('val_acc', val_acc, prog_bar = True)    
    return {'val_loss': loss, 'val_acc': val_acc}

  def optimizer_step(self, epoch_nb, batch_nb, optimizer, optimizer_i, opt_closure = None, on_tpu=False,
    using_native_amp=False, using_lbfgs=False):
        if self.trainer.global_step < 1000:
            lr_scale = min(1., float(self.trainer.global_step + 1) / 1000.)
            for pg in optimizer.param_groups:
                pg['lr'] = lr_scale * self.hparams.learning_rate
        else:
            for pg in optimizer.param_groups:
                pg['lr'] = polynomial(self.hparams.learning_rate, self.trainer.global_step, max_iter = self.max_steps)
        optimizer.step(opt_closure)
        optimizer.zero_grad()

  def validation_epoch_end(self, outputs):       
        val_loss = torch.stack([x['val_loss'] for x in outputs]).mean()
        val_acc = torch.stack([x['val_acc'] for x in outputs]).mean()
        self.log('val_loss_epoch_end', val_loss, on_epoch=True, sync_dist=True)
        self.log('val_acc_epoch_end', val_acc, on_epoch=True, sync_dist=True)
        
        self.val_prediction = []

model=LaTrForVQA(config)

```

and this is for trainer

```auto
trainer = pl.Trainer(
    max_steps = max_steps,
    default_root_dir="runs",
    gpus=2,
    deterministic=True,
) 

```

Now I fit my model

```auto
datamodule = DataModule(train_ds, val_ds)
trainer.fit(model,datamodule)   

```

If you have any other questions, be sure to ask…  
I change my code according some suggestion to don’t use parallel and only use pl.trainer(gpus=2) but with this code I still can’t use both of GPU in parallel and only one GPU work in source code and I dont know what other change I should apply in this source.

---

<div class="post-metadata">

**Author:** ![fduwjj](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/fduwjj/32/47855_2.png) [@fduwjj](https://discuss.pytorch.org/u/fduwjj)\
**Post date:** [November 15, 2022, 8:04pm UTC](https://discuss.pytorch.org/t/error-in-use-multiple-gpu-in-my-source/165622/4 "2022-11-15T20:04:15Z")

</div>

> [@setare](#):
>
> `LightningDataModule`

To enable parallelization in Lightning training, you need to use strategy here:  
[accelerators — PyTorch Lightning 2.1.2 documentation](https://pytorch-lightning.readthedocs.io/en/stable/api_references.html#strategies).

---

<div class="post-metadata">

**Author:** ![setare](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/b782af/32.png) [@setare](https://discuss.pytorch.org/u/setare)\
**Post date:** [November 16, 2022, 5:07am UTC](https://discuss.pytorch.org/t/error-in-use-multiple-gpu-in-my-source/165622/5 "2022-11-16T05:07:56Z")

</div>

Hi,  
I use different type of strategy and anyone don’t work and give me different type of errors and I don’t know what to do and can’t find any solution  
I mention some of this error

When I set strategy=“ddp\_spawn” (or ddp).It gives me this error

MisconfigurationException: `Trainer(strategy='ddp_spawn')` is not compatible with an interactive environment. Run your code as a script, or choose one of the compatible strategies: Trainer(strategy=None|dp|ddp\_fork). In case you are spawning processes yourself, make sure to include the Trainer creation inside the worker function

When I set strategy=“ddp\_fork” .It gives me this error  
"Cannot re-initialize CUDA in forked subprocess. To use CUDA with "  
RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the ‘spawn’ start method

When I set strategy=“dp” . It gives me this error  
AssertionError: Gather function not implemented for CPU tensors

When I set strategy=" ddp\_fork " (or ddp\_notebook).It gives me this error  
ValueError: ‘ddp\_notebook’ is not a valid DistributedType

If it is my wrong in source code what to do?  
I’d so appreciate it if you could help me with this wrong.

---

<div class="post-metadata">

**Author:** ![fduwjj](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/fduwjj/32/47855_2.png) [@fduwjj](https://discuss.pytorch.org/u/fduwjj)\
**Post date:** [November 16, 2022, 5:37am UTC](https://discuss.pytorch.org/t/error-in-use-multiple-gpu-in-my-source/165622/6 "2022-11-16T05:37:02Z")

</div>

How did you set your multi-process env? I mean from these errors looks it is complaining the way you set up multi-process env.

---

<div class="post-metadata">

**Author:** ![setare](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/b782af/32.png) [@setare](https://discuss.pytorch.org/u/setare)\
**Post date:** [November 16, 2022, 6:03am UTC](https://discuss.pytorch.org/t/error-in-use-multiple-gpu-in-my-source/165622/7 "2022-11-16T06:03:56Z")

</div>

If I have understood your point, first of all I create a virtual environment in anaconda navigator with python 3.7 and my source could identify both of GPU

import torch  
print(‘\_\_CUDNN VERSION:’, torch.backends.cudnn.version())  
print(‘\_\_Number CUDA Devices:’, torch.cuda.device\_count())  
print(‘\_\_CUDA Device Name 1:’,torch.cuda.get\_device\_name(0))  
print(‘\_\_CUDA Device Name 2:’,torch.cuda.get\_device\_name(1))  
print(‘\_\_CUDA Device current:’,torch.cuda.get\_device\_name(torch.cuda.current\_device()))  
print(‘\_\_CUDA Device Total Memory [GB]:’,torch.cuda.get\_device\_properties(0).total\_memory/1e9)  
print(‘\_\_CUDA Device Total Memory [GB]:’,torch.cuda.get\_device\_properties(1).total\_memory/1e9)  
print(‘\_\_CUDA Device Total Memory [GB]:’,torch.cuda.get\_device\_properties(0).total\_memory/1e9+torch.cuda.get\_device\_properties(1).total\_memory/1e9)

\_CUDNN VERSION: 8302  
\_\_Number CUDA Devices: 2  
\_\_CUDA Device Name 1: NVIDIA GeForce GTX 1080 Ti  
\_\_CUDA Device Name 2: NVIDIA GeForce GTX 1070 Ti  
\_\_CUDA Device Total Memory [GB]: 11.721572352  
\_\_CUDA Device Total Memory [GB]: 8.514043904  
\_\_CUDA Device Total Memory [GB]: 20.235616256  
​  
version of pytorch-lightning is 1.8.0.post1

and I set for different kinds of pytorch package . All of them have these errors.  
It is worth mentioning that with the same settings I can use both of my GPU in dataparallel only but I can’t use multi gpu in pytorch-lightning .

If I have not understood your point, may you give more guidance.

---

<div class="post-metadata">

**Author:** ![setare](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/b782af/32.png) [@setare](https://discuss.pytorch.org/u/setare)\
**Post date:** [November 19, 2022, 7:09am UTC](https://discuss.pytorch.org/t/error-in-use-multiple-gpu-in-my-source/165622/8 "2022-11-19T07:09:53Z")

</div>

Hi @fduwjj ,  
This problem is very important for me. I would appreciate it if you gave your opinion about this problem. 🙏

---

<div class="post-metadata">

**Author:** ![harshas](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/harshas/32/57107_2.png) [@harshas](https://discuss.pytorch.org/u/harshas)\
**Post date:** [February 7, 2023, 9:13pm UTC](https://discuss.pytorch.org/t/error-in-use-multiple-gpu-in-my-source/165622/10 "2023-02-07T21:13:03Z")

</div>

try replacing pl.LightningDataModule in class DataModule() with pl.LightningModule

---

<div class="post-metadata">

**Author:** ![Tali\_M](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tali_m/32/58288_2.png) [@Tali\_M](https://discuss.pytorch.org/u/Tali_M)\
**Post date:** [March 13, 2023, 7:44am UTC](https://discuss.pytorch.org/t/error-in-use-multiple-gpu-in-my-source/165622/11 "2023-03-13T07:44:23Z")

</div>

Did you find an answer? I’m having similar problems…

---

<div class="post-metadata">

**Author:** ![setare](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/b782af/32.png) [@setare](https://discuss.pytorch.org/u/setare)\
**Post date:** [July 11, 2023, 4:31am UTC](https://discuss.pytorch.org/t/error-in-use-multiple-gpu-in-my-source/165622/12 "2023-07-11T04:31:25Z")

</div>

Unfortunately, no. I use gpu that have more RAM.