# Extreme single thread cpu kernel usage while training on GPU

**URL:** <https://discuss.pytorch.org/t/extreme-single-thread-cpu-kernel-usage-while-training-on-gpu/172380>\
**Category:** Uncategorized\
**Created:** [February 12, 2023, 7:25am UTC](https://discuss.pytorch.org/t/extreme-single-thread-cpu-kernel-usage-while-training-on-gpu/172380 "2023-02-12T07:25:02Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![Yuyao\_Huang](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/yuyao_huang/32/57240_2.png) [@Yuyao\_Huang](https://discuss.pytorch.org/u/Yuyao_Huang)\
**Post date:** [February 12, 2023, 7:25am UTC](https://discuss.pytorch.org/t/extreme-single-thread-cpu-kernel-usage-while-training-on-gpu/172380/1 "2023-02-12T07:25:02Z")

</div>

Hello there,  
Just into deep learning, and currently I am facing some weird issues regarding the model I am working on. As the title and images below suggest, during training, one of the cpu cores is experiencing extreme kernel usage while other cores barely moves. Despite I am using GPU as the accelerator.

 ![kernel useage](https://discuss.pytorch.org/uploads/default/original/3X/8/a/8aaf6b8c3e137e42a090bb8afd27b0fe6c0b9043.png)  
 ![overtime](https://discuss.pytorch.org/uploads/default/original/3X/8/a/8ad32b84347e9aff3959396a2f231a16272f7d80.png)  
I had read quite a few discussions regarding similar issues, but none fixed my problem.  
e.g. [CPU usage extremely high](https://discuss.pytorch.org/t/cpu-usage-extremely-high/52172)  
and [Very high CPU utilization with pin\_memory=True and num\_workers \> 0 · Issue #25010 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/issues/25010)  
I had tried to set the num\_workers in the dataloader to 2 or more, make no difference. Set pin\_memory to True or False doesn’t seem to do anything as well.  
torch.get\_num\_threads() returns 6, but torch.set\_num\_threads() = 6 doesn’t help with the issue.  
If I switch the accelerator from cuda to cpu, the model also only uses one cpu core for training, which is extremely inefficient.  
The model itself is a Half-UNet for image segmentation, 16bit grey scale image data are converted into tensors in the Dataset, before a variety of transformations got applied and send into the model for training.

Computer Specs:  
CPU: Ryzen 5500  
GPU: 1050ti  
Disks: Data are stored in two 1TB Barracuda HDD in raid 1 using AMD Raid, Operating system and Pycharm are stored in a SSD  
Memory: 32GB

Pytorch Version: 1.13.1  
Cuda Version: 11.7  
Operating System: Linux mint cinnamon

Something interesting I noticed is when I train with my laptop, which has an 11th gen intel i5 (with no discrete graphics) and is running on Windows. The same piece of codes will correctly uses all of the cpu cores of the laptops.

Here is the Dataset class I used:

```auto
class Train_Dataset(torch.utils.data.Dataset):
    def __init__ (self, images_dir, labels_dir):
        # Get a list of file paths for images and labels
        self.file_list = make_dataset_train(images_dir, labels_dir)
        self.num_files = len(self.file_list)
        self.img_tensors = []
        self.lab_tensors = []
        # Pre-allocating the memory for img_tensors and lab_tensors
        for idx in range(self.num_files):
            # Convert the image and label path to tensors
            img_tensor = path_to_tensor(self.file_list[idx][0], label=False)
            lab_tensor = path_to_tensor(self.file_list[idx][1], label=True)
            lab_tensor = lab_tensor.long()
            # Append the tensors to the list
            self.img_tensors.append(img_tensor)
            self.lab_tensors.append(lab_tensor)
        super(). __init__ ()

    def __len__ (self):
        return self.num_files

    def __getitem__ (self, idx):
        # Get the image and label tensors at the specified index
        img_tensor = self.img_tensors[idx]
        lab_tensor = self.lab_tensors[idx]

        # Decide whether to apply each random transformation
        random_transforms = [random.random() < 0.5 for i in range(6)]

        # Apply vertical flip
        if random_transforms[0]:
            img_tensor, lab_tensor = T_F.vflip(img_tensor), T_F.vflip(lab_tensor)
        # Apply horizontal flip
        if random_transforms[1]:
            img_tensor, lab_tensor = T_F.hflip(img_tensor), T_F.hflip(lab_tensor)
        # Apply Gaussian blur
        if random_transforms[2]:
            img_tensor = gaussian_blur_3d(img_tensor, random.randint(1, 2) * 2 + 1)
        # Apply padding and cropping
        if random_transforms[3]:
            # Randomly generate padding sizes
            left_pad_size = random.randint(2, 64)
            right_pad_size = random.randint(2, 64)
            up_pad_size = random.randint(2, 64)
            down_pad_size = random.randint(2, 64)
            img_tensor = img_tensor[:, up_pad_size:-down_pad_size, left_pad_size:-right_pad_size]
            lab_tensor = lab_tensor[:, up_pad_size:-down_pad_size, left_pad_size:-right_pad_size]
            padding = (left_pad_size, right_pad_size,
                       up_pad_size, down_pad_size,
                       0, 0)
            img_tensor = F.pad(img_tensor, padding, mode="constant")
            lab_tensor = F.pad(lab_tensor, padding, mode="constant")
        # Apply rotations
        if random_transforms[4]:
            rotation_angle = random.uniform(-90, 90)
            img_tensor = T_F.rotate(img_tensor, rotation_angle)
            lab_tensor = T_F.rotate(lab_tensor, rotation_angle)
        # Apply contrast adjustments
        if random_transforms[5]:
            img_tensor = adjust_contrast_3d(img_tensor)
        img_tensor = img_tensor[None, :].to(torch.float16)
        return img_tensor, lab_tensor

```

The model itself and training process:  
(I know it’s handled by pytorch lightning but just in case it’s indeed a bug at pytorch end)

```auto
class HalfNet(nn.Module):

    def __init__ (self):
        super(HalfNet, self). __init__ ()
        channel_base = 18
        self.inc = ModuleComponents.DoubleConv3D(1, channel_base, ghost=False, kernel_size=(1, 3, 3))
        self.down1 = ModuleComponents.Down3D(channel_base, channel_base, ghost=False, kernel_size=(1, 3, 3))
        self.merge1 = ModuleComponents.Merge3D(2)
        self.down2 = ModuleComponents.Down3D(channel_base, channel_base, ghost=False, kernel_size=(1, 3, 3))
        self.merge2 = ModuleComponents.Merge3D(4)
        self.down3 = ModuleComponents.Down3D(channel_base, channel_base, ghost=False, kernel_size=(1, 3, 3))
        self.merge3 = ModuleComponents.Merge3D(8)
        self.down4 = ModuleComponents.Down3D(channel_base, channel_base, ghost=False, kernel_size=(1, 3, 3))
        self.merge4 = ModuleComponents.Merge3D(16)
        self.down5 = ModuleComponents.Down3D(channel_base, channel_base, ghost=False, kernel_size=(1, 3, 3))
        self.merge5 = ModuleComponents.Merge3D(32)
        self.z_work = ModuleComponents.DoubleConv3D(channel_base, channel_base, ghost=False, kernel_size=(3, 1, 1))
        self.outc = ModuleComponents.OutConv3D(channel_base, 2)

    def forward(self, x):
        x = self.inc(x)
        x2 = self.down1(x)
        x = self.merge1(x, x2)
        x2 = self.down2(x2)
        x = self.merge2(x, x2)
        x2 = self.down3(x2)
        x = self.merge3(x, x2)
        x2 = self.down4(x2)
        x = self.merge4(x, x2)
        x2 = self.down5(x2)
        x = self.merge5(x, x2)
        x = self.z_work(x)
        x = self.outc(x)
        return x

class HalfNetPL(pl.LightningModule):

    def __init__ (self, learning_rate=0.001):
        super(HalfNetPL, self). __init__ ()
        self.model = HalfNet()
        self.learning_rate = learning_rate
        #self.automatic_optimization = False

    def forward(self, image):
        return self.model(image)

    def training_step(self, batch, batch_idx):
        #loss = self._step(batch, batch_idx)
        #self.log("train_loss", loss, on_step=False, on_epoch=True, prog_bar=True, logger=True)
        return {'loss': self._step(batch, batch_idx)}

    def validation_step(self, batch, batch_idx):
        loss = self._step(batch, batch_idx)
        self.log("val_loss", loss, on_step=False, on_epoch=True, prog_bar=False, logger=False)
        return {'val_loss': loss}

    def configure_optimizers(self):
        optimizer = torch.optim.Adam(self.parameters(), lr=self.learning_rate)
        scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, mode='min', factor=0.5, patience=30,
                                                               threshold=0.0001, threshold_mode='rel',
                                                               cooldown=0, min_lr=0.00025, verbose=True)
        return {"optimizer": optimizer,
                "lr_scheduler": {"scheduler": scheduler, "monitor": "val_loss"},}

    def lr_scheduler_step(self, scheduler, optimizer_idx, metric):
        if metric is None:
            scheduler.step()
        else:
            scheduler.step(metric)

    def predict_step(self, batch, batch_idx: int, dataloader_idx: int = 0):
        x = batch
        return torch.argmax(self(x), dim=1)

    def validation_epoch_end(self, outputs):
        avg_loss = torch.stack([x['val_loss'] for x in outputs]).mean()
        self.logger.experiment.add_scalar("Loss/Val", avg_loss, self.current_epoch)
        self.val_loss = avg_loss
    
    def training_epoch_end(self, outputs):
        #if(self.current_epoch==0):
            #sampleImg = torch.rand((1,1,5,128,128))
            #self.logger.experiment.add_graph(HalfNet(), sampleImg)
        avg_loss = torch.stack([x['loss'] for x in outputs]).mean()
        self.logger.experiment.add_scalar("Loss/Train", avg_loss, self.current_epoch)

    def test_step(self, batch, batch_idx):
        loss = self._step(batch, batch_idx)
        self.log("test_loss", loss, on_step=False, on_epoch=True, prog_bar=True, logger=True)
        return {'test_loss': loss}
        
    def _step(self, batch, batch_idx):
        x, y = batch
        y_hat = self.forward(x)
        #weight = torch.tensor(data=(1, 1), device=self.device)
        #weight = weight.to(torch.float)
        loss = F.cross_entropy(input=y_hat, target=y)
        return loss

if __name__ == " __main__":
    train_dataset = DataComponents.Train_Dataset('datasets/train/img',
                                                 'datasets/train/lab',)
    train_loader = torch.utils.data.DataLoader(dataset=train_dataset, batch_size=2, shuffle=True, num_workers=2, pin_memory=False)
    # Setting up training parameters
    torch.set_float32_matmul_precision('medium')
    trainer = pl.Trainer(max_epochs=200, log_every_n_steps=1, logger=logger,
                         accelerator="gpu", enable_checkpointing=False,
                         precision=16, auto_lr_find=True, gradient_clip_val=0.5,)
    model = HalfNetPL()
    # Train the model
    trainer.fit(model,
                train_dataloaders=train_loader)

```

Thank you.

---

<div class="post-metadata">

**Author:** ![eqy](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/eqy/32/46233_2.png) [@eqy](https://discuss.pytorch.org/u/eqy)\
**Post date:** [February 12, 2023, 10:20pm UTC](https://discuss.pytorch.org/t/extreme-single-thread-cpu-kernel-usage-while-training-on-gpu/172380/2 "2023-02-12T22:20:05Z")

</div>

I would check if changing `OMP_NUM_THREADS` changes the behavior. Additionally, it may be useful to check if e.g., running the model on the same (already transferred to GPU) data over and over again changes the behavior to rule out other potential issues.

---

<div class="post-metadata">

**Author:** ![Yuyao\_Huang](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/yuyao_huang/32/57240_2.png) [@Yuyao\_Huang](https://discuss.pytorch.org/u/Yuyao_Huang)\
**Post date:** [February 13, 2023, 4:25am UTC](https://discuss.pytorch.org/t/extreme-single-thread-cpu-kernel-usage-while-training-on-gpu/172380/3 "2023-02-13T04:25:11Z")

</div>

> [@eqy](#):
>
> OMP\_NUM\_THREADS

Hello eqy,  
I tired `export OMP_NUM_THREADS=6`, doesn’t seem to do anything. I also tried transferring all the data in the dataloader to cuda device in the ` __init__ ` function as well, it marginally improved the rate of training (if any), but doesn’t help with my issue.

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [February 13, 2023, 5:29am UTC](https://discuss.pytorch.org/t/extreme-single-thread-cpu-kernel-usage-while-training-on-gpu/172380/4 "2023-02-13T05:29:40Z")

</div>

Could you check your plain PyTorch model (i.e. without the Lightning wrapper) and see if you are still using a single CPU core?

---

<div class="post-metadata">

**Author:** ![Yuyao\_Huang](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/yuyao_huang/32/57240_2.png) [@Yuyao\_Huang](https://discuss.pytorch.org/u/Yuyao_Huang)\
**Post date:** [February 13, 2023, 5:36am UTC](https://discuss.pytorch.org/t/extreme-single-thread-cpu-kernel-usage-while-training-on-gpu/172380/5 "2023-02-13T05:36:58Z")

</div>

I honestly don’t know how to use a plain pytorch model for training… Guess I will look into it.

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [February 13, 2023, 5:40am UTC](https://discuss.pytorch.org/t/extreme-single-thread-cpu-kernel-usage-while-training-on-gpu/172380/6 "2023-02-13T05:40:09Z")

</div>

You could also try to opposite and check if all Lightning models suffer from this issue in your setup or in case it’s only a specific set of models what might be the common pattern.

---

<div class="post-metadata">

**Author:** ![Yuyao\_Huang](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/yuyao_huang/32/57240_2.png) [@Yuyao\_Huang](https://discuss.pytorch.org/u/Yuyao_Huang)\
**Post date:** [February 13, 2023, 6:10am UTC](https://discuss.pytorch.org/t/extreme-single-thread-cpu-kernel-usage-while-training-on-gpu/172380/7 "2023-02-13T06:10:01Z")

</div>

Just check with the lightning MNIST example, the issue persists, so I guess all Lightning models will suffer from this issue…

---

<div class="post-metadata">

**Author:** ![Yuyao\_Huang](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/yuyao_huang/32/57240_2.png) [@Yuyao\_Huang](https://discuss.pytorch.org/u/Yuyao_Huang)\
**Post date:** [February 13, 2023, 6:18am UTC](https://discuss.pytorch.org/t/extreme-single-thread-cpu-kernel-usage-while-training-on-gpu/172380/8 "2023-02-13T06:18:30Z")

</div>

Tried with the pure pytorch mnist example from [MNIST Handwritten Digit Recognition in PyTorch - Nextjournal](https://nextjournal.com/gkoehler/pytorch-mnist) , this one correctly utilise all my cpu cores, so I guess this issue is more on pytorch lightning side.  
PS: Issue submitted on Lightning’s Github as [Extreme single thread cpu kernel usage while training on GPU · Issue #16737 · Lightning-AI/lightning (github.com)](https://github.com/Lightning-AI/lightning/issues/16737)

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [February 13, 2023, 8:24pm UTC](https://discuss.pytorch.org/t/extreme-single-thread-cpu-kernel-usage-while-training-on-gpu/172380/9 "2023-02-13T20:24:16Z")

</div>

Thanks for sharing the related GitHub issue as I would also be interested in the root cause.

---

<div class="post-metadata">

**Author:** ![adrianwaelchli](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/adrianwaelchli/32/38136_2.png) [@adrianwaelchli](https://discuss.pytorch.org/u/adrianwaelchli)\
**Post date:** [February 14, 2023, 3:38am UTC](https://discuss.pytorch.org/t/extreme-single-thread-cpu-kernel-usage-while-training-on-gpu/172380/10 "2023-02-14T03:38:36Z")

</div>

I converted @Yuyao_Huang’s code from Lightning to raw PyTorch and am able to reproduce the observations on CPU usage. See my comment here plus the full code: [Extreme single thread cpu kernel usage while training on GPU · Issue #16737 · Lightning-AI/lightning · GitHub](https://github.com/Lightning-AI/lightning/issues/16737#issuecomment-1429060866)

The difference in CPU behavior shows when enabling and disabling the autocast context manager. Not an expert, but I would guess mixed precision on CPU is perhaps not as optimized as on CUDA and exists more for debugging purposes, so perhaps it isn’t so surprising to see this happen?

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [February 14, 2023, 6:39am UTC](https://discuss.pytorch.org/t/extreme-single-thread-cpu-kernel-usage-while-training-on-gpu/172380/11 "2023-02-14T06:39:57Z")

</div>

That’s an interesting observation!  
Based on your comment:

> When using the autocast (mixed precision) context manager, we can see that the CPU usage is mainly concentrated in a single core. When setting `use_autocast=False` however, the CPU usage is more spread across several cores.

it seems you are using `autocast` on the CPU and then observe this behavior?  
Also, it seems you are “mainly” seeing a single core usage but other cores are still used?

---

<div class="post-metadata">

**Author:** ![adrianwaelchli](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/adrianwaelchli/32/38136_2.png) [@adrianwaelchli](https://discuss.pytorch.org/u/adrianwaelchli)\
**Post date:** [February 17, 2023, 1:19am UTC](https://discuss.pytorch.org/t/extreme-single-thread-cpu-kernel-usage-while-training-on-gpu/172380/12 "2023-02-17T01:19:00Z")

</div>

Sorry for the delayed response. Yes @ptrblck, that’s exactly right. That single core heavy behavior isn’t visible when removing the autocast and the load is more spread out across cores.

---

<div class="post-metadata">

**Author:** ![DanGrahn](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/dangrahn/32/59892_2.png) [@DanGrahn](https://discuss.pytorch.org/u/DanGrahn)\
**Post date:** [April 24, 2023, 9:46pm UTC](https://discuss.pytorch.org/t/extreme-single-thread-cpu-kernel-usage-while-training-on-gpu/172380/13 "2023-04-24T21:46:36Z")

</div>

Hi all,  
I’m encountering the same issue. How would one disable autocast using only the Lightning API?

Thanks!

---

<div class="post-metadata">

**Author:** ![bekhzod](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/bekhzod/32/56334_2.png) [@bekhzod](https://discuss.pytorch.org/u/bekhzod)\
**Post date:** [April 27, 2023, 7:21am UTC](https://discuss.pytorch.org/t/extreme-single-thread-cpu-kernel-usage-while-training-on-gpu/172380/14 "2023-04-27T07:21:54Z")

</div>

@Yuyao_Huang I have the same issue. Have you solved the problem?

---

<div class="post-metadata">

**Author:** ![Yuyao\_Huang](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/yuyao_huang/32/57240_2.png) [@Yuyao\_Huang](https://discuss.pytorch.org/u/Yuyao_Huang)\
**Post date:** [April 27, 2023, 7:22am UTC](https://discuss.pytorch.org/t/extreme-single-thread-cpu-kernel-usage-while-training-on-gpu/172380/15 "2023-04-27T07:22:55Z")

</div>

In the end, I just endured it and finished my project with it.