Very low GPU utilization when training YOLOV8

vision
1

I’m training a model on a small dataset (139 images, total size of 14MB, stored on HDD) for an object detection project. I have a GTX 1650 with 4GB memory on my laptop and I’m using YOLOV8 pre-trained models. I start the training using yolo detect train [arguments] with these arguments:

  • data=config.yaml
  • model=yolov8m.pt
  • epochs=3
  • imgsz=640
  • batch=4 (can’t set to more than 4, memory issue)
  • workers=8
  • device=0 (no difference if it’s omitted or not)

I’ve checked whether PyTorch recognizes my GPU using the code below, and it does:

import torch
torch.cuda.is_available() // True

But during the training process, the GPU utilization is 0-2%, GPU memory 3/4GB, CPU usage 10-16%, and RAM usage 14/16GB. Why is GPU utilization so low? Am I doing somehting wrong?

  • Windows 10
  • CUDA 12.1
  • PyTorch 2.4.0
  • Ultralytics 8.2.77
  • torchvision 0.19.0

A screenshot of task manager during training:

Capture
Capture834×742 30.7 KB

2

Not sure if I can pinpoint but older GPUs may not be able to maximally utilise the newer versions I guess.

Plus your data is quite small as well.

Another thing could be your torch environment itself?

Look into the issues mentioned on the YOLOv8 repo. Quite a few people have faced this and there isn’t any one solution unfortunately.

3

You need to either enable the Compute view in your Task Manager or use nvidia-smi to see the GPU utilization on Windows.

5

Not a windows user — but why does one have explicitly open the compute view? Is it not automatically tracked?

6

As you can see in the posted picture, it’s not as only 3D, Copy, Video Encode/Decode are shown. I’m also not using Windows, but know that the default view does not show any CUDA compute kernels.
Based on this post it seems a CUDA view might also be available, but unsure.

I would recommend sticking to nvidia-smi as it will show the GPU utilization properly.

7

Ah okay. Thanks! nvidia-smi (and nvitop) is my goto anyway.

8

It seems a few months after my post, a new feature was added to windows 10 (also 11) called " Hardware-Accelerated GPU Scheduling ."
When this feature is on, CUDA is not visible as one of the dropdown menu options in GPU tab in Task Manager (Source). Thus, turning off this feature by following these steps (in windows 11) will probably return CUDA back to Task Manager:

  1. Open Windows Setting > System > Display> Graphics
  2. Select Change default graphics settings
  3. Turn off " Hardware-Accelerated GPU Scheduling ."
  4. Reboot the system.

Note: These steps might be different in windows 10. Googling will easily show you how to turn it off.