Sanity-checking a first-pass input-bound vs compute-bound check for low GPU utilization

data
1

Sanity-checking a first-pass input-bound vs compute-bound check for low GPU utilization

Hi all,

Disclosure: I am working on TraceML, a small tracing tool for PyTorch training, and this question comes from that work. I am posting for sanity-check the methodology.

A pattern I keep running into is:

  • GPU utilization is low
  • training step time is high
  • model compute is not obviously the bottleneck
  • the GPU appears to be waiting on the input pipeline

Examples could be slow decoding, tokenization, disk/network I/O, collation, or DataLoader worker behavior.

I built a small test case that compares the same training setup in two modes:

  1. a compute-heavy baseline
  2. the same setup with a deliberately delayed input path

The idea is to time input fetch separately from the forward/backward/optimizer phase and use that as a cheap first-pass classification before reaching for heavier profilers.

Metric Fast baseline Slow input path
Diagnosis Compute-bound Input-bound
Total step time 37.3 ms 563.3 ms
Input loading 1.9 ms 531.8 ms
Model compute 35.0 ms 31.0 ms
GPU utilization 67% 7%

Here, model compute stays roughly the same, but input loading takes over the step and GPU utilization drops.

I am curious how people who regularly tune PyTorch training performance think about this early debugging stage:

  1. When you notice low GPU utilization, what is your usual escalation order?

    • nvidia-smi / dmon
    • manual timers around the DataLoader
    • torch.profiler
    • Nsight Systems
    • something else?

  2. Is an early “input-bound vs compute-bound” split useful in practice?
    Or do you usually need a deeper breakdown immediately, such as storage I/O vs CPU transforms vs collation vs H2D transfer?

  3. For distributed jobs, do you think this framing is still useful, or does rank skew / synchronization wait make the binary framing too simplistic?

My current thinking is that this kind of check is useful as a smoke test, but not a replacement for deeper profiling. It tells you where to aim the heavier tools.

Feedback on whether this abstraction is useful, misleading, or missing an important category would be helpful.

Full context and reproducible write-up: