Hi everyone,
In traditional application workloads (e.g., Java/Golang), there are generally two approaches to profiling:
- On-demand Profiling: If performance issues are reproducible, we profile for a short duration (e.g., 10s or 1m) to analyze the root cause. The overhead exists only during this window.
- Continuous Profiling: If issues are intermittent or hard to predict (e.g., a specific user request at midnight), we enable a profiler that continuously collects data and saves it to rotating files. Once an issue is reproduced, we stop the profiler and analyze the specific time range.
While the PyTorch Profiler is excellent for short-term tracing (as shown in tutorials like Hugging Face’s guide: Profiling in PyTorch (Part 1): A Beginner's Guide to torch.profiler ), the documentation of PyTorch Profiler explicitly warns: “Tracing all of the execution can be slow and result in very large trace files.” Inference engines like vLLM also emphasize this limitation.
I would like to propose adding Continuous Profiling support to the PyTorch Profiler. This would require two main improvements:
1.Streaming Data to Disk
- Currently, all data is buffered in memory, which causes RSS to grow significantly during long runs.
- Proposal: The profiler should stream Python stack traces, CPU ops, and CUDA kernels directly to files (preferably in a compact binary format) with rotation support.
- Optimization: To mitigate overhead, we could support sampling tracing. For example, sampling Python stacks could reduce overhead from >15% to <5%, trading off some accuracy for stability. Even without sampling, simply flushing data periodically to keep RSS steady would be a huge improvement.
2.Post-processing and Visualization
We need a tool or script to filter data based on specific timestamps and convert the binary logs into consumable formats (like Chrome Trace JSON or tabular data) for analysis.
Context:
Polar Signals has developed a continuous CUDA profiling system (see their blog post: Continuous NVIDIA CUDA Profiling In Production | Polar Signals ), but it is a standalone tool.
I am planning to implement a prototype for this in my own copy of PyTorch. I searched the issue tracker and found no related discussions.
Is there any existing plan for this?
If not, are there specific architectural reasons why this hasn’t been done?
Any thoughts?
Thanks!