PyTorch NCCL Flight Recorder

gjit_JNPR · August 1, 2024, 6:46pm

The recently published LLaMa-3 paper (https://arxiv.org/pdf/2407.21783) mentions the use of a PyTorch built-in feature called NCCL Flight Recorder as shown below:. On further exploring PyTorch, I did not find any such feature that can record NCCL events.

Question:

Is this feature open sourced?
If yes, is there any documentation or summary of it?

gnadathur · August 20, 2024, 8:42pm

Thanks for reaching out.
Flight recorder will be released as prototype API in 2.5 release with documentation.
cc: @c-p-i-o

c-p-i-o · August 20, 2024, 8:48pm

Hey Jit! Yes, flight recorder is open sourced and available in the PyTorch branch. We just haven’t broadly made any announcements as we’ve been testing and changing some of the underlying implementation.

We’re writing up some documentation next few days on how to enable it and make use of the generated data to detect stuck jobs. Please stand by. As @gnadathur mentioned, FR will be marked Prototype in upcoming 2.5 release.

gjit_JNPR · August 20, 2024, 8:58pm

Hi Chiraj and Gokul,

Thanks for your responses! Looking forward to being able to use the NCCL Flight Recorder.

Jay_you_Dosen · August 30, 2024, 12:41pm

Could you please tell me which branch the flight recorder is in? thanks

c-p-i-o · August 30, 2024, 1:42pm

Flight recorder collection portion has been available in 2.4 and in the main branch.
The tool to analyze FR traces is under /tools/flight_recorder/fr_trace.py in the main branch.

c-p-i-o · August 30, 2024, 1:43pm

Also, I’m finishing a tutorial here. Almost done.

github.com

pytorch/tutorials/blob/cpio/flight_recorder/prototype_source/flight_recorder_tutorial.rst

(prototype) Flight Recorder for Debugging
=========================================
**Author**: `Chirag Pandya <https://github.com/c-p-i-o>`, `Junjie Wang <https://github.com/fduwjj>`

This tutorial introduces a new tool for debugging stuck jobs during distributed training.

Background and Motivation
--------------------------
An AI distributed training job refers to the process of training a machine learning model using multiple devices, such
as GPUs or CPUs, connected in a network. This approach allows for faster and more efficient training of large models
that require significant computational resources.
An engineer’s goal is to complete an AI training job as fast as possible and make continuous improvements such that
subsequent training can be done faster. A trained usable model is the final desired outcome.
One of the biggest impediment to completing training is the concept of a "stuck job".

A distributed AI training job is considered "stuck" when it stops making meaningful progress for an extended period of
time.

A job can get stuck for various reasons:
    - Data Starvation: This happens when the training job is not receiving data at the expected rate. This could be due to

This file has been truncated. show original

Jay_you_Dosen · September 26, 2024, 6:40am

Thanks. It’s very helpful.

jinqinn · September 27, 2024, 2:04am

Thanks. its very helpful.

c-p-i-o · October 10, 2024, 8:36pm

Hey @jinqinn, the official tutorial is published here. Let us know if you’ve had any success with Flight Recorder.