Printing C++ stack traces

tushar00jain · October 11, 2025, 12:53pm

How do we get C++ to print stack traces? I’m getting the following error from ProcessGroupNCCL.cpp

vector::_M_range_check: __n (which is 1044) >= this->size() (which is 0)

Doing TORCH_SHOW_CPP_STACKTRACES=1 doesn’t help print the stack trace.

Ahsen_Zuhal · October 13, 2025, 3:41pm

You can use the built-in GNU backtrace mechanism:

export TORCH_SHOW_CPP_STACKTRACES=1
export TORCH_CPP_DEBUG=1

tushar00jain · October 13, 2025, 5:46pm

I still don’t get anything. Particularly, this is the error.

[ProcessGroupNCCL.cpp:2092] Process group watchdog thread terminated with exception: vector::_M_range_check: __n (which is 1049) >= this->size() (which is 0)

The code only prints what() on the exception. And no stack trace gets printed, just the error message.

github.com/pytorch/pytorch

torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp

70ec464c1


      
            VLOG(2)
                << pg_->logPrefix()
                << "main process destroyed cuda before watchdog loop exited, terminating watchdog."
                << " (Watchdog caught exception: " << e.what();
          
          } else {
            // Append error message reported from runLoop
            const auto exitMsg = c10::str(
                pg_->logPrefix(),
                "Process group watchdog thread terminated with exception: ",
                e.what());
            LOG(ERROR) << exitMsg;
            if (C10_LIKELY(rethrowCUDAErrors_) ||
                !(std::string(e.what()).find("CUDA Error"))) {
              // TODO(whc) clean up the rethrow - why is it stored in a class var and
              // rethrown?
              watchDogException_ =
                  std::make_exception_ptr(C10_BUILD_ERROR(DistBackendError, exitMsg));
              std::rethrow_exception(watchDogException_);
            }
          }