While reviewing the NCCL symmetric-memory cleanup path, I noticed a possible cleanup and lifetime issue in ProcessGroupNCCL::shutdown(). This is currently a code-review question;
ProcessGroupNCCL::shutdown() may destroy the communicator with symmetric-memory segement registeration and their cached handles(ncclCommMemPoolMap and registeredSegmentHandles_) are still alive. The following code in shutdown() is for none-symmetric memory pool:
// Deregister memory pool after finalizing all collectives
if (memPool_) {
try {
deregisterMemPool(memPool_.get());
} catch (...) {
LOG(ERROR) << logPrefix() << "Failed to deregister memory pool, ignoring";
}
}
I also noticed that NCCLComm::abort() explicitly deregisters registered window handles, while the normal destroy() path does not perform equivalent cleanup.