How to print out logs with LOG(INFO) in Python in PyTorch

jianyuhuang · April 10, 2020, 12:23am

How can we print out the GLOG info level log when running Python code in PyTorch?

For example,

pytorch/pytorch/blob/master/torch/lib/c10d/ProcessGroupNCCL.cpp#L236


      // exception to users, after this, ncclCommWatchdog can detect nccl
      // communicators are aborted and clean up devNCCLCommMap_ accordingly.
      // if throwing timed out excepiton without aborting nccl communicators
      // here, it was observed that CUDA GPU will have 100% utilization and
      // can not run new events successfully.
      for (const auto& ncclComm : ncclComms_) {
        ncclComm->ncclCommAbort();
        const auto& storeKey = getNcclAbortedCommStoreKey(
            buildNcclUniqueIdStr(ncclComm->getNcclId()));
        store_->set(storeKey, {});
        LOG(INFO) << "Wrote aborted communicator id to store: " << storeKey;
      }
      throw std::runtime_error("Operation timed out!");
    }
    // Check for errors and throw appropriate exception.
    checkAndThrowException();
    std::this_thread::sleep_for(
        std::chrono::milliseconds(kSynchronizeBusyWaitMillis));
  }
  checkAndThrowException();
}

Checking https://github.com/pytorch/pytorch/blob/master/c10/util/Logging.cpp

It appears that we can set “caffe2_log_level” in c10 log.

Basically,

from caffe2.python import workspace
workspace.GlobalInit(['caffe2', '--caffe2_log_level=2'])

However, this only works for Caffe2 code but not c10 code in PyTorch backend.

Zhesi_Huang · November 15, 2020, 7:07am

bad impl … logging in c10, but it relys on the FLAGS_caffe2_log_level ? …

but i found that it can print the LOG(INFO) in python frontend when building pytorch with USE_GLOG=1 (default GOOGLE_STRIP_LOG = 0)

for example

export CMAKE_PREFIX_PATH=${CONDA_PREFIX:-"$(dirname $(which conda))/../"}

MACOSX_DEPLOYMENT_TARGET=10.9 CC=clang CXX=clang++ MAX_JOBS=32 BUILD_CAFFE2=0 BUILD_CAFFE2_OPS=0 USE_GLOG=1 USE_DISTRIBUTED=1 USE_MKLDNN=0 USE_CUDA=0 USE_FBGEMM=0 USE_NNPACK=0 USE_QNNPACK=0 USE_XNNPACK=0 python setup.py develop