INFO process shutting down 2024-12-21 20:51:22,045
INFO process shutting down 2024-12-21 20:51:22,045
INFO process exiting with exitcode 0 2024-12-21 20:51:22,045
INFO process exiting with exitcode 0 2024-12-21 20:51:22,046
INFO process shutting down 2024-12-21 20:51:22,046
INFO process exiting with exitcode 0 2024-12-21 20:51:22,046
INFO process shutting down 2024-12-21 20:51:22,046
INFO process shutting down 2024-12-21 20:51:22,046
Second type which comes from torch.distributed
sc:14361:15083 [0] NCCL INFO AllReduce: opCount 0 sendbuff 0x7af275e06400 recvbuff 0x7af275e06400 count 3921 datatype 7 op 0 root 0 comm 0x14e99380 [nranks=1] stream 0x13d72800
I have tried to set log level like this:
torch._logging.set_logs(all = logging.WARN)
but it doesn’t work.
Question is how to get rid of these two types log?
Thanks,NCCL log gone by following your suggestion,
would you please give some suggestions about that Data
Loader log?
INFO process shutting down 2024-12-21 20:51:22,045
INFO process shutting down 2024-12-21 20:51:22,045
INFO process exiting with exitcode 0 2024-12-21 20:51:22,045
INFO process exiting with exitcode 0 2024-12-21 20:51:22,046
INFO process shutting down 2024-12-21 20:51:22,046
INFO process exiting with exitcode 0 2024-12-21 20:51:22,046
INFO process shutting down 2024-12-21 20:51:22,046
INFO process shutting down 2024-12-21 20:51:22,046
I don’t kow where these logs come from as I haven’t seen them in PyTorch. A quick forum search suggests you might have enabled them via logger = multiprocessing.log_to_stderr().