How to disable these two types of log output?

1

Hello,I got two types of log output:

First type which comes from Dataloader:

INFO     process shutting down 2024-12-21 20:51:22,045
INFO     process shutting down 2024-12-21 20:51:22,045
INFO     process exiting with exitcode 0 2024-12-21 20:51:22,045
INFO     process exiting with exitcode 0 2024-12-21 20:51:22,046
INFO     process shutting down 2024-12-21 20:51:22,046
INFO     process exiting with exitcode 0 2024-12-21 20:51:22,046
INFO     process shutting down 2024-12-21 20:51:22,046
INFO     process shutting down 2024-12-21 20:51:22,046

Second type which comes from torch.distributed

sc:14361:15083 [0] NCCL INFO AllReduce: opCount 0 sendbuff 0x7af275e06400 recvbuff 0x7af275e06400 count 3921 datatype 7 op 0 root 0 comm 0x14e99380 [nranks=1] stream 0x13d72800

I have tried to set log level like this:

torch._logging.set_logs(all = logging.WARN)

but it doesn’t work.

Question is how to get rid of these two types log?

2

The NCCL log is printed since you’ve exported the NCCL_DEBUG env variable. unset it and the logs will disappear.

3

Thanks,NCCL log gone by following your suggestion,
would you please give some suggestions about that Data

Loader log?
INFO     process shutting down 2024-12-21 20:51:22,045
INFO     process shutting down 2024-12-21 20:51:22,045
INFO     process exiting with exitcode 0 2024-12-21 20:51:22,045
INFO     process exiting with exitcode 0 2024-12-21 20:51:22,046
INFO     process shutting down 2024-12-21 20:51:22,046
INFO     process exiting with exitcode 0 2024-12-21 20:51:22,046
INFO     process shutting down 2024-12-21 20:51:22,046
INFO     process shutting down 2024-12-21 20:51:22,046
4

I don’t kow where these logs come from as I haven’t seen them in PyTorch. A quick forum search suggests you might have enabled them via logger = multiprocessing.log_to_stderr().

5

Thanks,these logs comes from the workers of DataLoader

DataLoader(trainDataSet, batch_size = len(x_train), num_workers = 1)

set
num_workers = 1
will print out one line, set
num_workers = n
will print out n lines