As I see it, this is not a problem if one is using one GPU per node as when submitting with torchrun one could specify what logging file to use for each process.
But what if one is using a node that use multiple GPUs? It seems like there is no way to redirect the outputs of different processes to different log files?