Hello, I have repeatedly attempted to train a model across two nodes using DDP and torchrun. For some unknown reason, my client node times out after 15 minutes of no connection. Thus far, I have been unable to resolve the issue. The only problem that I can think of is that perhaps DDP requires ports behind the scenes (besides 29500) that I have not opened. Is this true?
Also note that while I should and do specify the port with torchrun, the client node apparently attempts to use 29500 regardless.