When setting up a framework to perform distributed training do the machines have to been in the same data center or on the same network or can it be done when the machines are in different data centers?
The only requirement is that the two machines are able to establish a connection between each other, although note that running training on machines in two different TCs will likely cause a significant slowdown compared to them running on the same rack.