Are model weights consistent at the initialization in distributed training?

1

Hello, are model weights consistent at the initialization in distributed training?
Suppose we have node1, node2, node3.
Are weigthts on these nodes are same?