Code stuck at cuda.synchrize()

[agu (Andrew Gu)]

Maybe setting TORCH_DISTRIBUTED_DEBUG=DETAIL can help?
https://pytorch.org/docs/master/distributed.html#torch-distributed-debug