DistributedDataParralled not support cuda?

liubanglan · January 3, 2020, 7:00am

there was not proper synchronization with the CUDA events that recored copies into this contents tensor before bucket contents tensor allreduce. it seems like not supporting cuda?

web link:

github.com

pytorch/pytorch/blob/master/torch/csrc/distributed/c10d/reducer.cpp#L404




// Keep going, until we either:
// - have kicked off reduction for all buckets, or
// - found a bucket that's not yet ready for reduction.
for (; next_bucket_ < buckets_.size() && buckets_[next_bucket_].pending == 0;
     next_bucket_++) {
  auto& bucket = buckets_[next_bucket_];
  std::vector<at::Tensor> tensors;
  tensors.reserve(bucket.replicas.size());
  for (const auto& replica : bucket.replicas) {
    // TODO(@pietern): Ensure proper synchronization with the CUDA events
    // that recorded copies into this contents tensor. If these copies are
    // executed on non-default streams, the current stream for the device
    // that holds the contents tensor must wait on these events.
    //
    // As long as autograd uses the default stream for every device,
    // these operations are implicitly sequenced, and we don't need to
    // do any extra synchronization here.
    //
    tensors.push_back(replica.contents);
  }

mrshenli · January 16, 2020, 5:02pm

DistributedDataParallel (DDP) does supports CUDA. The comment suggests extra care might be necessary when backward run on non-default stream. Actually, even if backward occurs on non-default streams it should be fine for most use cases. Below is why:

background: I learned from @albanD that autograd engine will use the same stream as the forward pass.

Let’s take a look at what could go wrong for the code you quoted.

1: the tensor is not ready when launching the allreduce operation
2: the tensor was destroyed too soon before the allreduce finishes.

We can rule out 2 for now, as all_reduce does recordStream() properly to prevent CUDA blocks to be freed too early.

Then the only thing left is 1. The operation on that tensor before allreduce is bucket_view.copy_(grad.view({-1}), /* non_blocking */ true); in mark_variable_ready_dense. The copy here happens on the same device (replica.contents and grad). And Reducer itself does not switch streams in between. So the only case that could hit race condition is when the application used different streams for different operators during the forward pass, and grads associated with those operators fall into the same bucket in reducer.