Why DistributedSampler older versions have different way of sampling indices?

vadimkantorov · October 29, 2021, 10:55am

Current PyTorch core one splits the whole dataset in interleaved fashion:

pytorch/pytorch/blob/d5988c5eca0221e9ef58918e4f0b504940cb926a/torch/utils/data/distributed.py#L116

    
      
                  if padding_size <= len(indices):
                      indices += indices[:padding_size]
                  else:
                      indices += (indices * math.ceil(padding_size / len(indices)))[:padding_size]
              else:
                  # remove tail of data to make it evenly divisible.
                  indices = indices[:self.total_size]
              assert len(indices) == self.total_size
          
          
    # subsample
              indices = indices[self.rank:self.total_size:self.num_replicas]
              assert len(indices) == self.num_samples
          
          
    return iter(indices)
          
          
def __len__(self) -> int:
              return self.num_samples
          
          
def set_epoch(self, epoch: int) -> None:
              r"""
              Sets the epoch for this sampler. When :attr:`shuffle=True`, this ensures all replicas

indices = indices[self.rank:self.total_size:self.num_replicas]

Code in github.com/nvlabs/wetectron (which is derived from older version of Detectron2):
wetectron/distributed.py at 69436c0c6bb77e8dddb6f3f2b5d912f0d1ecd8fb · NVlabs/wetectron · GitHub
indices = indices[offset : offset + self.num_samples]

Why did older Detectron2 use a different way of sampling?