Distributed Data Parallel Overlap batch Training

I be overlap when try to use DDP (distributed data parallel). This image, show each batch on each epoch used on both 2 GPU 0 and GPU 1.