DDP and iterable datasets

How are folks using iterable datasets with DDP? The example for splitting an IterableDataset across workers (or DDP processes) seems a little silly – if I had random access to my dataset (iter_start), I wouldn’t be using an iterable dataset in the first place.

Has anyone come across / built a better solution?

cc @vincentqb for data loader question :slight_smile:

This is a recurring requestm, e.g. here or here. Please feel free to suggest a mechanism :slight_smile: