DataLoader produces NaNs with DDP

Seems to be the same problem as 120733