Are the indices generated in the main process and sent to the workers one by one or does each worker have its own Sampler that it iterates through independently?
DataLoader should use one
sampler which then sends the indices to each worker. If each worker would have a custom sampler they could load duplicated samples or the sampler would need to have multiprocessing capabilities, which is not the case.
Are you sure? Because this:
def __init__(self, data_source):
self.data_source = data_source
yield from torch.randperm(len(self))
doesn’t return the same expected behavior as the default
As in, inexplicably, it makes my MacBook crash after about 20k learning steps and the other doesn’t. I’ve ablated them 3 times now.
Well, I’m comparing to no sampler +
shuffle=True. Is that the same as
Yes, not providing a
sampler and using
shuffle=True will create a
RandomSampler as seen here.
I don’t understand your doubt as you are just asking if I’m sure without explaining what might indicate I’m wrong.