Datapipe reset?

1

Hi there,
I have a datapipe object, say datapipe. I want to get some shape out of it because it turns out I don’t know it beforehand.

For this, I do, say :
dim = int(next(iter(datapipe)).shape[1]

Then, I do some training with a dataloader on top of this datapipe. This gets me as many of the following warning as I have workers:

UserWarning: Some child DataPipes are not exhausted when iter is called. We are resetting the buffer and each child DataPipe will read from the start again.

I guess it’s because my datapipe is sent anew to each worker.
Do you see a way I could get rid of that warning gracefully ?

2

Currently, there isn’t a way to suppress that warning with torchdata's API.

Depending on your use case, you can use with warnings.catch_warnings().

Feel free to open a PR and suggest a way to suppress that warning. We will be happy to discuss.

3

thanks for the swift reply

4

is there maybe a way to reset manually ? I didn’t see that option

5

It should reset as needed if you call iter(child_datapipe) again. If you have a code snippet where something is not working as expected. I can have a look.