Torch DataPipe contains a list of lists, how do I split it to make it iterate over the first list?

data
1

dp_sen1_item_lists is a torch.utils.data.datapipes.iter.callable.MapperIterDataPipe

next(iter(dp_sen1_item_lists))

returns

[<Item id=sen12floods_s1_source_62_2019_04_01>,
 <Item id=sen12floods_s1_source_19_2019_04_01>,
 <Item id=sen12floods_s1_source_0250_2019_04_01>,
 <Item id=sen12floods_s1_source_0249_2019_04_01>,
 <Item id=sen12floods_s1_source_0248_2019_04_01>,
 <Item id=sen12floods_s1_source_0247_2019_04_01>,
 <Item id=sen12floods_s1_source_0246_2019_04_01>,
 <Item id=sen12floods_s1_source_0245_2019_04_01>]

but I want it to return just the first element

<Item id=sen12floods_s1_source_62_2019_04_01>

how do I make it iterate over that list rather than a list of lists? II want to split up the datapipe somehow. I’ve tried dp_sen1_item_lists.header(1) but that returns a child data pipe that still contains a list but doesn’t iterate over that list

next(iter(dp_sen1_item_lists.header(1)))

[<Item id=sen12floods_s1_source_62_2019_04_01>,
 <Item id=sen12floods_s1_source_19_2019_04_01>,
 <Item id=sen12floods_s1_source_0250_2019_04_01>,
 <Item id=sen12floods_s1_source_0249_2019_04_01>,
 <Item id=sen12floods_s1_source_0248_2019_04_01>,
 <Item id=sen12floods_s1_source_0247_2019_04_01>,
 <Item id=sen12floods_s1_source_0246_2019_04_01>,
 <Item id=sen12floods_s1_source_0245_2019_04_01>]

I’m definitely new to datapipes and having trouble finding an example or reference, any help is much appreciated

2

I’ve opted to use dp_sen1_item_lists.batch(8) for now to go to a nested list but this requires knowing the length of the iterable up front.

3

If you use flatmap with no argument to iterate through the inner lists (see the second example). If you only want the first list, the do dp.header(1).flatmap(). Let me know if that does not work as expected.

While the no-op behavior will be in the next release, for 0.4.0, you may need to pass in a no-op function. Note that you will need to install the nightly version or from source for that to work.