Convert map-style Dataset to Iterable?

I want to Chain two datasets together, one of which is a normal Dataset and the other is an IterableDataset. But Chain only accepts IterableDatasets.

So, I thought I’d just create a simple Iterable version of my original map-style dataset… but it’s not working. Sample code below.

I see a few questions related to this, such as this and this SO question with no answer, but no answers.

How do we do this? Here’s my attempt:

class MapStyleDataset(
    def __init__(self, path, **kwargs):
        self.filenames = glob.glob(path)

    def __getitem__(self, idx):
        data = somehow_read_from( self.filenames[idx] )
        return data

    def __len__(self):
         return len(self.filenames)

class MyIterableDataset(
    def __init__(self, *args, **kwargs):
        self.this = MapStyleDataset(args, kwargs)
        self.len = len(self.this)

    def __iter__(self):
        return self.this.__getitem__(random.randint(0, self.len))

Let’s assume MapStyleDataset works fine. You can do next(iter(....)) on it properly.

But when I try to do next(iter(....)) on the Iterable dataset, over and over I get error messages that read:

TypeError: iter() returned non-iterator of type 'Tensor'

How do do this conversion properly? Thanks.

ANSWER: So close! Just need to use yield instead of return to return data from __iter__. :slight_smile:

1 Like