I was able to get a Dataset-inheriting class to work with streaming data by effectively just ignoring the index in __getitem__
.
class MyDataset(Dataset):
...
def __getitem__(idx):
return self.get_data()
This works, so I’m wondering if there’s any advantage to using IterableDataset?
My data is streaming and doesn’t always have a fixed length.