Loading huge data functionality

[lan2720]

@apaszke

class MyDataset(torch.utils.Dataset):
def init(self):
self.data_files = os.listdir(‘data_dir’)
sort(self.data_files)

def __getindex__(self, idx):
    return load_file(self.data_files[idx])

def __len__(self):
    return len(self.data_files)

dset = MyDataset()
loader = torch.utils.DataLoader(dset, num_workers=8)

This way is good for images but isn’t fit for text. In NLP, data is usually in one file with multiple lines instead of one image in one file. So how to customize Dataset accordingly?