OSError: [Errno 24] Too many open files

[klory (Fangda Han)]

Too many open files when using dataLoader

Hi, When I use the data loader, I have met the following error: Too many open files. In my implementation of the Dataset, I use torch.load(‘xxx’) to load the data files (which are all tensors stored on disks), and when call getitem(self, index), it will take the corresponding items from the tensor, and return it. I construct the dalaloader in the following manner: dataset = Dataset(xxxxx) dataLoader = torch.utils.data.DataLoader(dataset=dataset, batch_size=batchSize, shuffle=shuffle, num…