# Custom Dataset trying to read invalid idx **URL:** \ **Category:** data\ **Created:** [July 8, 2024, 12:28am UTC](https://discuss.pytorch.org/t/custom-dataset-trying-to-read-invalid-idx/205901 "2024-07-08T00:28:48Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![vhrn](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vhrn/32/70074_2.png) [@vhrn](https://discuss.pytorch.org/u/vhrn)\ **Post date:** [July 8, 2024, 12:28am UTC](https://discuss.pytorch.org/t/custom-dataset-trying-to-read-invalid-idx/205901/1 "2024-07-08T00:28:48Z")
I currently have the following Dataset class: ```auto import numpy as np import torch class CVRPGraphDataset(torch.utils.data.Dataset): def __init__ (self, data_file, sparse_factor=-1): self.data_file = data_file self.sparse_factor = sparse_factor self.file_lines = open(data_file).read().splitlines() print(f'Loaded "{data_file}" with {len(self.file_lines)} lines') def __len__ (self): return len(self.file_lines) def get_example(self, idx): line = self.file_lines[idx] line = line.strip() capacity = float(line.split()[0]) # Extract points points = line.split(" points ")[1].split(" demands ")[0] points = points.split() # Extract demands demands = line.split(" demands ")[1].split(" output ")[0] demands = demands.split() if len(demands) != len(points) / 2: raise ValueError(f"Number of demands {len(demands)} are different from number of points {len(points)}") points = np.array( [[float(points[i]), float(points[i + 1]), float(demands[i // 2])] for i in range(0, len(points), 2)] ) # Extract route full_route = line.split(" output ")[1] full_route = full_route.split() full_route = np.array([int(t) for t in full_route[:-1]]) if min(full_route) == 1: full_route -= 1 return points, full_route def __getitem__ (self, idx): points, route = self.get_example(idx) # Return a densely connected graph adj_matrix = np.zeros((points.shape[0], points.shape[0])) for i in range(route.shape[0] - 1): adj_matrix[route[i], route[i + 1]] = 1 # return points, adj_matrix, route max_route_size = 2*points.shape[0] + 1 pad_size = max_route_size - len(route) route = np.pad(route, pad_width=(0,pad_size), mode="constant", constant_values=-1) route_tensor = torch.from_numpy(route).long() result = ( torch.LongTensor(np.array([idx], dtype=np.int64)), torch.from_numpy(points).float(), torch.from_numpy(adj_matrix).float(), route_tensor, ) return result ``` After an update on a seemingly unrelated location (test\_step of my LightningModule), I got an IndexError exception at the line: ```auto def get_example(self, idx): line = self.file_lines[idx] ``` because idx is greater than the size of my Dataset. **My question is: How can I debug this?** The print in the **init** shows that the right number of samples were loaded from the files: ```auto print(f'Loaded "{data_file}" with {len(self.file_lines)} lines') ``` I also printed the contents of my files and they are ok. My DataLoaders uses the default sampler, with 12 workers and a batch\_size=10 (I load 60/20/20 train/test/val samples). ---
**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\ **Post date:** [July 9, 2024, 4:38pm UTC](https://discuss.pytorch.org/t/custom-dataset-trying-to-read-invalid-idx/205901/2 "2024-07-09T16:38:51Z")
What does `len(dataset)` return as its ` __len__ ` will define the max. indices in the sampler? ---
**Author:** ![vhrn](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vhrn/32/70074_2.png) [@vhrn](https://discuss.pytorch.org/u/vhrn)\ **Post date:** [July 9, 2024, 5:19pm UTC](https://discuss.pytorch.org/t/custom-dataset-trying-to-read-invalid-idx/205901/3 "2024-07-09T17:19:00Z")
I printed the len at the ``` **init** `` of my model, and it seems correct, 60, 20 and 20. ```auto def __init__ (self, param_args=None): super(CVRPModel, self). __init__ (param_args=param_args, node_feature_only=False) self.train_dataset = CVRPGraphDataset( data_file=os.path.join(self.args.storage_path, self.args.training_split), sparse_factor=self.args.sparse_factor, ) print(f"len train dataset = {len(self.train_dataset)}") self.test_dataset = CVRPGraphDataset( data_file=os.path.join(self.args.storage_path, self.args.test_split), sparse_factor=self.args.sparse_factor, ) print(f"len test dataset = {len(self.test_dataset)}") self.validation_dataset = CVRPGraphDataset( data_file=os.path.join(self.args.storage_path, self.args.validation_split), sparse_factor=self.args.sparse_factor, ) print(f"len val dataset = {len(self.validation_dataset)}") ``` Thanks for answering ---
**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\ **Post date:** [July 9, 2024, 5:33pm UTC](https://discuss.pytorch.org/t/custom-dataset-trying-to-read-invalid-idx/205901/4 "2024-07-09T17:33:05Z")
In this case I wouldn’t know what might be causing the issue since the `len()` of the dataset will be used in the sampler as seen [here](https://github.com/pytorch/pytorch/blob/cb4bec311ad80d913ab2957efa8ad3cce541a0b8/torch/utils/data/sampler.py#L172) unless you specify a custom `num_samples` while creating the sampler explicitly. ---
**Author:** ![vhrn](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vhrn/32/70074_2.png) [@vhrn](https://discuss.pytorch.org/u/vhrn)\ **Post date:** [July 9, 2024, 6:04pm UTC](https://discuss.pytorch.org/t/custom-dataset-trying-to-read-invalid-idx/205901/5 "2024-07-09T18:04:41Z")
Thank you. I guess I found the issue. The validation DataLoader of my model was actually creating a subset of the validation dataset with a size parameter (self.args.validation\_examples) that was greater than the validation dataset size. ```auto def val_dataloader(self): batch_size = 1 val_dataset = torch.utils.data.Subset( self.validation_dataset, range(self.args.validation_examples) ) print("Validation dataset size:", len(val_dataset)) val_dataloader = GraphDataLoader( val_dataset, batch_size=batch_size, shuffle=False, num_workers=self.args.num_workers ) return val_dataloader ``` The len() of the Subset was considering the bigger list of indices. Perhaps an error could be thrown if the user passes and indices list greater than the dataset size in the Subset class. That would prevent errors. ```auto class Subset(Dataset[T_co]): r""" Subset of a dataset at specified indices. Args: dataset (Dataset): The whole Dataset indices (sequence): Indices in the whole set selected for subset """ dataset: Dataset[T_co] indices: Sequence[int] def __init__ (self, dataset: Dataset[T_co], indices: Sequence[int]) -> None: self.dataset = dataset self.indices = indices ``` ---
**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\ **Post date:** [July 10, 2024, 12:55am UTC](https://discuss.pytorch.org/t/custom-dataset-trying-to-read-invalid-idx/205901/6 "2024-07-10T00:55:20Z")
> [@vhrn](#): > > Perhaps an error could be thrown if the user passes and indices list greater than the dataset size in the Subset class. That would prevent errors. Not necessarily, as the passed `indices` can contain any duplicates (e.g. if you want to oversample a minority class). Adding an error would be too strict in these cases. A check for valid indices, however, might be a good idea in case you want to contribute this.