I have a large(3GB) dictionary to load in init of the Dataset class. I want to use this dictionary in getitem
but I am afraid that loading such a large file in init method would slowdown my training.
is there a better way to handle it?
Below is the sudo code of what I am trying to do…
class DataSetClass(Dataset): def __init__(self, dict_file_pth): self.large_dict = load_pkl(dict_file_pth) def __getitem__(self, index): data = self.large_dict[index] return data def __len__(self,): return len(self.large_dict)