Data loader with large ram variable


I have a large(3GB) dictionary to load in init of the Dataset class. I want to use this dictionary in getitem
but I am afraid that loading such a large file in init method would slowdown my training.
is there a better way to handle it?

Below is the sudo code of what I am trying to do…

class DataSetClass(Dataset):
    def __init__(self, dict_file_pth):
        self.large_dict = load_pkl(dict_file_pth)

    def __getitem__(self, index):
        data = self.large_dict[index] 
        return data
    def __len__(self,):
        return len(self.large_dict)