Hello,
I have a large(3GB) dictionary to load in init of the Dataset class. I want to use this dictionary in getitem
but I am afraid that loading such a large file in init method would slowdown my training.
is there a better way to handle it?
Below is the sudo code of what I am trying to do…
class DataSetClass(Dataset):
def __init__(self, dict_file_pth):
self.large_dict = load_pkl(dict_file_pth)
def __getitem__(self, index):
data = self.large_dict[index]
return data
def __len__(self,):
return len(self.large_dict)