Hi, I have a problem with some memory leak (?).
When I load my dataset the usage of memory increase to 100 processes with 30 GB of RES memory. I think that the problem is my Dataset function. So I have a question how to load dataloader part by part (is it possible to reduce in this way memory necessary in my computations)? I attach my function below.
class LoadDataset(Dataset):
def __init__(self): self.images = [] self.targets = [] img_path, ann_path = ( "path", "ann", ) coco_ds = torchvision.datasets.CocoDetection(img_path, ann_path) for i in range(0, len(coco_ds)): img, ann = coco_ds[i] images, targets = collate( [img.copy(), img.copy()], [ann, ann], coco_ds.coco ) for t in targets: self.targets.append(t) for image in images: self.images.append(image) def __len__(self): return len(self.images) def __getitem__(self, idx): img = self.images[idx] target = self.targets[idx] return ( img, target, )
Later I load data before iteration over epochs like:
train_loader = DataLoader(LoadDataset(), batch_size=24, shuffle=True)