Memory leaking for dataloader

Yuyang_Hu · July 2, 2024, 1:51pm

When I tried to use the dataloader provided in “https://github.com/PatrickTUM/UnCRtainTS/blob/main/data/dataLoader.py,” I noticed a memory leak issue. The dataloader gradually consumes all my RAM, eventually causing my desktop to freeze.

some experiments I did to fix it

(1) I tried to transfer all the python list to np.array but It does not work.
(2) I tried to just use the dataloader without any training but still get memory leaking. Like:
sen12mscr_test = SEN12MSCR(opt.data_root, split=‘test’, cloud_masks=‘s2cloudless_map_mask’)
test_dataloader = torch.utils.data.DataLoader(sen12mscr_test, batch_size=2, shuffle=False, num_workers=0, pin_memory=True)
for pdx, samples in enumerate(test_dataloader):
print(pdx)

anantguptadbl · July 3, 2024, 6:44am

@Yuyang_Hu the file is not available at the mentioned link. Can you re-post the file in the github link?

Yuyang_Hu · July 3, 2024, 2:09pm

https://github.com/PatrickTUM/UnCRtainTS/blob/main/data/dataLoader.py It is the correct link.

Yuyang_Hu · July 3, 2024, 2:09pm

[quote=“Yuyang_Hu, post:3, topic:205626”]

github.com

PatrickTUM/UnCRtainTS/blob/main/data/dataLoader.py

import os
import glob
import warnings
import numpy as np
from tqdm import tqdm
from natsort import natsorted

from datetime import datetime
to_date   = lambda string: datetime.strptime(string, '%Y-%m-%d')
S1_LAUNCH = to_date('2014-04-03')

# s2cloudless: see https://github.com/sentinel-hub/sentinel2-cloud-detector
from s2cloudless import S2PixelCloudDetector

import rasterio
from rasterio.merge import merge
from scipy.ndimage import gaussian_filter
from torch.utils.data import Dataset

from util.detect_cloudshadow import get_cloud_mask, get_shadow_mask

This file has been truncated. show original

[/quote] Sorry, It is the correct one. Thanks.