Memory Fills Up Reading Data From HDF5 File From Pytorch Dataloader

1

I created the getitem method of the DataLoader class in the Pytorch library as follows.

    with h5py.File(self.path_video,'r') as hf:
        sample = hf[video_id][...]

HDF5 files were also created as key-value pairs.

Sometimes, when reading data, the data in the file is read to the end with a memory occupancy rate of 30-40%. Sometimes this memory starts to fill up, and when it is completely full, the program crashes.

When the memory is full, I get the following error and the program crashes.

3

How can i solve this problem.