I have a problem I am seeing for the first time. It does not relate to gpu memory, since it occurs even if I use my dataloader code in an empty enumerate(loader) loop.
The code is this:
original_image_sequence = self.img_dict[self.index_dict[index]]
saliency_image_sequence = self.img_dict[self.index_dict[index]]
label_sequence = self.img_dict[self.index_dict[index]] img_ =  img_s_ =  for i in range(len(original_image_sequence)): img = self.tun(PIL.Image.open(original_image_sequence[i])) img.unsqueeze_(0) img_.append(img) img_s = self.tun(PIL.Image.open(saliency_image_sequence[i])) img_s.unsqueeze_(0) img_s_.append(img_s) ori_tensor = torch.cat(img_) sal_tensor = torch.cat(img_s_) return ori_tensor, sal_tensor, label_sequence
If I do this and return something random like
then there is no memory problem. However when I return the tensors, after every epoch, my “modified” memory keeps increasing without bound (its contents have to be written to disk), while used memory stays the same.
I have tried using gc for garbage collection or just del to delete the references, but it is weird since I am not referencing anything outside the function but it still keeps updating the memory size.