Sir, you saved my day! For those who might be interested, here’s my case:
I computed the loss and stored it in a dictionary. I updated the dictionary using the following code:
loss_dict.update({loss_name: loss})
...
return loss, loss_dict
However, I didn’t write any code to process or log the returned loss_dict, which resulted in GPU memory leaks. The issue was resolved after I added detach() to each loss term when updating the loss_dict.