Last night I put my model on training. A transformer model for Lyft5 Kaggle competition. Lo and behold, the next morning my SSD (Biostar m500 1TB) is dead (I mean dead as a brick and sent back to use warranty). The dataloader had memory leak despite my efforts to tune workers and batch size. I don’t know what happened. Whether it was swap memory overload or checkpoint model saving that lead to this event, I have no idea.
I was wondering if any of you guys had such an experience. Thank you.
PC spec:
core i9 10980xe
64gb of RAM
RTX 2080TI