Disk space and Training time

So when I ran my code around 2 days ago it was taking 1 min per epoch, and when I ran exactly the same code today it was taking around 10 mins per epoch. In fact, it started very fast and after about 30-50% of the epoch is done, it suddenly slows down.

I noticed that my disk space (4tb hdd) had only about 70 gb left. So I cleared out around 150gb of data. After this, the training time again took around 1 min epoch.

Any idea why disk space is so much related to the training time?

Based on these claims it’s apparently a valid assumption that your HDD/SSD slows down once it’s almost filled.

1 Like

I feared something similar but nice to see some sources on the same. Thank you.