What is accumulating in /tmp/torchinductor_{USER}/triton?

torch.compile
1

Hi,

My training crashed because I ran out of disk space. After further inspection it seems like there are hundreds of GBs stored in my /tmp/torchinductor_azureuser/triton

For context — I am training my model with torch.compile and DDP

What is being stored there? Is there any way for me to prevent running out of storage in long (weeks) training runs?

TIA

2

This folder is the default cache location according to this code snippet.