Compile source code, code navigation

My short question is where can I find the implementation of some of the compiler features described here. Specifically, how the different modes (default, max-autotune, reduce-overhead, max-autotune-no-cudagraph) affect the behavior of the compiler.

My broader, and perhaps more important question is if there are any tips for navigation a code base like this. In my experience, the documentation has not been super clear about where to find the implementations of these features. For example, I struggle to find where this code can be found, and how it fits into the structure of the compiler. I would very much appreciate any tips for navigating a codebase such as this, without needed to email the specific developer with every question.


When you import torch for the first time it’ll run an which exposes a function called compile() which just passes those calls to torch._dynamo.optimize()

When you pass in compile(mode=“some mode”)` the mode will set some configs here pytorch/torch/_inductor/ at main · pytorch/pytorch · GitHub

So for example here we learn that reduce-overhead sets a config variable called triton.cudagraphs

grep for triton.cudagraphs and you’ll find when it’s set and not in a file called _inductor/

Hard to give general tips but what’s worked for me is alternating between reading the code like a book and then writing toy examples and stepping through them with a debugger, fixing random bugs also really helps because you’re more likely to get the attention of a maintainer

1 Like

I recently found this, which has been incredibly helpful. Linked as I think it could save people a lot of time “reading the code like a book” :slight_smile:

1 Like