My short question is where can I find the implementation of some of the compiler features described here. Specifically, how the different modes (default, max-autotune, reduce-overhead, max-autotune-no-cudagraph) affect the behavior of the compiler.
My broader, and perhaps more important question is if there are any tips for navigation a code base like this. In my experience, the documentation has not been super clear about where to find the implementations of these features. For example, I struggle to find where this code can be found, and how it fits into the structure of the compiler. I would very much appreciate any tips for navigating a codebase such as this, without needed to email the specific developer with every question.
Thanks!
When you import torch for the first time it’ll run an __init__.py which exposes a function called compile() which just passes those calls to torch._dynamo.optimize()
When you pass in compile(mode=“some mode”)` the mode will set some configs here pytorch/torch/_inductor/__init__.py at main · pytorch/pytorch · GitHub
So for example here we learn that reduce-overhead sets a config variable called triton.cudagraphs
grep for triton.cudagraphs and you’ll find when it’s set and not in a file called _inductor/compile_fx.py
Hard to give general tips but what’s worked for me is alternating between reading the code like a book and then writing toy examples and stepping through them with a debugger, fixing random bugs also really helps because you’re more likely to get the attention of a maintainer
1 Like
I recently found this, which has been incredibly helpful. Linked as I think it could save people a lot of time “reading the code like a book” 
1 Like