My short question is where can I find the implementation of some of the compiler features described here. Specifically, how the different modes (default, max-autotune, reduce-overhead, max-autotune-no-cudagraph) affect the behavior of the compiler.
My broader, and perhaps more important question is if there are any tips for navigation a code base like this. In my experience, the documentation has not been super clear about where to find the implementations of these features. For example, I struggle to find where this code can be found, and how it fits into the structure of the compiler. I would very much appreciate any tips for navigating a codebase such as this, without needed to email the specific developer with every question.
import torch for the first time it’ll run an
__init__.py which exposes a function called
compile() which just passes those calls to
When you pass in
compile(mode=“some mode”)` the mode will set some configs here pytorch/torch/_inductor/__init__.py at main · pytorch/pytorch · GitHub
So for example here we learn that
reduce-overhead sets a config variable called
triton.cudagraphs and you’ll find when it’s set and not in a file called
Hard to give general tips but what’s worked for me is alternating between reading the code like a book and then writing toy examples and stepping through them with a debugger, fixing random bugs also really helps because you’re more likely to get the attention of a maintainer
I recently found this, which has been incredibly helpful. Linked as I think it could save people a lot of time “reading the code like a book”