Torch._dynamo.run vs torch.compile

torch.compile
1

I understand that if you want to use PyTorch 2.0’s torch.compile feature, you wrap your module with torch.compile and you shall get the benefits.

I was going through PyTorch Benchmark Suite, and in the speedup experiments there I found a call to:
torch._dynamo.run()

The definition of the torch._dynamo.run() function is as follows:

I find the doc string:

Don’t do any dynamic compiles, just use prior optimizations

Just want to clarify, is this API latest. And is it at par (performance wise) with torch.compile (after a torch compiled model say has run certain number of warmups?)

Please correct me if my understanding is mistaken or not, but what I feel is that, we can torch.compile a module, which shall do certain optimizations and keep it aside, and then when we use this torch._dynamo.run API, it somehow makes use of those optimizations which we did previously?

If so, I am unable to get, how is it done? Because the

function, takes in a regular model handle, and the function which is wrappered with torch._dynamo.run is not model but rather model_iter_fn which happens to be like either of the following function:

which defines the logic of how to run the model.

Can there be performance gap if we use torch._dynamo.run as used in the above example (where the model was previously torch.compiled). Or does torch.compiled module after initial warmups, use the same torch._dynamo.run? And both of them are equivalent after a certain number of warmups?

2

I don’t know the exact difference and why this call is used in the benchmarks (@marksaroufim might know), but wanted to point out that torch._dynamo is an internal module and should thus not be directly used (note the underscore at the beginning of the module).

3

Yeah so as Piotr mentioned this is an internal function you want to instead be using functions that are in the torch.compiler namespace

First off torch.compile() just calls another function called torch._dynamo.optimize() I would not use run in your code because it disables dynamic recompiles which I believe would have some silent correctness issues

Although I’m still not sure where your interest in run() stems from, are you trying to do serialization?

4

I was just trying to reuse some of the performance benchmark scripts from pytorch benchmark suite (the one which is used in the TorchInductor Performance Dashboard).

I started from

I thought that reusing most of the code from PyTorch itself shall be a good start.
The above script does basically something like:

./benchmarks/dynamo/torchbench.py --performance --training --amp --backend=inductor --output=torchbench_training.csv

This torchbench.py during the speedup calculation uses the torch._dynamo.run() function in the common.py file.

That is why I am curious about it. (I am not doing serialization)

Since you said that, is the same torch._dynamo.run function still being used in the latest PyTorch CI Dashboard workflow? Like is it safe to use the run() as in the common.py file — given the situation in that file? Why or why not?

I guess with changing inputs, we might hit a different branch (say), and we might have to compile some part dynamically, but if inputs remain the same (as in the common.py file) then this function is alright in that situation…