Two code patterns that caused every torch.compile graph break in Keras's PyTorch backend

Over the summer I removed the torch.compile graph breaks from Keras’s PyTorch backend. In my two test models, every break came from one of two small patterns. Both are easy to write into framework code without noticing, so here they are.

A graph break is a spot where Dynamo stops tracing and falls back to Python. I counted them with torch._dynamo.explain on torch 2.13.0.

Pattern 1: a C++ object built inside a capability check

Keras’s attention op asked “can flash attention run here?” on every call, by building a pybind11 object:

spda_params = SDPAParams(query, key, value, mask, 0.0, is_causal, False)

Dynamo can’t trace into that constructor, so every attention layer broke the graph.

The fix (keras#23327): while Dynamo is tracing, don’t build the object at all, and leave the kernel choice to scaled_dot_product_attention. Asking for flash_attention=True explicitly still runs the check.

On a 2-layer transformer: 11 breaks became 0, and the compiled forward went from 2.50 ms to 0.33 ms.

Pattern 2: id() and is checks in tree code

The helpers that walk nested inputs used identity checks like this one:

is_leaf=lambda x: id(x) != structure_id

Dynamo treats id() as compile-time only (“Comparison on compile-time-only id or hash value”), so it can’t branch on it.

  • keras#23201 rewrote the dict-ordering helper to recurse directly. It was a correctness fix, and the id() check went with it. A small CNN went from 9 breaks to 0.
  • keras#23647 removed the last identity checks from traverse, which now compiles with fullgraph=True. Breaks were already 0, yet compiled 32-token generation went from 57.2 ms to 7.82 ms on that merge.

What I’d check first next time

  • Constructors of C++ objects on the hot path, especially inside “is this supported?” checks.
  • Any id(), is or is not between Python objects in code Dynamo traces.

Every number was timed on the commit just before and just after each merge. The full write-up, with all 15 PRs: how compiled Keras on PyTorch got 11.6× faster at generating text

Have you found a cleaner way to keep capability checks out of a traced region?

I’m not sure if I understand the root cause completely but would simply calling torch.cuda.get_device_capability() not work?

Could you describe what exactly building the pybind11 object means here?