Over the summer I removed the torch.compile graph breaks from Keras’s PyTorch backend. In my two test models, every break came from one of two small patterns. Both are easy to write into framework code without noticing, so here they are.
A graph break is a spot where Dynamo stops tracing and falls back to Python. I counted them with torch._dynamo.explain on torch 2.13.0.
Pattern 1: a C++ object built inside a capability check
Keras’s attention op asked “can flash attention run here?” on every call, by building a pybind11 object:
spda_params = SDPAParams(query, key, value, mask, 0.0, is_causal, False)
Dynamo can’t trace into that constructor, so every attention layer broke the graph.
The fix (keras#23327): while Dynamo is tracing, don’t build the object at all, and leave the kernel choice to scaled_dot_product_attention. Asking for flash_attention=True explicitly still runs the check.
On a 2-layer transformer: 11 breaks became 0, and the compiled forward went from 2.50 ms to 0.33 ms.
Pattern 2: id() and is checks in tree code
The helpers that walk nested inputs used identity checks like this one:
is_leaf=lambda x: id(x) != structure_id
Dynamo treats id() as compile-time only (“Comparison on compile-time-only id or hash value”), so it can’t branch on it.
- keras#23201 rewrote the dict-ordering helper to recurse directly. It was a correctness fix, and the
id()check went with it. A small CNN went from 9 breaks to 0. - keras#23647 removed the last identity checks from
traverse, which now compiles withfullgraph=True. Breaks were already 0, yet compiled 32-token generation went from 57.2 ms to 7.82 ms on that merge.
What I’d check first next time
- Constructors of C++ objects on the hot path, especially inside “is this supported?” checks.
- Any
id(),isoris notbetween Python objects in code Dynamo traces.
Every number was timed on the commit just before and just after each merge. The full write-up, with all 15 PRs: how compiled Keras on PyTorch got 11.6× faster at generating text
Have you found a cleaner way to keep capability checks out of a traced region?
