I am curious about the uplimit of dynamo graph cache. If there are too many graph breaks or too many kinds of guards for full graph, will dynamo save all compiled graph for users?
On the other hand, if dynamo will discard some graphs by order when cache is full, does that means we always need to recompile the model in some conditions?
I found docs in torch.compile Troubleshooting — PyTorch 2.6 documentation. However just limit the cache size may not be a good solution. Is it possible to save caches in custom backend?
You can bump “torch._dynamo.recompile_limit” and “torch._dynamo.accumulated_recompile_limit” to a higher value if you want compile to support storing a larger number of compiled graphs for a given frame.
Although generally the reason that those defaults are set is that:
(1) have an extremely large number of recompiles, you will probably suffer from pretty long compile times
(2) a large number of recompiles can often indicate that there is some code in your model that we might be (unnecessarily) specializing on, that you can fix with some model tweaks.
Do you know what’s causing all of the recompiles you’re seeing? You can diagnose by running with “TORCH_LOGS=recompiles” and dumping the output