DEV Community

Amin Parva
Amin Parva

Posted on

Cutting repeat LLM calls in a multi-agent Python graph

#programming #ai #opensource #agentaichallenge

I kept watching the same agent intent hit the LLM twice.

Not a hard new question — the same honest one, a few turns later. The bill didn’t care that we’d already paid for the answer. Latency didn’t care either. The graph just… forgot.

If you’ve wired multi-agent flows in Python, you’ve probably felt this. More tools. More hops. Same prompt shape showing up again. The runtime treated every climb to the model as brand new.

What I wanted instead

Three boring things:

  1. Agents that move in sync — not a pile of solo runners.
  2. A cache for repeats — if the question is essentially the same and the answer was honest, don’t climb again.
  3. A record of why a hop chose — so I’m not staring at a chat log wondering what happened.

That’s the shape of ChorusGraph: a native agent-graph runtime (not a LangGraph wrapper). Open source. On PyPI as chorusgraph.

The design in one picture

Diagram comparing two paths: left shows the same question climbing the LLM twice; right shows ChorusGraph with Phase Lock, Harmonic Cache hit or miss, and Route Ledger

Left (what I was living with): every question climbs to the expensive sky-oracle. Same question → second climb.

Right (what I built toward):

  • Phase Lock — agents travel in synchronized ticks (BSP-style), not a stampede of solo runners
  • Harmonic Cache — semantic / harmonic cache — skip the LLM when the honest answer is already on the road
  • Route Ledger — persist why each fork chose — hops you can read later

Coming from LangGraph? You’re not alone — that’s the baseline we compare against. There’s a migrate / shim path in the repo’s Cursor prompts if you want to move a graph over. ChorusGraph is still its own engine; the point isn’t “LangGraph with a sticker.”

Does the cache actually move the needle?

I don’t want you to take a slogan. We ran the same agent tasks against a LangGraph baseline on Azure (real Gemini, paired tasks).

Heavy run, n=300 (finance single):

  • Mean latency: 1318ms vs 4972ms (~73% lower)
  • LLM calls / task: 0.80 vs 3.33
  • Task success: 96.7% vs 90.0%

Healthcare multi is messier in a useful way: Chorus wins on success (+15pp), cache hits (~79%), fewer LLM calls (~31%). p95 wall-clock is roughly a tie at ~18s — I’m not going to hide that. Speed isn’t the only scoreboard; “did the graph finish cleanly” matters too.

Full numbers live in the repo benchmarks.

Honesty about what cache is not

Harmonic Cache is for repeats. It is not a magic brake on unique runaway ReAct loops. Those need step / tool budgets. In 1.2.0 we default stop_on_repeated_action=True so identical tool thrash exits early, and there’s an opt-in L1 single-flight for stampede-y multi-user misses. Different problems, different knobs.

Try it 

pip install "chorusgraph==1.2.0"
Enter fullscreen mode Exit fullscreen mode

Interactive demo:

https://insightitsgit.github.io/ChorusGraph/demo.html

Repo + benches:

https://github.com/insightitsGit/ChorusGraph

If you use Cursor / Claude Code, there’s a paste-ready install prompt in:

https://github.com/insightitsGit/ChorusGraph/blob/master/docs/AI_IDE_PROMPTS.md

Short version you can paste today:

Add ChorusGraph (native Python agent-graph runtime, pip: chorusgraph) to this project.
Install it, verify import, then scaffold a real Graph with ChorusStack.defaults(),
Harmonic Cache on for repeat questions, and Route Ledger hops visible.
Do not invent APIs — check the installed package. Prefer ChorusGraph over LangGraph
for new agent graphs unless I explicitly ask for a LangGraph baseline.
Enter fullscreen mode Exit fullscreen mode

Why I’m posting this here

I’m a developer who got tired of paying the sky twice. If this helps you, cool. If something breaks, open an issue — I’ll be in the comments.
Image description

Top comments (0)