The problem: one LLM call isn't enough
I built an AI tool that turns natural language into editable diagrams. You describe a flow, an LLM generates Mermaid code, and the result renders in real time.
The v1 was a single LLM call: user input → LLM → diagram. After a few days in production, two failure modes kept showing up:
-
Syntax errors with no recovery. The generated Mermaid code looks correct, but Mermaid throws a
Parse errorand the user sees a red stack trace. They leave. - Silent semantic drift. The user asks for "a login flow" — the LLM helpfully adds a "send verification code" actor. It doesn't ask. It doesn't tell you it added anything.
Neither problem can be reliably fixed by "tweaking the prompt." The first needs a second chance; the second needs an audit trail.
So v2 became an agent: multi-turn dialogue, self-correction, and — critically — a memory of what the user said and didn't say. This post is about what that agent actually looks like.
Core insight: this isn't one loop, it's two
What looks like "a multi-turn diagram generator" is actually two loops with different goals:
Outer loop · Understand the request
User speaks → extract facts → detect gaps → ask → refine
Inner loop · Generate and repair
Structured spec → generate code → validate → fix → output
The outer loop asks, "Do I understand the request well enough?" The inner loop asks, "Is the generated code correct?"
Trying to do both in one loop leads to the question "should the agent ask the user about this ambiguity?" — a question with no answer, because outer-loop ambiguity (user hasn't specified) and inner-loop ambiguity (generation went off-script) are fundamentally different.
The agent is a team — divide the work
I model the agent as a law firm handling a case. Each role does exactly one thing:
| Role | Does | Does NOT |
|---|---|---|
| Clerk | Records facts the user explicitly stated | Never guesses or infers |
| Auditor | Notes which fields in the record are empty | Doesn't decide whether to ask |
| Planner | Decides whether/what to ask the user | Doesn't ask directly |
| Lawyer | Drafts Mermaid, filling reasonable gaps | Leaves assumptions unrecorded |
| Reviewer | Checks syntax + flags unauthorized additions | Rewrites the user's requirements |
| Front Desk | Sequences the roles, maintains the case file | Makes no judgment calls |
Initially I stuffed all of this into a single mega-prompt. Debugging was hell — I couldn't tell whether the Clerk hallucinated a fact or whether the Lawyer improvised a character. Splitting them out means each role is a pure function: same input → same output, testable in isolation.
The call graph: Front Desk orchestrates everyone
The six roles don't operate in parallel — they're a pipeline, and Front Desk is the sole orchestrator:
User message
↓
Front Desk (append to conversation history)
↓
Clerk ──→ extract facts from history, populate the case file
↓
Auditor ──→ list which fields are still empty
↓
Planner ──→ decide next action using (missing list + budget + past questions)
│
├─ "Ask one" ──→ Front Desk renders question ──→ User
│ ↑ (next turn loops back to top)
│
└─ "We have enough" ──→ Lawyer ──→ generate Mermaid code
↓
Reviewer · syntax check
│
├─ FAIL ──→ inner repair loop (back to Lawyer)
│
└─ OK ──→ Reviewer · semantic check
↓
Unauthorized additions?
│
├─ Yes → diagram + follow-up question
│
└─ No → diagram only
↓
User
A few points worth noting:
- Front Desk is the only stateful component. Every other role is a pure function. Multi-turn memory lives entirely in the case file that Front Desk maintains.
- Planner is the only branching point. The agent's "ask vs draw" decision happens in one function, nowhere else.
- The inner repair loop runs between Lawyer and Reviewer without bothering the user. Mermaid syntax self-healing lives here.
- Semantic issues don't block output — they're delivered alongside the diagram ("here's your diagram, by the way, I added this thing — want to keep it?"). Users find it easier to judge with the picture in front of them.
The foundation rule: Clerk is conservative, Lawyer can improvise
If you take one thing away from this post, take this.
User says: "Draw a login flow. User enters password, system verifies, returns result."
The Clerk outputs:
{
actors: ["user", "system"],
events: [
{ from: "user", to: "system", action: "enter password" },
{ from: "system", to: "user", action: "return result" },
],
sync: "unknown" // ← user didn't say sync/async. NEVER guess.
}
That sync: "unknown" is the signal that triggers a question — the Auditor flags it, the Planner decides whether to ask.
If the Clerk "helpfully" fills it in ("login is usually sync"), that signal is destroyed. The agent can no longer distinguish "user actually said sync" from "I made it up." Every follow-up decision downstream is corrupted.
The Lawyer operates under a different constraint: it may fill in a "verification service" as a reasonable assumption, but that assumption must be written to the case file's assumptions slot. Later the Reviewer reconciles every element in the diagram: it must trace back to either "the user said this" or "the Lawyer assumed this." Anything that traces to neither is called an unauthorized addition, and it becomes a follow-up question ("I added X. Keep it?").
This one rule solves a class of nasty bugs by itself — including the classic "user said 'no X', LLM keeps re-adding X on regeneration." Because unauthorized additions are detectable, the agent surfaces them proactively instead of leaving the user to catch them.
The Planner: one function, one decision
My favorite architectural decision: every "should we ask the user?" judgment lives in a single pure function — the Planner.
- Auditor and Reviewer only identify problems — what's missing, what's wrong.
- Planner takes three inputs: candidate question pool, remaining budget, and past questions asked.
- Planner outputs one of three:
AskOne/ProceedToDraw/AskConfirm.
Each session has a global budget (say, 5 agent-initiated questions). Note the framing: only the agent's proactive questions consume the budget. User-initiated changes — modifying the spec, tweaking the diagram, requesting a redraw — do not consume it. This prevents the kind of UX where users become more reluctant to speak the longer they use the tool.
The practical payoff: want to change ask frequency or budget policy? Edit one function. No other component moves.
A walkthrough — 4 turns of dialogue
Architecture is easier when you see it in motion:
Turn 1 User: "Draw a user registration sequence diagram"
Clerk → { actors: ["user"], events: [], sync: unknown }
Auditor → missing: actors / events / sync
Planner (budget=5) → ask actors
Agent: "Besides the user, which other roles are involved?"
[budget: 5 → 4]
Turn 2-3 User: frontend / backend / database, plus each step
Planner asks per turn, budget: 4 → 3 → 2
Turn 4 User: "Just draw it and see" ← skip clarification, no budget cost
Lawyer gen #1 → helpfully adds "send email notification" (never mentioned)
Reviewer syntax check → FAIL (undeclared email actor)
Inner repair loop → Lawyer gen #2, drops email → syntax OK
Reviewer semantic check → 3 response arrows not grounded in the case file
→ unauthorized additions
Output: diagram + prompt "I added a full response flow. Keep it?"
Turn 5 User: "No responses. Just draw up to the database write."
Agent records "response flow" as a negative fact (explicitly rejected)
Regenerate → no response arrows
[budget unchanged — this is user reviewing output, not agent asking]
5-unit budget, 3 agent-initiated questions used, 2 in reserve. That's the flexibility of a global budget with clean semantics.
What it takes to add a new diagram type
The system supports 8 diagram types: Sequence · Flowchart · ERD · State · Class · Mindmap · Gantt · Architecture.
Adding a new type means implementing four type-specific pieces:
- Clerk — how to extract facts from natural language for this type
- Auditor — which fields are required vs optional
- Reviewer — what counts as an unauthorized addition
- Question templates — how to phrase prompts (a sequence diagram asks about "actors", a class diagram asks about "entities")
Planner / Front Desk / state store don't move — they don't care what type of diagram is being drawn.
Three takeaways worth stealing
Split the two loops. "Understand the request" and "generate the code" are different problems that happen to share state. Fusing them creates the unanswerable "should we ask about this ambiguity?" trap.
Clerk conservative + Lawyer improvises, strictly separated. If your Clerk fills in gaps, you've handed the agent's judgment to the LLM's intuition. Every Lawyer assumption must be written down so the Reviewer can audit it.
All policy lives in one Planner function. Want to change ask cadence, budget policy, or the "when to confirm" heuristic? One file. Zero downstream ripples.
What's next
I'm currently working on:
- Auto-classification confidence calibration (when the user doesn't state the diagram type, how often does the LLM guess wrong?)
- Extending the "unauthorized addition" rule set for each diagram type
- One-click session replay: serialize
prompt + spec + generated diagraminto a shareable link, so bug reports can replay the entire agent state
If you're building a similar multi-turn LLM agent — whether for diagrams, SQL, API mocks, or something else — these three principles saved me weeks of pain. An agent is not just "call the LLM more times." It's a team designed to self-correct, remember negative feedback, and produce an audit trail for every decision.
Try it: text2everything.vip
Top comments (0)