Throughout the AI System Design series we follow Mailmind, a fictional AI email assistant that reads an inbox, triages messages, and drafts replies. It is our running example.
Before you read: Prompt Engineering for Agents, Production Grade and Evaluating Prompts: Beyond 'Looks Good to Me'.
A self-improving loop lets Mailmind refine its email drafting and meeting scheduling by tracking what worked, comparing outcomes to targets, then updating the prompts and few-shot examples it uses. This cycle of action, outcome capture, evaluation, and guarded strategy update has the shape of reinforcement learning (RL), without gradient descent or any change to the model's weights.
Mailmind doesn't need a reinforcement learning algorithm. The same natural language prompts that control it also serve as its policy, so improvements are just edited text. You can open the updated prompt and see exactly why behavior changed.
After Context Window Management for Big Jobs tackled work that never fits in one window, we turn to improvement: once the agent acts, how does it get better with as little human effort as possible?
Why does an agent need to look back at what it did?
Think of a salesperson who sends out a hundred emails every week. They keep a private note of which subject lines got replies. When they notice that "Quick question about [topic]" gets far more replies than "Following up," they switch. The next batch performs better. No expensive training run. Just a person reading the receipts and adjusting the playbook.
Mailmind does the same job mechanically. It drafts replies and meeting proposals every day. Some drafts the user sends without editing. Some get heavy rewrites. Some meeting slots get accepted within minutes; others sit unanswered. Those outcomes are the signal Mailmind uses to tune its own behavior. The loop is not about backpropagating through a loss function. It's about mining the best-performing outputs, finding what they have in common, and making the next prompt a little more like them.
flowchart LR
A[Draft reply / Propose slot] --> B["Capture outcome<br/>(edit distance, acceptance, latency)"]
B --> C["Accumulate until<br/>minimum sample window"]
C --> D["Evaluate:<br/>compare against targets,<br/>mine patterns in wins"]
D --> E["Update strategy:<br/>prompt edits, new examples,<br/>threshold adjustments"]
E --> AWhat are the four stages of Mailmind's self-improving loop?
Every pass through the loop has the same shape.
Action. Mailmind performs a task: draft a reply to a vendor, propose three meeting slots to a colleague, file a receipt. The action leaves a trace: the generated text and the context in which it was produced (recipient type, time of day, urgency signal).
Outcome tracking. For a draft, the outcome is the edit distance between what Mailmind wrote and what the user actually sent: how many words the user had to add, delete or change. A low edit distance means the draft was close to what the user wanted. For a meeting slot, the outcome is binary (accepted or declined) plus time to accept. These metrics are stored with tags that allow attribution: which prompt produced this draft, which example bank was active, which slot-selection rule was used.
Evaluation. Weekly, Mailmind pools the outcomes. It compares actual edit distances against a target (say, under 20% of words changed). It also looks at meeting acceptance rates against a target (say, more than 70% accepted the same day). The wins are then mined: for high-quality drafts, what tone, length, and structure do they share? Are they grouped by recipient type? The evaluation surfaces patterns the previous prompt didn't capture.
Strategy update. The findings become concrete changes. The drafting prompt gets a new line: "For external vendors, keep the tone direct and under 120 words." A few winning drafts become new few-shot examples, after they are scrubbed of names, amounts and other personal details, so one contact's information never leaks into a draft for another. The slot-selection rule might be tweaked to prefer Tuesday 10 AM slots for recipients in distant time zones. These changes are versioned, recorded, and rolled out gradually.
How do you design a feedback loop that doesn't accidentally make things worse?
Self-modifying prompts are dangerous if the loop isn't careful. Four design rules keep Mailmind from drifting.
Explicit outcome capture. You can't improve what you don't measure. Mailmind always logs the exact output, the user action (sent as is, rewritten, ignored), and the timestamp. For meeting proposals, it tracks the invitee, time zone, proposed slots, and acceptance status. Without this detail, attribution is guesswork.
Attribution. Knowing which prompt version and example bank generated which output is critical. Mailmind tags every action with a strategy version ID. When evaluation later finds a drop in acceptance rate, the operator can see which update caused it and roll back that single version.
Lag awareness. Outcomes take time. A meeting slot proposed on Friday might be accepted on Monday. Mailmind's evaluation windows are configurable: for meetings, a 72-hour wait before scoring. The loop never updates on an incomplete window. It also never punishes a proposal that gets a slow acceptance. It records the latency and uses it to adjust the slot-timing rules.
Guardrails on self-modification. Every update passes the same gates, with one value for each:
- Minimum sample: no update fires with fewer than 100 observations for a given prompt template and segment.
- One change at a time: each iteration changes at most one rule or one example. Counting changed tokens would be a poor brake: turning "never" into "always" is a one-word edit that flips behavior.
- Held-out check: every candidate is re-run on a held-out set of past cases before it reaches any user. Evaluating Agents: Golden Sets, Judges, and Shadow Mode covers how to build that set.
- Human approval: any change that touches a locked rule, or that moves the main metric by more than 10% on the held-out set (in either direction), waits for a person to approve it.
- Staged rollout: a new strategy goes to 10% of traffic first and expands only when the outcomes hold.
- Automatic revert: if the new strategy's acceptance rate falls more than 15% relative to the old one, and the drop is larger than the noise, the system switches back on its own and logs why.
- Versioned strategies: every prompt and example bank change lives in a registry, so a rollback is a single call.
graph TD
A["Candidate update"] --> B{"At least 100 observations?"}
B -->|No| W["Wait for more data"]
B -->|Yes| C{"One rule or example changed?"}
C -->|No| S["Split into separate updates"]
C -->|Yes| D["Re-run on held-out cases"]
D --> E{"Locked rule touched, or metric moved over 10%?"}
E -->|Yes| H["Hold for human approval"]
E -->|No| F["Roll out to 10% of traffic"]
H -->|Approved| F
F --> G{"Significant drop over 15%?"}
G -->|Yes| R["Revert automatically and log"]
G -->|No| X["Expand to all traffic"]How do you tell a real change from noise?
A loop like this only works if it has enough data, and one user's inbox rarely does. So Mailmind pools outcomes across many users, per segment (the same email type, language and recipient kind), and keeps a separate loop for each segment.
Pooling matters because small samples are noisy. Suppose a strategy's true acceptance rate is 70%. Measure it on 30 proposals and the 95% confidence interval runs from about 54% to 86%, so a "drop to 60%" on a sample that small may be pure chance. To tell a real 70% from a real 60% reliably (at the usual 5% significance level, with an 80% chance of catching a real drop), you need roughly 350 proposals in each group. So the staged rollout expands, or reverts, only once the 10% group has that many outcomes and a significance test or confidence interval says the difference is real. If a segment produces 2,000 proposals a day, its 10% group gathers about 200 a day, and the decision takes two days. A segment ten times smaller takes about two and a half weeks.
A loop for a single user must run much slower. A person who sends a few dozen meeting proposals a month cannot support a weekly update, so a per-user loop changes only small personal preferences, slowly, and leans on what the pooled loop has already learned.
What failure patterns break self-improving loops in production?
Even with guardrails, loops can misbehave.
Metric drift. Optimize for opens and Mailmind learns to write clickbait subject lines that annoy recipients. The loop must optimize for the user's true goal (low-effort replies, accepted meeting slots, few manual rewrites). Mailmind ties its primary metric to edit distance on drafts, not cosmetic signals like open rate. Edit distance can be gamed too: short, bland drafts get fewer edits. So it is paired with a check that the reply achieved its goal, such as the meeting getting booked or the vendor confirming.
Distribution shift. The user's email mix changes: suddenly 60% of incoming messages are from a new client's French-speaking team. The loop keeps reinforcing patterns from the old, mostly English mix. Mailmind runs separate loops per language segment and flags when segment proportions shift by more than 30% over a month. It then re-evaluates whether the existing examples still apply.
Overfitting. All drafts start sounding identical: same opener, same structure, same sign-off. The variety in the original example bank gets squeezed out. Mailmind adds a diversity constraint: each updated example bank must include at least three distinct tones or lengths. A separate "freshness" score triggers re-seeding from the original base prompt if all drafts cluster too tightly.
Lag corruption. A batch of meeting proposals for a holiday period shows low acceptance. The loop interprets that as a bad strategy and downgrades the slot-selection rule, when the real cause was that people were away. Mailmind marks known slow-response periods (company-wide holidays, weekends) and either excludes those windows or uses them for a separate "seasonal" rule.
How does Mailmind handle rare email types when most feedback is from the common ones?
The loop naturally gets great at the majority cases (quick replies to internal colleagues) and gradually worse at rare ones (refund disputes in German). The feedback signal is dominated by common instances, so the strategy update slowly forgets what the minority needed.
Mailmind tackles this with three techniques. First, it stratifies outcomes: each email type (by topic label and language) gets its own feedback pool. The update step weights minority-type feedback higher: a single German refund draft that the user sent unchanged counts as much as 20 regular internal drafts. Weighting cannot create evidence, though. Five German refund drafts weighted 20 times are still five data points, so rare types update more slowly and pass the same sample and noise checks as everything else. Second, it keeps separate example banks per type, so the "refund-German" bank is never diluted by the "status-update-English" bank. Third, it seeds rare types by hand: a person provides 5 to 10 high-quality examples that live as a fixed, protected bank. The loop can add to it but never remove or down-weight those seeds. This stops the rare types from dying out.
How does this compare to reinforcement learning?
The shape is the same: act, receive a reward, update the policy. But instead of numerical rewards flowing through a gradient update over a neural network's weights, Mailmind's "policy" is a set of natural language prompts and examples. The "update" is a text edit, grounded in evaluation rules a person can read.
You don't have to build this from scratch. Researchers have studied the same idea under the names automatic prompt optimization and reflection. In Reflexion (Shinn et al. 2023, Reflexion: Language Agents with Verbal Reinforcement Learning), an agent writes reflections on task feedback and keeps them in a memory that guides its next attempts, with no weight updates. OPRO (Yang et al. 2023, Large Language Models as Optimizers) has a model propose new instructions from a list of earlier instructions and their scores. DSPy (Khattab et al. 2023, DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines) turns a pipeline's prompts and examples into parts an optimizer tunes against a metric you choose. Each of them automates the evaluate and update steps of the loop above. The guardrails are still yours to add.
That difference from RL matters in production. You can read the updated prompt and understand the exact rule change: "We added a line that tells the model to ask for confirmation before proposing a slot to external attendees." In a gradient-based system, you can't audit a weight change the same way. The RL loop is opaque; the self-improving loop is a decision record. For email drafting, where tone errors can damage relationships, that auditability matters a great deal.
{
"type": "comparison",
"title": "Reinforcement Learning vs Mailmind",
"caption": "Mailmind replaces numerical rewards and gradient updates with natural language feedback and prompt edits.",
"before": {
"label": "Reinforcement Learning",
"points": [
"Numerical reward signals",
"Gradient updates over neural network weights",
"Policy is a black box",
"Hard to interpret rule changes"
]
},
"after": {
"label": "Mailmind's Prompt Loop",
"points": [
"Natural language feedback",
"Prompt edits based on outcome analysis",
"Policy is a readable prompt",
"Exact rule change is visible"
]
}
}
Quick Reference
These are Mailmind's design choices, not universal constants. Set your own from your traffic and your tolerance for risk.
| Guardrail | Mailmind's value |
|---|---|
| Minimum sample before an update | 100 observations per template and segment |
| Size of one update | One rule or one example |
| Check before any rollout | Re-run on a held-out set of past cases |
| Human approval | Any change to a locked rule, or a held-out metric move over 10% |
| Staged rollout | 10% of traffic until the result is clear |
| Automatic revert | A significant relative drop of more than 15% |
| Rollback | One call (versioned prompt registry) |
| Rare-type weight | Up to 20 times per instance, same sample checks |
| Meeting outcome window | 72 hours before scoring |
Frequently Asked Questions
Q: Can Mailmind write its own prompt improvements, or does a human still do it?
Mailmind's evaluation step produces a structured diff: for example, that drafts under 120 words needed less editing, and that the phrase "let me know if that works" went with lower acceptance. The prompt edit itself is drafted by a tightly constrained "strategy writer" agent. Every edit passes the same gates: one change at a time, a held-out check, and a staged rollout. Edits that touch a locked rule or move the main metric by more than 10% on the held-out set also wait for a person.
Q: What stops the loop from amplifying a tone that a minority of recipients dislike?
Segment-specific evaluation. If a tone change hurts a segment that makes up 5% of traffic, the loop sees that segment's edit-distance metric worsen. Because rare types are weighted higher and evaluated separately, the signal isn't drowned out. The automatic revert also catches significant drops.
Q: Could the loop learn to just copy the user's past replies and lose generality?
The diversity constraint and the held-out check guard against it. If the drafts merely repeat a template, they fail when a new recipient type appears. The loop measures generalization on held-out recent weeks, not on the same data it learned from; Evaluating Agents: Golden Sets, Judges, and Shadow Mode covers held-out sets in depth.
Q: Why not just use actual reinforcement learning with reward models?
You could. But for an inbox assistant, the "reward" is complex (tone, timeliness, user edit friction) and hard to reduce to one number. The natural language policy is easier to debug, modify, and explain. For many production agent systems, that operational clarity beats a black-box score.
Q: Does Mailmind ever revert its own updates automatically?
Yes. If an updated prompt version is serving 10% of traffic and its acceptance rate falls more than 15% relative to the control, and the drop is statistically significant, the system switches back to the previous version and logs the event.
Test yourself
Mailmind proposes meeting slots for a mixed team across three time zones. You want it to learn which slot characteristics lead to the fastest acceptance, without breaking the existing rule that it never proposes slots before 9 AM local time. Design the outcome tracking and update loop, including the guardrails.
Answer: Log each proposal with the invitee's time zone, day, slot position, acceptance and time to accept, pooled across users in the segment. Score only after the 72-hour window. Compare acceptance and median time to accept by time zone and day, for example finding that Tuesday 10 AM works far better than Friday 4 PM for one group. Turn the finding into one change at a time to the slot preferences. Lock the 9 AM rule, so any candidate that touches it stops at human review instead of shipping. Require 100 observations per time zone before an update, re-run each candidate on held-out past proposals, and send it to a person if it moves acceptance by more than 10%. Roll it out to 10% of proposals and expand only when that group has enough outcomes (roughly 350 to tell 70% from 60%) to show the change is real. Revert automatically on a significant relative drop of more than 15%.
Next: Guardrails: Input, Process, Output. It covers the checks around an agent that limit what it can read, do and send, even when it is fooled.
If you want this kind of breakdown every week, how real systems actually work under the hood, subscribe to Internals Decoded at internalsdecoded.com.
Originally published at Internals Decoded. AI internals, explained conversationally.
Top comments (0)