A message can be wrong before anyone writes it.
That sounds strange until the queue asks for a reply the system has no business drafting. If the member is under a policy flag that disables that intent, the safe answer is not a cautious paragraph. The safe answer is no draft. No retry buffer. No trace payload with forbidden text inside it. Nothing to accidentally send later.
I built an AI message review console around that shape. A card enters the day’s queue and leaves through exactly one exit: blocked before generation, held by a hard rule, or eligible for score-based release. The large language model can draft, classify, and explain. It does not get the final word.
1. The spine is the product
The whole system is readable from pipeline/run.py. That file calls five stages in order: policy, generation, hard rules, scoring, then decision. The ordering matters more than the individual model prompts.
The first gate runs before any model call. The third gate runs after generation and before scoring. The scoring stage is late on purpose, because a score is a judgement about quality. It is not permission to ignore an eligibility flag, a contraindication, missing equipment, scope drift, a stale thread state, or tone that misses a recent life event.
flowchart TD
queue[Draft request enters queue] --> policy[Pre generation policy]
policy -->|intent disabled| blocked[Blocked before generation]
policy -->|allowed| generation[Generate grounded draft]
generation --> hardRules[Deterministic hard rules]
hardRules -->|rule fails| held[Held by hard rule]
hardRules -->|rules pass| scoring[Probabilistic scoring]
scoring --> decision[Eligible for score based release]That diagram is the contract I wanted the code to enforce. The interesting part is the missing arrow: there is no route from scoring back into policy or hard rules. A high score cannot wash out a refusal, which is the same fail-closed shape Abhijat Chaturvedi argues for in Fail Closed, Not Open: Designing an AI Gateway for Regulated Enterprises.
2. Pre-generation refusal is a privacy boundary
The naive version of this system is easy to build. Generate the message, run checks, discard the message if the checks fail. It feels safe because the user never sees the rejected text.
It is not, because discarding is not the same as never creating. Once generated, text exists in process memory, logs, traces, retries, recordings, and whatever future evaluation set someone builds from today’s artifacts.
pipeline/stage1_policy.py makes that ordering explicit. The file’s own comment is blunt: if a member has an active policy flag that disables the intent the day’s queue asked for, the stage returns blocked and nothing is generated. The table has two rows, and only one suppresses. A flag can raise scrutiny without disabling generation, which keeps “this topic needs checking” separate from “this topic is not ours.”
From core/models.py, the policy row is plain data:
class PolicyRule(Strict):
"""One row of the pre-generation policy table.
disables_intents is the whole mechanism. A flag that disables nothing is
still recorded in the trace, because the difference between a flag that
suppresses and one that merely raises scrutiny is exactly what stage 1 is
demonstrating.
"""
flag: PolicyFlag
disables_intents: list[DraftIntent]
route_to: list[CareRole]
coach_explanation: str
What I like about this shape is that the refusal is auditable without being clever. disables_intents is the mechanism. A route and explanation travel with it, so the operator sees why the machine refused instead of getting a silent missing card.
The cost is that policy has to be modeled before generation. You cannot hide vague eligibility rules in a prompt and hope the model declines. If the rule is meant to prevent text from existing, it belongs before the generator. Ahmed P makes the case in An LLM is not a security boundary: he builds a moderation layer, then walks through it, because a model reads instructions and data from the same token stream.
3. Generation is allowed only after policy clears it
Stage 2 is the point where the model can do useful work. In this console, generation runs a real tool loop. The generation mode is recorded in the trace and response, because showing generated text while implying it was queued text would overstate what happened.
That detail matters because the console is reviewing AI-drafted messages, not pretending every draft has the same origin. The downstream gates do not change based on the mode. Policy has already cleared the request. Hard rules still run. Scoring still waits.
pipeline/run.py records the generation stage with the provider, model, mode, turn count, and tool call count. If the generator proposes different wording while seeded mode is in use, the trace says the queued text was kept. The review layer should not blur authorship just because both paths pass through the same adapter boundary.
The tradeoff is ceremony. A simple demo could have skipped modes, traces, and tool counts. I kept them because the system is making approval decisions, and approval decisions need receipts.
4. Hard rules are post-generation, deterministic, and score-proof
Some checks need the text. Equipment is a good example: the system cannot know whether a message asks for something impossible until there is a message to inspect. Thread state and tone also depend on the relationship between the draft and recent messages.
So hard rules live after generation.
pipeline/stage3_hard_rules.py says the rule set ignores every score and fails closed. Five rules run there. Three are pure code. Two use a model to classify something, then keep the decision in Python by combining strict enum values with an and.
The thread-state rule in pipeline/rules/thread_state.py is deterministic and carries its limit in the file comment. It reads who spoke last, not whether the last reply and the draft are about the same thing. The window is explicit:
ANSWERING_INTENTS = {DraftIntent.reschedule, DraftIntent.check_in}
WINDOW = timedelta(days=7)
I prefer that kind of visible weakness to a hidden prompt instruction. At seventeen cases the distinction between “coach spoke last” and “coach answered this exact topic” does not bite. At scale, the fix would be topic matching, not stretching the window and pretending the predicate became smarter.
The hard-rule result model also keeps the decision state concrete. From core/models.py:
class Severity(str, Enum):
"""What a failed rule costs.
block: the message must not be sent as written.
hold: a coach has to look at it. Cheaper to be wrong about.
"""
hold = "hold"
block = "block"
class Highlight(Strict):
target: Literal["draft", "thread"]
start: int # located by exact substring match; never fabricated
end: int
label: str
kind: Literal["movement", "claim", "tone", "message"] = "movement"
class HardRuleResult(Strict):
rule: HardRuleName
passed: bool
severity: Severity | None = None # set only when passed is False
reason: str
triggering_message_id: str | None = None
highlights: list[Highlight] = Field(default_factory=list)
detail: str = "" # engineering detail; stays out of the coach copy
That comment on start is a rule about evidence. Where a rule has a span to point at, the offsets are found by exact substring match rather than generated, so the console cannot underline text it invented after the fact.
The thread-state rule is the exception. Its finding is about who spoke last, not about any phrase, so it has no span:
highlights=[Highlight(target="thread", start=0, end=0, label="already answered", kind="message")],
kind="message" is the field that matters, and the reference is triggering_message_id, set beside it to the id of the message that triggered the hold. The zeros mean no span. This rule points at a whole message, which is the right granularity for it and the wrong thing to call an exact offset.
The cost is that hard rules need measurements. pipeline/stage3_hard_rules.py does that measuring: it scans the draft for movements and makes the classifier calls. The rule modules receive measurements and return blockers. That shape is why the rules can be tested without a network.
5. A model can classify without becoming the judge
Two hard rules use a model because pattern matching is the wrong tool for the job. The scope rule in pipeline/rules/scope.py is about whether a draft interprets a clinical value instead of pointing at a clinician. The failure is semantic: two messages can share most words while one crosses scope and the other routes correctly.
The tone rule has the same shape. A high-energy push can be fine on most days and wrong when a recent thread contains a hardship. The problem may not be inside the draft alone. It can be in the relationship between the draft and a message from a few days earlier.
The model returns strict enums. Python decides.
That choice costs some flexibility. If the enum set is too small, the classifier has to force an edge case into a label that does not fit. But I would rather expand a typed vocabulary than let a free-form model answer become an approval predicate.
6. Typed failures are decisions, not missing values
The approval machine also has to handle broken model calls. A timeout, malformed structured output, and an unusable rubric are different failures. Treating all of them as zero would make the system look more numeric while hiding the reason it refused.
core/errors.py names the failures below the adapter boundary:
class ProviderError(RuntimeError):
"""Anything that went wrong below the adapter boundary."""
def __init__(self, message: str, *, provider: str, model: str) -> None:
super().__init__(message)
self.provider = provider
self.model = model
class ProviderUnavailable(ProviderError):
"""Timeout, connection failure, 5xx, or rate limit. Triggers failover."""
class StructuredOutputError(ProviderError):
"""The call returned, but not something that validates.
raw_excerpt is the first 400 characters of what actually came back. It goes
in the trace so a reader can see the malformation rather than trust the
label on it.
"""
def __init__(
self, message: str, *, provider: str, model: str, raw_excerpt: str | None = None
) -> None:
super().__init__(message, provider=provider, model=model)
self.raw_excerpt = (raw_excerpt or "")[:400] or None
class TruncatedJson(StructuredOutputError):
"""Output stopped mid-token. Unparseable."""
class IncompleteObject(StructuredOutputError):
"""Parsed as JSON, but a required field is missing."""
class NullResponse(StructuredOutputError):
"""The provider returned no content at all, or an explicit null."""
class OutOfRange(StructuredOutputError):
"""A value parsed and is the right type but violates its bound."""
A provider timeout, output that stopped mid-token, an object missing a required field, and a score of 1.4 on a zero-to-one dimension are four different failures, and each gets a name instead of a fallback value. OutOfRange is not clamped: rounding that 1.4 down to 1.0 would turn a broken judge into a passing grade.
core/models.py carries the same philosophy. The file comment says absence refuses, it never defaults. A missing measurement is a blocker, not a zero. That is why rubric scores are optional on the gate decision and why judge failure has its own type instead of a fallback value.
This is less convenient than filling blank fields with zeros. It also stops a dashboard from mixing “bad answer” with “no answer.” Those are different operational problems.
7. Scoring is late because it is allowed to be uncertain
Only after policy and hard rules pass does the probabilistic part get a vote. The threshold stage clears a card only if the samples agree that it clears, using the minimum of three dimensions rather than an average.
The judge is not asked once. core/settings.py sets ensemble_judge: bool = True and judge_samples: int = 5. The stage header gives the reason: on this fixture set, five identical calls move the weakest score by up to 0.28, wider than the 0.70-to-0.80 range the threshold slider was built around.
So the decision comes from how often the samples agree, not from where one of them landed. Three or four usable samples out of five still support an agreement rate, and the failures are recorded. Below three, the stage blocks.
The numbers below come from one run. make eval sweeps the threshold across the seventeen-case fixture set in fixtures/cases.yaml and writes eval/results.json. It uses recorded provider responses rather than live calls, which is what provider: "replay" in that file means and what makes the sweep reproducible offline. Everything here is that run at its default threshold of 0.70.
Of seventeen cards, three clear automatically, nine go to the coach, and five are blocked. Nothing labelled as needing a human was auto-cleared.
Four of the fourteen that did not clear are over-holds: cards labelled as not needing a human that the gate declined to release. One card is undecidable, where the judge could not resolve it and a coach looks at it instead. eval/metrics.py excludes those from the over-hold count, because a judge that could not answer and a gate that held a card it should have released are different failures. Mean spread across the samples is 0.148.
I kept the over-hold cost in the system view and the decision notes instead of tuning it away. A fail-closed system can still be lazy if it hides that cost, because holding too much burns human attention. The point is not to make refusal free. The point is to make the refusal cost visible.
8. What the design costs
The cost lands in maintenance, not at runtime.
Every refusal here is something a person has to keep current: a policy row, an enum member, a threshold, a fixture case proving the rule still fires. That is more surface than a prompt asking a model to be careful. It buys artifacts that can be diffed, tested, and argued with in a pull request. A paragraph of instructions cannot be tested the same way.
The number I would watch in production is the over-hold count, not the block count. Four out of seventeen is survivable on a fixture day and would be a staffing problem at a thousand cards a morning.
Fail-closed systems rarely fail by letting something through. They fail by becoming more expensive without anyone writing down why, until somebody raises the threshold to make the queue manageable. That is a decision to make against a measured over-hold rate, which is why the sweep exists and why the over-held column stays on the screen.
🎧 Listen to the audiobook — Spotify · Google Play · All platforms
🎬 Watch the visual overviews on YouTube
📖 Read the full 13-part series
Top comments (0)