When you run a 3-billion-parameter model like Llama 3.2 locally on commodity hardware, the initial experience feels like magic: it is fast, private, and runs completely offline with zero API costs.
Then you put it in front of real operational data—like invoicing or ledger billing—and reality immediately catches up with you.
A Small Language Model (SLM) can compose a remarkably polite, natural email draft, and in the very same paragraph quietly invent a $1,990 balance on a $1,975 invoice, fabricate an imaginary transaction ID like TX-9999, or guess randomly when faced with missing records.
When probabilistic language models meet consequential business workflows, four failure modes appear almost immediately:
- Math & Ledger Hallucinations: Even models with solid reasoning capability will randomly botch arithmetic, round numbers inconsistently, or invent transaction line items.
- The "Vibes" Evaluation Trap: Teams often evaluate systems by eyeball-testing 5 outputs or asking another LLM "Rate this response from 1 to 5" without measuring statistical rater agreement or testing for length bias.
-
Missing Exit Ramps: When a model is uncertain or encounters ambiguous data, default prompting pushes it to guess rather than cleanly refusing (
Abstain) or escalating to an expert (Escalate). - Static, Disconnected Knowledge: Human operators spend time reviewing and correcting agent drafts, but those edits disappear into an email client instead of continuously training the system.
To tackle these challenges, I built and open-sourced closed-loop-slm-agent: a production-oriented, local-first agent architecture built around Llama 3.2 3B. It enforces deterministic verification before any text ships, provides explicit tri-state routing, calibrates evaluation judges using Quadratic Weighted Kappa (QWK), and continuously learns from human feedback in-context.
Here is the complete architectural blueprint, the failure modes we encountered along the way, and the code patterns that solved them.
The Core Philosophy: "Numbers Owned by Code, Words Owned by Models"
The foundational dividing line in this system is simple:
Never let a language model calculate numbers or verify ground truth in consequential domains.
Language models are probabilistic token predictors, not algebraic solvers. In this architecture, the local SLM is responsible solely for language generation: prose flow, tone, and readability.
Ground truth arithmetic, invoice line-item verification, ledger balance calculations, and date checks are strictly owned by deterministic Python code.
┌────────────────────────────────────────────────────────┐
│ INCOMING DRAFT REQUEST │
└──────────────────────────┬─────────────────────────────┘
│
▼
┌──────────────────────────┐
│ Drafter Agent (SLM) │
│ (Llama 3.2 3B / Local) │
└─────────────┬────────────┘
│
┌─────────────────────┴─────────────────────┐
▼ ▼
[No Billable Data] [Draft Generated]
┌─────────────────┐ │
│ ABSTAIN │ ▼
│ (Safe Refusal) │ ┌───────────────────┐
└─────────────────┘ │ Hybrid Verifier │
└─────────┬─────────┘
│
┌─────────────────────────────┴─────────────────────────────┐
▼ ▼
┌─────────────────────────┐ ┌─────────────────────────┐
│ Deterministic Code │ │ SLM Tone Evaluator │
│ (src/verifier/claim) │ │ (Only if Code Passes) │
├─────────────────────────┤ └────────────┬────────────┘
│ • Integer-cents math │ │
│ • Ledger tx matching │ │
│ • Zero hallucinated $ │ │
└────────────┬────────────┘ │
│ │
└─────────────────────────────┬─────────────────────────────┘
│
┌───────────────────────┴───────────────────────┐
▼ ▼
[Fails Check / Retries > N] [Passed Verifier]
┌─────────────────────────┐ │
│ ESCALATE │ ▼
│ (Hand off to Human) │ ┌───────────────────┐
└─────────────────────────┘ │ Human Review Hub │
│ (Approve / Edit) │
└─────────┬─────────┘
│
▼
┌───────────────────┐
│ Continuous Memory │
│ (QWK Eval Harness)│
└───────────────────┘
1. The Self-Checking Agent & Tri-State Guardrail
Most agent designs implement a linear flow: Input -> LLM -> Output. If the output contains subtle errors, it ships directly to the user or database.
In src/agents/graph.py, we implement an autonomous Generate-Verify-Decide state machine with three explicit terminal states:
The Tri-State Routing Logic
-
Revise(Local Self-Correction): If a generated draft fails formatting, tone, or math checks, the verifier intercepts it before any human or client sees it. The error is structured as an actionable critique and routed back to the Drafter node with a strict retry budget (N ≤ 2). -
Abstain(Graceful Refusal): If a client has zero unpaid transactions or ambiguous records, the agent must refuse to draft. Most commercial agents hallucinate a boilerplate message or claim unbilled amounts. In our system, the Drafter immediately routes to a terminalAbstainstate and emits a structuredcapability_gaprecord. -
Escalate(Human-in-the-Loop Fallback): If a draft fails multiple verifier passes or detects irreconcilable ledger discrepancies, the agent immediately bails out to anEscalatestate, handing the case to a human specialist with full debugging telemetry.
The Hybrid Verifier: Python AST + SLM Gate
The verification layer in src/verifier/claim.py runs a two-stage sequential gate:
# Deterministic checks run FIRST in pure Python (0ms latency, zero hallucination)
def verify_draft_claims(draft: str, ledger: ClientLedger) -> VerificationResult:
# 1. Integer-cents arithmetic check
expected_cents = ledger.total_unpaid_cents
found_numbers = extract_currency_amounts(draft)
if expected_cents not in found_numbers:
return VerificationResult(valid=False, reason="Math discrepancy: Total balance mismatch")
# 2. Grounding verification: Every referenced invoice must exist in database
for tx_id in extract_referenced_invoices(draft):
if not ledger.has_transaction(tx_id):
return VerificationResult(valid=False, reason=f"Fabricated invoice ID: {tx_id}")
# 3. Structural rules: Prose word limits, line-item completeness, banned punctuation
if contains_em_dashes(draft) or contains_unfilled_placeholders(draft):
return VerificationResult(valid=False, reason="Unfilled placeholder or disallowed syntax")
# Only if ALL deterministic assertions pass do we invoke the SLM for tone analysis
return run_slm_tone_check(draft)
By enforcing that every Python assertion must pass before the SLM tone check is even invoked, we eliminate 100% of mathematical errors at zero API cost.
2. Injected-Error Test Suite ("Verifier Teeth")
An agent verifier is only as good as the errors it actually catches. To prove our verifier wasn't just "vibes," we built a dedicated injected-error test suite (tests/evaluation/run_teeth_tests.py):
| Scenario | Injected Corruption | Verifier Decision | Final State |
|---|---|---|---|
client_err_math |
Tampered invoice total ($1,990 vs $1,975) | CAUGHT | Self-Revised, then Passed |
client_err_missing |
Omitted mandatory line item | CAUGHT | Escalated to Human |
client_err_invalid |
Fabricated synthetic invoice ID (TX-9999) |
CAUGHT | Escalated to Human |
client_err_ungrounded |
Added ungrounded billing month (July) | CAUGHT | Escalated to Human |
client_err_semantic |
Plausible but unsupported "legal fee" claim | MISSED* | Shipped as Success |
client_empty |
Zero billable transactions | ABSTAINED | Clean Abstain + Logged Gap |
*Engineering transparency note: The semantic miss (client_err_semantic) is a documented edge case: when numbers and IDs match, a pure deterministic verifier cannot detect subtle semantic fabrications (e.g., claiming a call took 60 minutes instead of 30) without an auxiliary Natural Language Inference (NLI) model like MiniCheck. Documenting your system's exact boundary is what separates toy projects from production engineering.
3. The Math of Trust: Calibrating LLM Judges with Quadratic Weighted Kappa (QWK)
How do you know if an LLM-as-a-Judge is actually reliable?
Most AI projects prompt GPT-4: "Score this draft from 1 to 5" and assume the score is gospel. But LLM judges suffer from notorious length bias (grading verbose responses higher than concise ones) and prompt drift.
Why Naive Accuracy Fails on Ordinal Scales
If human experts grade on a 1-to-5 rubric:
- A human rates a draft 5 (Excellent), and the LLM rates it 4 (Good). This is a minor nuance.
- A human rates a draft 5 (Excellent), and the LLM rates it 1 (Severe Hallucination). This is a catastrophic failure.
Simple accuracy treats both mistakes as an identical failure.
To solve this, we measure inter-rater reliability using Quadratic Weighted Kappa (QWK) in src/evaluation/metrics.py:
Where the quadratic penalty weight w_{i,j} between ratings i and j is:
A disagreement of |5 - 1| = 4 receives a penalty of 4² = 16, whereas a disagreement of |5 - 4| = 1 receives a penalty of 1² = 1.
Live Calibration Benchmark
We cross-validated our automated judge prompt against 15 hand-labeled expert ground-truth examples (data/calibration_labels.json):
- Quadratic Weighted Kappa (QWK): 0.942 (Near-perfect inter-rater agreement)
- Mean Absolute Error (MAE): 0.067 (Average score error < 0.1 on a 5-point scale)
- Pearson Correlation (r): 0.974
- Cross-Validation Agreement: 100% (15/15 samples agreed within ≤ 1 point)
Furthermore, we explicitly neutralized length bias: the rubric prompt measures prose word count exclusively (ignoring required invoice tables), ensuring concise, efficient drafts are never penalized.
4. Closing the Loop: Continuous Learning from Human Edits
Fine-tuning a model or retraining LoRAs every time an operator edits a draft is costly, slow, and prone to catastrophic forgetting.
Instead, we built a real-time closed-loop retrieval engine (src/memory/retrieval.py):
-
Structured Telemetry Ingestion: Every operator action in the CLI (
src/cli/app.py) is logged to a JSONL trace:APPROVED_AS_ISAPPROVED_WITH_EDITSDECLINEDABSTAINED
- PII Sanitization: Human-edited drafts are stripped of client-specific identifiers and currency numbers before indexing.
- Dynamic Few-Shot Injection: When the Drafter prepares an email for that client (or a similar client), the retrieval layer injects the most relevant historical human edit into the prompt as an in-context exemplar.
Empirical Proof: Before vs. After Feedback
Using our QWK-calibrated evaluation harness, we executed live before/after trials across multiple sample runs (N = 1, 3, 5):
================================================================================
EVALUATION RESULTS: BEFORE vs. AFTER LEARNING LOOP
================================================================================
Metric Before Loop After Loop Delta (Δ)
--------------------------------------------------------------------------------
factual_grounding 5.00 5.00 +0.00 (Pinned by code)
completeness 5.00 5.00 +0.00 (Pinned by code)
absence_of_invented_numbers 5.00 5.00 +0.00 (Pinned by code)
tone 4.60 4.80 +0.20
style_adherence 3.40 5.00 +1.60 (+47%)
actionability 1.80 5.00 +3.20 (+177%)
--------------------------------------------------------------------------------
OVERALL AVERAGE 4.13 4.93 +0.80 (+19.4%)
================================================================================
Why Did Factual Grounding Stay at +0.00?
Notice that factual_grounding, completeness, and absence_of_invented_numbers were 5.00 before and 5.00 after.
This is the signature of disciplined system architecture. Because numbers are owned by Python code and gated by the verifier, factual correctness was already saturated. The learning loop did not have to waste capacity learning basic math; instead, 100% of the learning delta landed on actionability (+3.20) and style_adherence (+1.60)—the subjective human nuances the agent was taught.
An Honest Limitation: Recency vs. Semantic Retrieval
Right now, if you look inside src/memory/retrieval.py, the StyleMemoryStore is intentionally lightweight: it reads data/traces.jsonl in reverse and picks the most recent human edits, prioritizing that specific client's history.
For a local prototype with no external database dependencies, this works great. But as a real system scales, recency alone has clear blind spots:
- If an operator just edited a polite, casual check-in email for Client A, and five minutes later the agent needs to draft a formal 90-day overdue legal notice for Client B, recency might pull the casual style simply because it's newer.
How to extend this:
-
Semantic Vector Search: The next step is upgrading
src/memory/retrieval.pyfrom a chronological scan to a lightweight local vector store (like ChromaDB or FAISS with an embedding model likeall-MiniLM-L6-v2). That way, the agent retrieves examples based on situation similarity (e.g. dispute vs. gentle reminder) rather than just timestamps. -
Offline Fine-Tuning (DPO / LoRA): Because every review is logged in
data/traces.jsonlwith both what the model drafted (original_draft) and what the human corrected (final_draft), we are naturally collecting a Direct Preference Optimization (DPO) dataset. Once you gather a few hundred traces, you don't even need to inject few-shot prompts—you can fine-tune a small LoRA adapter directly on your team's real edits.
5. Quickstart: Run It Locally in 30 Seconds (Zero Credentials)
The entire test harness and stub mode require no external API keys, no cloud credentials, and no GPU.
# 1. Clone the repository
git clone https://github.com/AkshatSoni26/closed-loop-slm-agent.git
cd closed-loop-slm-agent
# 2. Install dependencies via uv
uv sync
# 3. Run the injected-error teeth suite (7/7 offline tests)
make verifier-tests
# 4. Run the full evaluation harness in stub mode
make eval-stub
# 5. Launch the interactive CLI review loop
make run
If you have Ollama installed locally with llama3.2:3b and a Groq API key, you can run the live model comparisons with:
make measure-learning CLIENT=client_101 N=5
Architecture Takeaways for Engineers
- Separate Prose from Math: Let your language model write sentences; let Python write numbers. Your hallucination rate will drop to near zero.
-
Build Explicit Exit Paths: If an agent cannot proceed safely, provide first-class
AbstainandEscalatestates. Graceful failure is the ultimate mark of production software. - Calibrate Your Judges with QWK: Stop using naive accuracy or vibe checks for ordinal rubrics. Use Quadratic Weighted Kappa to penalize catastrophic disagreements.
- Learn In-Context Before Fine-Tuning: A clean telemetry pipeline paired with dynamic few-shot retrieval solves 90% of domain adaptation problems without the cost or complexity of model retraining.
Code & Resources
The entire codebase—including fixtures, schemas, prompts, and evaluation runbooks—is open-source:
- GitHub Repository: https://github.com/AkshatSoni26/closed-loop-slm-agent
- Architecture Documentation: architecture.md
- Decision Journal & Lessons: decision_journal.md
- Evaluation Runbook: before_after_runbook.md
An Open Invitation to the Community
I designed this project as a practical, local-first blueprint for running small language models where precision and operational reliability cannot be compromised. At the same time, agent system design and automated evaluation are rapidly evolving areas, and I am sharing this implementation as part of learning in public.
If you spot flaws in the verification logic, see edge cases in the QWK calibration assumptions, or have discovered alternative guardrail patterns that handle semantic grounding better than what I've documented here—please point them out in the comments or open an issue on GitHub.
Rigorous critique, alternative viewpoints, and shared failure stories are how all of us build more reliable software. If you find something in the code that could be improved, I'd genuinely value your feedback and perspective.
Top comments (0)