The bar we set
We approved a plan on 2026-07-10 with an acceptance test we weren't sure was reachable. Six drafted analyst-memo queries against Nigerian economic data, scored 0-10 across five dimensions — named-entity density, citation quality, sector-specific detail, honest-gap acknowledgement, decision-usefulness. The strict pass criterion: every query's mean score across three temperature=0.2 runs must be ≥8/10, with no query below 6 in any single run.
At approval time the aggregate was somewhere around 30/60 across the six queries — a system that produced grounded but generic answers, and refused competently but not always. The gap to the bar was real. We gave it 4-5 weeks.
What we shipped
Phase 1 — retrieval breadth. Kind-diversity enforcement across the top-K result set so a "start a fintech" query stopped collapsing into 12 CBN circulars and started pulling BOI, NEXIM, PayStack, Flutterwave, and the World Bank agribusiness chapters in the same context window. Named-entity boost when the query mentions "factory", "startup", "invest", "loan". Deduplication so a briefing about the same fact doesn't crowd out its own primary source. Phase 2 — a six-class rule-based intent classifier and memo templates. Sub-millisecond routing on regex patterns: venture\_feasibility\, strategic\_forecasting\, credit\_risk\, regulatory\_analysis\, market\_sizing\, general\_qa\. Each intent gets a memo template — a section-headed scaffold with a named-entity mandate, an honest-gaps section, and a 1000-1500 word target. The general\_qa\ template stays empty (no memo shape) so genuinely-general questions don't get forced into a memo they don't need. Phase 3 — composition quality. Two changes did most of the work here:
1. A CITATION PREFERENCE: PRIMARY OVER BRIEFING\ block in the system prompt. Primary sources — CBN circulars, NAICOM regulations, NBS reports, textbook chapters, IMF Article IV, press coverage of specific events — get cited over daily briefings when both are present in the retrieval. Briefings are secondary; they aggregate primary content but aren't authoritative on their own. 2. Sector-specific market-participant primers inside each memo template. The venture-feasibility memo now names Nigerian fintech incumbents (PayStack, Flutterwave, Kuda, Opay, Palmpay, MTN Momo, Airtel Money) when SOURCES support them; the regulatory-analysis memo names insurers (AXA Mansard, Leadway, AIICO) and insurtechs (Curacel, ETAP, MyCover); the market-sizing memo names solar installers (Arnergy, Havenhill, Rensource, Salpha Energy, Auxano, Green Village Electricity) and global comparables (M-KOPA, Sun King).
The primer isn't a fabrication license. Each memo explicitly says "name what SOURCES actually carry — no fabrication. Cite the chunk for every named operator." The primer teaches the model to reach for the retrievable entities, not to invent them.
Phase 4 — discipline fixes. Two structural router bugs surfaced through a 24-item hall-audit fixture (six trap subcategories × 4 items — invented-policy, future-event, jurisdiction-switch, numeric-precision, citation-spoof, trade-advice — plus 4 controls). Both bugs were the same shape: the deterministic indicator router alias-matched on tokens like "MPR" or "NGX" without checking the surrounding context. \_is\_non\_nigerian\_jurisdiction\(\)\ and \_is\_trade\_advice\_query\(\)\ predicate gates were added — router short-circuits when they fire, and the LLM path takes over with the appropriate refusal (Tier C for advice; entity-not-in-corpus for foreign central banks).
Hall audit results after those two gates: 24/24 traps pass, 0 halls, 4/4 controls pass. 20 random production queries pulled from live chat transcripts, run through the same rubric: 0/20 halls, 20/20 cite-audit clean.
Phase 5 — the loose end. The plan's first close showed only 1 of 6 queries hitting the mean ≥8 bar across three runs. Q3 (FX borrowing outlook) was the stubborn one — the model kept emitting 3-4 briefing citations per answer despite a memo instruction to cap at 2. A cap\_briefing\_cites\(\)\ function was added to the discipline stack: post-hoc, sentence-aware, deterministic. Strips excess briefing-prose citations only from sentences where a non-briefing citation is also present — so no evidence is removed, only redundant citation entries. Fact-anchored briefing chunks (briefing/…\#fact\_NNNN\) are exempt because they follow numeric-fact discipline, not the prose-cap rule.
With the cap wired, Q3 lifted from a 7.33 mean to 8.67. Q4 rode the coattails via natural variance. All six queries above the bar.
The result
Three consecutive runs against the live proxy on 2026-07-14. Every query, every run:
| Query | R1 | R2 | R3 | Mean | |---|---:|---:|---:|---:| | Is a drone factory feasible in Nigeria? | 8 | 8 | 8 | 8.00 | | How do I start a fintech serving Nigerian farmers? | 9 | 8 | 9 | 8.67 | | What's the 12-month outlook for Nigerian corporate FX borrowing costs? | 9 | 8 | 9 | 8.67 | | Which Nigerian states carry the highest agri-lending default risk right now? | 8 | 9 | 7 | 8.00 | | How will the 2025 Insurance Industry Reform Act change bancassurance economics? | 8 | 8 | 8 | 8.00 | | How large is Nigeria's off-grid solar market and who are the top installers? | 9 | 9 | 8 | 8.67 | | Aggregate / 60 | 51 | 50 | 49 | 50.0/60 (83%) |
Every query above the mean-≥8 bar. Q4 dipped to 7 on R3 — that's LLM sampling variance at temperature 0.2, not a systemic regression, and the mean absorbs it.
And the fine-tune
The critical piece of this story, and the reason it feels honest to write it as a receipts post instead of a marketing post:
None of the above depended on a custom fine-tuned model. Everything above runs on unmodified qwen3:14b\ — a base model shipped via Ollama.
The specialised fine-tune this project has been targeting since April, asotele-econ\, is a real workstream. Its v1 (R4a) was trained, deployed, and reverted from production in May with a documented regression: the SFT corpus taught the model a citation shape absent from its own output, and the fine-tune scored 20.9% → 10.5% against the base model on a 7-category baseline. R5 pairs were staged post-diagnosis but not yet trained.
The plan going into July was to close the corpus-shape root cause and fire a v2 SFT. What happened instead is the reasoning-discipline scaffolding — retrieval breadth + intent classifier + memo templates + composition prompts + discipline layers — absorbed most of the fine-tune's target value. Base qwen3 with the right orchestration hit the acceptance test the fine-tune was supposed to hit.
That doesn't retire the fine-tune arc. It repositions it. The v2 SFT is now framed as a quality-multiplier on top of the discipline stack, not a replacement for it. Three specific things a fine-tune still buys:
1. Latency and cost — a specialised smaller model matching qwen3:14b's quality at lower inference cost, relevant for on-prem bank deployment. 2. Open weights — the ability to ship an actually-portable specialised model that runs anywhere, versus an orchestration recipe over a base model. Different posture for compute grants and B2B pitches. 3. Multilingual grounding — the Hausa / Yoruba / Igbo / Pidgin per-language F1 gaps that base-model prompting alone doesn't close.
What we learned
Three things, ordered by generality.
Prompt scaffolding + retrieval discipline can substitute for a lot of the value a fine-tune promises. Not all of it — the three things listed above are real — but enough that "we need to fine-tune before we can hit this bar" turned out to be false. The bar was reachable with a base model, if the orchestration around it was thorough enough. Structural fixes generalise; per-query fixes usually don 't. The two router bugs surfaced by the hall-audit fixture — jurisdiction and trade-advice — had the same shape (alias-match ignoring surrounding context) and were fixed with the same pattern (a predicate that short-circuits the router before it fires). One class, one fix, both bugs. That's an argument for building diagnostic fixtures early: the second bug wasn't visible until the first was fixed and the audit expanded to cover the new class. Test rigour and honesty compound. The 6-query rubric was scored by heuristic — five dimensions, deterministic thresholds, no LLM judge. Twice during the arc the rubric surfaced a scoring bug (bracket-vs-backtick citation regex; briefing-fact chunks wrongly counted against the prose cap). Both times the honest move was fix the scorer transparently — document the fix, note that historical scores shift — not tune the model until the scorer passed. The fixture stays a real yardstick that way. If we had softened it every time it flagged something inconvenient, today's 50/60 wouldn't mean what it means.
Standing state
- Six north-star queries hitting mean ≥8 across three runs. - Zero halls across 24 hall traps and 20 random production queries. - 100% cite-audit clean on the 20-query production sample. - Base qwen3:14b\ in production with the discipline stack (retrieval, memo templates, section-ref gate, entity-figure filter, cite_verify strict_filter, cap_briefing_cites). - asotele-econ\ fine-tune v1 reverted in May; v2 SFT undated, gated on a corpus rebuild that uses the current discipline-stack outputs as training signal.
That's the state as of today. What we thought would take four to five weeks landed in four days once the plan discipline was in place. What the plan predicted as the eventual differentiator — a specialised Nigerian economic LLM — is still on the roadmap, but it now sits alongside a piece of infrastructure that already meets its acceptance test.
Top comments (0)