DEV Community: Francis Oyakhire

Six queries, three runs, every mean 8 — and the fine-tune wasn't why

Francis Oyakhire — Thu, 23 Jul 2026 21:10:43 +0000

The bar we set

We approved a plan on 2026-07-10 with an acceptance test we weren't sure was reachable. Six drafted analyst-memo queries against Nigerian economic data, scored 0-10 across five dimensions — named-entity density, citation quality, sector-specific detail, honest-gap acknowledgement, decision-usefulness. The strict pass criterion: every query's mean score across three temperature=0.2 runs must be ≥8/10, with no query below 6 in any single run.

At approval time the aggregate was somewhere around 30/60 across the six queries — a system that produced grounded but generic answers, and refused competently but not always. The gap to the bar was real. We gave it 4-5 weeks.

What we shipped

Phase 1 — retrieval breadth. Kind-diversity enforcement across the top-K result set so a "start a fintech" query stopped collapsing into 12 CBN circulars and started pulling BOI, NEXIM, PayStack, Flutterwave, and the World Bank agribusiness chapters in the same context window. Named-entity boost when the query mentions "factory", "startup", "invest", "loan". Deduplication so a briefing about the same fact doesn't crowd out its own primary source. Phase 2 — a six-class rule-based intent classifier and memo templates. Sub-millisecond routing on regex patterns: venture\_feasibility\, strategic\_forecasting\, credit\_risk\, regulatory\_analysis\, market\_sizing\, general\_qa\. Each intent gets a memo template — a section-headed scaffold with a named-entity mandate, an honest-gaps section, and a 1000-1500 word target. The general\_qa\ template stays empty (no memo shape) so genuinely-general questions don't get forced into a memo they don't need. Phase 3 — composition quality. Two changes did most of the work here:

1. A CITATION PREFERENCE: PRIMARY OVER BRIEFING\ block in the system prompt. Primary sources — CBN circulars, NAICOM regulations, NBS reports, textbook chapters, IMF Article IV, press coverage of specific events — get cited over daily briefings when both are present in the retrieval. Briefings are secondary; they aggregate primary content but aren't authoritative on their own. 2. Sector-specific market-participant primers inside each memo template. The venture-feasibility memo now names Nigerian fintech incumbents (PayStack, Flutterwave, Kuda, Opay, Palmpay, MTN Momo, Airtel Money) when SOURCES support them; the regulatory-analysis memo names insurers (AXA Mansard, Leadway, AIICO) and insurtechs (Curacel, ETAP, MyCover); the market-sizing memo names solar installers (Arnergy, Havenhill, Rensource, Salpha Energy, Auxano, Green Village Electricity) and global comparables (M-KOPA, Sun King).

The primer isn't a fabrication license. Each memo explicitly says "name what SOURCES actually carry — no fabrication. Cite the chunk for every named operator." The primer teaches the model to reach for the retrievable entities, not to invent them.

Phase 4 — discipline fixes. Two structural router bugs surfaced through a 24-item hall-audit fixture (six trap subcategories × 4 items — invented-policy, future-event, jurisdiction-switch, numeric-precision, citation-spoof, trade-advice — plus 4 controls). Both bugs were the same shape: the deterministic indicator router alias-matched on tokens like "MPR" or "NGX" without checking the surrounding context. \_is\_non\_nigerian\_jurisdiction\(\)\ and \_is\_trade\_advice\_query\(\)\ predicate gates were added — router short-circuits when they fire, and the LLM path takes over with the appropriate refusal (Tier C for advice; entity-not-in-corpus for foreign central banks).

Hall audit results after those two gates: 24/24 traps pass, 0 halls, 4/4 controls pass. 20 random production queries pulled from live chat transcripts, run through the same rubric: 0/20 halls, 20/20 cite-audit clean.

Phase 5 — the loose end. The plan's first close showed only 1 of 6 queries hitting the mean ≥8 bar across three runs. Q3 (FX borrowing outlook) was the stubborn one — the model kept emitting 3-4 briefing citations per answer despite a memo instruction to cap at 2. A cap\_briefing\_cites\(\)\ function was added to the discipline stack: post-hoc, sentence-aware, deterministic. Strips excess briefing-prose citations only from sentences where a non-briefing citation is also present — so no evidence is removed, only redundant citation entries. Fact-anchored briefing chunks (briefing/…\#fact\_NNNN\) are exempt because they follow numeric-fact discipline, not the prose-cap rule.

With the cap wired, Q3 lifted from a 7.33 mean to 8.67. Q4 rode the coattails via natural variance. All six queries above the bar.

The result

Three consecutive runs against the live proxy on 2026-07-14. Every query, every run:

| Query | R1 | R2 | R3 | Mean | |---|---:|---:|---:|---:| | Is a drone factory feasible in Nigeria? | 8 | 8 | 8 | 8.00 | | How do I start a fintech serving Nigerian farmers? | 9 | 8 | 9 | 8.67 | | What's the 12-month outlook for Nigerian corporate FX borrowing costs? | 9 | 8 | 9 | 8.67 | | Which Nigerian states carry the highest agri-lending default risk right now? | 8 | 9 | 7 | 8.00 | | How will the 2025 Insurance Industry Reform Act change bancassurance economics? | 8 | 8 | 8 | 8.00 | | How large is Nigeria's off-grid solar market and who are the top installers? | 9 | 9 | 8 | 8.67 | | Aggregate / 60 | 51 | 50 | 49 | 50.0/60 (83%) |

Every query above the mean-≥8 bar. Q4 dipped to 7 on R3 — that's LLM sampling variance at temperature 0.2, not a systemic regression, and the mean absorbs it.

And the fine-tune

The critical piece of this story, and the reason it feels honest to write it as a receipts post instead of a marketing post:

None of the above depended on a custom fine-tuned model. Everything above runs on unmodified qwen3:14b\ — a base model shipped via Ollama.

The specialised fine-tune this project has been targeting since April, asotele-econ\, is a real workstream. Its v1 (R4a) was trained, deployed, and reverted from production in May with a documented regression: the SFT corpus taught the model a citation shape absent from its own output, and the fine-tune scored 20.9% → 10.5% against the base model on a 7-category baseline. R5 pairs were staged post-diagnosis but not yet trained.

The plan going into July was to close the corpus-shape root cause and fire a v2 SFT. What happened instead is the reasoning-discipline scaffolding — retrieval breadth + intent classifier + memo templates + composition prompts + discipline layers — absorbed most of the fine-tune's target value. Base qwen3 with the right orchestration hit the acceptance test the fine-tune was supposed to hit.

That doesn't retire the fine-tune arc. It repositions it. The v2 SFT is now framed as a quality-multiplier on top of the discipline stack, not a replacement for it. Three specific things a fine-tune still buys:

1. Latency and cost — a specialised smaller model matching qwen3:14b's quality at lower inference cost, relevant for on-prem bank deployment. 2. Open weights — the ability to ship an actually-portable specialised model that runs anywhere, versus an orchestration recipe over a base model. Different posture for compute grants and B2B pitches. 3. Multilingual grounding — the Hausa / Yoruba / Igbo / Pidgin per-language F1 gaps that base-model prompting alone doesn't close.

What we learned

Three things, ordered by generality.

Prompt scaffolding + retrieval discipline can substitute for a lot of the value a fine-tune promises. Not all of it — the three things listed above are real — but enough that "we need to fine-tune before we can hit this bar" turned out to be false. The bar was reachable with a base model, if the orchestration around it was thorough enough. Structural fixes generalise; per-query fixes usually don 't. The two router bugs surfaced by the hall-audit fixture — jurisdiction and trade-advice — had the same shape (alias-match ignoring surrounding context) and were fixed with the same pattern (a predicate that short-circuits the router before it fires). One class, one fix, both bugs. That's an argument for building diagnostic fixtures early: the second bug wasn't visible until the first was fixed and the audit expanded to cover the new class. Test rigour and honesty compound. The 6-query rubric was scored by heuristic — five dimensions, deterministic thresholds, no LLM judge. Twice during the arc the rubric surfaced a scoring bug (bracket-vs-backtick citation regex; briefing-fact chunks wrongly counted against the prose cap). Both times the honest move was fix the scorer transparently — document the fix, note that historical scores shift — not tune the model until the scorer passed. The fixture stays a real yardstick that way. If we had softened it every time it flagged something inconvenient, today's 50/60 wouldn't mean what it means.

Standing state

- Six north-star queries hitting mean ≥8 across three runs. - Zero halls across 24 hall traps and 20 random production queries. - 100% cite-audit clean on the 20-query production sample. - Base qwen3:14b\ in production with the discipline stack (retrieval, memo templates, section-ref gate, entity-figure filter, cite_verify strict_filter, cap_briefing_cites). - asotele-econ\ fine-tune v1 reverted in May; v2 SFT undated, gated on a corpus rebuild that uses the current discipline-stack outputs as training signal.

That's the state as of today. What we thought would take four to five weeks landed in four days once the plan discipline was in place. What the plan predicted as the eventual differentiator — a specialised Nigerian economic LLM — is still on the roadmap, but it now sits alongside a piece of infrastructure that already meets its acceptance test.

The CC-BY 4.0 we never had

Francis Oyakhire — Thu, 23 Jul 2026 21:01:07 +0000

The day started with the word "status."

I typed it into the model without qualification, hoping the answer would be short. The reply came back with the shape of a project dashboard: RAG index at 695,337 chunks, sector-audit halls at 13 out of 136, three advisors landed, port-activity ingest freshly wired the previous evening. The last item on the list was a follow-up bullet from yesterday's session: " Founder unblocked #3 HDX port activity — ingest/port\_activity\_to\_duckdb.py\ landed: imf\_portwatch\_daily\ 78,880 rows · 29 ports · 2019-01 → 2026-06 + 11 transport\_port\_activity\ narrative chunks. Kind added to NIGERIAN_KINDS. RAG: 695,326 → 695,337. DHS + IATI still pending founder call."

The port-activity note was written at yesterday's end. Two dataset decisions — DHS Subnational and IATI Nigeria, both flagged as HDX "Other" licence — had been left open. The intended plan was to address them, close them out, and move on.

I picked the DHS + IATI thread first. Both packages had been sitting in data/forecasts/hdx/\ since 2026-06-20, downloaded but not indexed, waiting for licence verification. The HDX API package\_show\ endpoint returned their real licence text, buried in a license\_other\ field that the summary listings don't surface. DHS came back as CC-BY-NoDerivatives 4.0 — a real CC licence, but not one on our whitelist, and — critically — a licence whose derivative-works clause makes any LLM-paraphrased briefing about the data legally ambiguous. IATI came back as " licence is delegated per publisher" — a mixed-licence pool where every activity carries whichever CC (or non-CC) terms its reporting organisation chose, with no per-activity field in the JSONL export to filter on.

Both firewalled. Tracker rows moved from ❌ pending to 🚫 firewalled. Two memory files written. The health-and-education blind spot that DHS was going to fill stays open; the fiscal-aid gap that IATI was going to fill is already partially covered by cleaner-licenced siblings in the same HDX batch (OCHA FTS, CBPF, CERF, IFRC — all under uniform terms).

That looked like the end of the story. It was actually the middle.

"That came from a government website — I didn't think that would have any issues"

The next message from the founder was casually declarative: let us double check port activity that came from the government website I did not think that would have any issues.

This is the discipline that produces the good catches — the reader who trusts nothing yet, including yesterday's confident bullet in yesterday's own notes. IMF PortWatch is not a Nigerian government website (it's the IMF's vessel-tracking dashboard for global port activity), but the founder's instinct read a deeper fact than the words: if we 've been quietly confident about a source, someone should look.

The HDX API for the port-activity package returned this, verbatim in license\_other\:

https://www.imf.org/en/about/copyright-and-terms\

Just a URL. No CC licence declared. Not CC-BY, not CC-BY-4.0, not CC-anything. A pointer to IMF's own copyright page — which, on retrieval (via Wayback, because www.imf.org\ is Akamai-blocked to our stack for reasons already documented), turns out to declare IMF 's own bespoke Data Terms, effective 2024-10-11. Not on any Creative Commons spectrum at all.

The Data Terms are substantively permissive for our use case — the "Use of IMF Data" section explicitly overrides IMF's general commercial-use prohibition and grants " download, extract, copy, create derivative works, publish, distribute, and use" subject to attribution and integrity. There's a genuinely ambiguous clause immediately following that says " For any potential commercial reuse of IMF Data, please email [email protected] to request permission," which reads either as a safety-belt-not-a-hard-gate or as a hard-gate depending on how conservatively you interpret the drafter. But the material fact remains: it is not CC-BY 4.0.

Yesterday's ingest note had said, with confidence: " HDX manifest flags 'Other' but the upstream IMF PortWatch (portwatch.imf.org) publishes CC-BY 4.0." That claim was unverified — I had assumed it based on context and phrasing that sounded reasonable. Three hardcoded strings in sources/ingest/port\_activity\_to\_duckdb.py\ carried the false CC-BY-4.0 declaration. Eleven narrative chunks in the RAG index each contained the sentence " Source: IMF PortWatch (portwatch.imf.org) via HDX. Licence: CC-BY 4.0." in their topline. The DuckDB imf\_portwatch\_daily\ catalog metadata had a "license": "CC-BY-4.0"\ field.

Any bank-facing brief that cited port-activity data would have shown that footer. Any lawyer reading the brief would have caught the mismatch — because IMF Data Terms have a fingerprint (they require the specific attribution string " Source: International Monetary Fund, Database <>") that a CC-BY 4.0 footer erases. The failure would not have been "we cited an unlicensed source"; it would have been "we cited the source under the wrong licence and mislead a reader about the terms of use." That's a different flavour of unforced error, and arguably a worse one for a fiduciary-facing product.

The fix landed in about twenty minutes: three source-string edits in the ingest script, a re-run that replaced-in-place the eleven chunks with the correct attribution (" Source: International Monetary Fund, IMF PortWatch Nigeria daily port activity … Terms: IMF Data Terms (imf.org/en/about/copyright-and-terms)"), and a tracker note that recorded the earlier-note-was-wrong finding openly.

The rule that made itself

The catch changed the shape of the rest of the day. Every remaining ❌ pending row on the coverage tracker went through the same verify-verbatim-first pass before being touched. HDX-published packages: pull license\_other\ directly from the API, don't trust the summary listing. Non-HDX Nigerian gov sites: read the actual /legal\ page and, if the artifacts are PDFs, sample the front matter for embedded engagement-letter clauses.

The next five items on the shortlist:

- HDX Nigeria health-care facilities (GRID3 publisher) — API returned license\_id: cc-by\, license\_title: Creative Commons Attribution International \(CC BY\)\, license\_url: http://www.opendefinition.org/licenses/cc-by\. Textbook CC-BY. On the whitelist. Ingested clean: 46,146 facilities across 37 states and 769 LGAs into DuckDB nga\_health\_facilities\, plus 38 per-state narrative RAG chunks. Every chunk carries the correct attribution string this time — no hardcoded CC-BY declaration written from memory. The data itself is bank-relevant: LGA-level facility inventory with ownership mix (public / private / mission), functional-status (74.3% functional, 25.5% unknown), and tier (95.3% primary, 2.9% secondary, 1.7% tertiary). Lagos state alone: 2,320 facilities across 20 LGAs, 49.6% private, 24.1% state PHCDA. This is the level of granularity that lets a bank underwrite a hospital-loan portfolio.

- Mozilla Common Voice — Nigerian languages. Licence verified verbatim from the common-voice/cv-dataset\ GitHub repository: CC-0 (public-domain dedication). Zero risk. But the Nigerian-language coverage tells its own honest story: Hausa 4.09 validated hours, Yoruba 5.57, Igbo 0.02, Pidgin absent from Common Voice entirely. The trajectory across releases v13 → v21 (2023-03 → 2025-03) shows Yoruba growing fastest, Hausa slow-steady, Igbo effectively stagnant. Metadata-landed as four RAG chunks; audio files not downloaded because we don't currently have a speech pipeline. This is future substrate for the accessibility workstream, catalogued honestly with the gaps stated.

- Nigerian Pidgin Bible (eBible.org, expected as a Pidgin-corpus filler). The details page for the only Pidgin translation on eBible reads: " Copyright © 2019 Wycliffe Bible Translators, Inc. … All rights reserved." Not CC-anything. Firewalled. The Pidgin gap has to close some other way — AfriSenti (CC-BY-4.0), WURA Pidgin split (Apache-2.0, partial already), NollySenti (CC-BY-SA-4.0) — all noted in an existing memory that the day's audit reaffirmed rather than replaced.

- NEITI — Nigeria Extractive Industries Transparency Initiative audit reports (row #99, oil-and-gas + solid-minerals + fiscal-allocation reconciliation audits). This one was subtle. The NEITI site was migrated to Next.js since our last probe, but SSR HTML still exposes all 55 audit-PDF URLs cleanly. The /legal\ page says " Content is protected by copyright and intellectual property laws." Traditional copyright — no CC grant. But the deeper distinction is on the PDFs themselves. Page 3 of the 2013 Oil and Gas audit report, embedded verbatim by the audit firm (Taju Audu & Co) before NEITI published the artifact: " Our report is solely for informing the NSWG on the matters set out in the Terms of Reference and is not to be used for any other purpose." NEITI publishes the reports under statutory mandate; they don't author them. The audit firms do — different firms in different years — and each firm attaches engagement-letter language that governs downstream use. Nigerian statutory-instrument no-copyright status (which unlocks tax laws and Presidency releases) does not extend to consulting deliverables published under statute. Catalog-only ingest : 55 audit-report titles with year + sector + URL, plus fair-use extraction of the three NEITI-authored press releases at /media/news/\*\. Seven RAG chunks total. Full-text ingest requires a written reuse licence from NEITI, which the founder can pursue via [\[email protected\]](/cdn-cgi/l/email-protection)\ if oil-transparency full-text becomes strategically critical later.

- PEBEC — Presidential Enabling Business Environment Council (row #91, 200 MB of reform reports harvested via Wayback since the live site was moved behind Clerk authentication). PEBEC is a Presidency body; its reports are self-authored federal-government public output. Same author class as State House policy releases already in the corpus, distinct from the NEITI audit-firm class. Fifteen text-extractable PDFs ingested: Executive Order 1 on Ease of Doing Business (2017), National Action Plans NAP-60 / 2.0 / 5.0 / 7.0, three EO1 compliance reports (2021, 2022, 2024-H1), the Business Facilitation Act 2022 text plus its 2023 compliance report, and the biggest single artifact: the January 2021 Subnational Ease of Doing Business Baseline Survey — 156 pages scoring all 36 states plus FCT on regulatory-friction metrics. 759 new RAG chunks. Two scanned PDFs deferred to the OCR backlog. One duplicate skipped.

The five decisions in sequence: catch-and-correct, firewall, land-clean, land-metadata-only, firewall, catalog-only, land-clean. The RAG index moved from 695,337 chunks at the start of the day to 696,145 at the end. Not dramatic. Two of the five items were bigger conversation-changers than any raw-chunk-count would suggest.

The distinction the audit made explicit

At the beginning of the day, the mental model was " Nigerian federal-government content is default safe to ingest with cite-with-attribution." That model was mostly right — it correctly handled State House releases, tax laws, the CBN corpus, the NAICOM insurance corpus, the NDIC bank-stability corpus, the NUPRC gas-sector rows.

What today's arc surfaced is that the model was too coarse. The load-bearing question isn't who published this? — it's who authored the specific artifact?

- Self-authored by the federal-government body: State House releases, tax laws, NAICOM guidelines, NDIC quarterly reports, PEBEC reform reports. Cite-with-attribution safe. - Third-party-authored, government-published under statute : NEITI audit reports (authored by Taju Audu & Co and other consulting firms), similar future items likely including NNPCL audits, DPR/NUPRC commissioned consultancy reports, and the AMCON annual reports (whose author-class we still need to verify before promising a date on that ingest). These carry the authoring firm's engagement-letter terms even when the government publishes them under a public-disclosure mandate. - Third-party-published under a bespoke non-CC licence : IMF PortWatch (IMF Data Terms), USAID DHS Program via HDX (CC-BY-ND), most WFP HDX packages (need per-package check because "usually CC-BY-IGO" was the assumption class that just broke on PortWatch). - Third-party-published under whitelist CC : Grid3 health facilities (CC-BY), Mozilla Common Voice (CC-0), WB WDI (CC-BY-4.0), Africa's Pulse (CC-BY-IGO).

The distinction changes what each remaining ❌ row on the tracker requires. It's no longer a licence audit on the publisher — it's a licence audit on the author of the artifact. That's a slower but more honest posture, and it's the one a fiduciary-facing product needs.

What this is not about

The Asotele product is a data product for Nigerian banks. Banks trust it — the intent is that they eventually will trust it — to inform credit committees and treasury decisions. The stack we're building is the stack such a bank would build itself if it had the time.

Nothing about today's arc was a licence-crisis story. No bank has been harmed. No brief was shipped with a false attribution — the CC-BY-4.0 fabrication was caught before it appeared on any advisor-facing surface, let alone a bank-facing one. The chunks were replaced in-place with the correct attribution about ninety minutes after the founder's initial " let us double check" message.

But the near-miss is the point. In eleven months of building this system, the failure mode I have come to trust least is the confident line in yesterday's own notes. Today it was licence text; earlier this week it was a hardcoded oil-benchmark constant; last month it was a WHO relevance-classifier that lifted country mentions into country-cases. Each of those was caught by the same shape of behaviour: someone reading a system output and refusing to accept the sentence just because the system produced it.

The audit-yesterday's-notes discipline compounds in a way that adding data does not. Four new ingests today added 808 chunks. The catch removed one class of latent failure that would have been embedded in every future bank-facing citation of a large recurring dataset.

The rule that came out of the day is not going to sound novel. Every HDX package with license\_id == hdx-other\ needs license\_other\ pulled verbatim before ingest. Every non-HDX Nigerian federal-government artifact needs the author of the specific document verified — not just the publisher. Every commit that hardcodes a licence string in an ingest script requires a citation to the verbatim source that the string came from.

None of that is beautiful engineering. It's tax on shipping. But it's exactly the tax a bank pays when it does its own version of this work, and it's exactly the tax that makes the difference between "a project someone built quickly" and "a system someone can rely on."

We didn't move the sector-audit needle today. Halls at 13 out of 136 this morning; halls likely at 13 tomorrow, until the next audit run comes in overnight. The compounded improvement is in the moving parts you don't see — the memory that now records why DHS is blocked, the ingest script that now emits the correct IMF attribution, the tracker row that now notes the audit-firm engagement-letter distinction that will apply to a dozen future items.

The CC-BY 4.0 that we never had is not in the system any more. That's the win.

— founder note, 2026-07-03

Hello Dev world

Francis Oyakhire — Thu, 23 Jul 2026 15:41:55 +0000

Hello Dev Word