Our last post ended on a problem we could not measure: answers that cited our corpus were eight times less consistent than answers that cited nothing, and every test we had was blind to it. A question about broomsticks cited an aviation procedure. The facts were often right; the citations were invented.
Thirteen days later, that workstream is closed. Not solved — closed, with a negative result, which is a different and less comfortable thing to publish.
First we built an instrument that could fail
The old measurement scored our answer against the question , which meant every sourced addition counted against us: 73 answers wearing an audit footer scored a median of 0.049 while 153 bare ones scored 0.874. It was measuring formatting.
The replacement scores each sentence against the text of the chunks that sentence actually cited , pulled from the corpus by id. Per sentence, not per answer — because a long well-grounded answer would otherwise score worse than a short one, reimporting the same length artefact we had just removed. And a sentence is the unit you can act on: it names which claim is unsupported.
That gave us a number that could move. Then we tried to move it.
Five techniques. Five controls. Nothing survived.
Each fix was shipped with a control designed to kill it — a decoy input, a size-matched random baseline, word-salad. This is the part we would ask anyone to copy.
The clearest failure was atomic decomposition: breaking each sentence into individual claims before scoring. Support went from 20.00% at sentence level to 32.81% atomic. Thirteen points. We were ready to call it a result.
Then the control came back. Splitting the text into random fragments of the same size scored 40.62% — better than the technique, on five questions out of five.
The gain was not the method. It was an artefact of chopping text into smaller pieces, and any chopping would do. Without that control we would have published a thirteen-point improvement that meant nothing.
All five techniques went the same way. The scoreboard is 0 for 5.
The finding underneath
Widening retrieval from 15 chunks to 100 — nearly seven times the evidence — rescued four sentences out of 138.
That is the whole result, and it points somewhere specific. If depth barely helps, the material was not missing. We went and checked: 84.8% of sentences had no supporting citation, but the chunks retrieved were about a different question. The corpus held the answer. The retrieval step fetched something adjacent and the generator wrote around it.
So the defect is upstream of everything we were tuning. Uncited, unsupported, and absent are three different failures , and we had been treating them as one. Every technique we tried sat downstream of a retrieval step that had already fetched the wrong thing — which is why polishing them changed nothing.
The one intervention that did work was not a grounding technique at all. A domain gate that routes by jurisdiction rather than topic — deciding whether a question is about Nigeria, not whether it sounds economic — cut wrong answers from 91 to 51 on the external benchmark. Maths, statistics and finance count as in-domain; the gate has produced no false refusals across 107 in-domain questions.
What we built while that was running
A negative result on one front is not a fortnight’s work. In the same period:
A feasibility engine with no language model in it. Ten slots — method, evidence, absence — where every figure comes from a SQL query and prints its source table. It cannot invent a number. Asked about vehicle assembly it returns a 32.7 percentage-point tariff wedge on HS8703 and effective protection of 70–168%, each traceable to a row.
We test it with a fictional industry. On its first run, “XXXYS” scored 0 out of 6 industry slots but still printed a national electricity price — country facts leaking in as industry findings. That is exactly the false-adjacency failure that produces confident nonsense, caught by an industry that does not exist.
A tariff schedule labelled line by line. All 6,515 lines of the ECOWAS Common External Tariff, 92% precision on a blind sample. The load-bearing rule turned out to be negation order : “not for assembly industry” sits beside “for assembly industry” on the same component, and testing them in the wrong order inverts the protection wedge on precisely the lines that matter.
A wage anchor from the ILO. Nigerian manufacturing earnings are ₦56,332/month (2024) — 26% below the economy-wide ₦76,489. Quoting the national average overstates a plant’s wage bill by a third. We had rejected an alternative source on licensing; when we finally got a clean one, we discovered that source also ran 4.4× too high. The method was the real problem, and we only learned it by getting the good data.
Where the numbers stand
Our six-query internal benchmark scored 50 out of 60 today, with every query at 8 or above — the first run where none falls below the bar. The previous run scored 51.
We are not going to tell you that 51 to 50 means anything. The measured noise floor is ±1 per query, and generation on this stack is not deterministic even at temperature zero with a fixed seed — we tested that directly: byte-identical prompt, same process, different answers. Any single run is a sample. Claiming a one-point movement as a trend would be the same error as the thirteen-point gain that turned out to be random.
The hallucination scorer did not produce a fresh number today at all: it collected all twelve answers and then failed to load its model after a dependency upgrade. The answers are checkpointed and will be scored when the loader is fixed. We would rather report that than a stale figure.
What this cost us to learn
Three things worth carrying out of it.
A control built from the same source as the thing it checks is not a control. We verified a fiscal deficit by confirming the published figure equalled revenue minus expenditure — using the same columns all three came from. It passed 71 times out of 71 with an error of exactly zero, and proved nothing. A check that cannot fail is not evidence, and a perfect pass rate is a warning.
One matching number is not a match. Given a figure that seemed to conflict with ours, we decided it belonged to a different year because one of its three numbers matched that year. The other two matched nothing. It did not belong to a different year.
When a figure is doubtful, the fix is an anchor, not a disclaimer. We had written a great many caveats. What actually resolved it was asking which series everyone else publishes, and whether ours ties to it. Our federal deficit figures now come from the Budget Office’s own Budget Implementation Reports — ₦13.51tn for 2024 against ₦20.98tn of revenue — because that is the number Nigerian newsrooms quote and a reader can check us against it. Where other official sources differ, and they do, we print the difference on the page rather than reconciling it away.
Next: the retrieval step, which is where the grounding work says the defect actually lives. We have spent a fortnight proving that the things downstream of it cannot be fixed from downstream.
Top comments (0)