ChatGPT for Financial Services arrived on 10 September with a benchmark score attached: a shade under seventy per cent correct across a large pile of treasury filings. Strong number, honestly reported by OpenAI.
It also tells you almost nothing about your documents.
That is not a complaint about the vendor. It is what benchmarks are. A score is an average over one corpus, and your corpus is a different corpus. If you are the engineer who has to answer "can we ship this", you need a number computed on your own material, and you need it before someone else picks one for you.
Here is a harness that produces one. It is deliberately small. No eval framework, no vector database, no orchestration layer. A spreadsheet, a loop, and a scoring function.
The shape of the thing
from dataclasses import dataclass
@dataclass
class Case:
qid: str
question: str
expected: str # the answer a human will defend
kind: str # extraction | arithmetic | definition | comparison
source_doc: str
source_page: int
difficulty: str # routine | ambiguous | contested
Five fields carry the whole design, and two of them are the ones teams skip.
kind exists because "document question answering" is four unrelated tasks wearing one label. Pulling a stated figure off a page is not the same skill as computing a ratio from three of them, which is not the same as deciding whether this filing's definition of adjusted earnings matches the one used last quarter. A system can be excellent at the first and useless at the third, and a pooled score will read as respectable either way.
difficulty exists because errors are not distributed evenly. They cluster on the items that were ambiguous or contested, which are exactly the items a junior colleague would have escalated instead of answering. If your test set is all routine questions, you have measured the easy half and learned nothing about your exposure.
Building the set
Fifty cases. Not five hundred. Fifty that a domain expert wrote, where each expected answer is one that person will defend in a meeting.
Pull them from your own documents, and deliberately include:
figures that were later restated, with the original still sitting in the file
a parent and a subsidiary with similar names and different numbers
at least one internally inconsistent document, because you have them
two questions whose correct answer is "the document does not say"
That last category is the most valuable and the most often omitted. A system that never declines to answer is not more capable, it is less honest, and you want that to appear in your numbers rather than in production.
Running it twice
def run_suite(cases, ask, show_sources: bool):
rows = []
for c in cases:
out = ask(c.question, with_citations=show_sources)
rows.append({
"qid": c.qid,
"kind": c.kind,
"difficulty": c.difficulty,
"answer": out.text,
"cited_doc": out.citation.doc if out.citation else None,
"cited_page": out.citation.page if out.citation else None,
"correct": None, # graded by a person, below
"cite_valid": (out.citation is not None
and out.citation.doc == c.source_doc),
})
return rows
Two passes matter, and they measure different systems.
Pass one, citations hidden. This measures the model.
Pass two, citations visible, graded by a reviewer who is allowed to open the source. This measures your review process. The delta between the two passes is the number nobody has: how many wrong answers your humans actually catch when the evidence is right in front of them.
Most organisations have never measured the second one. They assume the review step works because it exists.
Note cite_valid is tracked separately from correct. This is the distinction the whole exercise turns on. A citation establishes provenance. It does not establish that the retrieved figure answers the question asked. A perfectly valid citation pointing at a superseded figure scores cite_valid=True, correct=False, and that combination is the interesting one. Count it explicitly:
def report(rows):
by = {}
for r in rows:
for axis in (("kind", r["kind"]), ("difficulty", r["difficulty"])):
b = by.setdefault(axis, {"n": 0, "ok": 0})
b["n"] += 1
b["ok"] += bool(r["correct"])
for (axis, val), b in sorted(by.items()):
print(f"{axis:10} {val:12} {b['ok']}/{b['n']} {b['ok']/b['n']:.0%}")
trap = [r for r in rows if r["cite_valid"] and not r["correct"]]
print(f"\nsourced but wrong: {len(trap)} <- the ones review will wave through")
Reading the output
Never read the total first. The total is the least informative line in the report, for the same reason a company's average salary tells you nothing about any individual.
Read the segment rows. If one kind is far below the others, you have found the workflow that cannot ship yet, and you can route those questions to a person while shipping the rest.
Then read the direction of the errors. Where the answer is numeric, record signed error rather than a pass/fail flag. If mistakes scatter both ways, that is noise and you can buffer against it. If they all lean the same way, that is bias, it will not average out with volume, and it will produce the identical mistake every time at scale.
This is not a theoretical distinction. In a forecasting system we run in the open, the pooled accuracy sat above where it needed to be for weeks while one slice inside it had stopped working entirely - it captured a twenty-fifth of outcomes where the interval was constructed for half, and the ones it missed were all beyond the same edge. The overall figure was computed correctly and hid the whole thing. We found it by cutting along an axis we had never reported.
The sourced but wrong counter deserves its own attention. Those cases pass every automated check you are likely to build. The link resolves, the page exists, the number appears on it. Only a person who understands the domain catches them, which means that count is effectively a measure of how much human review your workflow genuinely requires.
Keeping it alive
Re-run monthly, pinned to a model version, and store the results. Models get updated without changing their name, your documents change, and a score from March is not evidence about September. A harness that runs once is a slide. A harness that runs monthly is a control.
Total cost: an afternoon for the code, a day of a domain expert's time for the fifty cases. In exchange you get to replace "the vendor reports roughly seventy per cent" with a number about your own work, broken out by task type, with the direction of the errors attached.
That is the difference between citing a benchmark and having evidence.
We run this discipline on our own forecasting models and publish the results, failures included, at neuportal.ai/experiment
Build the harness before somebody else picks the number for you.
Educational content - not financial advice.
Top comments (0)