Every agent that reads financial text has to decide which company a name means. Here is how we graded ours - and why we had to grade the answer key first.
Every AI agent that reads financial text has to answer one question before anything else: which company is this?
"Apple" in a filing. "MS" in a headline. A name that changed three years ago, or a ticker that now belongs to someone
else. Get it wrong and everything downstream is confidently wrong.
We have been building a lookup for exactly that: give it a company name as it was written, in a filing or an article,
and a date, and it returns the company that name meant on that date - including renamed and delisted ones. This week
we measured it properly. Three things surprised me.
1. We had to check the answer key first
To grade a matcher you need an answer key: a set of names where careful labellers have written down the right company.
We split ours in two - one half to tune on, one half held out and never looked at while tuning.
Before trusting any score, we had a second model act as a referee on 891 items of the key, without ever seeing what our
lookup had answered. It cost about $3.50 of API usage. What it found:
- 485 confirmed as they stood.
- 226 widened: the original answer was right, but the same company also existed as a second record, and the two had been offered as rivals. One company, two entries - both are now accepted.
- 101 settled: items the labellers had marked "can't tell" now have an answer.
- 2 reversed: outright wrong labels.
So the key was rarely wrong; it was incomplete. Every change is stored as its own record with a reason, on top of the
original, never as a silent edit.
On the checked key, the lookup scores 0.927 on the held-out half (629 matches; lower bound 0.904). The part I find
most useful: answers the lookup gave where the key said "can't tell" fell from 27 to 10 - most of them turned out
to be the same company listed twice. Real wrong answers barely moved (37 to 36). Checking the key changed what we
thought was a mistake, not how many mistakes we make.
If you grade an entity matcher against an answer key nobody has audited, you are partly grading the key.
2. Same question, three different orderings
Some names legitimately point to several entities - Morgan Stanley is a good example. When the lookup returned those
tied alternatives, their order depended on a random seed that changes every time a program starts. Same question,
three runs, three orderings. The answer itself never changed - only the order of the alternatives. For a person that is
a shrug; for an agent that takes the first item in a list, it is a coin flip. Ties now break on a stable identifier, and
the same question gets the same list every time.
3. News is not filings
Filings are formal; company names sit in predictable places. News prose is not. On FiNER-ORD, a public, human-labelled
set of financial news text (Shah et al., 2023), rule-based name finding scored an F1 of 0.350. A free open model
(GLiNER), fine-tuned on our filings data, scored 0.820. Rules that work on filings fall apart on headlines.
What I don't know yet
- How well matching (not just finding) works on news text - there is no human-labelled answer key for that yet.
- Whether 0.927 holds outside SEC filings.
What's next
We are working on letting AI agents ask this lookup directly - "which company did this name mean on this date?" -
with the answer key's score attached to every answer. If you are building an agent that needs that, reply or write to
AlexB@AltDataForge.com and I'll let you know when there is something to try.
Next post: when a ticker changes hands - how one company's history quietly leaks into another's when a symbol is
reused, and what it takes to stop it.
A question for you
If you run entity resolution for agents or research: how do you grade it - and has your answer key ever turned out to
be incomplete rather than wrong? If someone has published a good open answer key for company matching, link it and I'll
credit it.
Want this checked on your own data or backtest? Backtest & Data Audit, one-week pilot: https://altdataforge.com/audit
FiNER-ORD: Shah et al., 2023, arXiv:2302.11157 (CC BY-NC 4.0; used here for evaluation only). The free SEC data
dictionaries this work sits on: https://alexbocio.github.io/us-gov-data-dictionary/
Top comments (0)