Every system that assigns a number to human output eventually teaches people how to produce that number. This is not cynicism about human nature, it is a measurable property of deployed software, and the finance industry has been running the longest and best-documented version of the experiment. The story of how executives quietly rewrote their vocabulary once algorithms started grading earnings calls is worth reading as an engineering postmortem rather than a market curiosity, because the failure mode it describes — a model that works beautifully until its subjects learn the rules — shows up in code review bots, abuse classifiers, fraud scores, and LLM evaluation harnesses with the exact same shape. If you ship anything that scores text, you are already inside this loop. The only question is whether you have instrumented it.
The Dictionary That Stopped Working Once It Was Public
The origin story is a measurement bug. Early attempts to score financial documents borrowed general-purpose psychology word lists, which counted words like liability, cost, and tax as negative. In a bank's annual report those are neutral bookkeeping vocabulary, so the scores were noise dressed up as signal. In 2011, Tim Loughran and Bill McDonald published a domain-specific negative word list that fixed the mislabeling, and the corrected tone measure predicted future earnings and stock returns. A working signal, cleanly validated.
Then the list was published. Within a few years, companies whose filings attracted heavy automated download traffic began stripping out precisely the tokens on that list. Not negative language in general. The specific vocabulary the dominant scorer penalized.
There is a rule in here that every engineer should internalize: a published rubric is a published exploit. The moment you document your linting heuristics, your spam features, your ranking factors, or your code-review checklist, you have handed the measured population a spec. They will build to it, usually without any intent to deceive, because building to the spec is what conscientious people do when a spec exists and consequences attach to it.
Bigger Models Raise the Price of Gaming, They Do Not End It
The standard response is to swap the lexicon for something that reads meaning instead of counting tokens. That works, for a while. When the BERT architecture was released in 2018, contextual embeddings made simple word substitution insufficient, because the model could tell the difference between a hedge and a synonym. The finance research found a second, measurable behavioral shift right after that release. Sophistication moved the equilibrium. It did not remove it.
The useful mental model is a cost curve, not immunity. Every upgrade to your scorer raises the effort required to satisfy it without genuinely changing the underlying thing you care about. Sometimes that effort is so high that gaming and honest improvement converge, which is the ideal outcome. Often it just filters out the unsophisticated and rewards whoever can afford better tooling.
Machines Are Now a Large Share of Your Readers
This is no longer a finance-only phenomenon, because the audience composition of the open web has changed. Cloudflare's breakdown of who is actually crawling websites found GPTBot's share of crawler traffic climbing from 2.2% to 7.7% in a single year, a 305% jump in raw requests, while several traditional indexers stayed flat. Your API docs, your changelogs, your error messages, and your incident write-ups are being parsed by systems that will summarize them for humans who never load your page.
That changes what "writing for your users" means. Documentation that is technically accurate but structurally hostile to parsing gets summarized badly. Documentation optimized purely for extraction gets thin and repetitive. Both failure modes are real, and the second one is the earnings-call trap arriving in your repo.
Six Practices That Keep a Scorer Honest
- Score residuals, not levels. What predicts anything is deviation from what the subject's circumstances already explain. Levels are easy to shift; residuals are expensive to fake.
- Keep a private holdout rubric. Publish the criteria that describe genuine quality, retain a scoring variant nobody outside the team has seen, and use divergence between the two as your alarm.
- Monitor feature distributions, not just accuracy. Accuracy on stale labels stays flat while the input distribution rots underneath it. Track the mean and variance of every feature your model weights heavily.
- Instrument the unscripted channel. In earnings calls, prepared remarks got sanitized while the analyst Q&A stayed comparatively honest. Every system has an equivalent: freeform commit messages, on-call chatter, support transcripts.
- Treat each scorer release as an intervention. Version the model, timestamp the deploy, and check for behavioral breaks afterward. You are not just observing a population, you are perturbing it.
- Budget for relabeling from day one. A scoring model has a half-life. Pretending otherwise means you will discover the decay through a business incident instead of a dashboard.
Instrument the Exposure Split
The single sharpest technique from the finance literature is a natural experiment: compare subjects heavily exposed to machine scrutiny against subjects barely exposed at all. If both cohorts drift together, that is real change in the world. If only the exposed cohort drifts, and it drifts specifically on the features your scorer rewards, you are watching adaptation.
That translates into maybe forty lines of production code:
def adaptation_signal(df, feature_cols, exposure_col, period_col):
"""Divergence in feature TRENDS between high- and low-exposure cohorts.
Large positive values = the scored population is moving on the
exact dimensions your model rewards. That's the alarm."""
hi_cut = df[exposure_col].quantile(0.75)
lo_cut = df[exposure_col].quantile(0.25)
high = df[df[exposure_col] >= hi_cut]
low = df[df[exposure_col] <= lo_cut]
signals = {}
for f in feature_cols:
h_trend = high.groupby(period_col)[f].mean().diff().mean()
l_trend = low.groupby(period_col)[f].mean().diff().mean()
signals[f] = h_trend - l_trend
return dict(sorted(signals.items(), key=lambda kv: -abs(kv[1])))
Run it weekly against whatever your system scores. Alert on the top features by absolute divergence. It will not tell you why the gap opened, but it will tell you where to look, which is more than most teams have.
The Cost That Never Reaches the Dashboard
There is a second-order effect worth naming, because it is the part that damages products rather than models. Text optimized against a parser is text drained of information. When every quarterly update, every postmortem, and every release note is written in the same carefully neutral register, the artifacts get longer, smoother, and less useful to the humans they were originally written for. The information does not vanish. It migrates into residuals, tone, timing, and the things people say when they forget the transcript exists — signals that only well-equipped observers can extract. Everyone else reads polished prose and learns nothing.
That is the real bill. Not a degraded AUC on some internal benchmark, but an organization that has slowly optimized its own writing into noise while its dashboards report improvement.
Build Like Your Scorer Will Be Read
Assume your criteria will leak. Assume the population you measure is intelligent, motivated, and reading carefully. Design so that the cheapest path to a high score is the path you actually wanted, keep a private check on the gap between measured and genuine quality, and set a review date for every model the moment you deploy it. The teams that get burned are not the ones whose metrics get gamed. Every metric gets gamed. They are the ones who never built the instrument that would have shown it happening.
Top comments (0)