DEV Community: INTFRAME

We measured the LLM signature in 398,296 Korean journal abstracts

INTFRAME — Tue, 01 Sep 2026 12:37:01 +0000

Since 2023 several studies have shown that the vocabulary of English scholarly writing changed after ChatGPT. Kobak et al. (Science Advances, 2025) called it excess vocabulary: a word's frequency above the trend it was on before 2023. English abstracts started overusing delve, underscore, intricate.

One Korean study exists: Koo, Kim and Kim (Language and Information, 2026) ran an excess-vocabulary comparison over ~880,000 KCI humanities and social-science abstracts, projecting 2024 from 2021-2022. They found a sharp rise in the English abstracts and only modest movement in the Korean ones, and their observation ends in 2024 — before the Korean onset we measure. Korean had not been measured at the point where it actually moves, and the method does not transfer for free: Korean is agglutinative, so whitespace tokens are the wrong unit, and the verbs that carry register are built by fusing a Sino-Korean noun with a verbalising suffix.

So we rebuilt the statistic over morphological lemma-tag units and ran it on the full openly harvestable record of the Korea Citation Index: 398,296 abstracts from 2,282 journals, 2018 to August 2026, with 47,165 Vietnamese abstracts as an exploratory second case.

What the numbers say

The verb 시사하다 (sisahada, "suggest, imply") appears in 21.4% of 2026 abstracts. Extrapolating its 2018–2022 trend predicts 5.3%.

In the other direction, plain-register expressions fell: 알아보다 ("look into") and 도움이 되다 ("be helpful") dropped to about a quarter of their trend.

The time course is sharp. Nothing in 2023. Onset in the second half of 2024. A steep rise through 2025. Flattening in mid-2026.

Under stated identification assumptions, the conditional lower bound on the share of LLM-processed Korean abstracts is:

Statistic	2024	2025	2026
Single word	3.5%	10.5%	16.1%
Split-half word set	7.8%	20.6%	33.0% (95% CI 30.8–35.4)

This is not a detector

It cannot tell you whether your paper was written by a model, and it is not built to. It measures a distribution over a corpus and reports a floor: the true share is at least this, under assumptions we state and test.

Ruling things out

A number this large invites obvious objections, so we tried to kill it several ways.

"The fields just changed what they write about." We paired every 2026 abstract with the base-period abstract from the same journal whose title is most similar (character-bigram cosine, IDF-weighted). The matched difference is 34.1 pp against 34.0 pp unmatched. Topic matching barely moves it. We also had three independent language-model annotators classify each marker lemma as style or topic; restricting the set to lemmas all three call style still leaves 14.7 pp.

"It's translation from English." The surface marks of translated Korean (the 것이다 construction, ~에 대한, explicit plural 들) fell over the same period while the markers rose.

"It's a statistical artefact of extrapolation." We ran the identical procedure with pre-ChatGPT target years, at every extrapolation horizon the data allow. The largest set-statistic value any placebo produced was 2.9 pp. The 2026 value is more than eleven times that.

"You got lucky with one split." The set is chosen on one half of the journals and measured on the other. We repeated the split 100 times with different journal hashes: median 32.4 pp, 5th–95th percentile 31.2–33.8, set sizes 38–56 lemmas.

The English abstracts of the same articles

Most KCI articles carry an English abstract alongside the Korean one. That gives an article-fixed comparison: same authors, same journal, same paper, two languages.

The English side moves a year earlier — the excess is already there in 2023. And among articles whose English abstract carries none of the period's English markers, the Korean shift is still present, at 30–66% of its uncorrected rate after correcting for the English indicator's own sensitivity.

So the Korean change is not simply a by-product of a model-written English abstract, though the size of the effect does shrink under that correction.

Everything reproduces

The paper (28 pages), the analysis code, per-year document-frequency tables for Korean, English and Vietnamese, the journal panels, and 1,437 control abstracts we generated with models from three providers are all deposited:

https://doi.org/10.5281/zenodo.22110398

Unzip it into an empty directory, run reproduce.sh, and every table and figure in the paper is regenerated from the shipped results. It ends in REPRODUCE_OK or it fails loudly. Raw abstract texts are not redistributed for copyright reasons, but the harvesting scripts against the public KCI and VJOL OAI-PMH endpoints are included.

Two free tools built from the same measurements, in Korean:

AI style dictionary — which words rose and fell, with base rates
Style checker — paste text, see which markers it contains (again: a base-rate lookup, not a verdict)

What this does not settle

The attribution assumption is the weak point, and we say so in the paper. Observational data cannot rule out that some contemporaneous stylistic drift, entirely unrelated to language models, produced part of this. What we can do is show the size of the change, its timing, its resistance to topic and translation controls, and the level of noise the method produces when nothing is happening. That is why the headline number is a conditional lower bound, not a prevalence.

The annotation of markers as style-versus-topic was itself done by language models, which is circular in a way we would rather fix with human annotators. That is the first thing on the list.

The manuscript is under review at Scientometrics. Comments, replications and objections are welcome — the data is right there.

Trust, then verify, then verify again

INTFRAME — Sat, 15 Aug 2026 10:50:06 +0000

Generative systems fail differently from normal software. They do not crash. They produce something plausible, fluent and occasionally wrong, at scale, with a straight face. Shipping their output to the outside world without machinery in between is negligence. Ours pass three walls.

Wall one: evidence gates

Every automated claim must carry its receipts, and the receipts are structured, not vibes:

{
  "claim": "business site has no online booking flow",
  "evidence": {
    "url": "https://example-biz.com/reserve",
    "fetched_at": "2026-03-11T02:14:09Z",
    "check": "booking CTA selectors absent in rendered DOM",
    "screenshot": "sha256:9f2c11ab..."
  },
  "verdict": "pass"
}

No evidence object, no pass. Fuzzy match where the rule demands exact, no pass. It is astonishing how many hallucinations die at the simple demand: show me where you read that.

Wall two: the identity gate, and the bug that taught us humility

A pipeline that writes about real businesses must first prove it is looking at the right business. Our first identity gate compared the business name against the domain name and skipped anything that did not match. Reasonable, and catastrophically wrong in a market where businesses have Korean names and Latin-alphabet domains. The gate was silently discarding 98% of legitimate targets. The fix was to stop trusting the domain string and verify identity from the page body instead, name, address, phone cross-checked against the record. Pass rate went from 2% to 71% with zero wrong-entity incidents since. The lesson generalized: a gate that fails closed can be just as wrong as a gate that fails open, and only measurement tells you which failure you have.

Wall three: adversarial second passes

The cheapest quality upgrade we know is asking a second model instance to attack the first one's output. Not improve. Refute. The reviewer prompt states that its job is to find the flaw, and that when uncertain it should call the finding refuted, because a missed defect costs us more than a discarded one. A reviewer whose only job is finding flaws finds the ones a collaborative reviewer politely overlooks. Findings that survive are the ones worth acting on.

Behind all three walls sits the regression suite, about 3,900 cases, expected failures zero, one case added per incident. None of this slows the pipeline meaningfully. All of it together is what lets us point generative machinery at real-world consequences and sleep.

Evals are the unit tests of prompts

INTFRAME — Sat, 15 Aug 2026 10:48:35 +0000

Prompt changes used to ship on vibes: run three examples, looks better, merge. Then a "small wording improvement" quietly broke number formatting in a pipeline for four days. Now a prompt is code, and code does not merge without tests.

The golden set grows out of scars

Each pipeline carries a labeled set of input/expected pairs. Nobody sat down to author it; it accreted, one case per incident, each test a way the system once embarrassed itself. Our largest text pipeline currently runs about 3,900 cases on every change with an expected failure count of exactly zero. The zero is non-negotiable. A suite that tolerates a few failures is a suite nobody reads.

LLM-as-judge, with handcuffs

Half our cases have no single right answer, so a model judges them. A model judge is a biased witness, and we treat it like one:

Verdict rules:
- Compare A and B for the stated criteria only. Do not reward length.
- TIE is a valid verdict. Prefer TIE over a coin flip.
- Output exactly one line: WINNER: A | B | TIE, then one reason.

(Runner: every pair is judged twice with A and B swapped.
 Only a verdict that survives the swap is recorded. ~11% do not.)

Position bias is real. Judges favor the first answer. The swap-and-agree protocol deletes that class of noise, at the cost of doubling judge calls. Worth it.
Length bias is real. Unconstrained judges reward word count. The rubric says so explicitly, and outputs are capped before judging.
Self-preference is real. A model grades its own family generously, so the judge is a different family than the generator wherever the stakes are high.

Agents get a harder bar

For agent tasks we do not measure pass@k, whether one of k attempts succeeds. We measure the opposite: run the same task k times and require every run to succeed before the agent earns autonomy on that task class. A workflow that succeeds four times out of five is not 80% done. It is a pager duty rotation you have not scheduled yet.

The payoff is boring in the best way: prompt refactors, model swaps and provider migrations became routine changes with a red/green answer, instead of leaps of faith followed by a week of watching logs.

Small models where they win

INTFRAME — Sat, 15 Aug 2026 10:45:00 +0000

The question "which model should we use" is usually malformed. For classification-shaped work, the answer is two models and a threshold.

The pattern: label big, serve small

When a pipeline needs to classify at volume, categories, spam, intent, language, we do not call a frontier model per item. We call it once per dataset: have the big model label 30,000 historical examples with a carefully engineered rubric, spot-check a sample by hand, then distill those labels into a small model that runs on our own box. The frontier model's judgment gets compiled into something that serves at 15 ms for the price of electricity.

The router handles the hard tail

const p = small.classify(x);            // ~15 ms, on-box, effectively free
if (p.confidence >= 0.92) return p.label;
return frontier.classify(x);            // ~1.5 s, the 7% that deserve it

On our distribution, 93% of items clear the threshold locally and agreement with the frontier model on that slice is above 99%. The ambiguous 7% get the expensive brain. Blended cost lands two orders of magnitude below calling the frontier model on everything, and p50 latency drops from seconds to milliseconds, which matters when the classifier sits inside a user-facing request.

Where fine-tuning actually loses

We keep re-testing fine-tuning and it keeps losing to prompting for our shapes of work. Style, format and policy compliance are handled better by a versioned prompt with few-shot examples riding a warm prefix cache: changes deploy in minutes, roll back in seconds, and A/B cleanly. A fine-tune is a build artifact with a days-long iteration loop, and it goes stale every time the base model improves, which lately is often. The distillation pattern above is the exception that earns it, and even there we distill into small open-weight models we host, so the artifact is ours.

The drift tax

The distilled model is frozen; the world is not. So the router double-writes a 1% shadow sample to the frontier model continuously, and disagreement above a threshold pages us to re-label and re-distill. Small models are cheap to run and easy to forget. The shadow sample is what keeps "we automated this last spring" from decaying into "we have been silently misclassifying since spring".

Our AI coworkers run on files

INTFRAME — Sat, 15 Aug 2026 10:38:19 +0000

The most consequential architecture decision we made for our agents is also the least glamorous: agent memory is plain files in a git-friendly tree. One note per fact, frontmatter for machines, prose for the model:

---
name: payment-retry-policy
description: "retry queue must never re-attempt a captured payment"
type: project
---
The retry worker treats 'captured' as terminal. The 2025-11 double-charge
incident came from retrying on a timeout after capture succeeded.
Related: [[refund-flow]], [[idempotency-keys]]

A session starts by reading a 40-line index, one line per note, not the 400 KB corpus. The model picks the two or three notes whose description matches the task and opens exactly those. Retrieval is a decision made by something that can reason, against descriptions a human curated, and it is fully explainable after the fact: here is the index line, here is the file, here is the sentence.

Why not embeddings

We prototyped the same corpus behind a vector store and killed it in two weeks. When an agent behaved oddly there was no file to open, only similarity scores to second-guess. Corrections meant re-embedding instead of editing a paragraph. And nothing about a few hundred curated notes needs approximate nearest neighbors; the whole index fits in one screen of context. Embeddings solve scale and fuzziness. Curated memory has neither problem.

Caps matter more than schemas

The rules file fits on one screen, and adding a line requires deleting a line.
The state file has a hard line cap; a cron sweeps anything stale into an archive that recall can still search.
Every note links related notes with [[name]] references, so the graph is walkable with grep alone.

Memory that grows without bound stops being read, and memory that is not read is decoration. The caps are what keep the system honest.

The boundary that makes it safe

Agents propose, gates dispose. An agent can draft anything into its tree, but irreversible actions, deploys, deletions, payments, outbound messages, pass through the same mechanical gates as our cron jobs: whitelists, caps, ledgers, human approval where it counts. We do not ask the model to remember to be careful. Care is compiled into the pipeline around it.

The night shift is a fleet of models

INTFRAME — Sat, 15 Aug 2026 10:36:48 +0000

At some point the number of scheduled jobs, monitors and pipelines crossed the line where a human could still be the router. Our answer was not one big autonomous agent. It was a fleet of small ones with a deterministic script holding the leash.

Code owns control flow, models own content

The orchestrator is ordinary code: it decides what runs, in what order, with what concurrency, and what happens on failure. Models fill in the steps. A planner turns an objective into a task list. Workers execute tasks inside tool sandboxes with step budgets. A verifier, prompted to refute rather than to help, attacks the workers' output. A reporter compresses the surviving results for humans. Every role writes a JSONL transcript, so any night's work can be replayed step by step the next morning.

The one architectural rule: models never talk to each other directly. They talk through the orchestrator, which means every hop is logged, budgeted and interruptible.

Outbound is the dangerous direction

The risky side of ops automation is not reading, it is writing. A bot that can message a human at 4 AM will, one day, message them forty times at 4 AM. Every outbound path in our fleet, human or API, goes through one gate:

function send(msg, to) {
  if (!WHITELIST.has(to))                return refuse('recipient');
  if (ledger.countToday(to) >= DAILY_CAP) return refuse('cap');
  ledger.append({ to, digest(msg), at: now() });   // written BEFORE sending
  return transport.send(msg, to);
}

A whitelist of recipients, a daily cap enforced in code, a ledger written before the send, and one gate shared by every sender. Ten scripts sharing one gate beat ten scripts with ten opinions about etiquette.

Silence detection over failure alerts

# the watchdog does not listen for errors. it demands proof of life.
find /opt/health -name '*.ok' -mmin +90 -print    # stale heartbeat = incident

Cron jobs fail loudly only in fairy tales. In production they fail by not running: a typo in a schedule, a dead virtualenv, a server clock in a timezone the author did not expect. Our watchdog expects heartbeats and treats absence as the alarm. Since that inversion, the class of incident where something died silently for a week has simply disappeared.

The rule of thumb we settled on: automation earns autonomy in proportion to how mechanically its blast radius is capped. Write the gate first, then let the fleet loose.

Grep first, embed when it earns it

INTFRAME — Sat, 15 Aug 2026 10:30:07 +0000

Retrieval shows up twice in our stack and the two cases got opposite architectures. Agent memory, a few hundred curated notes, runs on files and grep. User-facing search, tens of thousands of multilingual records, runs on hybrid embeddings. The dividing line is worth spelling out, because the industry default is to buy a vector database for both.

Where grep wins

Agent memory is small, curated, and read by something that can already reason. Each note carries a one-line description in an index file; the agent reads the 40-line index, decides what is relevant, and opens two or three files. Debugging is opening the exact file the agent read. Correcting is editing a paragraph. When we prototyped the same corpus behind embeddings, retrieval became a probability distribution we had to interrogate, updates required re-embedding, and nobody could answer the only question that matters in an incident: why did it read that?

Where embeddings earn it

Search across 40,000 records in mixed Korean, English and Vietnamese is a different animal. Users type "cheap spa near the river" and the record says "riverside massage, budget". Keywords miss; vectors catch. But pure dense retrieval lost to a hybrid on our own eval set, so hybrid is what runs:

const kw = bm25(query, 40);              // exact terms, names, numbers
const dn = cosine(embed(query), 40);     // paraphrase, cross-lingual
const pool = dedupe([...kw, ...dn]);
const ranked = await llmRerank(query, pool.slice(0, 30));  // one cheap model call
return ranked.slice(0, 8);               // what actually enters the context

On a 400-query labeled set built from real search logs, recall@20 was 0.71 for keyword alone, 0.76 for dense alone, 0.90 for the hybrid with rerank. The rerank call costs a fraction of a cent and fixed the last mile better than any embedding model swap we tried.

Chunking is where quality actually lives

Fixed 512-token windows sever tables from their headers and answers from their questions. We chunk by document structure, headings and list boundaries, and every chunk carries its source path and byte offsets, so anything the model quotes is one click from the sentence it came from. A retrieval system that cannot cite its sources is a rumor generator with good latency.

Write prompts for the cache, not the reader

INTFRAME — Sat, 15 Aug 2026 10:28:36 +0000

Modern LLM APIs cache the key-value state of a prompt prefix. Send the same opening tokens twice and the second call is billed at roughly a tenth of the input price and returns noticeably faster. This one fact reorganized how we write every high-volume prompt.

The rule

Everything stable goes first, byte-identical on every call: system rules, output schema, few-shot examples, reference tables. Everything volatile goes last: the task, the retrieved documents, the user text. The cache breaks at the first differing byte, so a single timestamp in the header poisons the whole prefix.

[ system rules          ]  stable, versioned      ─┐
[ output schema         ]  stable, versioned       ├─ cached prefix, ~91% of tokens
[ 12 few-shot examples  ]  stable, versioned      ─┘
[ retrieved context     ]  volatile ──────────────── paid in full
[ the actual task       ]  volatile

Measured on one of our extraction pipelines

prompt layout	prefix reuse	cost per 1k calls	p50 latency
task first, rules inline, timestamp in header	0%	$31.40	4.1 s
stable prefix, task last	91%	$6.90	1.8 s

Same model, same accuracy, 78% cheaper and twice as fast, for a diff that only moved paragraphs.

Operational consequences

The prefix is versioned like an API. Editing it is a deliberate release, because every edit is a fleet-wide cache flush and a visible cost spike. We tag prompts with a version constant and roll them like schema migrations.
Batch queues are sorted by prefix. Requests sharing a prefix run adjacently so the cache stays warm instead of being evicted between hits.
Hit rate is a logged metric. The API reports cached token counts per call; we graph them. A silent drop in cache hit rate has caught two accidental prompt edits before the invoice did.
Randomness is banned from the prefix. No timestamps, no request IDs, no dict-ordering roulette. Serialization is canonical, keys sorted, for the same reason your build system wants deterministic inputs.

None of this touches quality. It is pure systems hygiene applied to a resource most teams do not know they are wasting.

An agent is a while loop with taste

INTFRAME — Sat, 15 Aug 2026 10:24:09 +0000

A meaningful share of our daily work is executed by AI agents: research, code changes, QA passes, report writing. People assume this requires an orchestration framework. It requires a loop:

let messages = [system, task];
for (let step = 0; step < MAX_STEPS; step++) {
  const res = await llm(messages, TOOLS);
  if (res.type === 'text') return res.text;        // the model decided it is done
  const tool = TOOLS[res.name];                     // no dynamic anything
  const result = await tool.run(res.args);          // sandboxed, permission-gated
  messages.push(res, clip(result, TOOL_BUDGET));    // clip. always.
}
return escalate(messages);                          // never loop forever

The model decides what to do next. The code owns whether and how long. That division is the entire design.

What the loop taught us

Clip every tool result. Ours are capped around 2,000 tokens; the full output goes to disk and the model gets the head, the tail and the path. Before the cap, a single verbose build log could eat an entire context window and lobotomize the rest of the session.
Few orthogonal tools beat many clever ones. Our agents carry six to nine tools with strict schemas. Every time we added a convenience tool that overlapped an existing one, tool-choice accuracy dropped. The model dithers exactly where a human would.
MAX_STEPS is a safety property, not a tuning knob. When the budget runs out, the transcript escalates to a human with state intact. An agent that can loop forever eventually will, at 4 AM, against a rate-limited API.
Transcripts are JSONL, and JSONL is replayable. Every step is appended to a log we can re-run against a new model or a fixed tool. Half of our debugging is replaying yesterday's transcript against today's code.
Reads are free, writes go through gates. The loop can read anything in its sandbox. Deploys, deletions, payments and outbound messages leave the loop and enter the same mechanical gates our cron jobs use. We do not ask the model to remember to be careful.

Four model swaps later, the loop is byte-for-byte the same. Everything around it improved. That is what a good abstraction boundary feels like: the thing that changes fast is quarantined from the thing that must not.

Structured output is a contract, not a request

INTFRAME — Sat, 15 Aug 2026 10:18:59 +0000

The first thing we standardized when model calls entered our pipelines was the boundary. Every call whose output feeds a machine returns JSON against a schema, and the schema is enforced by an ordinary validator. Not by asking nicely in the prompt. By rejecting the output and making the model try again with the validator's error in its face.

The loop

async function extract(input, schema, tries = 3) {
  let feedback = '';
  for (let i = 0; i < tries; i++) {
    const raw = await llm({
      system: RULES + feedback,
      user: input,
      response_format: schema,   // constrained decoding where the API supports it
      temperature: 0,
    });
    const errors = validate(schema, raw);   // plain JSON Schema, same lib as our forms
    if (errors.length === 0) return raw;
    feedback = ' Previous output failed validation: ' + errors.join('; ');
  }
  return quarantine(input);   // a review queue. never a crash, never a guess.
}

Three details carry most of the weight:

Enums beat free text. The single biggest reliability jump came from replacing string fields with closed enums. One extraction pipeline went from a 4.1% rejection rate to 0.2% the week we enum-ified its category fields. A model that must choose from seven values cannot invent an eighth.
Feed the validator error back verbatim. A bare "try again" retries the same mistake. The exact message, "items[2].price must be integer", fixes it on the next attempt almost every time.
Temperature 0 is necessary, not sufficient. Greedy decoding removes randomness, not wrongness. The validator is still the one telling the truth.

Quarantine is a feature

The item that fails three attempts is the interesting one. It goes into a review queue, a human labels it, and the labeled case joins the regression suite. Our schemas have been hardened by two years of their own rejects. Which points at the real lesson: the schema is half of the prompt. Most of our prompt engineering time is spent deleting fields, tightening types and closing enums, because every degree of freedom you remove from the output is a hallucination that can no longer happen.

One box is enough

INTFRAME — Sat, 15 Aug 2026 10:18:57 +0000

Everyone we meet assumes we run a fleet of cloud instances. We run one dedicated box in a Falkenstein data center, and it hosts every production site, database and bot we operate.

This is not nostalgia. It is a forcing function. When everything lives on one machine, you cannot hide sloppy engineering behind autoscaling. A memory leak is a neighbor problem within the hour. A runaway cron shows up in everyone's latency. So the discipline has to live in three places, and all three are enforced by config, not by promises.

1. Every service gets a ceiling

# systemd override, applied to every unit, no exceptions
[Service]
MemoryMax=512M
CPUQuota=80%
Restart=on-failure
RestartSec=5

The ceiling is not there to be hit. It is there so that when something does go wrong, the kernel kills one service instead of the OOM killer choosing a victim at random among forty.

2. Every scheduled job gets a lock, a timeout, and a heartbeat

*/5 * * * * flock -n /run/lock/sync.lock timeout 240 /opt/jobs/sync.sh \
            && date > /opt/health/sync.ok

The lock stops a slow run from stacking behind itself. The timeout stops a hung run from holding the lock forever. The heartbeat file is the interesting one: a separate watchdog checks the mtime of every .ok file and treats staleness itself as the alarm. Jobs do not fail loudly in production. They fail by not running.

3. Routing is one file you can read

example.com {
  encode zstd gzip
  root * /srv/sites/example
  file_server
  handle /api/* { reverse_proxy 127.0.0.1:3160 }
}

Caddy terminates TLS for every domain and routes by hostname. Behind it, apps are containers or static directories. Databases are plain Postgres and MySQL, one instance per engine, many schemas. Nothing exotic, everything inspectable with standard tools.

What one box buys you: latency you can reason about (every internal hop is a loopback), a single filesystem of truth (debugging is grep, not a distributed tracing seminar), and honest capacity planning (htop is the dashboard). The catch is that the box becomes precious, and precious things need paranoia. That paranoia, backups we actually restore, silence detection, blast-radius caps, is most of what the rest of these notes are about.

Design QA at 1:1

INTFRAME — Fri, 14 Aug 2026 18:15:44 +0000

Two screens can look identical at a glance and be twenty percent apart in every measurement that matters. Font size, line height, spacing rhythm, contrast. The glance is the least reliable instrument in the building, and for months it was the only one we used.

Measure the original, not your memory of it

When the brief is "make it feel like this reference", the first step is no longer a mood board. It is a probe run in both pages' consoles:

const probe = (sel) => {
  const cs = getComputedStyle(document.querySelector(sel));
  return [sel, cs.fontSize, cs.lineHeight, cs.letterSpacing,
          cs.fontWeight, cs.color].join('  ');
};
['h1', 'h2', 'p', 'nav a', 'button'].map(probe).join('\n');

Run it on the reference, run it on ours, diff the two tables. Design conversations change completely when both sides are looking at numbers. "It feels lighter" becomes "their h1 is 88/0.94 at weight 900, ours is 72/1.1 at 800", which is a ticket, not a debate.

Compare crops, not impressions

Crop the same region from ours and theirs, stack them at 1:1, and look at the pair. Differences that vanish in side-by-side thumbnails become obvious.
Do it on mobile first. Desktop hides crimes that a 390-pixel viewport confesses immediately.
Automate the mechanical half: a script walks the page and flags text overflow, unexpected horizontal scroll and elements taller than their containers, before a human spends taste on it.
Record scroll behavior too. Easing curves and pin durations are design decisions, and they are measurable in the performance timeline, not just feelable.

Taste still decides

None of this replaces judgment. It feeds judgment with facts. The measurements tell you where you diverge from the reference. Whether the divergence is a bug or a choice is still a human call, and making that call consciously is the entire difference between a site that is almost right and one that is right.

Originally published on the INTFRAME engineering blog. The companion code lives at github.com/intframe/scroll-qa.