DEV Community: Omer Hochman

Your most active user is your test suite.

Omer Hochman — Sat, 05 Sep 2026 02:55:06 +0000

Originally published at nlqdb.com/blog

Before launch, your end-to-end suite is often the only thing exercising production. Ours registers fixture users, creates databases, and asks real natural-language queries through the exact pipeline a stranger would hit — which is the point of an e2e suite, and also the problem: every dashboard we built quietly measured our robots. Here are the three places it bit us, and the fix for each.

1. Your web analytics count headless browsers as visits

Our weekly funnel said 120 visits. Real-browser visits were about 41. The rest were our own walkers: headless Chromium lands in RUM analytics with userAgentBrowser: "Unknown" — 76 of the 120 — plus one that identified as headless outright. The fix is a pinned client-class cut: count named browsers (Chrome, Mobile Safari, Edge…) as the filtered number, and report both numbers side by side. Raw tells you the instrument is alive; filtered tells you whether humans showed up. Reporting only one of them is how you lie to yourself in either direction.

2. Your product KPIs saturate on fixture accounts

Our headline onboarding KPI — the share of a new account's first ten queries that succeed — dropped 8 points in one night. No deploy, no incident. A failing e2e run had burned its asks against the same saturating counters real users increment; the suite's fixture account looked like our worst-onboarded user, because it was.

The tempting fix is to skip the counters when the caller is a test account — and it's wrong. The write path is the thing under test; fork it for fixtures and your suite stops exercising the pipeline you ship. Filter at read time instead: keep an explicit founder/test identity list and join it against every user-scoped metric when you pull the number. Writes stay honest, reads stay clean, and the filter is one place to audit instead of a flag threaded through every emit site.

3. The hardest one: accepting “not yet measurable”

Once the joins were in place, our first-ten-queries KPI read: stranger sample size zero, not yet measurable. That's a worse-feeling number than the unfiltered 5/22 it replaced — and a strictly better one. A KPI computed over founder and fixture traffic isn't a pessimistic estimate of the real number; it's a number about a different population wearing the KPI's name. Shipping it to a scorecard anchors decisions to noise. An honest “N=0” at least tells you what the actual bottleneck is: distribution, not product.

The rule of thumb

A metric that doesn't name its population is measuring your robots. Every user-scoped number on a pre-launch dashboard should say which of three populations it covers — everyone, humans-only, or strangers-only — and the cut that produces it should be pinned in one greppable place (a UA class list, an identity list) rather than re-derived per query. Synthetic traffic through the production pipeline is a feature; synthetic traffic in your KPIs is a bug, and the boundary between them is attribution at read time.

Your five fallback models are one point of failure.

Omer Hochman — Fri, 04 Sep 2026 01:33:48 +0000

Originally published at nlqdb.com/blog

Our agentic CI suite drives a real browser with an LLM. The driver model comes from an ordered five-model fallback list — health-probed before each run with real tool-call probes, re-ranked by measured agent competence. Textbook redundancy. It still failed 13 dispatches in a row.

Model-diverse, provider-identical

The list was diverse in the dimension we could see — five different model families, five different weights — and identical in the dimension that failed: every slug resolved to the same gateway's free pool. That pool saturates as a unit. When model #1 came back 429, models #2 through #5 were rate-limited too, at 04:37 UTC and at 22:14 alike, because the limit lives on the shared pool, not on any model. Five models on one provider are five servers in one rack.

The probes made it worse, not better. A three-probe health gate picked whichever model answered its probes that minute — and then the pick starved 216 seconds into the run, because passing three probes measures a moment of pool weather, not a claim on future capacity. A flapping pool passes probes all day.

The tell in the traces

What finally broke the diagnosis open was a latency split in the Playwright traces: the product under test answered its API call in 4 seconds with a 200 — while three tests burned their entire 240-second budget waiting for the driver to produce the next click. The app was green; the thing testing the app was starving; the suite reported "product red." If your E2E harness and your product share a failure domain, a harness outage is indistinguishable from a product regression until you split them.

Redundancy has to cross the failure-domain boundary

The fix is a lane, not a longer list. A lane is the real unit of failure: base URL + API key + candidate models. Fall back between lanes only when the whole primary lane fails its probes — and make the first fallback candidate the same weights hosted by a different provider, which keeps agent competence constant while moving to an independent pool:

// The fallback unit is the lane, not the model. Walking candidates
// inside one lane retries the same saturated pool with a new name.
type Lane = { baseUrl: string; apiKey: string; candidates: string[] };

const LANES: Lane[] = [
  // Primary: an independent pool. Same weights as the old list’s best
  // performer, hosted elsewhere — competence constant, failure domain new.
  { baseUrl: NIM_URL, apiKey: NIM_KEY, candidates: ["openai/gpt-oss-120b"] },
  // Fallback: the gateway’s free pool — five candidates, ONE domain.
  { baseUrl: GW_URL, apiKey: GW_KEY, candidates: FREE_SLUGS },
];

for (const lane of LANES) {
  const model = await probeLane(lane); // 3 real tool-call probes
  if (model) return { ...lane, model };
  // Whole lane failed — only now is trying elsewhere informative.
}

One more boundary matters: budgets. The fallback lane must not raid the quota the application under test runs on — if the rescue driver and the product share a pool, the rescue causes the next outage. Keep the driver's spend and the product's spend on separate keys, and treat "the driver ran out" as a harness failure, never a product failure.

On the first dispatch with the swapped lanes, the test that had starved at 216 seconds passed in 14.9 — same suite, same app, same weights. Nothing about the product changed; only the failure domain did. (This is a build note from nlqdb, the database you query in plain English. Honest split: a CI/infra lesson from our E2E harness, not a product feature.)

An "open question" that's already decided is worse than one that's still open.

Omer Hochman — Thu, 03 Sep 2026 01:28:05 +0000

Originally published at nlqdb.com/blog

Every long-lived codebase grows a decision log — ADRs, a DECISIONS.md, per-feature records — and every one of those grows an Open questions section. The failure mode isn't the open questions. It's the entries that were quietly answered — in a PR, a standup, a founder's one-line reply — and never re-labelled. A bullet that reads "we should probably cap the queue at 7,000 ops/day" is a decision wearing an open question's clothes.

What a decided-but-unmarked bullet does to a reader

A reader who hits that bullet does one of two bad things. Either they treat the settled call as unsettled and re-litigate it — burning a design discussion on a question someone already closed — or they build on top of a "maybe" that was actually a "yes," and their design inherits a hedge that no longer exists. Both cost more than an honestly open question would, because the log looks authoritative while pointing nowhere. A vague decision documented is worse than none.

Two moves fix it

First, make resolved a first-class, greppable state. A decided bullet keeps its line but gains a marker word — Resolved, Decided:, Parked, or a strikethrough — plus a pointer to the decision's canonical home. The questions list is where a question's lifecycle is visible; it is never where the decision's body lives. When you answer an open question, the same commit moves the body to the canonical record and leaves only the marked pointer behind — never a second copy that can drift.

Second, count the unmarked ones as debt. If resolved is greppable, so is unresolved — and a number you can compute is a number you can drive to zero:

# Ambiguity debt: open-question bullets with no resolution marker.
# -h suppresses the filename prefix grep -A puts on context lines —
# without it the bullet match never fires and the count is always 0.
# Case-INSENSITIVE on the marker set — a case-sensitive grep counts
# "resolved" and "Resolved" differently and the number drifts.
grep -hA9999 '^## Open questions' docs/features/*/FEATURE.md \
  | grep '^\s*- ' \
  | grep -vciE 'resolved|decided:|parked|deferred|~~'

The case-insensitivity is not a nitpick — it's the difference between a metric and a mood. We track this count as a scorecard row, and the first version drifted for exactly that reason: two agents marked resolutions with different capitalization, the grep saw only one, and the "debt" number moved without any question changing state. Pin the counting method next to the number it produces.

Why a count, and not a cleanup day

A one-off sweep fixes today's rot and leaves the seam that produced it. The count makes the seam visible continuously: every answered-but-unmarked bullet is +1 debt that someone will notice, and driving one bullet to a marked resolution is a small, complete, verifiable unit of work — research the answer, write it in the canonical home, mark the pointer. A question only a human with prod access or a checkbook can answer moves to a separate blocked-on-human list and off the count, so the number only tracks debt an engineer can actually retire.

(This is a build note from nlqdb, the database you query in plain English. Our decision log is agent-operated, which makes the rot mechanical instead of cultural — but the fix is the same one a human team needs. Honest split: a decision-hygiene lesson, not a product feature.)

Your metric is only as honest as the layer you emit it from.

Omer Hochman — Wed, 02 Sep 2026 01:19:21 +0000

Originally published at nlqdb.com/blog

We wanted one number: the destructive-op retry rate — the share of write previews a user abandons instead of confirming. Our pipeline renders every destructive statement as a diff first; the user re-sends with a confirm flag to commit. So the rate is 1 - committed / preview_rendered, and it takes exactly two counter events to compute. The only real design decision is where those two events get emitted — and the obvious answer is wrong.

The route doesn't know what it's confirming

The obvious emit site is the HTTP route. It already has everything an event wants: the principal, the surface, the request, the response status. Emit preview_rendered when the response asks for confirmation, committed when a request carrying confirm: true returns 200. Done — except the route does not actually know whether the statement was a write. That fact is decided two layers down, where the generated SQL is inspected and the preview-vs-commit branch is taken. The route sees a flag and a status code.

Here is the failure that makes it concrete: a client sends a read with a stray confirm: true — a retried request, an over-eager SDK default, a copy-pasted call. The read succeeds, the route emits committed, and there is no matching preview_rendered because reads never preview. The numerator now exceeds the denominator, and your retry rate goes negative — a number that cannot exist. Nothing errored. Every layer behaved correctly. The metric was simply emitted above the layer where its defining distinction is known.

// Route layer — the tempting emit site. It has the principal, the
// surface, the status. It does NOT know whether the plan was a write:
// emit "committed" on any confirm:true 200 and a stray-confirm read
// inflates the numerator. The rate goes negative.

// Orchestrator — both facts are certain on exactly this boundary.
if (!req.confirm && isWriteVerb(planSql)) {
  // A write plan rendered as a diff, nothing executed: the denominator.
  emit({ name: "feature.destructive.preview_rendered", surface: req.surface });
  return { requires_confirm: true, diff };
}
// ...execution happens only past this point...
if (req.confirm && isWriteVerb(planSql)) {
  // Exec success on an approved diff: the numerator.
  emit({ name: "feature.destructive.committed", surface: req.surface });
}

Thread facts down, don't pull decisions up

The fix is not teaching the route to re-detect writes — that duplicates the write-detection logic in a second place, and the two copies will drift. The fix is moving the emission down to the layer where isWriteVerb(sql) and the preview-vs-commit branch are already decided. There, both events fire on exactly the boundary they measure, and a stray-confirm read emits nothing at all.

Moving down costs you one thing: the lower layer didn't know the surface the request came from — web app, CLI, MCP — and the events are only useful sliced by surface. So thread that one field down through the request. Passing one known fact down is cheap and drift-free; hoisting a whole decision up is neither. And when a caller doesn't thread the field — an internal path with no surface — skip the emit rather than fabricating a value. A gap in the data is honest; a guessed label is not.

The rule

A metric's emit point is the lowest layer where the distinction it encodes is certain. Above that layer you are guessing, and a guessed metric is worse than no metric, because it looks precise while being structurally wrong — negative rates, inflated numerators, denominators that undercount. The same reasoning fixes the dedup key: a volume event like this one keys on the request, not on a per-principal-per-day bucket, because collapsing repeated previews from one user would erase exactly the retries the metric exists to count. Decide what distinction the number encodes, find the lowest layer where that distinction is a fact, and emit there.

(This is a build note from nlqdb, the database you query in plain English — where destructive statements render as a diff you approve before anything runs. Honest split: this is an instrumentation-design lesson, not a product feature.)

You need to rotate an encryption key. You don't need a key-version column.

Omer Hochman — Tue, 01 Sep 2026 01:19:56 +0000

Originally published at nlqdb.com/blog

We seal secrets at rest — a customer's database DSN, a stored API key — under a key-encryption key (KEK). Sooner or later you have to rotate that KEK: it's been in an env var too long, it might have leaked, or a policy just says every key expires. And the moment you plan the rotation, a reflex kicks in: add a key_version column so you know which key sealed which row. Skip it. That column is a second source of truth for something the ciphertext already has to tell you, and the two will drift.

The ciphertext is already self-describing

You cannot decrypt AES-GCM without the IV, so the IV was never a secret and it already lives inside the stored blob — every sealed value is a self-describing string, not opaque bytes. A key version is exactly the same kind of fact: metadata you need in hand before you can decrypt. So it belongs in the same place the IV does — the blob — not in a sibling column that every write path has to keep in sync. One token of prefix carries it:

// The sealed blob already carries its IV. Give it a version tag too.
//   nbe2.<v>.<iv>.<ciphertext>
// Bumping the KEK is a one-token change: v1 -> v2. No schema migration,
// no key_version column, no ALTER TABLE, no backfill job on write.
function seal(plaintext: string, kek: Kek): string {
  const iv = randomIv();
  const ct = aesGcmSeal(plaintext, kek.material, iv);
  return `nbe2.${kek.version}.${b64(iv)}.${b64(ct)}`; // version IS the tag
}

function open(blob: string, keyring: Keyring): string {
  const [, version, iv, ct] = blob.split(".");
  const kek = keyring.byVersion(version); // pick the key the blob names
  return aesGcmOpen(fromB64(ct), kek.material, fromB64(iv));
}

What the prefix buys you that a column doesn't

Rotation becomes a sequence of cheap, interruptible steps instead of a migration. You add the new key to the keyring alongside the old one — a two-key overlap window. Reads pick the right key by the tag the blob carries, so nothing breaks the instant you flip. Writes always seal under the newest version, so every row that gets touched migrates itself. Then you do one sweep for the cold rows that nobody wrote, and drop the old key when no blob still carries its version.

The part people reach for the column to solve — "find the rows still on the old key" — the prefix already solves, and solves better: stale rows are filterable by their tag without decrypting a single one. A WHERE blob LIKE 'nbe1.%' finds every row that needs re-wrapping straight off the index. The key_version column only earns its keep if that sweep has to find stale rows blind — and it never does, because the tag is right there in the value.

The one nuance: derived keys vs. stored DEKs

If each row stores its own data-encryption key (DEK) wrapped by the KEK, "re-wrap" is cheap and never touches plaintext: decrypt the little DEK with the old KEK, re-encrypt it with the new one, leave the bulk ciphertext untouched. If instead your content key is HKDF-derived straight from the KEK — no stored DEK — then re-wrap means decrypt-then-reseal the secret itself. That sounds heavier, but for the short secrets this pattern guards (a DSN, an API key — tens of bytes, not gigabytes) it's trivial. Either way the version tag is what tells the sweep which branch a given row needs.

The rule

The encrypted blob is the source of truth for how to decrypt it — key version included. Anything you also store in a column is a second copy that can drift from the first, and the day it drifts is the day you can't decrypt a row you thought you could. Put the version where the IV already lives, rotate by bumping a prefix, and let the tag drive both the read path and the sweep. Reach for a column only when you have a query that genuinely can't be answered from the value itself — and key rotation isn't one of them.

(This is a build note from nlqdb, the database you query in plain English — where connection secrets are sealed at rest and the SQL is generated, validated, and shown to you before it runs. Honest split: this is a security-architecture lesson, not a product feature.)

You added a second SQL engine. Your text-to-SQL model is still being told it's the first one.

Omer Hochman — Mon, 31 Aug 2026 01:26:31 +0000

Originally published at nlqdb.com/blog

A text-to-SQL model is dialect-aware by design. You hand it a target dialect, and it obliges — name Postgres, get Postgres. So when we added a second engine, ClickHouse alongside Postgres, the model kept doing exactly what it does well: writing SQL for the dialect it was told. The trouble is what it was told. The happy path compiled, the Postgres queries still worked, and a class of analytical questions on the ClickHouse databases started coming back subtly wrong. The model wasn't confused. It was confidently writing Postgres for a database that speaks ClickHouse, because one line told it to.

The planner was never the problem

The prompt already carries a Dialect: line — the system prompt says "emit SQL valid for the named dialect," the few-shot exemplars are tagged with theirs. The model reads that field and complies. So the bug isn't in the model or the prompt. It's in the one place that fills the field, and it looks like the most innocent line in the file:

// The request type — grew for the first two engines, then stopped.
type PlanRequest = {
  goal: string;
  dialect: "postgres" | "sqlite"; // never grew a "clickhouse" member
};

// The call site — a literal that was correct exactly once.
const plan = await llm.plan({
  goal: req.goal,
  dialect: "postgres", // hardcoded; db.engine is right there, unused
});

That dialect: "postgres" was true on the day it was written, when Postgres was the only engine. It is a fact frozen into a literal. The database row already knows its real engine — db.engine is sitting one field away — but nothing carries that value the few inches into the request. The model is downstream of a lie it has no way to detect.

Why it hides

Nothing logs "wrong dialect," because from every layer's point of view nothing went wrong. The type-checker is happy — "postgres" is a valid member of the union. The planner is happy — it got a dialect and emitted valid SQL for it. Every Postgres database on the platform keeps working, so the whole happy path stays green. The failure only surfaces on the analytical grammar that is the entire reason you added the second engine — LIMIT n BY, quantile(0.5)(x), ARRAY JOIN, WITH ROLLUP — none of which the model will ever reach for while it believes it's writing Postgres. The feature that justified the new engine is precisely the feature that silently degrades.

The fix is a value, not a transpile layer

The reflex is to reach for a translation layer — write Postgres, run it through SQLGlot or an ANTLR grammar, transpile to ClickHouse. Don't. Those pull a parser-and-grammar bundle that busts an edge/Workers deploy, and they solve a problem you don't have: the model can already write ClickHouse. It just has to be asked. The fix is to make the dialect flow from the row instead of a literal, and — this is the load-bearing half — to make the type refuse to compile until every call site does so.

// 1. Widen the type. This is what turns the bug into a compile error:
//    every hardcoded `dialect: "postgres"` call site now fails to build
//    until it proves it handles the new member.
type Dialect = "postgres" | "sqlite" | "clickhouse";

// 2. Map the engine the row already carries to its dialect — one place.
const dialectFor = (engine: DbEngine): Dialect =>
  engine === "clickhouse" ? "clickhouse" : "postgres";

// 3. Thread the value, not a literal.
const plan = await llm.plan({
  goal: req.goal,
  dialect: dialectFor(db.engine),
});

Widening the type first is the trick that makes this safe. Add "clickhouse" to the union and the compiler walks you to every call site that still hardcodes the first engine — the two plan sites, the retry-repair site, anywhere a literal slipped in. A runtime bug you'd have to catch with a live ClickHouse database and a discerning eye becomes a build error you can't merge past. You are not trusting yourself to remember every place; you are making the type remember for you.

The generator and the validator are twins

There's a matching bug one layer down, and it's worth fixing in the same breath. The SQL validator makes the same first-engine assumption: a Postgres-pinned parser false-rejects valid ClickHouse grammar as a parse failure. Fix the generator alone and you've just relocated the damage — now the planner correctly emits ClickHouse SQL and the validator vetoes it as invalid. The generator and the validator both silently assume engine #1, and they have to stop assuming it together, or you trade a wrong-answer bug for a rejected-query bug.

The rule

When you add a second engine, the dialect stops being config you set once and becomes a value that must flow from the row on every request. Grep for every place the first engine's name appears as a literal — that list is your bug list. Then widen the dialect type before you touch anything else, so the compiler converts the ones you'd have missed from silent wrong answers into loud build failures. A frozen literal is a fact that was true once; the type is what keeps it from staying true after it isn't.

(This is a build note from nlqdb, the database you query in plain English — where the SQL is generated, validated, and shown to you before it runs. Honest split: this is an architecture lesson about multi-engine prompting, not a product feature.)

You added ClickHouse. Your Postgres SQL validator now rejects valid queries — quietly.

Omer Hochman — Sun, 30 Aug 2026 01:15:57 +0000

Originally published at nlqdb.com/blog

We generate SQL from plain English, and before any of it touches a database we validate it. So when we added a second engine — ClickHouse alongside Postgres — the tempting move was obvious: reuse the validator we already trust. It ran clean, the tests were green, and a class of real user questions started coming back parse_failed for no reason anyone could see. The validator wasn't broken. It was doing exactly what it was told, on the wrong engine.

One parser, pinned to one dialect

Our validator is an AST parse — node-sql-parser, configured database: "PostgreSQL". Point that at a ClickHouse database and valid analytical grammar the query planner correctly emits — LIMIT n BY, quantile(0.5)(x), ARRAY JOIN, WITH ROLLUP — fails the parse. The user asked a perfectly good question, the model wrote perfectly good ClickHouse, and the guardrail in the middle rejected it. Nothing in the logs said "wrong dialect." It just said the query was invalid, which it wasn't.

The seductive part is that it never looks like a dialect bug. Every Postgres query still validates. The happy path is green. The failure only shows up on the analytical grammar that is the entire reason you added ClickHouse in the first place — so the feature that motivated the second engine is precisely the feature the validator quietly vetoes.

The fix is not a second parser

The reflex is to go find a ClickHouse parser. Don't. node-sql-parser has no ClickHouse dialect, and the ANTLR4 grammars that do are far too heavy for an edge/Workers bundle. Chasing a per-engine parser is chasing the wrong problem — because the validator was never really one job.

Look at what it actually does and it splits cleanly in two. First, it enforces a destructive-verb allowlist — no DROP, no DELETE, no TRUNCATE reaching a read path. Second, it walks the AST to catch a dangerous verb smuggled inside a CTE or subquery. Only the second job needs a parser. The first is a leading-verb check — dialect-agnostic, already correct on every engine, because DROP is DROP in ClickHouse and Postgres alike.

// Security-load-bearing: dialect-agnostic, authoritative on EVERY engine.
// A leading destructive verb is a hard reject regardless of parser support.
const DESTRUCTIVE = /^\s*(drop|delete|truncate|alter|update|insert|grant|revoke)\b/i;
if (DESTRUCTIVE.test(sql)) return reject("destructive_verb");

// Defense-in-depth: AST walk for a verb smuggled into a CTE/subquery.
// Best-effort PER ENGINE — a parse failure here means "wrong parser for
// this dialect", NOT "dangerous query". It must never veto a valid query.
try {
  const ast = parse(sql, { database: dialectFor(engine) });
  if (hasNestedDestructive(ast)) return reject("destructive_verb");
} catch {
  // ClickHouse grammar the PG parser can't read lands here. That is not
  // a security signal. The allowlist above already made the safety call.
}
return allow();

Why the layering matters

Once the two jobs are separated, the security-load-bearing layer works on every engine and the dialect-locked layer degrades gracefully. A parse failure from the wrong dialect stops being a verdict on whether the query is safe and becomes what it always was: a statement about which parser you happened to load. The allowlist already made the safety call, up front, on grammar that never varies. So you lose nothing on security and you stop rejecting correct SQL.

This is the same instinct behind layered guardrails everywhere: no single rule is trusted to be both complete and correct on every input. The widely-reported agent-that-dropped-a-production-database incident had multiple safeguards and still lost data — the lesson isn't "add one perfect check," it's that the authoritative safety layer has to hold even when a best-effort layer above it can't run.

The rule

In a multi-engine guardrail, a parse failure from the wrong dialect means "wrong parser," not "dangerous query" — never let it decide whether a valid query runs. Keep the dialect-agnostic allowlist authoritative on every engine; make the AST parse best-effort per engine. When you add a second dialect, the bug won't announce itself as a dialect bug — it'll look like a validator that's merely stricter than you remembered. Check what your guardrail is actually pinned to before you trust its rejections.

(This is a build note from nlqdb, the database you query in plain English — where the SQL is generated, validated, and shown to you before it runs. Honest split: this is an engineering lesson about multi-engine guardrails, not a product feature.)

We read the agent-memory benchmarks. Almost none measure analysis.

Omer Hochman — Sat, 29 Aug 2026 01:20:29 +0000

Originally published at nlqdb.com/blog

We're building nlqdb as analytical memory for AI agents — a real database an agent queries with GROUP BY and JOIN, not a fuzzy fact store. So before writing our own memory-quality benchmark, we read the ones the field already uses: LoCoMo, LongMemEval, and Mem0's and Zep's evaluations. The short version: almost none of them measure what we assumed, and almost every headline number is self-reported.

What the benchmarks actually score

The canonical benchmarks are end-to-end question-answering suites. LoCoMo (Snap Research, ACL 2024) builds very long multi-session dialogues — around 300 turns over up to 35 sessions — and asks single-hop, multi-hop, temporal, commonsense, and adversarial questions, graded by string-match F1. LongMemEval (ICLR 2025) embeds 500 questions across five separately-scored abilities: information extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention. Both grade whether the final answer is right — not whether the right memory was retrieved.

That distinction matters. In classic retrieval you measure recall@k, precision@k, MRR, and nDCG against labeled-relevant items. We went looking for those numbers in agent-memory evaluations and mostly didn't find them: the field grades QA accuracy (or an LLM-as-judge), not component-level retrieval. There is no agreed ground-truth-relevance standard for agent memory. If you want to measure retrieval quality directly, you have to label relevance yourself.

Almost every headline number is self-reported

Mem0 reports a 26% relative improvement over OpenAI's memory on LoCoMo; Zep reports 94.8% on Deep Memory Retrieval and up to 18.5% on LongMemEval. Read the author lists: these are the vendors' own papers. None has been cleanly reproduced by a neutral third party, and the two vendors publicly dispute each other's LoCoMo methodology. The benchmark itself is soft — an independent audit found roughly 6.4% of LoCoMo's answer key is simply wrong, and re-scoring with a corrected judge prompt swung one system from 84% to 58%. Treat every leaderboard number as directional, not settled.

The thing nobody measures: analysis over memory

Here's the gap that matters for us. Every system in the field stores facts and retrieves them. A vector store can recall "Alice has a $50k deal." What none of them benchmark — and most structurally can't do — is analysis over that memory: "show the top 5 deals by value, grouped by stage, for enterprise accounts only." That's a GROUP BY with a HAVING and a JOIN, and a fuzzy fact store has no query planner to run it. We could not find a single benchmark that isolates analytical queries over episodic memory against vector or graph memory on identical data. The field measures recall of facts, not reasoning across them.

There's even supporting evidence hiding in LoCoMo's own results: restructuring raw dialogue into a "database of assertions" lifted temporal-question F1 from 21.3 to 41.9. Structure helps most exactly where LLMs are weakest — temporal and multi-hop reasoning.

Where a database does not win (the honest part)

A real database is not a free win everywhere, and pretending otherwise would be dishonest. For fuzzy semantic recall over unstructured text — "find the thing I said that's kind of like this" — embedding similarity still beats exact SQL. Every serious system relies on it, including the most database-native academic proposal we found, which is Postgres-based and still falls back to pgvector nearest-neighbor. Pure analytical SQL cannot replace vector search for unstructured recall. The honest wedge is analytical memory, not memory without embeddings — which is why hybrid recall is on our roadmap, not our marketing.

What we're building

We're adding an agent-memory-quality eval to the same benchmark harness we already run for text-to-SQL accuracy. It scores four axes: retrieval precision and recall (against relevance labels we define, since the field has none), temporal reasoning, forgetting and contradiction resolution, and consolidation. And it includes the analytical-memory task nobody else runs: aggregation and ordering over episodic memory, head-to-head against a vector-recall baseline on the same data — reported honestly, including the questions where a pure-SQL store loses. We'll publish the numbers here as they land, reproducible, not self-graded.

If you're building an agent and you want its memory to answer questions, not just echo facts back, that's the whole idea. You can point an agent at a live nlqdb database and start asking in plain English.

We shipped 18 SEO pages and got 1 referral. The links only pointed one way.

Omer Hochman — Thu, 27 Aug 2026 01:32:48 +0000

Originally published at nlqdb.com/blog

For a few weeks the surface count was the number that moved. Comparison pages, how-to pages, engineering posts — the sitemap kept growing. Referral traffic did not. It sat at roughly one external visit a week the whole time. More pages, same trickle. When volume climbs and yield doesn't, the pages aren't the problem; the graph connecting them is.

A tree, not a mesh

We drew our own internal links and the shape was obvious in hindsight. Every blog post linked down to the deep guide it anchored — a post about duplicate-row detection linked to /solve/find-duplicate-rows-in-my-data, a comparison post linked to /vs/mode. Nothing linked back up. /blog → /solve, never /solve → /blog. The graph was a tree with the deep pages as leaves, and the freshest content — the posts we most needed crawled and indexed — hung off the ends with zero internal inbound links.

That one-way arrow costs you twice, once for machines and once for humans.

Cost one: authority flows the way the link points

Links pass authority in the direction they point. Every forward link we shipped fed the deep evergreen page and starved the new post. The URL that most needed a crawl signal — the one published yesterday — was the one with no internal links pointing at it.
A page with zero internal inbound links is nearly invisible to a crawler. The sitemap tells a bot the URL exists; internal links tell it the URL matters. We were publishing pages that only the first told anyone about.
Fresh pages have the shortest runway. An evergreen guide accrues links over months. A post has days to prove it's worth indexing. Pointing all the authority the other way is exactly backwards.

Cost two: a one-way link is a dead end for a reader

The human cost is simpler and you can feel it. A searcher lands on /solve/find-top-n-rows-per-group from Google, reads the answer, and… stops. There is no next hop. The session is one page long, the reader bounces, and the analytics tell you the page 'didn't engage' when really it just had nowhere to send anyone. A page that answers a question and then offers the obvious follow-up keeps the reader on the site; a page that answers and then goes quiet hands them back to the search results.

The fix was a field we already had

Each post already declared the guide it anchored — a typed anchor field with a label and a path, used to render the forward link. That field is bidirectional information; we were only reading it one way. The deep page knew nothing about the posts pointing at it, but the posts knew exactly which deep page they belonged to. So we inverted the same data: at build time, for each /solve and /vs page, collect the posts whose anchor names it and render a reciprocal "Further reading" link back. No new content, no new data model — one derived link per relationship, generated from the field that already drove the forward one.

// The forward link already existed on each post:
//   anchor: { label: 'Find top-N rows per group', path: '/solve/...' }
//
// Invert it once, at build time, to get every backlink for free:
const backlinks = new Map<string, BlogPost[]>();
for (const post of BLOG_POSTS) {
  if (!post.anchor) continue;
  const list = backlinks.get(post.anchor.path) ?? [];
  list.push(post);
  backlinks.set(post.anchor.path, list);
}
// /solve/[slug] and /vs/[slug] now render backlinks.get(currentPath)
// as a 'Further reading' block — the mesh closes with zero new data.

Every anchored deep page now has at least one internal inbound link, and every post has a two-way relationship with the guide it belongs to. The link count went up without a single new page; the graph went from a tree to a mesh. (nlqdb is the database you talk to; this is one of the notes from building it in the open.)

The lesson: measure the graph, not the count

Publishing volume is not distribution. A pile of pages that only link one direction is a stack of dead ends wearing a sitemap. The metric that actually predicts yield is the shape of the internal link graph — how many pages a reader (or a crawler) can reach from any starting point, and whether authority can flow to the pages that need it. Before you write the next page, check that the last one links both ways. The cheapest distribution win is usually a link you already have the data to draw.

The timeout that looked like a hallucination

Omer Hochman — Wed, 26 Aug 2026 01:53:26 +0000

Originally published at nlqdb.com/blog

Our NL→SQL benchmark reported that a frontier model "emitted junk" — non-SQL, unparseable — on 5 of 150 hard questions, dragging its score below a floor a frontier model has no business sitting under. The number was real. The explanation was wrong. The model never hallucinated on those five. It was answering fine and we killed it mid-sentence, then filed the crime under the wrong charge.

The clamp you measure instead of the model

The request layer aborted every model call at a 5-second timeout — a budget inherited from the production hot path, where a user is staring at a spinner and 5 seconds is already too long. That budget is correct for what ships. It is exactly wrong for a benchmark, whose entire job is to measure what the model can do, not what fits in a latency SLA. A reasoning model on a hard schema wants seconds of chain-of-thought; a 5-second clamp truncates it. So the benchmark wasn't scoring the model. It was scoring the clamp — and reporting the clamp's verdict as the model's.

The rule that fell out of this: a benchmark's timeout budget is a measurement instrument, not a config value to reuse. Separate "what ships" (the prod hot-path budget) from "what the model can do" (the eval budget). Point the frontier eval lane at its own generous budget and the score jumps to reflect the model, not the SLA. That single change lifted the lane off its false floor.

An aborted read is not a parse error

The second bug is why the truncation disguised itself. When the timeout fired, an AbortController cancelled the in-flight fetch. The code was mid-res.json(), so that read rejected — and the surrounding catch did the natural, wrong thing: no JSON, therefore the model returned something unparseable, therefore classify it parse (model emitted non-SQL). But the read didn't fail because the body was junk. It failed because we pulled the plug. The abort and the parse failure land in the same catch, and only one of them is the model's fault.

try {
  const json = await res.json();
  return classify(json);
} catch (err) {
  // WRONG: every failure here reads as "model returned non-SQL"
  return { failure: "parse" };
}

// Right: the signal tells you who aborted the read.
try {
  const json = await res.json();
  return classify(json);
} catch (err) {
  if (controller.signal.aborted) return { failure: "timeout" }; // infra, not model
  return { failure: "parse" };                                   // genuinely unparseable
}

Classify by the signal state, not by "the JSON didn't parse." An AbortError when your own controller fired is a timeout — an infra event you own — not a model-quality datapoint. Conflate them and every timeout inflates your hallucination rate and deflates the model.

The tell was in the latencies

We should have caught this without reading a line of code, because the evidence was sitting in the numbers. Every single "hallucination" had a latency of 5000–5004 ms. Real model failures scatter across the latency distribution; these were pinned to the wall, all five clustered at exactly the timeout. A failure mode that always takes the same round number of milliseconds is never the model — it's a clock. Log per-attempt latency next to every failure, or you will stare at a bogus quality regression and never see the fingerprint.

The honest split

This is eval-harness measurement integrity, not a claim that the model is better than you thought — though in this case it was. None of it makes a weak model strong. It stops an infra artifact from being scored as incompetence. If you benchmark any model behind a timeout: give the eval its own budget separate from prod, classify aborts by the signal not the exception, and log the latency next to the verdict. Otherwise the first thing your benchmark measures is your own patience.

(This is how we score the frontier lane for nlqdb, the data layer you ask in English — the eval harness runs a separate capability budget from the production ask path, so the number reflects the model, not the SLA.)

Your "best model" toggle quietly serves the cheap model. Ship a 409 instead.

Omer Hochman — Tue, 25 Aug 2026 01:34:38 +0000

Originally published at nlqdb.com/blog

Every AI product has a model picker now — Fast / Balanced / Best, or a dropdown of names. You add one to your app. Then you hit the branch that decides how good a knob it is: the user picked Best, but the premium lane isn't available right now. No API key on file, no paid plan, the metered lane is dark. What do you do?

The tempting answer is to quietly serve the default chain. The request succeeds, an answer comes back, nobody sees an error. It feels graceful. It is the single worst thing you can do, and it took us a review pass to stop doing it ourselves.

A placebo knob is worse than no knob

When "Best" silently downgrades to "cheapest", the user asked for one thing and got another, and the only signal they ever get is quality variance. Their answers are a little worse than they expected, intermittently, for reasons they can't see. They won't file a bug that says "the Best toggle is a no-op" — they'll conclude your product is unreliable and churn. You've spent UI real estate and a database column building a control that lies, and the lie is expensive precisely because it's invisible.

A product with no picker at all is more honest: it makes no promise it can't keep. So if you ship the knob, the knob has to never lie about which chain answered.

The honest contract is three lines

fast pins the cheap chain — always. Even when a stored provider key would auto-upgrade the request. An explicit per-request instruction beats an ambient credential: the caller who wrote model: fast into a nightly CI job means it, and silently spending their premium budget because a key happens to be on file is the same betrayal in the other direction.
best gets a real frontier lane, or it fails loud. If the premium lane is live for this caller, route to it. If not, return a 409 with a machine-branchable error code (model_unavailable) and a fix-it link — not a 200 with a trace note nobody reads. An error the caller can branch on is a contract; a buried warning is a shrug.
auto / absent stays the default chain. No opinion expressed, no promise made, cheapest-that-works — the behavior you already had before the knob existed.

// one pure function, shared by every surface
function selectDispatchLane(preset, caller) {
  if (preset === "fast") return { lane: "free" };        // pin, ignore stored keys
  if (preset === "best") {
    return caller.hasFrontierLane
      ? { lane: "frontier" }
      : { error: "model_unavailable", status: 409, link: FIX_IT_URL };
  }
  return { lane: caller.defaultLane };                    // auto / absent
}

The refusal is your demand signal

Here's the bonus that makes failing loud strictly better than degrading quietly. Every model_unavailable you return is a user telling you, in the clearest possible terms, that they want a paid lane you haven't lit yet. Count them and you have a demand curve with an honest denominator — real requests for the upgrade, not survey wishes. A silent downgrade throws that signal away; a 409 files it for you.

Make it one function, or your surfaces will drift

The trap after you get the contract right is re-implementing it per surface. Your HTTP API, SDK, CLI, MCP server, and web app each resolve the preset, and the fourth one you write will subtly disagree with the first — one of them forgets the anonymous short-circuit and serves best off the free path before the lane check ever runs. Put the precedence in one pure function every surface calls, cover it with one test file, and the knob means the same thing everywhere by construction.

The rule

A model knob is a contract, not a capability. The free chain still answers most questions fine — this was never about having the best model. It's about the toggle never lying about which chain answered. Pin when told to pin, upgrade when you can, and fail loud with a code the caller can act on when you can't. Anything else is a placebo, and placebos cost you the trust you built the picker to earn.

(This is how the model preset works on nlqdb, the data layer you ask in English: fast pins the free chain, best gets a frontier lane or a 409 model_unavailable with a fix-it link, and one shared function resolves it across the API, SDK, CLI, MCP server, and the <nlq-data> element — so the knob can't drift and can't lie.)

Your LLM health probe passed. Your agent still starved.

Omer Hochman — Sat, 22 Aug 2026 20:13:32 +0000

Originally published at nlqdb.com/blog

We gate an end-to-end suite on a live free LLM: before the agent test runs, a pre-flight probe checks the model is up. The probe was green. The suite failed anyway — six runs in a row. Every failure was the agent starving mid-task on a model the probe had just certified healthy. Here is what a health probe in front of an LLM provider has to actually check, learned one red run at a time.

1. Probe the exact shape you'll use — not a 1-token ping

The cheap probe is a one-token completion: send "hi", get a token back, call it healthy. But your agent doesn't send "hi" — it does a tool-call round trip: system prompt, tool schemas, a turn that must emit a well-formed tool call. A model (or a saturated free pool) can return a token for "hi" and still fail to produce a valid tool call under load. Probe the shape you depend on. Ours now does a real tool-call round trip and asserts the response parses as a tool invocation, because that is the capability the suite needs — not liveness.

2. Check the body, not the status

This one cost us the most time. The probe checked res.ok and moved on. But an AI gateway in front of the provider wraps an upstream 429 in an HTTP 200 envelope — the transport succeeded, the body carries the error. res.ok is necessary, never sufficient, for anything behind a gateway.

// Looks healthy. Isn't.
const res = await fetch(gatewayUrl, { ... });
if (res.ok) return "healthy"; // 200 wrapping an upstream 429

// The upstream status lives in the body.
const json = await res.json();
if (json.error || json.choices?.[0]?.finish_reason === "error") {
  return "unhealthy";
}

3. Saturated free pools flap — require N consecutive healthy probes

A single healthy probe against a free model pool is a coin flip when the pool is busy: the next request lands on a different, throttled backend. One green probe means "a backend was free 40 ms ago," not "the pool is healthy." We now require three consecutive healthy probes before the gate opens — enough to distinguish a stable pool from a flapping one, cheap enough to stay fast.

4. Probe-time health can't promise a 15-minute window

Even three green probes only certify the moment they ran. A suite that takes fifteen minutes will outlive that certification — the pool that was healthy at minute zero can throttle at minute nine. A pre-flight gate reduces the odds of starting a doomed run; it cannot guarantee the run finishes. Pair the gate with client-side backoff and retry inside the run, so a mid-run throttle is absorbed instead of failing the suite. The gate and the backoff do different jobs; you need both.

5. Never hard-code one model — walk an ordered candidate list

Our six red runs all pointed at one hard-coded free model. When that single model's pool got busy, there was no fallback and no signal beyond a failed suite. The fix is an ordered candidate list: probe the first, fall through to the next on an unhealthy verdict, and run against the first that clears. One model is a single point of failure you control — so don't build one.

6. Probe-healthy is not agent-competent

The subtlest lesson: a model can ace the probe and still be useless to the agent. Trace-verified in our runs, a probe-healthy free model spammed forbidden tools, leaked raw <|tool_call_id|> framing tokens into its tool arguments, and burned a 240-second budget without ever reaching a verdict. Health and competence are different axes. Gate CI on health — is the pool up — but rank your candidate list by demonstrated competence — which model actually finishes the task. A model that flunks the task never earns the top slot no matter how green its probe.

The honest split

This is CI reliability engineering, not a claim about model quality. None of it makes a weak model strong; it stops a busy free pool from being scored as a broken product. If you gate any pipeline on a live LLM provider: probe the real shape, read the body not the status, require consecutive greens, pair the gate with in-run backoff, walk a candidate list, and rank that list by competence — not by whichever model answered your ping first.

(These probes guard the agent E2E suite for nlqdb, the data layer you ask in English — we run the free-model chain in CI on purpose, so the harness has to tell a saturated pool apart from a real regression.)