DEV Community: Na'aman Hirschfeld (Goldziher)

300+ languages doesn't mean what you think: three tiers of code understanding

Na'aman Hirschfeld (Goldziher) — Sat, 08 Aug 2026 09:28:50 +0000

Every tool in this space leads with a language count. basemind's README says 300+. Serena says 40+.
aider says 130+. codebase-memory-mcp says 158.

Those numbers are not comparable, because they are not measuring the same thing. Most of them are
counting grammars - how many languages the tool can parse into a syntax tree. Parsing is the
cheap part. It tells you where the functions are. It does not tell you which get this get is.

So here is the honest version of basemind's number, split by what you actually get.

Tier	Languages	Engine	What you get
Full resolution	JavaScript, TypeScript	oxc	Scope + import/export resolution, intra- and cross-file
Full resolution	Python, Java	stack-graphs `.tsg` rules	Name binding over the parse tree, cross-file via specifier resolution
Scope only	everything else	tree-sitter `locals` query	Intra-file lexical binding, no imports
Symbols + calls	~100 grammars	tree-sitter `tags.scm`	Symbol and call extraction
Parse only	371	tree-sitter-language-pack	A file map. JSON, YAML and TOML produce no symbols at all

Three languages get real resolution. About a hundred get symbols. The rest parse. That is a much
less impressive sentence than "300+ languages," and it is the one that survives contact with a user.

Why not just run a language server

The obvious objection: language servers already solve this, correctly, with a real type checker.
rust-analyzer knows things about your code that no amount of tree-sitter will ever recover.

That is true, and I want to concede it up front rather than have it raised in a comment thread.
For generics, macros, dynamic dispatch, dependency injection and monkey-patching, an LSP-backed tool
like Serena - or an SCIP-backed one like Sourcegraph - will be right where basemind is wrong.

What you pay for that is a build. A language server needs your project to compile, or close to it: a
resolved dependency tree, a populated node_modules, a cargo build that has actually run. It needs
a server process per language, each with its own startup cost and its own memory. On a polyglot
monorepo that means several daemons, several indexes, several cold starts, and a hard failure the
moment the build is broken - which, if an agent is halfway through editing, it usually is.

basemind is optimizing for a different constraint: an agent asking a hundred questions per session
about a repo that may not compile, in a language mix nobody planned for. So the design bet is no
build step, no language server, no per-language daemon - and the honest cost of that bet is that
resolution is an approximation.

The part nobody else shipped

For JavaScript and TypeScript, the answer is easy and good: oxc. oxc_semantic
builds a real scope tree, symbol table and reference bindings; oxc_resolver does Node and tsconfig
module resolution, including solution-style monorepo path aliases. It is fast, actively developed,
and funded. A monorepo rarely imports by relative path - it writes @app/hooks/useSettings - and
oxc_resolver handles that natively.

For Python and Java there was no equivalent, so basemind uses stack-graphs.

Stack graphs come out of GitHub research: Douglas Creager's
Name resolution at scale. The idea is that name binding can be
expressed as a graph-reachability problem, with per-language rules written declaratively in a DSL
(.tsg) that runs against a tree-sitter parse tree. No type checker, no build, and - critically -
the per-file work is independent, so it caches.

It is a genuinely good idea, and it is also abandoned. GitHub archived the repository on
2025-09-09 with the note that it is no longer supported and that you should fork it if you want to
keep going. The tree-sitter-stack-graphs crate has not had a release since v0.10.0 in December 2024.

I am not the first person to look at this. aider evaluated stack-graphs and went with tree-sitter
tags plus PageRank instead (#534). So did
OpenHands (#742) and
SWE-agent (#38). Three serious teams reached the
same conclusion: too much work, unmaintained upstream, not worth it.

They were probably right about the cost. Here is what taking it on actually involved.

Four rule bugs that silently deleted a file's entire resolution

The upstream Python rules abort the whole file's stack-graph build on several perfectly ordinary
constructs. Not "resolve that one line wrong" - the file produces no resolution at all, quietly. A
sweep over a real Python codebase surfaced four:

typed_parameter matched too broadly. A typed splat parameter - **kwargs: T, *args: T - bound the splat pattern as if it were a plain name. Fixed by restricting the rule to identifier-named params.
Keyword arguments in a class header were treated as superclasses. class X(TypedDict, total=False) and anything with metaclass=… broke. Fixed by restricting the superclass list to identifiers, attributes and subscripts.
Parameter-less lambdas got no call node. The combined function/lambda stanza required a parameters field, so lambda: x was skipped entirely.
Chained assignment referenced an undefined output. a = b = c nests, and the inner assignment had no .output to flow from. Fixed by having every assignment carry one.

With those four fixed, that codebase goes to 100% of files building.

Two panics on the stitching hot path

A panic while resolving one file must never take down a scan of an 82k-file repository, and two
upstream ones could:

Database::get_incoming_path_degree indexed a lazily-grown supplemental arena directly, which panics for any node that is not already the end node of some partial path in the database. It now reports Degree::Zero.
ForwardPartialPathStitcher::extend called .expect() on the cycle detector. The cycle test replays a suffix of the path against freshly minted stack variables, which can legitimately fail to unify. An undecidable cycle test now discontinues the path - the same conservative outcome as a detected cycle - instead of panicking.

Both have regression tests. Alongside that, the scanner persists the code map before running the
optional lanes and wraps each lane in catch_unwind, so a per-file panic in the resolve pass costs
that file's edges and nothing else. It used to cost the entire map, which meant gigabytes of blobs
sitting behind a file_count of 0 and a full re-scan on every launch.

The fork is four workspace crates - stack-graphs, tree-sitter-graph, tree-sitter-stack-graphs,
lsp-positions - carrying upstream copyright, ported to tree-sitter 0.26 and edition 2024, with the
C FFI, serde, visualization and storage modules stripped because none of that surface is reachable
when you build the graph in memory and stitch it in-process.

Cross-file is where the interesting caching lives

Intra-file resolution links a use to its definition inside one file. That result is deterministic in
the file's bytes, so it is cached in a content-addressed blob next to the code map: unchanged file,
no re-analysis.

Cross-file resolution is the opposite. It resolves an import's module specifier - Node/tsconfig for
JS/TS, dotted and relative path arithmetic for Python, fully-qualified names over Maven and Gradle
source roots for Java - then joins the imported name against the target file's exports, following
re-export chains through package __init__.py files and TypeScript barrels up to eight hops.

That result is not a function of one file's bytes. So it is deliberately not cached in the
per-file blob. If a file is unchanged but its dependency moved, the edges still get re-stitched. The
incremental path restages only the changed files' intra-file facts and re-stitches only the affected
importers - the changed files plus everything that imports one.

Caching the thing that is safe to cache, and refusing to cache the thing that looks similar but
isn't, is most of what makes this fast enough to sit under an agent.

The contract that matters more than the resolution

Here is the design decision I would defend hardest, and it is a decision to do less.

callers never narrows to the resolved subset.

It returns every call site a plain name scan would return - the complete, no-scope floor - and
annotates each hit with whether resolution proved it binds to this definition, plus a
resolved_total count. It does not filter down to the proven ones.

That looks wrong. Filtering to proven hits is more precise, and precision is the whole point of
building a resolver.

It is wrong because resolution cannot see through a module-object import - from pkg import mod
then mod.f() - or an unresolvable path alias. Those are real callers that resolution will not
prove. Filter to the proven subset and you return a smaller set with no indication anything is
missing.

We know exactly how that fails, because it did. Resolution saw 2 of 172 real call sites and the
tool reported total: 2, with no truncation flag. A confident, complete-looking, wrong answer. A
Python case returned 7 of roughly 4,000. An agent acting on either would conclude a function had
almost no callers and refactor accordingly.

resolved: false is not evidence a hit is not a real caller. It is evidence resolution could not
prove it either way. So the sound floor is the answer, and precision is an annotation on top. Filter
on resolved when you want confidence; trust total when you need completeness.

The same principle runs through the provenance model on the code graph: every edge carries a
confidence - extracted, inferred, or ambiguous - derived on read, and it degrades downward. A
binding whose proof is unreachable is reported as inferred, never as falsely extracted. Import and
inheritance edges are name-resolved by construction, so they can never claim to be extracted at all.

What this still doesn't do

Java qualified member calls on an imported type - Foo.greet() - are unresolved, and
import static is not distinguished from a type import. Both are misses, never wrong answers, which
is the failure mode the contract above is built to guarantee.

basemind is read-only. It will tell you every caller of a symbol; it will not rename it for you.
Serena does symbol-level editing and refactoring, and its users lean on that heavily.

And the PageRank-over-a-tree-sitter-symbol-graph idea for ranking repo architecture is not mine -
aider published that in October 2023. basemind blends
git churn into the weighting, which is an increment on their work, not a replacement for it.

The number on the box is 371 languages. The number that describes what the tool understands is
three, plus about a hundred that give up symbols, plus a long tail that parses. Both are true. Only
one of them is useful when you are deciding whether to install something.

basemind is MIT, pure Rust, and lives at github.com/Goldziher/basemind.

Scythe: SQL as a first-class language (and where that leaves your ORM)

Na'aman Hirschfeld (Goldziher) — Wed, 05 Aug 2026 10:42:22 +0000

I build and maintain scythe (MIT). This is the argument behind it: why SQL deserves the same tooling as the rest of your code, what generating your data access layer from it replaces, and the one case where it doesn't.

Every app has a data access layer. The question is who writes it

Whether or not anyone on your team calls it that, your application has a data access layer: the code that maps parameters in, maps result rows out, keeps the types on both sides aligned, and gets rewritten every time a query or the schema changes. It is tedious, it is where a lot of quiet bugs live, and nobody enjoys maintaining it.

There are roughly five ways teams get one. A full ORM writes it for you at runtime and hides it. A data mapper writes most of it and asks you to be explicit. A query builder gives you the pieces to assemble — this is what people usually mean when they say "we skipped the ORM and wrote our own DAL". Raw SQL plus manual mapping means you write every line by hand. And then there is the option this article is about: generate the layer from the SQL itself, at build time.

sqlc did this cleanly for Go. You write plain .sql files, it reads your schema and your queries, and it generates typed Go functions and structs before anything compiles. No runtime ORM, no DSL, just SQL in and a typed access layer out. It is one of the nicer developer experiences in the Go ecosystem, and it is the tool scythe is most directly inspired by.

The obvious question is why that idea should be locked to one language. Scythe is my answer: the same SQL-first, compile-time approach, generating an idiomatic layer for 10 languages, with a real type-inference engine and a SQL quality toolchain underneath it.

SQL as a first-class language

Here is the thing that bothers me about how most codebases treat SQL. Your Python gets a formatter, a linter, a type checker, and a pre-commit hook that blocks the bad version from ever reaching a branch. Your .sql files, holding the queries that decide whether the product is fast or correct, get none of that. They sit in a directory as untyped strings, reviewed by eye.

SQL is a 50-year-old language every database speaks, expressive and heavily optimized. Treating it as a first-class language in your repo means giving it exactly what everything else already has:

A formatter. scythe fmt normalizes style so SQL diffs show intent instead of whitespace.
A linter. 23 schema-aware rules — UPDATE without WHERE, = NULL instead of IS NULL, ambiguous columns in joins, leading-wildcard LIKE, unbounded ORDER BY — plus sqruff's 69 style rules underneath.
A security auditor. scythe audit is static analysis for SQL: GRANT ALL, grants to PUBLIC, SECURITY DEFINER without a pinned search_path, literal passwords, SELECT * over PII. 35 rules, emitting SARIF with CWE tags so findings land in GitHub code scanning like any other scanner's.
Pre-commit hooks. scythe-fmt, scythe-lint, and scythe-audit ship as hooks, so bad SQL fails at commit time rather than in review or production.
Types. Which is the part that needs a real example.

The boundary between a query and your program's types is where this has always been weakest, so that is where the interesting work is. Here is the case that shows why the inference has to be genuine — a query with a LEFT JOIN:

-- @name GetUserOrders
-- @returns :many
SELECT u.id, u.name, o.total, o.notes
FROM users u
LEFT JOIN orders o ON u.id = o.user_id
WHERE u.status = $1;

The right side of a LEFT JOIN is nullable. A user with no orders still produces a row, with total and notes as NULL. If your generated types say those columns are non-null, you have a crash waiting for the first user without an order. Scythe knows this, and encodes it in every language it targets:

#[derive(Debug, sqlx::FromRow)]
pub struct GetUserOrdersRow {
    pub id: i32,
    pub name: String,
    pub total: Option<rust_decimal::Decimal>,
    pub notes: Option<String>,
}

@dataclass
class GetUserOrdersRow:
    id: int
    name: str
    total: decimal.Decimal | None
    notes: str | None

interface GetUserOrdersRow {
    id: number;
    name: string;
    total: string | null;
    notes: string | null;
}

Same query, three languages, and in each one total and notes come out correctly optional. You did not annotate anything. The nullability fell out of reading the SQL.

That inference is the core of the tool, and it goes well past joins: COALESCE, CASE branch widening, window functions (ROW_NUMBER, LAG, LEAD, and friends each with their own nullability), aggregates, CTEs including recursive ones, and RETURNING clauses. Postgres enums, arrays, and composite types map to native language types too.

Where scythe goes past sqlc

sqlc is excellent and I want to be precise about the difference. Scythe is wider on two axes.

Breadth: it generates native, idiomatic code for 10 languages (Rust, Python, TypeScript, Go, Java, Kotlin, C#, Elixir, Ruby, PHP) across 10 databases (PostgreSQL, MySQL, MariaDB, SQLite, DuckDB, CockroachDB, Redshift, SQL Server, Oracle, Snowflake) with 70+ backend drivers. Adding a language or a driver is mostly a declarative type-mapping manifest, not a plugin in a separate repo.

Depth on the SQL itself: sqlc infers nullability primarily from column constraints. Scythe infers it from the query structure (the join and expression analysis above). And it treats SQL as code you lint, format and audit, which sqlc leaves to other tools: 58 built-in rules (23 lint, 35 audit) with sqruff's 69 style rules layered underneath, plus scythe inspect, which runs operational health checks against a live database — foreign keys with no covering index, tables with policies but RLS disabled, duplicate indexes. All of it runs in the compile pipeline, so bad SQL is caught before code generation rather than at runtime.

The one job ORMs still do better

There are ORMs I genuinely like. SQLAlchemy is probably the most sophisticated data-access library in any language, largely because Core and ORM are separable layers rather than one monolith. Drizzle fits its ecosystem well. Ecto is a better-designed thing than the category it gets filed under. This is not an argument that the category was a mistake.

It is an argument about which layer you want, for this app. And there is exactly one job an ORM does that scythe does not answer: bring-your-own-database portability. If your product lets the user pick the database, so the same application code has to run on PostgreSQL or MySQL or SQLite depending on deployment, an ORM abstracts those dialect differences at runtime and that is genuinely valuable.

Scythe takes the opposite bet on purpose. You write SQL for a specific engine, which means you get that engine's real features (Postgres arrays, MySQL JSON functions, whatever you actually deploy on) and let its optimizer do its job. The cost is that targeting multiple engines means multiple SQL files.

So the honest split is simple. If you must run on whatever database the customer brings, use an ORM. If you control the database, which most teams building most products do, then a full ORM is mostly overhead: N+1 queries from lazy loading, SQL you cannot see or predict, model-versus-schema drift, and weak types exactly where SQL gets interesting — aggregates, conditional expressions, nullable joins.

There is also a related decision people conflate with this one, and shouldn't: how thick a layer you put over whatever you chose. Repository, DAO, Unit of Work, or nothing. That is orthogonal — you can put a repository over raw SQL or over Hibernate, and teams do both. It is also where a lot of wasted effort lives, since wrapping a full ORM in a repository is largely rebuilding what a Data Mapper with a Unit of Work already is. The pattern earns its keep further down, where you are the one deciding what a "user" is when it comes back from three joined tables.

For a single-engine architecture on Postgres, generating the layer removes the boilerplate and the runtime dependency, kills a class of hidden query-generation bugs, and leaves you with code you can actually read.

Why this matters more with coding agents

There is a newer reason this approach wins, and it is about who reads the code now. A coding agent working in your repo reasons about plain, explicit, statically typed code far better than it reasons about ORM runtime behavior. With scythe the SQL is right there in a file, the generated function has concrete argument and return types, and the compiler and linter catch a mistake immediately. There is no lazy-loading, no identity map, no query builder assembling strings at runtime for the agent (or you) to simulate in its head. Legible, static, deterministic output is easier for a human to review and easier for a model to get right. SQL-first codegen produces exactly that.

What scythe deliberately does not do

Static queries are the trade. Scythe does not build queries dynamically at runtime the way jOOQ's DSL does. If you need conditional composition you write separate queries, or push the condition into SQL (WHERE ($1 IS NULL OR status = $1)). That is a deliberate choice: static queries are the ones you can lint, analyze, and optimize. If runtime query composition is central to your app, jOOQ or an ORM query builder is the better fit, and if you are a Go-only shop already invested in sqlc, staying there is completely reasonable.

Coming from sqlc, scythe migrate sqlc.yaml reads your existing config and emits a scythe.toml to get you started.

Try it

cargo install scythe-cli
# or prebuilt binaries:
cargo binstall scythe-cli
# or
brew install Goldziher/tap/scythe

To hold the line at commit time, in .pre-commit-config.yaml:

- repo: https://github.com/Goldziher/scythe
  rev: v0.11.0
  hooks:
    - id: scythe-fmt
    - id: scythe-lint
    - id: scythe-audit

Repo: https://github.com/Goldziher/scythe
Docs: https://goldziher.github.io/scythe
Discord: https://discord.gg/xt9WY3GnKR

If you try it against a gnarly query, a chain of CTEs, a pile of window functions, a join graph that should confuse the nullability, I want to know where it gets the types wrong. That is the feedback the inference engine gets built from.

Knip Keeps My JS/TS Dependencies Honest (I Wish Python Had It)

Na'aman Hirschfeld (Goldziher) — Fri, 24 Jul 2026 12:32:53 +0000

Every long-lived JS/TS project I've worked on accumulates the same three kinds of rot:

Dependencies in package.json that nothing imports anymore. You added moment, migrated off it, and the line stayed.
Exports nothing consumes. A function was public once, the last caller was deleted, the export is still there advertising an API with no users.
Whole files that fell out of the import graph but never got deleted, so every new engineer reads them trying to understand code that runs nowhere.

None of this breaks the build. That's exactly why it survives. The compiler is happy, the tests pass, and the dead weight compounds quietly until onboarding takes a week and your bundle ships code no user will ever execute.

The tool I've settled on for this is knip (by Lars Kappert). I run it on the enterprise codebase I maintain and on basically every other TS project I touch. One command:

$ npx knip

Unused files (2)
  src/legacy/formatValue.ts
  src/hooks/useLegacyModal.ts
Unused dependencies (3)
  lodash        package.json
  moment        package.json
  @types/uuid   package.json
Unused exports (5)
  parseLegacy   src/parse.ts:42:14
  toLegacyDate  src/date.ts:9:14
  ...

Files, dependencies, and exports in one pass, cross-referenced against the actual import graph. It's the first tool I've used that treats all three as the same problem, which they are: something is declared, nothing uses it, delete it.

The honest caveat

Knip is not zero-config on a real codebase. Anything resolved dynamically, runtime import(), plugin systems, framework entry points it doesn't recognize (Next.js pages, a CLI bin, config files loaded by string), gets flagged as unused when it isn't. You will get false positives on day one.

The fix is a knip.json that names your real entry points, and after that it's accurate. But budget an afternoon to tune it before you trust the output enough to delete on it. Anyone who tells you it's instant hasn't run it on a large app.

What I actually want

Here's the part that bugs me. This problem is not specific to JavaScript. Python pyproject.toml files rot the same way. Go modules accumulate unused requires. Rust Cargo.toml grows dependencies that no use references anymore.

Knip is genuinely good, and it's stuck being JS/TS-only because it leans on the TypeScript compiler's understanding of the module graph. I'd trade a lot for the same three-in-one report, unused files, unused deps, unused exports, in Python and everywhere else. There are partial tools (deptry, vulture, cargo-udeps), but nothing that does all three with knip's precision and single-command ergonomics.

If you write JS/TS and you haven't run knip on your repo, do it this week. If you're building tooling and looking for a real gap: this, for every other language.

Stop Writing Specs for Your Agents

Na'aman Hirschfeld (Goldziher) — Thu, 23 Jul 2026 07:24:23 +0000

Spec-driven development is having a moment. Write a detailed spec, hand it to the agent, get working code. It's a clean story and it maps neatly onto how we already think about requirements.

I think it's the wrong shape, and the failure mode isn't obvious until you're six months in.

Specs rot at four different speeds

Here's the actual problem with a spec: it isn't one document. It's four documents wearing a trench coat.

A typical spec contains, all interleaved:

Why we're building this — the product requirement.
How the system is structured — the architectural decision.
The order of operations — the implementation plan.
What "done" looks like — the contract.

Each of these has a completely different half-life.

The product requirement changes when a customer complains. The architectural decision holds for years — and when it doesn't, the record of it still matters, because someone will ask why the old thing looked like that. The implementation plan is dead the moment the PR merges. The contract only matters if it's executable, because anything else is a lie waiting to happen.

Put them in one file and the file decays at the speed of its fastest-rotting component. Within a couple of sprints, that document is 30% accurate. It's the worst possible state — confidently wrong, and now indistinguishable from the parts that were right.

That was bad enough when only humans read it. Now you have agents ingesting it as ground truth. A stale spec doesn't just confuse a new hire who'll ask a follow-up question in Slack. It actively steers an agent into building against a system that no longer exists, and the agent won't push back, because it has no way to know.

Documentation should be layered by lifetime, not by topic.

Here's what I actually run.

Layer 1: Product requirements → the ticket system

GitHub Issues, Linear, Jira. Inside larger orgs, PRDs and RFCs.

These are supposed to churn. That's not a defect, it's the job. Tickets have closed states, owners, and timestamps, which makes their staleness legible — a ticket from eight months ago is visibly a ticket from eight months ago.

Critically: this layer does not belong in the repo. The moment you mirror product requirements into markdown next to your code, you've created two sources of truth that will diverge, and the one in git will lose — because nobody closes a markdown file.

Layer 2: Architectural decisions → ADRs in git

This is the layer that actually earns its place in version control.

I use MADR. Each decision gets a numbered file: context, options considered, decision, consequences. Decisions are never edited to reflect new reality — they're superseded by a new record that references the old one.

Two things fall out of this that matter enormously for agents:

It's append-only, so it can't rot. An ADR describes a decision made at a point in time. It's accurate forever, by construction. "We chose Postgres over DynamoDB in March 2024 for these reasons" doesn't stop being true when you migrate — you just add ADR-0042 marking 0017 as superseded.

It captures the why, which is the thing agents cannot infer. An agent can read your code and reconstruct what it does. It cannot reconstruct why you rejected the obvious alternative. That's the highest-value context you can possibly hand it, and it's exactly what specs bury under implementation detail.

When an agent asks "should I add a queue here?", the ADR directory is the answer.

Layer 3: Implementation plans → ephemeral, never committed

This is where I differ most sharply from the spec-driven crowd.

The implementation plan is scaffolding. It exists to get from "we agreed on the approach" to "the code works." Then it should die.

Modern agents handle this natively now — Claude and Codex both maintain working plans internally and iterate against them. Before that, I'd write a todo.md, work through it, and delete it. Same thing, worse ergonomics.

The important part is that it stays out of git. A committed implementation plan is a fossil that looks like a map. Six months later someone — or some agent — reads it as a description of the system, when it was only ever a description of an afternoon.

Layer 4: Contracts → black-box integration tests

Prose cannot define a contract. Prose describes a contract, and the description drifts.

What actually defines my code at the contract level is a suite of integration tests exercising it from the outside in — no internal mocks, no reaching into private state. Call the public surface, assert on observable behavior.

This is interface-first design. It gives you three things at once:

A specification that cannot go stale, because CI fails when it does.
A refactoring boundary — internals are free to change as long as the black box holds.
A verification loop an agent can run itself, without a human in the middle.

That last one is underrated. An agent with executable contracts can self-correct. An agent with a prose spec can only guess whether it complied, and will report success either way.

This is not TDD. TDD means writing tests and defining correctness up front, before the implementation. Sometimes that's right. Often you don't know the correct interface until you've built enough to feel where it's wrong, and committing to a test suite prematurely just means you refactor tests instead of code. What I'm claiming is narrower: by the time it's merged, the contract is expressed as executable black-box tests. How you got there is your business.

What this looks like in practice

Layer	Lives in	Lifetime	Who reads it
Product requirements	Ticket system	Churns constantly	Humans, mostly
Architectural decisions	Git (MADR)	Permanent, append-only	Humans + agents
Implementation plan	Agent context / scratch file	Dies with the PR	Agent, briefly
Contract	Integration test suite	Enforced by CI	CI, agents, humans

Four artifacts. Four locations. Four decay rates, each matched to a medium that handles it.

Nothing here is novel on its own — ADRs are a decade old, black-box testing is older than that. The claim is about the separation itself: that the layering is the methodology, and that collapsing it into a single spec document is the thing quietly poisoning your agent's context window.

The uncomfortable part

The pitch for spec-driven development is that you write one document and the agent does the rest. What I'm describing is more work up front. You maintain a ticket system, you write ADRs, you build real integration tests.

But you were going to do all three anyway. Spec-driven development doesn't remove that work — it duplicates it into a fifth artifact that has no owner and no expiry date, and then feeds that artifact to a machine that can't tell it's out of date.

Write less documentation. Put each piece where it can't lie to you.

Debugging basemind with Claude Code

Na'aman Hirschfeld (Goldziher) — Wed, 08 Jul 2026 16:53:10 +0000

basemind is my code-intelligence tool: a pure-Rust code map and scanner, tree-sitter across 300+ languages, a content-addressed blob store, a Fjall-backed inverted index. I harden it by pointing it at a repo big enough to hurt, a 20M-line, 200k-commit monorepo, and watching what it does.

I did this debugging session with Claude Code. It started as "which version is installed?" and ended with a 7x speedup, two upstream PRs, and a feature turned off. Here is how I worked, and the methods that made it effective.

The symptom is not the bug

The report: basemind's cache grew from 1.8 GB to 4 GB while CPU pinned for a long time. I did not open the scanner code. I looked at the live system. ps and lsof found a process at ~70% CPU holding the index open, and it was a serve process, not a scan. Different bug entirely.

That process was an old release. So I had Claude dig through git instead of the debugger: git log on the watcher, then git merge-base --is-ancestor and commit dates. A commit titled "fix: stop watcher CPU runaway" was dated four days after the running binary shipped. The bug (a watcher re-scanning its own writes forever) was already fixed. It was a stale process left over from another editor session.

Two rules, both applied before reading a line of source:

Verify the running artifact, not a proxy. Homebrew, the PATH, and the editor plugin each reported a different version. Find the one that actually runs.
"Live bug or old build?" is the first question. Dating the fix against the running version can end an investigation before it starts.

Measurement tells a scan from a leak

I killed the ghost, and a fresh scan appeared at 488% CPU, nearly five cores. That is rayon pegging every core, and it is indistinguishable from a runaway loop on a single reading.

So I sampled three signals over time: CPU%, cache size, blob count. Disk climbed then plateaued. Blob writes burst then stopped. CPU fell to one core. That is a one-time rebuild converging, not a leak. A leak keeps allocating, a loop keeps writing, a scan converges. The derivative is the diagnosis, not the number.

To watch without babysitting, I had Claude write background monitors: shell loops sampling every 30s, printing only on a state transition. One fired on "SCAN COMPLETE + IDLE," another on "0% CPU and no disk growth for 4 minutes." The alert encodes the hypothesis: a hang is low CPU and no progress, a healthy scan is high CPU or growing disk.

Isolate the confound

A multi-worktree test then hung: scans wedged at 0% CPU, and lsof on a stuck process itself hung for two minutes. A hanging lsof means a stuck socket. The last log line was Registering VLM OCR backend. The OCR tier made a network call with no timeout, a firewall dropped it, and the connection blocked forever.

I did not debug the hang in place. That tier is opt-in behind a cargo feature, so I built basemind with default features, a binary that cannot make the call, and studied the scanner in isolation. When two failures tangle, change one variable.

Read for the mechanism, then measure the fix

On a clean binary I took a baseline. A fresh worktree scan took 181s, re-parsed all 67,700 files though their blobs already existed, and rebuilt the git-history index per worktree at ~160s. Now I read code, to confirm the mechanism, not guess a patch.

The "unchanged file" fast paths keyed off the per-worktree index, empty on a fresh worktree. So every file fell through to a full re-parse, despite its blob already sitting in the shared store.
The git-history index opened at the worktree's own cache dir, though it derives entirely from the shared .git.

Both fixes followed patterns already in the code, and each got a regression test written to fail first. Re-measured: 181s to 25s, ~7x, all 67,700 files reused, peak RSS about halved. The re-measurement is the point. It turns "should be faster" into "is 7x faster."

Follow the failure across the boundary

The network hang still deserved a fix. I traced it: the OCR and embedding models download through hf_hub, whose builder in 0.4 and 0.5 constructs a ureq agent with no timeout. A firewalled download blocks forever. The root cause was in the dependency.

I forked it and had Claude map every download call site across two crates, then add a watchdog: run the blocking fetch on a detached thread, error on a tunable deadline, degrade by skipping the model instead of hanging. The mechanical multi-file edit went to a subagent with a tight spec. Then I read the diff and reran the build and tests myself. The subagent even corrected my assumption about a type not being Clone. That shipped as an upstream PR plus an honest bug report. Delegate the mechanical work, verify with diffs and reruns, never a summary.

Sometimes the fix is deletion

Monitoring caught the embedding pass stuck on a general-purpose model, an empty model dir with a stale lock, spinning a core. The sharper question was not why it was slow, but whether code embeddings should exist at all.

I traced what gets embedded (symbol, signature, doc, body) against what embeds it (a general-English model), and basemind already builds a BM25 index over the same text. A prose model embeds code weakly, the one real win (natural language to symbol) is already covered by the keyword lane, and the cost is a 100 MB pull plus gigabytes of vectors. Verdict: code embeddings off by default, kept for docs and images. A feature that is off cannot hang, leak, or pin a core.

The methodology

Observe the live system (ps, lsof, du) before reading source.
Separate stale-artifact bugs from live bugs by dating the fix against the running version.
Measure dynamics, not snapshots. The derivative separates converging from leaking.
Encode the hypothesis into a monitor so the machine watches for you.
Isolate confounds by changing one variable.
Read code for the mechanism, then write the smallest fix.
Prove it with a test that fails without the fix, and re-measure the metric you started from.
Follow failures into dependencies, and report them honestly.
Ask whether the costly feature should exist, not just why it broke.
Delegate the mechanical work to an agent, but verify every diff yourself.

What shipped

Worktree cache reuse: cold scan 181s to 25s (~7x), 67,700 files reused, peak RSS about halved.
Code embeddings off by default, per-file embed_exclude, safe preset swapping.
An upstream timeout guardrail in the dependency, plus a bug report.

The method I kept coming back to: verify the artifact, watch the live system, measure the change over time, isolate the confound, read for the mechanism, and prove the fix by the number you started from. basemind lives at github.com/Goldziher/basemind.

One Rust core, native SDKs in 16 languages: how we stopped hand-writing bindings

Na'aman Hirschfeld (Goldziher) — Mon, 06 Jul 2026 08:47:33 +0000

Building xberg, a document-intelligence engine with a Rust core, we needed the same engine callable from Python, TypeScript, Ruby, Go, PHP, and a dozen more. This is how we ship native SDKs in every one of them without hand-writing a single binding layer.

The polyglot binding tax

A Rust core is a good decision right up until you have to ship it to other languages. Then you inherit a second, quieter job: keep a PyO3 wrapper, a NAPI-RS addon, a Magnus extension, a cgo layer, and a handful more in lockstep with a core that moves every week.

That is where polyglot projects rot. The core adds a field, one language's wrapper lags, and nobody notices until a Ruby user files a bug that a Python user never sees. Multiply by sixteen languages and the binding layer becomes the most expensive code in the project, none of which is the actual product.

We did not want to write that code, let alone maintain it. So we generate it.

One config, every target

The tool is alef. It extracts the Rust API surface and emits language-native bindings, package scaffolding, type stubs, docs, and e2e tests from one alef.toml:

[workspace]
languages = ["python", "node", "ffi", "go"]
alef_version = "0.32.7"

[[crates]]
name = "sample_core"
sources = ["src/lib.rs"]
version_from = "Cargo.toml"

alef generate --format   # bindings + type stubs
alef scaffold            # package manifests + native build files
alef verify --exit-code  # fail CI if generated files are stale

Each target gets an idiomatic package, not a lowest-common-denominator shim:

Language	Backend
Python	PyO3 + type stubs
TypeScript / Node	NAPI-RS addon with `.d.ts`
Ruby	Magnus extension
Go	cgo over a generated C FFI layer
PHP	native extension
Elixir / Gleam	Rustler NIF
Swift, Kotlin, C#, Dart, R, Zig, WASM	native bridge per platform

Sixteen languages, eighteen targets (C FFI and a JNI shim round out the list). The Rust author writes Rust; the Python user gets a dataclass with type stubs, the Go user gets a cgo package, and neither knows the other exists.

The part that keeps them honest

Generating bindings is the easy half. The hard half is trust: how do you know all sixteen behave the same after you change the core?

Alef's answer is shared fixtures. You write one set of JSON fixtures describing inputs and expected outputs, and it generates a cross-language e2e suite plus standalone registry-mode test apps from them:

alef e2e        # generate the cross-language suite from shared fixtures
alef test-apps  # standalone apps that install the published package and run it

Now a change in the Rust core does not surface as a runtime surprise in one binding. It surfaces as the same failing assertion in every language at once, in CI, before release. The fixtures are the contract, and every SDK is held to it.

Generated files also carry an Alef hash, so alef verify --exit-code fails the build if someone hand-edits a generated binding or forgets to regenerate after a core change. The generated layer stays generated.

The honest tradeoffs

This is not free, and it is not magic.

The core has to be designed for it. A stable, extractable API surface is a constraint you accept up front. Exotic Rust signatures do not always have a clean home in every target language, and the ones that matter live behind adapters rather than leaking into the core.
Idioms live in the adapter, not the core. Async wrappers, context managers, callbacks, error types: these are per-language, generated per-language, and configured in [crates.<language>] sections. Shared semantics stay in the core so they cannot drift; ergonomics stay local.
When codegen is not enough, you extend it. Domain-specific generation (an HTTP service API, a plugin registry) is an Extension trait rather than a fork, shipped as a linked binary, a dynamic library, or template-only blocks in alef.toml.

We build xberg this way because the alternative, sixteen hand-maintained binding layers chasing one moving core, is exactly the kind of work that reads productive and quietly sinks a polyglot project. Generate the boring layer, test it against one fixture set, and spend the human time on the engine that is actually the product.

Alef is MIT and standalone: github.com/xberg-io/alef. If you maintain a Rust library that people keep asking for bindings to, it is worth a look.

Code review can't keep up with AI. Build a verification layer instead.

Na'aman Hirschfeld (Goldziher) — Sun, 05 Jul 2026 10:48:07 +0000

There is a popular argument going around that code review no longer matters. The reasoning is that a model wrote the code, another model can check it, and a human reading the diff is a slow, expensive step you can now skip. I think that gets the problem exactly backwards.

Code review did not stop mattering. It stopped scaling. You can now generate code faster than any human can read it, let alone reason about it. The old model, where an engineer reading the diff is the last line of defense, quietly broke the moment a single afternoon started producing more change than a person can hold in their head. Skipping review does not fix that. It just removes the one check that was already failing and puts nothing in its place.

The answer is to move verification onto things that keep up. I have been building xberg, a document-intelligence engine with a Rust core and a dozen language SDKs in one repo, and this is the setup that lets me ship at generation speed without shipping garbage.

Static analysis and types do the cheap work

The first layer is everything a machine can check without running your program: linting, static analysis, type checks. This is the cheapest verification you will ever buy, and it catches whole classes of errors before a test ever runs. A type error, an unused binding, a dangerous cast, a lint rule you agreed to as a team, none of these need a human to notice.

For a polyglot repo this is its own project. I ended up building polylint, two self-contained Rust binaries driven by one config, because wiring a separate linter and formatter per language into pre-commit was itself a maintenance tax. The point is not the tool. The point is that mechanical checks should run on every change, locally and in CI, and they should be fast enough that nobody is tempted to skip them.

Comprehensive tests prove behavior

Static analysis tells you the code is well formed. It does not tell you the code is correct. That is what tests are for, and at generation speed the test suite is doing far more of the reviewing than any human is.

The tests that earn their keep here are end-to-end and integration tests. Unit tests are useful, but a wall of them can be green while the system as a whole does the wrong thing. What you want is tests that exercise real paths through the software the way a user would: real inputs, real boundaries, real failure modes. When I add a format to xberg's extraction pipeline, the test that matters is the one that runs a real file of that format end to end and checks the output, not the one that mocks the parser and asserts it was called.

Comprehensive coverage of behavior is what lets you accept a large generated change with confidence. The suite is the reviewer that never gets tired and never skims.

Agents review against written guidelines

Here is where the "another model can check it" idea is actually right, as long as you give it something to check against. An agent doing a first-pass review is fast, tireless, and consistent, but only if the standard it reviews against is explicit and written down. Vague instructions produce vague reviews.

So write the guidelines. The conventions, the security rules, the patterns you want and the ones you have banned, the things a reviewer on your team would flag. I keep those rules in one place and generate the per-tool configs from it with ai-rulez, so Cursor, Claude, and Copilot all review against the same standard instead of three slightly different ones. An agent with clear guidelines catches a real fraction of what a human reviewer would, at a speed a human never will, and it frees the human to look at the things that actually need judgment.

Then you actually test it, by hand

None of the above removes the need to run the software yourself. This is the step people are most tempted to skip now, and it is the one that catches what everything else misses.

A green pipeline is a signal. It is not proof that your software does what you think it does. Automated tests only check the things you thought to check. Running the thing yourself, exercising the real paths, poking at the edges, is how you find the assumption the whole test suite quietly shares with the bug. QA is not a box you tick after the checks pass. At generation speed it becomes one of the highest-leverage things an engineer does, because it is the only layer that brings human judgment to bear on software no human wrote line by line.

Teams that treat QA as a core engineering function, rather than a phase they graduated out of, are the ones this speed will not break.

Open source makes the code better, for free

There is one more layer, and it is the strongest one I know of. Open source your code and it gets tested at a scale you could never reproduce in-house. Thousands of people run it in environments you never imagined, on inputs you never thought to generate, and every bug they hit and report comes back and makes the code more robust. No internal test suite competes with real usage at that volume.

Auditability and good citizenship are real, but the benefit that changes the code itself is simpler. Public code is massively tested code, and massively tested code is better code. Every one of xberg's libraries is open source partly for this reason.

The shape of it

Put together, the verification layer looks like this: static analysis and types catch the cheap errors, comprehensive end-to-end and integration tests prove the behavior, agents do first-pass review against written guidelines, you test it by hand to catch what the machines missed, and open source turns your users into the largest test suite you will ever have.

Code review did not become optional. It became a system instead of a person. Build that system, and you can move as fast as the models let you without lying to yourself about whether it works.

Deleting the toolchain: linting a polyglot monorepo with one Rust binary

Na'aman Hirschfeld (Goldziher) — Sat, 04 Jul 2026 08:02:42 +0000

A repository that mixes languages pays a tax most single-language repos never see. Python wants ruff, and ruff wants a Python. JavaScript and TypeScript want oxlint or prettier, and those want a Node. Shell wants shfmt and shellcheck. Then taplo for TOML, rumdl for Markdown, sqruff for SQL, hadolint for Dockerfiles, and one more for whatever you added last month. Each tool is installed separately, pinned separately, and wired into .pre-commit-config.yaml separately.

The result is slow setup, version drift, CI images fat with language runtimes you only need for linting, and a hook stack that breaks the day a system dependency goes missing or bumps a version. I hit this hard while building xberg, a document-intelligence engine with a Rust core and a dozen language SDKs in one repo, which is where the itch to fix it came from.

polylint is the fix I landed on. It ships two self-contained Rust binaries, poly lint and poly fmt, driven by one poly.toml. No language runtime is required on the default path. The goal was not to add one more tool beside the stack. It was to delete the stack.

What using it looks like

$ poly fmt --check
would format crates/example/src/main.rs

$ poly fmt --fix
formatted 1 file

$ poly lint --format toon
path: crates/example/src/main.rs
diagnostics[0]: engine=ruff, code=F401, severity=warning, title="`os` imported but unused"

$ poly hooks install
installed git hooks: pre-commit, commit-msg

poly fmt is a dry run by default, which is what you want in CI. Add --fix to write changes. poly hooks install wires the git hooks once, and lint, format, and commit checks then run on every git commit.

How it avoids the runtimes

There are three tiers, and a file falls through them until something can handle it.

Curated Rust backends, in process. The Rust ecosystem has grown real, high-quality linters and formatters that are libraries, not just CLIs. polylint links them directly: oxc for JS/TS, ruff internals for Python, taplo for TOML, rumdl for Markdown, sqruff for SQL, malva and markup_fmt for CSS and HTML, mago for PHP. Every backend implements one trait and returns one diagnostic shape, so reporting, caching, and the MCP server stay uniform no matter which tool did the work.

pub trait Engine: Send + Sync {
    fn name(&self) -> &'static str;
    fn languages(&self) -> &[Language];
    fn capabilities(&self) -> Capabilities;   // lint / format / fix
    fn version(&self) -> &str;                 // folded into the cache key
    fn lint(&self, src: &SourceFile, cfg: &EngineConfig) -> Result<Vec<Diagnostic>>;
    fn format(&self, src: &SourceFile, cfg: &EngineConfig) -> Result<FormatOutput>;
}

A tree-sitter fallback for everything else. For languages without a dedicated backend, polylint uses tree-sitter-language-pack, parses the source, and re-emits it with structural reindentation and whitespace normalization. This covers hundreds of grammars, including Java, using each language's indents.scm queries. So you get consistent indentation and clean whitespace on a Java file with no JDK on the machine. Grammars where whitespace carries meaning are left untouched on purpose, since a reindent there would corrupt the file rather than tidy it.

An opt-in catalog for the rest. Some tools have no Rust equivalent yet. For those, polylint vendors a tool catalog derived from the mdsf project, which maps a tool to its binary, argument vector, and languages. It is data only, so polylint does not depend on or shell out to mdsf itself. Each catalog tool is probed on PATH and skipped when absent, so enabling one never makes a run fail on a machine that lacks it.

Configuration for all of this lives in one file:

[defaults]
line_length = 120
line_ending = "lf"

[lint.python.ruff]
select = ["E", "F", "W"]

[per-file-ignores]
"tests/**" = ["F401"]

[hooks.builtin]
polylint = true
polyfmt = true
commit = { stages = ["commit-msg"] }
file_safety = true

That [hooks] table replaces .pre-commit-config.yaml outright. polylint dogfoods it: the commit-msg hook runs Conventional Commit checks and strips AI-attribution trailers, and the file-safety builtin catches merge-conflict markers, large files, leaked private keys, and case collisions, all in pure Rust.

Why it is fast enough to run on every commit

Two things do the work. Nothing spawns a subprocess per file on the default path, so there is no process-startup cost multiplied across a large tree, and the whole run is a rayon parallel loop over files.

On top of that sits a blake3 result cache. The cache key is not just the file contents. It folds in the engine name, the engine's own version, and the resolved config, alongside the file bytes:

key = blake3(namespace, engine, engine_version, resolved_config, file_bytes)

That detail matters more than it looks. A plain content hash would happily serve you a stale result after you upgraded a formatter or edited a rule, because the bytes did not change. Folding the tool version and config into the key means an upgrade or a config edit invalidates exactly the affected entries and nothing else.

In CI, the binary itself is cached by version, OS, and architecture, so a run downloads it once and reuses it until you bump the version.

name: Lint and Format
on: [push, pull_request]
jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4      - uses: Goldziher/polylint@v1      - run: poly lint .
      - run: poly fmt --check .

The tradeoffs

I would rather name these than have you find them.

The binary is large, around 70 MB per platform, because it bundles several big parsers and linters. For a tool you install once and cache in CI, I think that is the right trade against carrying zero system dependencies, but if you expected a 3 MB download, that is not this.

The coverage is uneven by design. Some backends lint only, some format only, so it is not true that every language gets every capability. The tree-sitter tier is a formatter, not a linter, and it deliberately does nothing on whitespace-sensitive grammars rather than risk a destructive rewrite.

And it is not fully subprocess-free in every case. rustfmt and gofmt are used when present, because reimplementing them in process is not worth it, and a few other native formatters are opt-in. The default path stays runtime-free; those are the documented exceptions.

Try it

polylint is MIT licensed.

pip install polylint
# or
brew install Goldziher/tap/polylint

Source and docs are at github.com/Goldziher/polylint. If you run a polyglot repo and your pre-commit config has quietly become its own maintenance project, point poly lint at it and see what falls out.

When an LLM response fails validation, feed the error back into the retry

Na'aman Hirschfeld (Goldziher) — Thu, 02 Jul 2026 05:17:08 +0000

If you ask an LLM for structured output and validate it against a schema, you already know the failure
mode: most of the time it is fine, and every so often it hands back something that does not parse or
misses a required field. The usual reflex is to wrap the call in a retry and move on.

The problem is that a plain retry is the same prompt, the same temperature, roughly the same odds. You
are paying for another round and hoping the dice land differently.

There is a better move, and it is almost free to add: when validation fails, put the validation error
and the model's own bad output back into the next prompt, and ask it to fix that specific thing.

Here is the core of the loop I used for this on a RAG platform I built (trimmed to the essential path):

while attempts < max_attempts:
    try:
        msgs = [messages] if isinstance(messages, str) else messages
        if error_message:
            msgs = [*msgs, error_message]        # last attempt's error rides along
        response = await make_completion_request(..., messages=msgs)
        if validator:
            validator(response)                  # raises ValidationError on bad output
        return response
    except ValidationError as e:
        attempts += 1
        error_message = f"""
        The last response from the API failed validation due to the following error:
            <error>{format_error_for_llm(e)}</error>
        Your task is to fix the error and return the corrected response data:
            <data>{serialize(response).decode()}</data>
        """
        response = None

Two details do the work:

The error is described for the model, not for a log. format_error_for_llm turns the raw validation exception into a plain instruction ("field X must be an integer, you sent a string"). The model is good at patching a concrete, named mistake; it is bad at guessing why an opaque retry keeps failing.
You hand back its own previous output as the thing to correct. It is not regenerating from scratch, it is editing. That keeps the parts that were already right and usually fixes the one field that was wrong on the first pass.

The tradeoffs, because there always are some:

It costs an extra call on a bad response, and the follow-up prompt is longer (it carries the error plus the prior payload). On a schema that fails often, that adds up. Cap the attempts.
It only works when the bad response is parseable enough to serialize back into the prompt. Truly empty or truncated output has nothing to correct, so you still need a normal retry underneath.
Prompt semantics do not always transfer if you also fail over between providers mid-loop. If you do that, do not count a provider swap as a real attempt.

That is the whole idea. It is not a framework, it is about ten lines around a call you already have. If
you are generating structured output at any volume, it turns a chunk of your "model was flaky" retries
into first-try-after-feedback successes.

Benchmark-Driven Development: let agents build the harness you never had time for

Na'aman Hirschfeld (Goldziher) — Tue, 30 Jun 2026 04:41:22 +0000

Most teams ship on two signals: does it compile, and do the tests pass. Both are correctness signals. Neither tells you whether the thing is fast, whether it got slower this week, or whether the output is actually good against some ground truth. We close that gap with a third signal, and we let it drive the work. We call it benchmark-driven development.

The loop is simple:

Write the plan and the spec first. Decide what the code should do before writing it.
Use TDD to drive it to correct. Red, green, refactor, the usual.
Test it for real with proper benchmarks, against ground truth, with profiling.

Correctness and performance are both requirements. So both get measured, on every change, not once at the end when a customer complains.

Why benchmarks used to be a luxury

Everyone agrees benchmarks are good. Almost no one builds a serious one. The reason is cost. A real harness is its own project: a fixture corpus with trustworthy ground truth, scoring that reflects quality and not just a checksum, resource sampling, a way to compare across implementations, and CI plumbing to run it all and publish results. That is weeks of work that ships no features. So it gets cut, and teams fall back to a stopwatch around a hot loop and a vibe about whether output looks right.

This is the part that changed. With an agent in the loop, the harness is no longer a quarter of engineering budget. It is a few focused sessions. You describe the corpus, the metrics, and the comparison you want, and you iterate on the harness the same way you iterate on a feature. The economics flipped. The thing that was too expensive to justify is now the cheap part.

A concrete harness

To make this concrete, here is the benchmark harness we run for xberg, our document-extraction engine. It is a Rust CLI, and it does the following.

It compares 13 language bindings of our own engine against 7 external reference frameworks, across a corpus of 318 fixtures that carry markdown ground truth, spanning 17 formats: PDF, HTML, DOCX, ODT, RTF, XLSX, CSV, EPUB, PPTX, and more.

It scores quality two ways, because text and structure fail differently:

Text F1 (TF1): token-level bag-of-words F1 between extracted text and ground truth, with a separate numeric-token score for number-heavy documents like financial and scientific PDFs.
Structural F1 (SF1): block-level matching between extracted markdown and ground-truth markdown. Headings, code blocks, formulas, tables, list items, and images are weighted by how much they matter, matched greedily with fuzzy cross-type compatibility, and scored for ordering with a longest-increasing-subsequence pass.

When markdown ground truth is present it combines them:

quality_score = 0.5 * f1_text + 0.2 * f1_numeric + 0.3 * f1_layout

It samples CPU and memory during each run, and behind a feature flag it generates flamegraphs so a regression in latency points you straight at the function that caused it. It runs each binding in both single-file mode for fair latency and batch mode for throughput. And it runs as a CI matrix: build, validate ground truth, fan out one job per binding, gate, run the external frameworks, then consolidate everything into percentiles and publish the aggregate as a release artifact.

None of that is exotic. All of it used to be too much work to justify for anything but a flagship. That is the point.

What "real benchmarks" actually means

A benchmark you can drive development with has to clear a higher bar than a stopwatch around a hot loop. Four properties matter.

Ground truth you trust. A number is only as good as what it is measured against. We keep ground truth in version control, validate its integrity on every CI run, and clean known artifacts so the target does not drift silently. If you cannot defend the reference, you cannot defend the score.

Metrics that reflect the goal. "Bytes match" is not quality for document extraction. A table flattened into a paragraph can have high text overlap and be useless. That is why structure gets its own score and its own weights. Pick metrics that punish the failures you actually care about.

Resource truth, not just wall-clock. Throughput hides memory blowups. We sample CPU and memory and keep flamegraphs around so "it got slower" turns into "this function got slower."

Comparison and regression gates. A score in isolation is trivia. The harness compares pipelines and implementations against each other and against history, and it can fail the build on a quality regression. That is what makes it a development driver rather than a report.

How to adopt it without a big-bang rewrite

You do not need 318 fixtures on day one.

Start with ten fixtures and one honest metric. Ten documents you understand beat a thousand you cannot defend.
Write the spec, then the failing test, then make it pass. Keep TDD for correctness. The benchmark is the layer on top, not a replacement.
Add a benchmark the first time you guess about performance. The moment you say "this is probably fine," measure it instead.
Put it in CI and gate on regressions once the metric is stable. A benchmark nobody runs is documentation.
Lean on an agent for the unglamorous parts: fixture wrangling, scoring code, CI matrix, percentile aggregation. This is exactly the work that used to make a harness too expensive, and it is exactly the work an agent is good at.

The payoff

When correctness and speed are both measured on every change, you stop guessing and you stop arguing from intuition. A refactor either held the line on quality and latency or it did not, and the harness tells you which before the diff merges. Spec-driven, TDD for correctness, real benchmarks for the rest. Build the harness you never had time for, because now you do.

Kreuzberg v4.0.0-RC.8 is Available

Na'aman Hirschfeld (Goldziher) — Mon, 15 Dec 2025 13:06:14 +0000

Hi Peeps,

I'm excited to announce that Kreuzberg v4.0.0 is coming very soon. We will release v4.0.0 at the beginning of next year - in just a couple of weeks time. For now, v4.0.0-rc.8 has been released to all channels.

What is Kreuzberg?

Kreuzberg is a document intelligence toolkit for extracting text, metadata, tables, images, and structured data from 56+ file formats. It was originally written in Python (v1-v3), where it demonstrated strong performance characteristics compared to alternatives in the ecosystem.

What's new in V4?

A Complete Rust Rewrite with Polyglot Bindings

The new version of Kreuzberg represents a massive architectural evolution. Kreuzberg has been completely rewritten in Rust - leveraging Rust's memory safety, zero-cost abstractions, and native performance. The new architecture consists of a high-performance Rust core with native bindings to multiple languages. That's right - it's no longer just a Python library.

Kreuzberg v4 is now available for 7 languages across 8 runtime bindings:

Rust (native library)
Python (PyO3 native bindings)
TypeScript - Node.js (NAPI-RS native bindings) + Deno/Browser/Edge (WASM)
Ruby (Magnus FFI)
Java 25+ (Panama Foreign Function & Memory API)
C# (P/Invoke)
Go (cgo bindings)

Post v4.0.0 roadmap includes:

PHP
Elixir (via Rustler - with Erlang and Gleam interop)

Additionally, it's available as a CLI (installable via cargo or homebrew), HTTP REST API server, Model Context Protocol (MCP) server for Claude Desktop/Continue.dev, and as public Docker images.

Why the Rust Rewrite? Performance and Architecture

The Rust rewrite wasn't just about performance - though that's a major benefit. It was an opportunity to fundamentally rethink the architecture:

Architectural improvements:

Zero-copy operations via Rust's ownership model
True async concurrency with Tokio runtime (no GIL limitations)
Streaming parsers for constant memory usage on multi-GB files
SIMD-accelerated text processing for token reduction and string operations
Memory-safe FFI boundaries for all language bindings
Plugin system with trait-based extensibility

v3 vs v4: What Changed?

Aspect	v3 (Python)	v4 (Rust Core)
Core Language	Pure Python	Rust 2024 edition
File Formats	30-40+ (via Pandoc)	56+ (native parsers)
Language Support	Python only	7 languages (Rust/Python/TS/Ruby/Java/Go/C#)
Dependencies	Requires Pandoc (system binary)	Zero system dependencies (all native)
Embeddings	Not supported	✓ FastEmbed with ONNX (3 presets + custom)
Semantic Chunking	Via semantic-text-splitter library	✓ Built-in (text + markdown-aware)
Token Reduction	Built-in (TF-IDF based)	✓ Enhanced with 3 modes
Language Detection	Optional (fast-langdetect)	✓ Built-in (68 languages)
Keyword Extraction	Optional (KeyBERT)	✓ Built-in (YAKE + RAKE algorithms)
OCR Backends	Tesseract/EasyOCR/PaddleOCR	Same + better integration
Plugin System	Limited extractor registry	Full trait-based (4 plugin types)
Page Tracking	Character-based indices	Byte-based with O(1) lookup
Servers	REST API (Litestar)	HTTP (Axum) + MCP + MCP-SSE
Installation Size	~100MB base	16-31 MB complete
Memory Model	Python heap management	RAII with streaming
Concurrency	asyncio (GIL-limited)	Tokio work-stealing

Replacement of Pandoc - Native Performance

Kreuzberg v3 relied on Pandoc - an amazing tool, but one that had to be invoked via subprocess because of its GPL license. This had significant impacts:

v3 Pandoc limitations:

System dependency (installation required)
Subprocess overhead on every document
No streaming support
Limited metadata extraction
~500MB+ installation footprint

v4 native parsers:

Zero external dependencies - everything is native Rust
Direct parsing with full control over extraction
Substantially more metadata extracted (e.g., DOCX document properties, section structure, style information)
Streaming support for massive files (tested on multi-GB XML documents with stable memory)
Example: PPTX extractor is now a fully streaming parser capable of handling gigabyte-scale presentations with constant memory usage and high throughput

New File Format Support

v4 expanded format support from ~20 to 56+ file formats, including:

Added legacy format support:

.doc (Word 97-2003)
.ppt (PowerPoint 97-2003)
.xls (Excel 97-2003)
.eml (Email messages)
.msg (Outlook messages)

Added academic/technical formats:

LaTeX (.tex)
BibTeX (.bib)
Typst (.typ)
JATS XML (scientific articles)
DocBook XML
FictionBook (.fb2)
OPML (.opml)

Better Office support:

XLSB, XLSM (Excel binary/macro formats)
Better structured metadata extraction from DOCX/PPTX/XLSX
Full table extraction from presentations
Image extraction with deduplication

New Features: Full Document Intelligence Solution

The v4 rewrite was also an opportunity to close gaps with commercial alternatives and add features specifically designed for RAG applications and LLM workflows:

1. Embeddings (NEW)

FastEmbed integration with full ONNX Runtime acceleration
Three presets: "fast" (384d), "balanced" (512d), "quality" (768d/1024d)
Custom model support (bring your own ONNX model)
Local generation (no API calls, no rate limits)
Automatic model downloading and caching
Per-chunk embedding generation

from kreuzberg import ExtractionConfig, EmbeddingConfig, EmbeddingModelType

config = ExtractionConfig(
    embeddings=EmbeddingConfig(
        model=EmbeddingModelType.preset("balanced"),
        normalize=True
    )
)
result = kreuzberg.extract_bytes(pdf_bytes, config=config)
# result.embeddings contains vectors for each chunk

2. Semantic Text Chunking (NOW BUILT-IN)

Now integrated directly into the core (v3 used external semantic-text-splitter library):

Structure-aware chunking that respects document semantics
Two strategies:
- Generic text chunker (whitespace/punctuation-aware)
- Markdown chunker (preserves headings, lists, code blocks, tables)
Configurable chunk size and overlap
Unicode-safe (handles CJK, emojis correctly)
Automatic chunk-to-page mapping
Per-chunk metadata with byte offsets

3. Byte-Accurate Page Tracking (BREAKING CHANGE)

This is a critical improvement for LLM applications:

v3: Character-based indices (char_start/char_end) - incorrect for UTF-8 multi-byte characters
v4: Byte-based indices (byte_start/byte_end) - correct for all string operations

Additional page features:

O(1) lookup: "which page is byte offset X on?" → instant answer
Per-page content extraction
Page markers in combined text (e.g., --- Page 5 ---)
Automatic chunk-to-page mapping for citations

4. Enhanced Token Reduction for LLM Context

Enhanced from v3 with three configurable modes to save on LLM costs:

Light mode: ~15% reduction (preserve most detail)
Moderate mode: ~30% reduction (balanced)
Aggressive mode: ~50% reduction (key information only)

Uses TF-IDF sentence scoring with position-aware weighting and language-specific stopword filtering. SIMD-accelerated for improved performance over v3.

5. Language Detection (NOW BUILT-IN)

68 language support with confidence scoring
Multi-language detection (documents with mixed languages)
ISO 639-1 and ISO 639-3 code support
Configurable confidence thresholds

6. Keyword Extraction (NOW BUILT-IN)

Now built into core (previously optional KeyBERT in v3):

YAKE (Yet Another Keyword Extractor): Unsupervised, language-independent
RAKE (Rapid Automatic Keyword Extraction): Fast statistical method
Configurable n-grams (1-3 word phrases)
Relevance scoring with language-specific stopwords

7. Plugin System (NEW)

Four extensible plugin types for customization:

DocumentExtractor - Custom file format handlers
OcrBackend - Custom OCR engines (integrate your own Python models)
PostProcessor - Data transformation and enrichment
Validator - Pre-extraction validation

Plugins defined in Rust work across all language bindings. Python/TypeScript can define custom plugins with thread-safe callbacks into the Rust core.

8. Production-Ready Servers (NEW)

HTTP REST API: Production-grade Axum server with OpenAPI docs
MCP Server: Direct integration with Claude Desktop, Continue.dev, and other MCP clients
MCP-SSE Transport (RC.8): Server-Sent Events for cloud deployments without WebSocket support
All three modes support the same feature set: extraction, batch processing, caching

Performance: Benchmarked Against the Competition

We maintain continuous benchmarks comparing Kreuzberg against the leading OSS alternatives:

Benchmark Setup

Platform: Ubuntu 22.04 (GitHub Actions)
Test Suite: 30+ documents covering all formats
Metrics: Latency (p50, p95), throughput (MB/s), memory usage, success rate
Competitors: Apache Tika, Docling, Unstructured, MarkItDown

How Kreuzberg Compares

Installation Size (critical for containers/serverless):

Kreuzberg: 16-31 MB complete (CLI: 16 MB, Python wheel: 22 MB, Java JAR: 31 MB - all features included)
MarkItDown: ~251 MB installed (58.3 KB wheel, 25 dependencies)
Unstructured: ~146 MB minimal (open source base) - several GB with ML models
Docling: ~1 GB base, 9.74GB Docker image (includes PyTorch CUDA)
Apache Tika: ~55 MB (tika-app JAR) + dependencies
GROBID: 500MB (CRF-only) to 8GB (full deep learning)

Performance Characteristics:

Library	Speed	Accuracy	Formats	Installation	Use Case
Kreuzberg	⚡ Fast (Rust-native)	Excellent	56+	16-31 MB	General-purpose, production-ready
Docling	⚡ Fast (3.1s/pg x86, 1.27s/pg ARM)	Best	7+	1-9.74 GB	Complex documents, when accuracy > size
GROBID	⚡⚡ Very Fast (10.6 PDF/s)	Best	PDF only	0.5-8 GB	Academic/scientific papers only
Unstructured	⚡ Moderate	Good	25-65+	146 MB-several GB	Python-native LLM pipelines
MarkItDown	⚡ Fast (small files)	Good	11+	~251 MB	Lightweight Markdown conversion
Apache Tika	⚡ Moderate	Excellent	1000+	~55 MB	Enterprise, broadest format support

Kreuzberg's sweet spot:

Smallest full-featured installation: 16-31 MB complete (vs 146 MB-9.74 GB for competitors)
5-15x smaller than Unstructured/MarkItDown, 30-300x smaller than Docling/GROBID
Rust-native performance without ML model overhead
Broad format support (56+ formats) with native parsers
Multi-language support unique in the space (7 languages vs Python-only for most)
Production-ready with general-purpose design (vs specialized tools like GROBID)

Is Kreuzberg a SaaS Product?

No. Kreuzberg is and will remain MIT-licensed open source.

However, we are building Kreuzberg.cloud - a commercial SaaS and self-hosted document intelligence solution built on top of Kreuzberg. This follows the proven open-core model: the library stays free and open, while we offer a cloud service for teams that want managed infrastructure, APIs, and enterprise features.

Will Kreuzberg become commercially licensed? Absolutely not. There is no BSL (Business Source License) in Kreuzberg's future. The library was MIT-licensed and will remain MIT-licensed. We're building the commercial offering as a separate product around the core library, not by restricting the library itself.

Target Audience

Any developer or data scientist who needs:

Document text extraction (PDF, Office, images, email, archives, etc.)
OCR (Tesseract, EasyOCR, PaddleOCR)
Metadata extraction (authors, dates, properties, EXIF)
Table and image extraction
Document pre-processing for RAG pipelines
Text chunking with embeddings
Token reduction for LLM context windows
Multi-language document intelligence in production systems

Ideal for:

RAG application developers
Data engineers building document pipelines
ML engineers preprocessing training data
Enterprise developers handling document workflows
DevOps teams needing lightweight, performant extraction in containers/serverless

Comparison with Alternatives

Open Source Python Libraries

Unstructured.io

Strengths: Established, modular, broad format support (25+ open source, 65+ enterprise), LLM-focused, good Python ecosystem integration
Trade-offs: Python GIL performance constraints, 146 MB minimal installation (several GB with ML models)
License: Apache-2.0
When to choose: Python-only projects where ecosystem fit > performance

MarkItDown (Microsoft)

Strengths: Fast for small files, Markdown-optimized, simple API
Trade-offs: Limited format support (11 formats), less structured metadata, ~251 MB installed (despite small wheel), requires OpenAI API for images
License: MIT
When to choose: Markdown-only conversion, LLM consumption

Docling (IBM)

Strengths: Excellent accuracy on complex documents (97.9% cell-level accuracy on tested sustainability report tables), state-of-the-art AI models for technical documents
Trade-offs: Massive installation (1-9.74 GB), high memory usage, GPU-optimized (underutilized on CPU)
License: MIT
When to choose: Accuracy on complex documents > deployment size/speed, have GPU infrastructure

Open Source Java/Academic Tools

Apache Tika

Strengths: Mature, stable, broadest format support (1000+ types), proven at scale, Apache Foundation backing
Trade-offs: Java/JVM required, slower on large files, older architecture, complex dependency management
License: Apache-2.0
When to choose: Enterprise environments with JVM infrastructure, need for maximum format coverage

GROBID

Strengths: Best-in-class for academic papers (F1 0.87-0.90), extremely fast (10.6 PDF/sec sustained), proven at scale (34M+ documents at CORE)
Trade-offs: Academic papers only, large installation (500MB-8GB), complex Java+Python setup
License: Apache-2.0
When to choose: Scientific/academic document processing exclusively

Commercial APIs

There are numerous commercial options from startups (LlamaIndex, Unstructured.io paid tiers) to big cloud providers (AWS Textract, Azure Form Recognizer, Google Document AI). These are not OSS but offer managed infrastructure.

Kreuzberg's position: As an open-source library, Kreuzberg provides a self-hosted alternative with no per-document API costs, making it suitable for high-volume workloads where cost efficiency matters.

Community & Resources

GitHub: Star us at https://github.com/kreuzberg-dev/kreuzberg
Discord: Join our community server at discord.gg/pXxagNK2zN
Subreddit: Join the discussion at r/kreuzberg_dev
Documentation: kreuzberg.dev

We'd love to hear your feedback, use cases, and contributions!

TL;DR: Kreuzberg v4 is a complete Rust rewrite of a document intelligence library, offering native bindings for 7 languages (8 runtime targets), 56+ file formats, Rust-native performance, embeddings, semantic chunking, and production-ready servers - all in a 16-31 MB complete package (5-15x smaller than alternatives). Releasing January 2026. MIT licensed forever.

I benchmarked 4 Python text extraction libraries (2025)

Na'aman Hirschfeld (Goldziher) — Sun, 06 Jul 2025 10:25:38 +0000

TL;DR: Comprehensive benchmarks of Kreuzberg, Docling, MarkItDown, and Unstructured across 94 real-world documents. Results might surprise you.

📊 Live Results: https://goldziher.github.io/python-text-extraction-libs-benchmarks/

Context

As the author of Kreuzberg, I wanted to create an honest, comprehensive benchmark of Python text extraction libraries. No cherry-picking, no marketing fluff - just real performance data across 94 documents (~210MB) ranging from tiny text files to 59MB academic papers.

Full disclosure: I built Kreuzberg, but these benchmarks are automated, reproducible, and the methodology is completely open-source.

🔬 What I Tested

Libraries Benchmarked:

Kreuzberg (71MB, 20 deps) - My library
Docling (1,032MB, 88 deps) - IBM's ML-powered solution
MarkItDown (251MB, 25 deps) - Microsoft's Markdown converter
Unstructured (146MB, 54 deps) - Enterprise document processing

Test Coverage:

94 real documents: PDFs, Word docs, HTML, images, spreadsheets
5 size categories: Tiny (<100KB) to Huge (>50MB)
6 languages: English, Hebrew, German, Chinese, Japanese, Korean
CPU-only processing: No GPU acceleration for fair comparison
Multiple metrics: Speed, memory usage, success rates, installation sizes

🏆 Results Summary

Speed Champions 🚀

Kreuzberg: 35+ files/second, handles everything
Unstructured: Moderate speed, excellent reliability
MarkItDown: Good on simple docs, struggles with complex files
Docling: Often 60+ minutes per file (!!)

Installation Footprint 📦

Kreuzberg: 71MB, 20 dependencies ⚡
Unstructured: 146MB, 54 dependencies
MarkItDown: 251MB, 25 dependencies (includes ONNX)
Docling: 1,032MB, 88 dependencies 🐘

Reality Check ⚠️

Docling: Frequently fails/times out on medium files (>1MB)
MarkItDown: Struggles with large/complex documents (>10MB)
Kreuzberg: Consistent across all document types and sizes
Unstructured: Most reliable overall (88%+ success rate)

🎯 When to Use What

⚡ Kreuzberg (Disclaimer: I built this)

Best for: Production workloads, edge computing, AWS Lambda
Why: Smallest footprint (71MB), fastest speed, handles everything
Bonus: Both sync/async APIs with OCR support

🏢 Unstructured

Best for: Enterprise applications, mixed document types
Why: Most reliable overall, good enterprise features
Trade-off: Moderate speed, larger installation

📝 MarkItDown

Best for: Simple documents, LLM preprocessing
Why: Good for basic PDFs/Office docs, optimized for Markdown
Limitation: Fails on large/complex files

🔬 Docling

Best for: Research environments (if you have patience)
Why: Advanced ML document understanding
Reality: Extremely slow, frequent timeouts, 1GB+ install

📈 Key Insights

Installation size matters: Kreuzberg's 71MB vs Docling's 1GB+ makes a huge difference for deployment
Performance varies dramatically: 35 files/second vs 60+ minutes per file
Document complexity is crucial: Simple PDFs vs complex layouts show very different results
Reliability vs features: Sometimes the simplest solution works best

🔧 Methodology

Automated CI/CD: GitHub Actions run benchmarks on every release
Real documents: Academic papers, business docs, multilingual content
Multiple iterations: 3 runs per document, statistical analysis
Open source: Full code, test documents, and results available
Memory profiling: psutil-based resource monitoring
Timeout handling: 5-minute limit per extraction

🤔 Why I Built This

Working on Kreuzberg, I worked on performance and stability, and then wanted a tool to see how it measures against other frameworks - which I could also use to further develop and improve Kreuzberg itself. I therefore created this benchmark. Since it was fun, I invested some time to pimp it out:

Uses real-world documents, not synthetic tests
Tests installation overhead (often ignored)
Includes failure analysis (libraries fail more than you think)
Is completely reproducible and open
Updates automatically with new releases

📊 Data Deep Dive

The interactive dashboard shows some fascinating patterns:

Kreuzberg dominates on speed and resource usage across all categories
Unstructured excels at complex layouts and has the best reliability
MarkItDown is useful for simple docs shows in the data
Docling's ML models create massive overhead for most use cases making it a hard sell

🚀 Try It Yourself

git clone https://github.com/Goldziher/python-text-extraction-libs-benchmarks.git
cd python-text-extraction-libs-benchmarks
uv sync --all-extras
uv run python -m src.cli benchmark --framework kreuzberg_sync --category small

Or just check the live results: https://goldziher.github.io/python-text-extraction-libs-benchmarks/

🔗 Links

📊 Live Benchmark Results: https://goldziher.github.io/python-text-extraction-libs-benchmarks/
📁 Benchmark Repository: https://github.com/Goldziher/python-text-extraction-libs-benchmarks
⚡ Kreuzberg (my library): https://github.com/Goldziher/kreuzberg
🔬 Docling: https://github.com/DS4SD/docling
📝 MarkItDown: https://github.com/microsoft/markitdown
🏢 Unstructured: https://github.com/Unstructured-IO/unstructured

🤝 Discussion

What's your experience with these libraries? Any others I should benchmark? I tried benchmarking marker, but the setup required a GPU.

Some important points regarding how I used these benchmarks for Kreuzberg:

I fine tuned the default settings for Kreuzberg.
I updated our docs to give recommendations on different settings for different use cases. E.g. Kreuzberg can actually get to 75% reliability, with about 15% slow-down.
I made a best effort to configure the frameworks following the best practices of their docs and using their out of the box defaults. If you think something is off or needs adjustment, feel free to let me know here or open an issue in the repository.