DEV Community: Na'aman Hirschfeld (Goldziher)

Debugging basemind with Claude Code

Na'aman Hirschfeld (Goldziher) — Wed, 08 Jul 2026 16:53:10 +0000

basemind is my code-intelligence tool: a pure-Rust code map and scanner, tree-sitter across 300+ languages, a content-addressed blob store, a Fjall-backed inverted index. I harden it by pointing it at a repo big enough to hurt, a 20M-line, 200k-commit monorepo, and watching what it does.

I did this debugging session with Claude Code. It started as "which version is installed?" and ended with a 7x speedup, two upstream PRs, and a feature turned off. Here is how I worked, and the methods that made it effective.

The symptom is not the bug

The report: basemind's cache grew from 1.8 GB to 4 GB while CPU pinned for a long time. I did not open the scanner code. I looked at the live system. ps and lsof found a process at ~70% CPU holding the index open, and it was a serve process, not a scan. Different bug entirely.

That process was an old release. So I had Claude dig through git instead of the debugger: git log on the watcher, then git merge-base --is-ancestor and commit dates. A commit titled "fix: stop watcher CPU runaway" was dated four days after the running binary shipped. The bug (a watcher re-scanning its own writes forever) was already fixed. It was a stale process left over from another editor session.

Two rules, both applied before reading a line of source:

Verify the running artifact, not a proxy. Homebrew, the PATH, and the editor plugin each reported a different version. Find the one that actually runs.
"Live bug or old build?" is the first question. Dating the fix against the running version can end an investigation before it starts.

Measurement tells a scan from a leak

I killed the ghost, and a fresh scan appeared at 488% CPU, nearly five cores. That is rayon pegging every core, and it is indistinguishable from a runaway loop on a single reading.

So I sampled three signals over time: CPU%, cache size, blob count. Disk climbed then plateaued. Blob writes burst then stopped. CPU fell to one core. That is a one-time rebuild converging, not a leak. A leak keeps allocating, a loop keeps writing, a scan converges. The derivative is the diagnosis, not the number.

To watch without babysitting, I had Claude write background monitors: shell loops sampling every 30s, printing only on a state transition. One fired on "SCAN COMPLETE + IDLE," another on "0% CPU and no disk growth for 4 minutes." The alert encodes the hypothesis: a hang is low CPU and no progress, a healthy scan is high CPU or growing disk.

Isolate the confound

A multi-worktree test then hung: scans wedged at 0% CPU, and lsof on a stuck process itself hung for two minutes. A hanging lsof means a stuck socket. The last log line was Registering VLM OCR backend. The OCR tier made a network call with no timeout, a firewall dropped it, and the connection blocked forever.

I did not debug the hang in place. That tier is opt-in behind a cargo feature, so I built basemind with default features, a binary that cannot make the call, and studied the scanner in isolation. When two failures tangle, change one variable.

Read for the mechanism, then measure the fix

On a clean binary I took a baseline. A fresh worktree scan took 181s, re-parsed all 67,700 files though their blobs already existed, and rebuilt the git-history index per worktree at ~160s. Now I read code, to confirm the mechanism, not guess a patch.

The "unchanged file" fast paths keyed off the per-worktree index, empty on a fresh worktree. So every file fell through to a full re-parse, despite its blob already sitting in the shared store.
The git-history index opened at the worktree's own cache dir, though it derives entirely from the shared .git.

Both fixes followed patterns already in the code, and each got a regression test written to fail first. Re-measured: 181s to 25s, ~7x, all 67,700 files reused, peak RSS about halved. The re-measurement is the point. It turns "should be faster" into "is 7x faster."

Follow the failure across the boundary

The network hang still deserved a fix. I traced it: the OCR and embedding models download through hf_hub, whose builder in 0.4 and 0.5 constructs a ureq agent with no timeout. A firewalled download blocks forever. The root cause was in the dependency.

I forked it and had Claude map every download call site across two crates, then add a watchdog: run the blocking fetch on a detached thread, error on a tunable deadline, degrade by skipping the model instead of hanging. The mechanical multi-file edit went to a subagent with a tight spec. Then I read the diff and reran the build and tests myself. The subagent even corrected my assumption about a type not being Clone. That shipped as an upstream PR plus an honest bug report. Delegate the mechanical work, verify with diffs and reruns, never a summary.

Sometimes the fix is deletion

Monitoring caught the embedding pass stuck on a general-purpose model, an empty model dir with a stale lock, spinning a core. The sharper question was not why it was slow, but whether code embeddings should exist at all.

I traced what gets embedded (symbol, signature, doc, body) against what embeds it (a general-English model), and basemind already builds a BM25 index over the same text. A prose model embeds code weakly, the one real win (natural language to symbol) is already covered by the keyword lane, and the cost is a 100 MB pull plus gigabytes of vectors. Verdict: code embeddings off by default, kept for docs and images. A feature that is off cannot hang, leak, or pin a core.

The methodology

Observe the live system (ps, lsof, du) before reading source.
Separate stale-artifact bugs from live bugs by dating the fix against the running version.
Measure dynamics, not snapshots. The derivative separates converging from leaking.
Encode the hypothesis into a monitor so the machine watches for you.
Isolate confounds by changing one variable.
Read code for the mechanism, then write the smallest fix.
Prove it with a test that fails without the fix, and re-measure the metric you started from.
Follow failures into dependencies, and report them honestly.
Ask whether the costly feature should exist, not just why it broke.
Delegate the mechanical work to an agent, but verify every diff yourself.

What shipped

Worktree cache reuse: cold scan 181s to 25s (~7x), 67,700 files reused, peak RSS about halved.
Code embeddings off by default, per-file embed_exclude, safe preset swapping.
An upstream timeout guardrail in the dependency, plus a bug report.

The method I kept coming back to: verify the artifact, watch the live system, measure the change over time, isolate the confound, read for the mechanism, and prove the fix by the number you started from. basemind lives at github.com/Goldziher/basemind.

One Rust core, native SDKs in 16 languages: how we stopped hand-writing bindings

Na'aman Hirschfeld (Goldziher) — Mon, 06 Jul 2026 08:47:33 +0000

Building xberg, a document-intelligence engine with a Rust core, we needed the same engine callable from Python, TypeScript, Ruby, Go, PHP, and a dozen more. This is how we ship native SDKs in every one of them without hand-writing a single binding layer.

The polyglot binding tax

A Rust core is a good decision right up until you have to ship it to other languages. Then you inherit a second, quieter job: keep a PyO3 wrapper, a NAPI-RS addon, a Magnus extension, a cgo layer, and a handful more in lockstep with a core that moves every week.

That is where polyglot projects rot. The core adds a field, one language's wrapper lags, and nobody notices until a Ruby user files a bug that a Python user never sees. Multiply by sixteen languages and the binding layer becomes the most expensive code in the project, none of which is the actual product.

We did not want to write that code, let alone maintain it. So we generate it.

One config, every target

The tool is alef. It extracts the Rust API surface and emits language-native bindings, package scaffolding, type stubs, docs, and e2e tests from one alef.toml:

[workspace]
languages = ["python", "node", "ffi", "go"]
alef_version = "0.32.7"

[[crates]]
name = "sample_core"
sources = ["src/lib.rs"]
version_from = "Cargo.toml"

alef generate --format   # bindings + type stubs
alef scaffold            # package manifests + native build files
alef verify --exit-code  # fail CI if generated files are stale

Each target gets an idiomatic package, not a lowest-common-denominator shim:

Language	Backend
Python	PyO3 + type stubs
TypeScript / Node	NAPI-RS addon with `.d.ts`
Ruby	Magnus extension
Go	cgo over a generated C FFI layer
PHP	native extension
Elixir / Gleam	Rustler NIF
Swift, Kotlin, C#, Dart, R, Zig, WASM	native bridge per platform

Sixteen languages, eighteen targets (C FFI and a JNI shim round out the list). The Rust author writes Rust; the Python user gets a dataclass with type stubs, the Go user gets a cgo package, and neither knows the other exists.

The part that keeps them honest

Generating bindings is the easy half. The hard half is trust: how do you know all sixteen behave the same after you change the core?

Alef's answer is shared fixtures. You write one set of JSON fixtures describing inputs and expected outputs, and it generates a cross-language e2e suite plus standalone registry-mode test apps from them:

alef e2e        # generate the cross-language suite from shared fixtures
alef test-apps  # standalone apps that install the published package and run it

Now a change in the Rust core does not surface as a runtime surprise in one binding. It surfaces as the same failing assertion in every language at once, in CI, before release. The fixtures are the contract, and every SDK is held to it.

Generated files also carry an Alef hash, so alef verify --exit-code fails the build if someone hand-edits a generated binding or forgets to regenerate after a core change. The generated layer stays generated.

The honest tradeoffs

This is not free, and it is not magic.

The core has to be designed for it. A stable, extractable API surface is a constraint you accept up front. Exotic Rust signatures do not always have a clean home in every target language, and the ones that matter live behind adapters rather than leaking into the core.
Idioms live in the adapter, not the core. Async wrappers, context managers, callbacks, error types: these are per-language, generated per-language, and configured in [crates.<language>] sections. Shared semantics stay in the core so they cannot drift; ergonomics stay local.
When codegen is not enough, you extend it. Domain-specific generation (an HTTP service API, a plugin registry) is an Extension trait rather than a fork, shipped as a linked binary, a dynamic library, or template-only blocks in alef.toml.

We build xberg this way because the alternative, sixteen hand-maintained binding layers chasing one moving core, is exactly the kind of work that reads productive and quietly sinks a polyglot project. Generate the boring layer, test it against one fixture set, and spend the human time on the engine that is actually the product.

Alef is MIT and standalone: github.com/xberg-io/alef. If you maintain a Rust library that people keep asking for bindings to, it is worth a look.

Code review can't keep up with AI. Build a verification layer instead.

Na'aman Hirschfeld (Goldziher) — Sun, 05 Jul 2026 10:48:07 +0000

There is a popular argument going around that code review no longer matters. The reasoning is that a model wrote the code, another model can check it, and a human reading the diff is a slow, expensive step you can now skip. I think that gets the problem exactly backwards.

Code review did not stop mattering. It stopped scaling. You can now generate code faster than any human can read it, let alone reason about it. The old model, where an engineer reading the diff is the last line of defense, quietly broke the moment a single afternoon started producing more change than a person can hold in their head. Skipping review does not fix that. It just removes the one check that was already failing and puts nothing in its place.

The answer is to move verification onto things that keep up. I have been building xberg, a document-intelligence engine with a Rust core and a dozen language SDKs in one repo, and this is the setup that lets me ship at generation speed without shipping garbage.

Static analysis and types do the cheap work

The first layer is everything a machine can check without running your program: linting, static analysis, type checks. This is the cheapest verification you will ever buy, and it catches whole classes of errors before a test ever runs. A type error, an unused binding, a dangerous cast, a lint rule you agreed to as a team, none of these need a human to notice.

For a polyglot repo this is its own project. I ended up building polylint, two self-contained Rust binaries driven by one config, because wiring a separate linter and formatter per language into pre-commit was itself a maintenance tax. The point is not the tool. The point is that mechanical checks should run on every change, locally and in CI, and they should be fast enough that nobody is tempted to skip them.

Comprehensive tests prove behavior

Static analysis tells you the code is well formed. It does not tell you the code is correct. That is what tests are for, and at generation speed the test suite is doing far more of the reviewing than any human is.

The tests that earn their keep here are end-to-end and integration tests. Unit tests are useful, but a wall of them can be green while the system as a whole does the wrong thing. What you want is tests that exercise real paths through the software the way a user would: real inputs, real boundaries, real failure modes. When I add a format to xberg's extraction pipeline, the test that matters is the one that runs a real file of that format end to end and checks the output, not the one that mocks the parser and asserts it was called.

Comprehensive coverage of behavior is what lets you accept a large generated change with confidence. The suite is the reviewer that never gets tired and never skims.

Agents review against written guidelines

Here is where the "another model can check it" idea is actually right, as long as you give it something to check against. An agent doing a first-pass review is fast, tireless, and consistent, but only if the standard it reviews against is explicit and written down. Vague instructions produce vague reviews.

So write the guidelines. The conventions, the security rules, the patterns you want and the ones you have banned, the things a reviewer on your team would flag. I keep those rules in one place and generate the per-tool configs from it with ai-rulez, so Cursor, Claude, and Copilot all review against the same standard instead of three slightly different ones. An agent with clear guidelines catches a real fraction of what a human reviewer would, at a speed a human never will, and it frees the human to look at the things that actually need judgment.

Then you actually test it, by hand

None of the above removes the need to run the software yourself. This is the step people are most tempted to skip now, and it is the one that catches what everything else misses.

A green pipeline is a signal. It is not proof that your software does what you think it does. Automated tests only check the things you thought to check. Running the thing yourself, exercising the real paths, poking at the edges, is how you find the assumption the whole test suite quietly shares with the bug. QA is not a box you tick after the checks pass. At generation speed it becomes one of the highest-leverage things an engineer does, because it is the only layer that brings human judgment to bear on software no human wrote line by line.

Teams that treat QA as a core engineering function, rather than a phase they graduated out of, are the ones this speed will not break.

Open source makes the code better, for free

There is one more layer, and it is the strongest one I know of. Open source your code and it gets tested at a scale you could never reproduce in-house. Thousands of people run it in environments you never imagined, on inputs you never thought to generate, and every bug they hit and report comes back and makes the code more robust. No internal test suite competes with real usage at that volume.

Auditability and good citizenship are real, but the benefit that changes the code itself is simpler. Public code is massively tested code, and massively tested code is better code. Every one of xberg's libraries is open source partly for this reason.

The shape of it

Put together, the verification layer looks like this: static analysis and types catch the cheap errors, comprehensive end-to-end and integration tests prove the behavior, agents do first-pass review against written guidelines, you test it by hand to catch what the machines missed, and open source turns your users into the largest test suite you will ever have.

Code review did not become optional. It became a system instead of a person. Build that system, and you can move as fast as the models let you without lying to yourself about whether it works.

Deleting the toolchain: linting a polyglot monorepo with one Rust binary

Na'aman Hirschfeld (Goldziher) — Sat, 04 Jul 2026 08:02:42 +0000

A repository that mixes languages pays a tax most single-language repos never see. Python wants ruff, and ruff wants a Python. JavaScript and TypeScript want oxlint or prettier, and those want a Node. Shell wants shfmt and shellcheck. Then taplo for TOML, rumdl for Markdown, sqruff for SQL, hadolint for Dockerfiles, and one more for whatever you added last month. Each tool is installed separately, pinned separately, and wired into .pre-commit-config.yaml separately.

The result is slow setup, version drift, CI images fat with language runtimes you only need for linting, and a hook stack that breaks the day a system dependency goes missing or bumps a version. I hit this hard while building xberg, a document-intelligence engine with a Rust core and a dozen language SDKs in one repo, which is where the itch to fix it came from.

polylint is the fix I landed on. It ships two self-contained Rust binaries, poly lint and poly fmt, driven by one poly.toml. No language runtime is required on the default path. The goal was not to add one more tool beside the stack. It was to delete the stack.

What using it looks like

$ poly fmt --check
would format crates/example/src/main.rs

$ poly fmt --fix
formatted 1 file

$ poly lint --format toon
path: crates/example/src/main.rs
diagnostics[0]: engine=ruff, code=F401, severity=warning, title="`os` imported but unused"

$ poly hooks install
installed git hooks: pre-commit, commit-msg

poly fmt is a dry run by default, which is what you want in CI. Add --fix to write changes. poly hooks install wires the git hooks once, and lint, format, and commit checks then run on every git commit.

How it avoids the runtimes

There are three tiers, and a file falls through them until something can handle it.

Curated Rust backends, in process. The Rust ecosystem has grown real, high-quality linters and formatters that are libraries, not just CLIs. polylint links them directly: oxc for JS/TS, ruff internals for Python, taplo for TOML, rumdl for Markdown, sqruff for SQL, malva and markup_fmt for CSS and HTML, mago for PHP. Every backend implements one trait and returns one diagnostic shape, so reporting, caching, and the MCP server stay uniform no matter which tool did the work.

pub trait Engine: Send + Sync {
    fn name(&self) -> &'static str;
    fn languages(&self) -> &[Language];
    fn capabilities(&self) -> Capabilities;   // lint / format / fix
    fn version(&self) -> &str;                 // folded into the cache key
    fn lint(&self, src: &SourceFile, cfg: &EngineConfig) -> Result<Vec<Diagnostic>>;
    fn format(&self, src: &SourceFile, cfg: &EngineConfig) -> Result<FormatOutput>;
}

A tree-sitter fallback for everything else. For languages without a dedicated backend, polylint uses tree-sitter-language-pack, parses the source, and re-emits it with structural reindentation and whitespace normalization. This covers hundreds of grammars, including Java, using each language's indents.scm queries. So you get consistent indentation and clean whitespace on a Java file with no JDK on the machine. Grammars where whitespace carries meaning are left untouched on purpose, since a reindent there would corrupt the file rather than tidy it.

An opt-in catalog for the rest. Some tools have no Rust equivalent yet. For those, polylint vendors a tool catalog derived from the mdsf project, which maps a tool to its binary, argument vector, and languages. It is data only, so polylint does not depend on or shell out to mdsf itself. Each catalog tool is probed on PATH and skipped when absent, so enabling one never makes a run fail on a machine that lacks it.

Configuration for all of this lives in one file:

[defaults]
line_length = 120
line_ending = "lf"

[lint.python.ruff]
select = ["E", "F", "W"]

[per-file-ignores]
"tests/**" = ["F401"]

[hooks.builtin]
polylint = true
polyfmt = true
commit = { stages = ["commit-msg"] }
file_safety = true

That [hooks] table replaces .pre-commit-config.yaml outright. polylint dogfoods it: the commit-msg hook runs Conventional Commit checks and strips AI-attribution trailers, and the file-safety builtin catches merge-conflict markers, large files, leaked private keys, and case collisions, all in pure Rust.

Why it is fast enough to run on every commit

Two things do the work. Nothing spawns a subprocess per file on the default path, so there is no process-startup cost multiplied across a large tree, and the whole run is a rayon parallel loop over files.

On top of that sits a blake3 result cache. The cache key is not just the file contents. It folds in the engine name, the engine's own version, and the resolved config, alongside the file bytes:

key = blake3(namespace, engine, engine_version, resolved_config, file_bytes)

That detail matters more than it looks. A plain content hash would happily serve you a stale result after you upgraded a formatter or edited a rule, because the bytes did not change. Folding the tool version and config into the key means an upgrade or a config edit invalidates exactly the affected entries and nothing else.

In CI, the binary itself is cached by version, OS, and architecture, so a run downloads it once and reuses it until you bump the version.

name: Lint and Format
on: [push, pull_request]
jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4      - uses: Goldziher/polylint@v1      - run: poly lint .
      - run: poly fmt --check .

The tradeoffs

I would rather name these than have you find them.

The binary is large, around 70 MB per platform, because it bundles several big parsers and linters. For a tool you install once and cache in CI, I think that is the right trade against carrying zero system dependencies, but if you expected a 3 MB download, that is not this.

The coverage is uneven by design. Some backends lint only, some format only, so it is not true that every language gets every capability. The tree-sitter tier is a formatter, not a linter, and it deliberately does nothing on whitespace-sensitive grammars rather than risk a destructive rewrite.

And it is not fully subprocess-free in every case. rustfmt and gofmt are used when present, because reimplementing them in process is not worth it, and a few other native formatters are opt-in. The default path stays runtime-free; those are the documented exceptions.

Try it

polylint is MIT licensed.

pip install polylint
# or
brew install Goldziher/tap/polylint

Source and docs are at github.com/Goldziher/polylint. If you run a polyglot repo and your pre-commit config has quietly become its own maintenance project, point poly lint at it and see what falls out.

When an LLM response fails validation, feed the error back into the retry

Na'aman Hirschfeld (Goldziher) — Thu, 02 Jul 2026 05:17:08 +0000

If you ask an LLM for structured output and validate it against a schema, you already know the failure
mode: most of the time it is fine, and every so often it hands back something that does not parse or
misses a required field. The usual reflex is to wrap the call in a retry and move on.

The problem is that a plain retry is the same prompt, the same temperature, roughly the same odds. You
are paying for another round and hoping the dice land differently.

There is a better move, and it is almost free to add: when validation fails, put the validation error
and the model's own bad output back into the next prompt, and ask it to fix that specific thing.

Here is the core of the loop I used for this on a RAG platform I built (trimmed to the essential path):

while attempts < max_attempts:
    try:
        msgs = [messages] if isinstance(messages, str) else messages
        if error_message:
            msgs = [*msgs, error_message]        # last attempt's error rides along
        response = await make_completion_request(..., messages=msgs)
        if validator:
            validator(response)                  # raises ValidationError on bad output
        return response
    except ValidationError as e:
        attempts += 1
        error_message = f"""
        The last response from the API failed validation due to the following error:
            <error>{format_error_for_llm(e)}</error>
        Your task is to fix the error and return the corrected response data:
            <data>{serialize(response).decode()}</data>
        """
        response = None

Two details do the work:

The error is described for the model, not for a log. format_error_for_llm turns the raw validation exception into a plain instruction ("field X must be an integer, you sent a string"). The model is good at patching a concrete, named mistake; it is bad at guessing why an opaque retry keeps failing.
You hand back its own previous output as the thing to correct. It is not regenerating from scratch, it is editing. That keeps the parts that were already right and usually fixes the one field that was wrong on the first pass.

The tradeoffs, because there always are some:

It costs an extra call on a bad response, and the follow-up prompt is longer (it carries the error plus the prior payload). On a schema that fails often, that adds up. Cap the attempts.
It only works when the bad response is parseable enough to serialize back into the prompt. Truly empty or truncated output has nothing to correct, so you still need a normal retry underneath.
Prompt semantics do not always transfer if you also fail over between providers mid-loop. If you do that, do not count a provider swap as a real attempt.

That is the whole idea. It is not a framework, it is about ten lines around a call you already have. If
you are generating structured output at any volume, it turns a chunk of your "model was flaky" retries
into first-try-after-feedback successes.

Benchmark-Driven Development: let agents build the harness you never had time for

Na'aman Hirschfeld (Goldziher) — Tue, 30 Jun 2026 04:41:22 +0000

Most teams ship on two signals: does it compile, and do the tests pass. Both are correctness signals. Neither tells you whether the thing is fast, whether it got slower this week, or whether the output is actually good against some ground truth. We close that gap with a third signal, and we let it drive the work. We call it benchmark-driven development.

The loop is simple:

Write the plan and the spec first. Decide what the code should do before writing it.
Use TDD to drive it to correct. Red, green, refactor, the usual.
Test it for real with proper benchmarks, against ground truth, with profiling.

Correctness and performance are both requirements. So both get measured, on every change, not once at the end when a customer complains.

Why benchmarks used to be a luxury

Everyone agrees benchmarks are good. Almost no one builds a serious one. The reason is cost. A real harness is its own project: a fixture corpus with trustworthy ground truth, scoring that reflects quality and not just a checksum, resource sampling, a way to compare across implementations, and CI plumbing to run it all and publish results. That is weeks of work that ships no features. So it gets cut, and teams fall back to a stopwatch around a hot loop and a vibe about whether output looks right.

This is the part that changed. With an agent in the loop, the harness is no longer a quarter of engineering budget. It is a few focused sessions. You describe the corpus, the metrics, and the comparison you want, and you iterate on the harness the same way you iterate on a feature. The economics flipped. The thing that was too expensive to justify is now the cheap part.

A concrete harness

To make this concrete, here is the benchmark harness we run for xberg, our document-extraction engine. It is a Rust CLI, and it does the following.

It compares 13 language bindings of our own engine against 7 external reference frameworks, across a corpus of 318 fixtures that carry markdown ground truth, spanning 17 formats: PDF, HTML, DOCX, ODT, RTF, XLSX, CSV, EPUB, PPTX, and more.

It scores quality two ways, because text and structure fail differently:

Text F1 (TF1): token-level bag-of-words F1 between extracted text and ground truth, with a separate numeric-token score for number-heavy documents like financial and scientific PDFs.
Structural F1 (SF1): block-level matching between extracted markdown and ground-truth markdown. Headings, code blocks, formulas, tables, list items, and images are weighted by how much they matter, matched greedily with fuzzy cross-type compatibility, and scored for ordering with a longest-increasing-subsequence pass.

When markdown ground truth is present it combines them:

quality_score = 0.5 * f1_text + 0.2 * f1_numeric + 0.3 * f1_layout

It samples CPU and memory during each run, and behind a feature flag it generates flamegraphs so a regression in latency points you straight at the function that caused it. It runs each binding in both single-file mode for fair latency and batch mode for throughput. And it runs as a CI matrix: build, validate ground truth, fan out one job per binding, gate, run the external frameworks, then consolidate everything into percentiles and publish the aggregate as a release artifact.

None of that is exotic. All of it used to be too much work to justify for anything but a flagship. That is the point.

What "real benchmarks" actually means

A benchmark you can drive development with has to clear a higher bar than a stopwatch around a hot loop. Four properties matter.

Ground truth you trust. A number is only as good as what it is measured against. We keep ground truth in version control, validate its integrity on every CI run, and clean known artifacts so the target does not drift silently. If you cannot defend the reference, you cannot defend the score.

Metrics that reflect the goal. "Bytes match" is not quality for document extraction. A table flattened into a paragraph can have high text overlap and be useless. That is why structure gets its own score and its own weights. Pick metrics that punish the failures you actually care about.

Resource truth, not just wall-clock. Throughput hides memory blowups. We sample CPU and memory and keep flamegraphs around so "it got slower" turns into "this function got slower."

Comparison and regression gates. A score in isolation is trivia. The harness compares pipelines and implementations against each other and against history, and it can fail the build on a quality regression. That is what makes it a development driver rather than a report.

How to adopt it without a big-bang rewrite

You do not need 318 fixtures on day one.

Start with ten fixtures and one honest metric. Ten documents you understand beat a thousand you cannot defend.
Write the spec, then the failing test, then make it pass. Keep TDD for correctness. The benchmark is the layer on top, not a replacement.
Add a benchmark the first time you guess about performance. The moment you say "this is probably fine," measure it instead.
Put it in CI and gate on regressions once the metric is stable. A benchmark nobody runs is documentation.
Lean on an agent for the unglamorous parts: fixture wrangling, scoring code, CI matrix, percentile aggregation. This is exactly the work that used to make a harness too expensive, and it is exactly the work an agent is good at.

The payoff

When correctness and speed are both measured on every change, you stop guessing and you stop arguing from intuition. A refactor either held the line on quality and latency or it did not, and the harness tells you which before the diff merges. Spec-driven, TDD for correctness, real benchmarks for the rest. Build the harness you never had time for, because now you do.

Kreuzberg v4.0.0-RC.8 is Available

Na'aman Hirschfeld (Goldziher) — Mon, 15 Dec 2025 13:06:14 +0000

Hi Peeps,

I'm excited to announce that Kreuzberg v4.0.0 is coming very soon. We will release v4.0.0 at the beginning of next year - in just a couple of weeks time. For now, v4.0.0-rc.8 has been released to all channels.

What is Kreuzberg?

Kreuzberg is a document intelligence toolkit for extracting text, metadata, tables, images, and structured data from 56+ file formats. It was originally written in Python (v1-v3), where it demonstrated strong performance characteristics compared to alternatives in the ecosystem.

What's new in V4?

A Complete Rust Rewrite with Polyglot Bindings

The new version of Kreuzberg represents a massive architectural evolution. Kreuzberg has been completely rewritten in Rust - leveraging Rust's memory safety, zero-cost abstractions, and native performance. The new architecture consists of a high-performance Rust core with native bindings to multiple languages. That's right - it's no longer just a Python library.

Kreuzberg v4 is now available for 7 languages across 8 runtime bindings:

Rust (native library)
Python (PyO3 native bindings)
TypeScript - Node.js (NAPI-RS native bindings) + Deno/Browser/Edge (WASM)
Ruby (Magnus FFI)
Java 25+ (Panama Foreign Function & Memory API)
C# (P/Invoke)
Go (cgo bindings)

Post v4.0.0 roadmap includes:

PHP
Elixir (via Rustler - with Erlang and Gleam interop)

Additionally, it's available as a CLI (installable via cargo or homebrew), HTTP REST API server, Model Context Protocol (MCP) server for Claude Desktop/Continue.dev, and as public Docker images.

Why the Rust Rewrite? Performance and Architecture

The Rust rewrite wasn't just about performance - though that's a major benefit. It was an opportunity to fundamentally rethink the architecture:

Architectural improvements:

Zero-copy operations via Rust's ownership model
True async concurrency with Tokio runtime (no GIL limitations)
Streaming parsers for constant memory usage on multi-GB files
SIMD-accelerated text processing for token reduction and string operations
Memory-safe FFI boundaries for all language bindings
Plugin system with trait-based extensibility

v3 vs v4: What Changed?

Aspect	v3 (Python)	v4 (Rust Core)
Core Language	Pure Python	Rust 2024 edition
File Formats	30-40+ (via Pandoc)	56+ (native parsers)
Language Support	Python only	7 languages (Rust/Python/TS/Ruby/Java/Go/C#)
Dependencies	Requires Pandoc (system binary)	Zero system dependencies (all native)
Embeddings	Not supported	✓ FastEmbed with ONNX (3 presets + custom)
Semantic Chunking	Via semantic-text-splitter library	✓ Built-in (text + markdown-aware)
Token Reduction	Built-in (TF-IDF based)	✓ Enhanced with 3 modes
Language Detection	Optional (fast-langdetect)	✓ Built-in (68 languages)
Keyword Extraction	Optional (KeyBERT)	✓ Built-in (YAKE + RAKE algorithms)
OCR Backends	Tesseract/EasyOCR/PaddleOCR	Same + better integration
Plugin System	Limited extractor registry	Full trait-based (4 plugin types)
Page Tracking	Character-based indices	Byte-based with O(1) lookup
Servers	REST API (Litestar)	HTTP (Axum) + MCP + MCP-SSE
Installation Size	~100MB base	16-31 MB complete
Memory Model	Python heap management	RAII with streaming
Concurrency	asyncio (GIL-limited)	Tokio work-stealing

Replacement of Pandoc - Native Performance

Kreuzberg v3 relied on Pandoc - an amazing tool, but one that had to be invoked via subprocess because of its GPL license. This had significant impacts:

v3 Pandoc limitations:

System dependency (installation required)
Subprocess overhead on every document
No streaming support
Limited metadata extraction
~500MB+ installation footprint

v4 native parsers:

Zero external dependencies - everything is native Rust
Direct parsing with full control over extraction
Substantially more metadata extracted (e.g., DOCX document properties, section structure, style information)
Streaming support for massive files (tested on multi-GB XML documents with stable memory)
Example: PPTX extractor is now a fully streaming parser capable of handling gigabyte-scale presentations with constant memory usage and high throughput

New File Format Support

v4 expanded format support from ~20 to 56+ file formats, including:

Added legacy format support:

.doc (Word 97-2003)
.ppt (PowerPoint 97-2003)
.xls (Excel 97-2003)
.eml (Email messages)
.msg (Outlook messages)

Added academic/technical formats:

LaTeX (.tex)
BibTeX (.bib)
Typst (.typ)
JATS XML (scientific articles)
DocBook XML
FictionBook (.fb2)
OPML (.opml)

Better Office support:

XLSB, XLSM (Excel binary/macro formats)
Better structured metadata extraction from DOCX/PPTX/XLSX
Full table extraction from presentations
Image extraction with deduplication

New Features: Full Document Intelligence Solution

The v4 rewrite was also an opportunity to close gaps with commercial alternatives and add features specifically designed for RAG applications and LLM workflows:

1. Embeddings (NEW)

FastEmbed integration with full ONNX Runtime acceleration
Three presets: "fast" (384d), "balanced" (512d), "quality" (768d/1024d)
Custom model support (bring your own ONNX model)
Local generation (no API calls, no rate limits)
Automatic model downloading and caching
Per-chunk embedding generation

from kreuzberg import ExtractionConfig, EmbeddingConfig, EmbeddingModelType

config = ExtractionConfig(
    embeddings=EmbeddingConfig(
        model=EmbeddingModelType.preset("balanced"),
        normalize=True
    )
)
result = kreuzberg.extract_bytes(pdf_bytes, config=config)
# result.embeddings contains vectors for each chunk

2. Semantic Text Chunking (NOW BUILT-IN)

Now integrated directly into the core (v3 used external semantic-text-splitter library):

Structure-aware chunking that respects document semantics
Two strategies:
- Generic text chunker (whitespace/punctuation-aware)
- Markdown chunker (preserves headings, lists, code blocks, tables)
Configurable chunk size and overlap
Unicode-safe (handles CJK, emojis correctly)
Automatic chunk-to-page mapping
Per-chunk metadata with byte offsets

3. Byte-Accurate Page Tracking (BREAKING CHANGE)

This is a critical improvement for LLM applications:

v3: Character-based indices (char_start/char_end) - incorrect for UTF-8 multi-byte characters
v4: Byte-based indices (byte_start/byte_end) - correct for all string operations

Additional page features:

O(1) lookup: "which page is byte offset X on?" → instant answer
Per-page content extraction
Page markers in combined text (e.g., --- Page 5 ---)
Automatic chunk-to-page mapping for citations

4. Enhanced Token Reduction for LLM Context

Enhanced from v3 with three configurable modes to save on LLM costs:

Light mode: ~15% reduction (preserve most detail)
Moderate mode: ~30% reduction (balanced)
Aggressive mode: ~50% reduction (key information only)

Uses TF-IDF sentence scoring with position-aware weighting and language-specific stopword filtering. SIMD-accelerated for improved performance over v3.

5. Language Detection (NOW BUILT-IN)

68 language support with confidence scoring
Multi-language detection (documents with mixed languages)
ISO 639-1 and ISO 639-3 code support
Configurable confidence thresholds

6. Keyword Extraction (NOW BUILT-IN)

Now built into core (previously optional KeyBERT in v3):

YAKE (Yet Another Keyword Extractor): Unsupervised, language-independent
RAKE (Rapid Automatic Keyword Extraction): Fast statistical method
Configurable n-grams (1-3 word phrases)
Relevance scoring with language-specific stopwords

7. Plugin System (NEW)

Four extensible plugin types for customization:

DocumentExtractor - Custom file format handlers
OcrBackend - Custom OCR engines (integrate your own Python models)
PostProcessor - Data transformation and enrichment
Validator - Pre-extraction validation

Plugins defined in Rust work across all language bindings. Python/TypeScript can define custom plugins with thread-safe callbacks into the Rust core.

8. Production-Ready Servers (NEW)

HTTP REST API: Production-grade Axum server with OpenAPI docs
MCP Server: Direct integration with Claude Desktop, Continue.dev, and other MCP clients
MCP-SSE Transport (RC.8): Server-Sent Events for cloud deployments without WebSocket support
All three modes support the same feature set: extraction, batch processing, caching

Performance: Benchmarked Against the Competition

We maintain continuous benchmarks comparing Kreuzberg against the leading OSS alternatives:

Benchmark Setup

Platform: Ubuntu 22.04 (GitHub Actions)
Test Suite: 30+ documents covering all formats
Metrics: Latency (p50, p95), throughput (MB/s), memory usage, success rate
Competitors: Apache Tika, Docling, Unstructured, MarkItDown

How Kreuzberg Compares

Installation Size (critical for containers/serverless):

Kreuzberg: 16-31 MB complete (CLI: 16 MB, Python wheel: 22 MB, Java JAR: 31 MB - all features included)
MarkItDown: ~251 MB installed (58.3 KB wheel, 25 dependencies)
Unstructured: ~146 MB minimal (open source base) - several GB with ML models
Docling: ~1 GB base, 9.74GB Docker image (includes PyTorch CUDA)
Apache Tika: ~55 MB (tika-app JAR) + dependencies
GROBID: 500MB (CRF-only) to 8GB (full deep learning)

Performance Characteristics:

Library	Speed	Accuracy	Formats	Installation	Use Case
Kreuzberg	⚡ Fast (Rust-native)	Excellent	56+	16-31 MB	General-purpose, production-ready
Docling	⚡ Fast (3.1s/pg x86, 1.27s/pg ARM)	Best	7+	1-9.74 GB	Complex documents, when accuracy > size
GROBID	⚡⚡ Very Fast (10.6 PDF/s)	Best	PDF only	0.5-8 GB	Academic/scientific papers only
Unstructured	⚡ Moderate	Good	25-65+	146 MB-several GB	Python-native LLM pipelines
MarkItDown	⚡ Fast (small files)	Good	11+	~251 MB	Lightweight Markdown conversion
Apache Tika	⚡ Moderate	Excellent	1000+	~55 MB	Enterprise, broadest format support

Kreuzberg's sweet spot:

Smallest full-featured installation: 16-31 MB complete (vs 146 MB-9.74 GB for competitors)
5-15x smaller than Unstructured/MarkItDown, 30-300x smaller than Docling/GROBID
Rust-native performance without ML model overhead
Broad format support (56+ formats) with native parsers
Multi-language support unique in the space (7 languages vs Python-only for most)
Production-ready with general-purpose design (vs specialized tools like GROBID)

Is Kreuzberg a SaaS Product?

No. Kreuzberg is and will remain MIT-licensed open source.

However, we are building Kreuzberg.cloud - a commercial SaaS and self-hosted document intelligence solution built on top of Kreuzberg. This follows the proven open-core model: the library stays free and open, while we offer a cloud service for teams that want managed infrastructure, APIs, and enterprise features.

Will Kreuzberg become commercially licensed? Absolutely not. There is no BSL (Business Source License) in Kreuzberg's future. The library was MIT-licensed and will remain MIT-licensed. We're building the commercial offering as a separate product around the core library, not by restricting the library itself.

Target Audience

Any developer or data scientist who needs:

Document text extraction (PDF, Office, images, email, archives, etc.)
OCR (Tesseract, EasyOCR, PaddleOCR)
Metadata extraction (authors, dates, properties, EXIF)
Table and image extraction
Document pre-processing for RAG pipelines
Text chunking with embeddings
Token reduction for LLM context windows
Multi-language document intelligence in production systems

Ideal for:

RAG application developers
Data engineers building document pipelines
ML engineers preprocessing training data
Enterprise developers handling document workflows
DevOps teams needing lightweight, performant extraction in containers/serverless

Comparison with Alternatives

Open Source Python Libraries

Unstructured.io

Strengths: Established, modular, broad format support (25+ open source, 65+ enterprise), LLM-focused, good Python ecosystem integration
Trade-offs: Python GIL performance constraints, 146 MB minimal installation (several GB with ML models)
License: Apache-2.0
When to choose: Python-only projects where ecosystem fit > performance

MarkItDown (Microsoft)

Strengths: Fast for small files, Markdown-optimized, simple API
Trade-offs: Limited format support (11 formats), less structured metadata, ~251 MB installed (despite small wheel), requires OpenAI API for images
License: MIT
When to choose: Markdown-only conversion, LLM consumption

Docling (IBM)

Strengths: Excellent accuracy on complex documents (97.9% cell-level accuracy on tested sustainability report tables), state-of-the-art AI models for technical documents
Trade-offs: Massive installation (1-9.74 GB), high memory usage, GPU-optimized (underutilized on CPU)
License: MIT
When to choose: Accuracy on complex documents > deployment size/speed, have GPU infrastructure

Open Source Java/Academic Tools

Apache Tika

Strengths: Mature, stable, broadest format support (1000+ types), proven at scale, Apache Foundation backing
Trade-offs: Java/JVM required, slower on large files, older architecture, complex dependency management
License: Apache-2.0
When to choose: Enterprise environments with JVM infrastructure, need for maximum format coverage

GROBID

Strengths: Best-in-class for academic papers (F1 0.87-0.90), extremely fast (10.6 PDF/sec sustained), proven at scale (34M+ documents at CORE)
Trade-offs: Academic papers only, large installation (500MB-8GB), complex Java+Python setup
License: Apache-2.0
When to choose: Scientific/academic document processing exclusively

Commercial APIs

There are numerous commercial options from startups (LlamaIndex, Unstructured.io paid tiers) to big cloud providers (AWS Textract, Azure Form Recognizer, Google Document AI). These are not OSS but offer managed infrastructure.

Kreuzberg's position: As an open-source library, Kreuzberg provides a self-hosted alternative with no per-document API costs, making it suitable for high-volume workloads where cost efficiency matters.

Community & Resources

GitHub: Star us at https://github.com/kreuzberg-dev/kreuzberg
Discord: Join our community server at discord.gg/pXxagNK2zN
Subreddit: Join the discussion at r/kreuzberg_dev
Documentation: kreuzberg.dev

We'd love to hear your feedback, use cases, and contributions!

TL;DR: Kreuzberg v4 is a complete Rust rewrite of a document intelligence library, offering native bindings for 7 languages (8 runtime targets), 56+ file formats, Rust-native performance, embeddings, semantic chunking, and production-ready servers - all in a 16-31 MB complete package (5-15x smaller than alternatives). Releasing January 2026. MIT licensed forever.

I benchmarked 4 Python text extraction libraries (2025)

Na'aman Hirschfeld (Goldziher) — Sun, 06 Jul 2025 10:25:38 +0000

TL;DR: Comprehensive benchmarks of Kreuzberg, Docling, MarkItDown, and Unstructured across 94 real-world documents. Results might surprise you.

📊 Live Results: https://goldziher.github.io/python-text-extraction-libs-benchmarks/

Context

As the author of Kreuzberg, I wanted to create an honest, comprehensive benchmark of Python text extraction libraries. No cherry-picking, no marketing fluff - just real performance data across 94 documents (~210MB) ranging from tiny text files to 59MB academic papers.

Full disclosure: I built Kreuzberg, but these benchmarks are automated, reproducible, and the methodology is completely open-source.

🔬 What I Tested

Libraries Benchmarked:

Kreuzberg (71MB, 20 deps) - My library
Docling (1,032MB, 88 deps) - IBM's ML-powered solution
MarkItDown (251MB, 25 deps) - Microsoft's Markdown converter
Unstructured (146MB, 54 deps) - Enterprise document processing

Test Coverage:

94 real documents: PDFs, Word docs, HTML, images, spreadsheets
5 size categories: Tiny (<100KB) to Huge (>50MB)
6 languages: English, Hebrew, German, Chinese, Japanese, Korean
CPU-only processing: No GPU acceleration for fair comparison
Multiple metrics: Speed, memory usage, success rates, installation sizes

🏆 Results Summary

Speed Champions 🚀

Kreuzberg: 35+ files/second, handles everything
Unstructured: Moderate speed, excellent reliability
MarkItDown: Good on simple docs, struggles with complex files
Docling: Often 60+ minutes per file (!!)

Installation Footprint 📦

Kreuzberg: 71MB, 20 dependencies ⚡
Unstructured: 146MB, 54 dependencies
MarkItDown: 251MB, 25 dependencies (includes ONNX)
Docling: 1,032MB, 88 dependencies 🐘

Reality Check ⚠️

Docling: Frequently fails/times out on medium files (>1MB)
MarkItDown: Struggles with large/complex documents (>10MB)
Kreuzberg: Consistent across all document types and sizes
Unstructured: Most reliable overall (88%+ success rate)

🎯 When to Use What

⚡ Kreuzberg (Disclaimer: I built this)

Best for: Production workloads, edge computing, AWS Lambda
Why: Smallest footprint (71MB), fastest speed, handles everything
Bonus: Both sync/async APIs with OCR support

🏢 Unstructured

Best for: Enterprise applications, mixed document types
Why: Most reliable overall, good enterprise features
Trade-off: Moderate speed, larger installation

📝 MarkItDown

Best for: Simple documents, LLM preprocessing
Why: Good for basic PDFs/Office docs, optimized for Markdown
Limitation: Fails on large/complex files

🔬 Docling

Best for: Research environments (if you have patience)
Why: Advanced ML document understanding
Reality: Extremely slow, frequent timeouts, 1GB+ install

📈 Key Insights

Installation size matters: Kreuzberg's 71MB vs Docling's 1GB+ makes a huge difference for deployment
Performance varies dramatically: 35 files/second vs 60+ minutes per file
Document complexity is crucial: Simple PDFs vs complex layouts show very different results
Reliability vs features: Sometimes the simplest solution works best

🔧 Methodology

Automated CI/CD: GitHub Actions run benchmarks on every release
Real documents: Academic papers, business docs, multilingual content
Multiple iterations: 3 runs per document, statistical analysis
Open source: Full code, test documents, and results available
Memory profiling: psutil-based resource monitoring
Timeout handling: 5-minute limit per extraction

🤔 Why I Built This

Working on Kreuzberg, I worked on performance and stability, and then wanted a tool to see how it measures against other frameworks - which I could also use to further develop and improve Kreuzberg itself. I therefore created this benchmark. Since it was fun, I invested some time to pimp it out:

Uses real-world documents, not synthetic tests
Tests installation overhead (often ignored)
Includes failure analysis (libraries fail more than you think)
Is completely reproducible and open
Updates automatically with new releases

📊 Data Deep Dive

The interactive dashboard shows some fascinating patterns:

Kreuzberg dominates on speed and resource usage across all categories
Unstructured excels at complex layouts and has the best reliability
MarkItDown is useful for simple docs shows in the data
Docling's ML models create massive overhead for most use cases making it a hard sell

🚀 Try It Yourself

git clone https://github.com/Goldziher/python-text-extraction-libs-benchmarks.git
cd python-text-extraction-libs-benchmarks
uv sync --all-extras
uv run python -m src.cli benchmark --framework kreuzberg_sync --category small

Or just check the live results: https://goldziher.github.io/python-text-extraction-libs-benchmarks/

🔗 Links

📊 Live Benchmark Results: https://goldziher.github.io/python-text-extraction-libs-benchmarks/
📁 Benchmark Repository: https://github.com/Goldziher/python-text-extraction-libs-benchmarks
⚡ Kreuzberg (my library): https://github.com/Goldziher/kreuzberg
🔬 Docling: https://github.com/DS4SD/docling
📝 MarkItDown: https://github.com/microsoft/markitdown
🏢 Unstructured: https://github.com/Unstructured-IO/unstructured

🤝 Discussion

What's your experience with these libraries? Any others I should benchmark? I tried benchmarking marker, but the setup required a GPU.

Some important points regarding how I used these benchmarks for Kreuzberg:

I fine tuned the default settings for Kreuzberg.
I updated our docs to give recommendations on different settings for different use cases. E.g. Kreuzberg can actually get to 75% reliability, with about 15% slow-down.
I made a best effort to configure the frameworks following the best practices of their docs and using their out of the box defaults. If you think something is off or needs adjustment, feel free to let me know here or open an issue in the repository.

Announcing Kreuzberg v2.0: A Lightweight, Modern Python Text Extraction library

Na'aman Hirschfeld (Goldziher) — Sat, 15 Feb 2025 18:44:30 +0000

🔍 What’s Kreuzberg?

Kreuzberg is a Python library that provides a unified async/sync interface for extracting text from PDFs, images, Office documents, and more.

Key Features

Async First: Optimized async using anyio and worker processes.
Minimal Dependencies: Much smaller footprint compared to alternatives.
Serverless-and-Docker Ready: Perfect for serverless functions and containerized deployments.
Local Processing: All processing is done locally, with no API calls or cloud services.
Modern Python: Built for Python 3.9+ with rigorous typing and extensive testing.
Versatile: Supports various formats, including PDFs, spreadsheets, Markdown, LaTeX, and more.

🚀 What’s New in Version 2.0?

Kreuzberg v2.0 brings significant enhancements to performance, usability, and feature set. Here’s what’s new:

Sync APIs: Kreuzberg supports synchronous extraction methods alongside async workflows.
Batch Processing: Efficiently process multiple files or byte streams in parallel.
Smart PDF Handling: Automatically fall back to OCR when direct text extraction fails.
Metadata Extraction: Retrieve metadata like document titles or creators using Pandoc.
Excel Multi-Sheet Support: Handle even the most complex spreadsheets.
Enhanced Performance: Worker processes for faster, resource-efficient extraction.

Check out the v2.0 changelog for more details.

🎯 Who’s It For?

Kreuzberg is ideal for developers building:

Retrieval-Augmented Generation (RAG) systems
LLM-powered applications
Document indexing, analysis, and automation tools

If you’re looking for a lightweight, efficient solution for text extraction, Kreuzberg is a great choice.

⚖️ How Kreuzberg Compares

Here’s how Kreuzberg stacks up against alternatives:

Python OSS Libraries
- Unstructured.io: Feature-rich but heavy, making it unsuitable for serverless or low-resource environments.
- Docling: Another strong alternative but larger and heavier—better suited for high-volume, GPU-based workloads.
Non-Python OSS Libraries
- Apache Tika: Requires a Java server running as a sidecar, with Python client libraries available.
- Grobid: Excellent for structured research text extraction but comes with a ~20GB Docker image.
Commercial APIs
Paid solutions like Azure Document Intelligence or AWS Textract offer best-in-class OCR and layout extraction. However, they come with pricing concerns and cloud dependencies, unlike Kreuzberg.

Staring ⭐ is Caring

If Kreuzberg sounds like the library you’ve been looking for, check it out on GitHub.

Please star the repo ⭐—it helps others discover the project and motivates me to keep improving it!

Functional Snippet: withSuppress

Na'aman Hirschfeld (Goldziher) — Sun, 01 Sep 2024 12:31:51 +0000

Sometimes, suppressing errors is handy.

E.g. you see this:

try {
  scrollableRef.update()
} catch {}

Let's create something to handle this in a cleaner fashion. We'll use a functional style that prefers wrap functions to extend functionality.

We "wrap" a function with error handling, returning a function with an identical signature:

import { isPromise } from '@tool-belt/type-predicates';

/**
 * Wraps a function with error suppression and optional logging.
 *
 * @param fn - The function to execute.
 * @param errorMessage - The error message to log if an exception is thrown.
 * @returns The wrapped function with error suppression.
 */
export function withSuppress<T extends (...args: any[]) => any>(
  fn: T,
  errorMessage?: string,
): (...args: Parameters<T>) => ReturnType<T> extends Promise<infer U>
  ? Promise<U | undefined>
  : ReturnType<T> | undefined {
  const log = errorMessage ? (msg: string) => console.error(msg) : null;

  return (...args: Parameters<T>) => {
    try {
      const result = fn(...args);

      if (isPromise(result)) {
        return result.catch((error) => {
          log?.(`${errorMessage}\n${error?.stack ?? error}`);
          return undefined;
        }) as ReturnType<T> extends Promise<infer U> ? Promise<U | undefined> : never;
      }

      return result;
    } catch (error) {
      log?.(`${errorMessage}\n${error?.stack ?? error}`);
      return undefined;
    }
  };
}

And now we can suppress errors elegantly.

Here are some example uses:

# generate content

Here is a Vitest test suite for the function:

import { withSuppress } from './functional';

describe('withSuppress', () => {
  it('should return the value from a synchronous function without errors', () => {
    const syncFn = () => 42;
    const wrappedFn = withSuppress(syncFn);
    expect(wrappedFn()).toBe(42);
  });

  it('should return undefined and log error when synchronous function throws', () => {
    const errorMessage = 'Sync error occurred';
    const syncFn = () => {
      throw new Error('Test Error');
    };
    const consoleErrorSpy = vi.spyOn(console, 'error').mockImplementation(() => {});

    const wrappedFn = withSuppress(syncFn, errorMessage);
    expect(wrappedFn()).toBeUndefined();
    expect(consoleErrorSpy).toHaveBeenCalledWith(
      expect.stringContaining(`${errorMessage}\nError: Test Error`),
    );

    consoleErrorSpy.mockRestore();
  });

  it('should return the resolved value from an asynchronous function without errors', async () => {
    const asyncFn = async () => await Promise.resolve(42);
    const wrappedFn = withSuppress(asyncFn);
    await expect(wrappedFn()).resolves.toBe(42);
  });

  it('should return undefined and log error when asynchronous function rejects', async () => {
    const errorMessage = 'Async error occurred';
    const asyncFn = async () => await Promise.reject(new Error('Test Error'));
    const consoleErrorSpy = vi.spyOn(console, 'error').mockImplementation(() => {});

    const wrappedFn = withSuppress(asyncFn, errorMessage);
    await expect(wrappedFn()).resolves.toBeUndefined();
    expect(consoleErrorSpy).toHaveBeenCalledWith(
      expect.stringContaining(`${errorMessage}\nError: Test Error`),
    );

    consoleErrorSpy.mockRestore();
  });

  it('should return undefined if no error message is provided and the function throws', () => {
    const syncFn = () => {
      throw new Error('Test Error');
    };
    const consoleErrorSpy = vi.spyOn(console, 'error').mockImplementation(() => {});

    const wrappedFn = withSuppress(syncFn);
    expect(wrappedFn()).toBeUndefined();
    expect(consoleErrorSpy).not.toHaveBeenCalled();

    consoleErrorSpy.mockRestore();
  });

  it('should return undefined if no error message is provided and the async function rejects', async () => {
    const asyncFn = async () => await Promise.reject(new Error('Test Error'));
    const consoleErrorSpy = vi.spyOn(console, 'error').mockImplementation(() => {});

    const wrappedFn = withSuppress(asyncFn);
    await expect(wrappedFn()).resolves.toBeUndefined();
    expect(consoleErrorSpy).not.toHaveBeenCalled();

    consoleErrorSpy.mockRestore();
  });

  it('should handle functions that return promises without errors', async () => {
    const asyncFn = async () => await Promise.resolve('async result');
    const wrappedFn = withSuppress(asyncFn);
    await expect(wrappedFn()).resolves.toBe('async result');
  });

  it('should handle synchronous functions returning undefined without errors', () => {
    const syncFn = () => undefined;
    const wrappedFn = withSuppress(syncFn);
    expect(wrappedFn()).toBeUndefined();
  });
});

Starlite December '22 Updates

Na'aman Hirschfeld (Goldziher) — Sun, 11 Dec 2022 13:31:27 +0000

Starlite December '22 Updates

Hi Pythonistas!

This post is an update on the development status of Starlite. Let me start, as usual, with a short intro about what is Starlite - for those of you unfamiliar with it.

Starlite is a Python ASGI API framework. ASGI is an async python API specification, originally from the good folk over at the Django project, and an ASGI API framework is a framework that follows this specification. You might be familiar with other ASGI frameworks, with the most famous one these days being FastAPI.

Starlite began as an alternative to FastAPI - it was originally built on the same foundation- Starlette, which offers a ready to use ASGI "tool-kit". The name of the framework - Starlite, was meant to highlight this relationship. Yet, over time Starlite grew in sophistication and complextiy, and we made the decision to drop Starlette as a dependency because this no longer made any sense.

Dropping Starlette and Benchmarks

Since version v1.39.0 Starlite (released on the 12.11.22) no longer has any dependency on Starlette. Between this version and v1.45.0 that was released today (11.12.22), we have invested significant effort into benchmarking and optimizing code. One of our maintainers, @provinzkraut (Janek Nouvertné), has done amazing work rewriting our benchmarking framework. You can read more about this in here and run the benchmarks on your own by cloning the benchmark repository. The results are pretty impressive if I may say so myself:

Other Important Changes

Contrib and Security Backends

Another important development is the inclusion of a starlite.contrib namespace, which we will be expanding on in the future.

This namespace includes an optional OpenTelemetry Integration, which was a long awaited feature.

We also added Security Backend support, which was discussed in some length here. The new security backend supports all of the different session backends Starlite supports, and there is also an optional JWT Security Backend as part of contrib.

Yield Based Dependencies

We made some updates to dependencies following a request on reddit - we now support yield based dependencies in the Starlite Dependency Injection framework. Additionally we made some optimizations to dependency injection which allows us to resolve dependencies in parallel and do some caching, the result is a significant boost in speed.

Call for Contributiors and Maintainers

The original imperative for creating Starlite was to create a community driven alternative to FastAPI. This was and remains a core pillar of Starlite- to have multiple maintainers and be as open, inviting and accessible for contributions as feasible. The project follows the all-contributors specification and we attribute all types of contribution - code, testing, refactoring, code reviews, documentation, design of the docs, writing and evangelizing etc.

We are a growing group of contributors and maintainers (5 maintainers at present), and we are always looking for more people to be involved. You're invited to join us on our discord server, or checkout our GitHub repository where you are welcome to write in both discussions and issues.