DEV Community: Vincent Tran

Acadia: Bringing Precise Types and Functional Programs to the Database

Vincent Tran — Wed, 19 Aug 2026 00:00:00 +0000

Database-backed applications usually contain several descriptions of the same fact.

A customer status may begin as a constrained value in a product model, become a TEXT column in SQLite, travel through a server record, cross an HTTP boundary as JSON, and finally arrive as a union type in a browser application. Each layer can be reasonable on its own while the complete chain remains fragile. Rename a case, change its nullability, or add a field, and the compiler only sees the pieces that belong to its language.

Acadia is an early attempt to make that chain one program. Created by Elm designer Evan Czaplicki, it combines an Elm-like functional language with database tables, transactions, generated SQL, and client/server integrations. Instead of treating SQL as a string embedded in an application, Acadia treats tables and endpoints as typed source that a compiler can inspect together.

The public alpha is deliberately narrow, but the design asks a broad question: what would database programming look like if schema changes, queries, transactions, and network contracts received the same compiler attention as ordinary application code?

The real problem is contract drift

Teams often describe their database problem as an ORM problem or a SQL ergonomics problem. Those labels point at real frustrations, but they miss the larger system.

The difficult boundary is not only between objects and rows. It is between several independently evolving contracts:

the domain types developers want to express;
the physical representation accepted by the database;
the queries and updates that operate on that representation;
the server functions that expose those operations;
the wire format sent over the network; and
the client types that consume the response.

Conventional tooling checks some edges. A typed query builder can confirm that a selected column exists. Generated API clients can synchronize an HTTP schema. Migration tools can order SQL files. Runtime validators can reject malformed payloads.

What they rarely provide is one place where changing a table field produces a useful error at every dependent endpoint and client call. The result is defensive translation: database rows become server models, server models become transport objects, and transport objects become client models. Every conversion is a chance to lose information.

Acadia’s central bet is that these are not separate contracts. They are different views of one typed program.

One compiler sees the data model and endpoint program, then derives the database and application-facing contracts together.

A table is a precise program value

An Acadia table begins with a record type rather than a CREATE TABLE string. A simplified inventory model might look like this:

type alias PantryItem =
  { id : PantryItemId
  , label : String
  , state : ItemState
  }

type PantryItemId = PantryItemId UInt64

type ItemState
  = Available
  | Reserved
  | Used

The wrapper around UInt64 matters. An item identifier and a customer identifier may share a physical representation, but they should not be interchangeable in application code. The custom ItemState type also carries more information than an unconstrained string.

The table declaration then connects the row type to database behavior:

pantry : Table Security.Unrestricted PantryItem
pantry :=
  Table.table
    { primary = .id
    , security = Security.unrestricted
    , indexes = []
    , constraints = []
    }

The same declaration identifies the primary key, indexes, constraints, and row-level security policy. The goal is not to make database design disappear. It is to turn that design into compiler-visible data rather than parallel configuration.

This is an important distinction from a typical active-record ORM. Acadia does not begin with mutable objects and try to preserve their identity inside a relational store. Its surface is closer to relational transformations expressed with functional operators. Rows remain rows; collections are filtered, mapped, joined, and selected.

Endpoints are compiled, not interpreted

A read endpoint can be assembled with familiar operations:

findLabel : Cookies -> PantryItemId -> Transaction String
findLabel _ wantedId =
  access pantry Security.Unrestricted
    |> filter (\item -> item.id == wantedId)
    |> map .label
    |> select

The compiler can lower that expression to parameterized SQL equivalent to:

SELECT item.label
FROM pantry AS item
WHERE item.id = $1;

That compilation boundary is the key. If a library loads every row and then applies filter in application memory, the pleasant syntax hides a serious performance bug. If it translates the whole expression at build time, the database still performs the set operation and its optimizer still chooses the execution plan.

Acadia prints the SQL it generates, which is essential for evaluating a young compiler. A typed abstraction should not ask for blind trust. Developers need to inspect joins, predicates, indexes, query plans, and row counts just as they would with hand-written SQL.

Compile-time generation also opens the door to optimizations across a larger expression. The compiler can see the complete endpoint rather than receiving a sequence of opaque runtime calls. That is how a functional interface can avoid degenerating into an N+1 query machine: the program must have a clear relational meaning before it runs.

Transactions become ordinary composition

Database work becomes awkward when later statements depend on values created by earlier statements. In raw SQL, developers may use common table expressions, RETURNING, stored-procedure variables, or several client round trips. In application code, the same workflow can accidentally hold a transaction open across unrelated work.

Acadia represents database effects with a Transaction type and offers a binding syntax similar in spirit to async/await. Consider starting a password-reset session:

beginReset : Cookies -> Email -> Transaction ResetSecret
beginReset _ email =
  let
    secret := Uuid.generate ResetSecret
    created := Time.now

    () :=
      insert resetSessions Security.Unrestricted
        { email = email
        , secret = secret
        , created = created
        }
  in
  Transaction.succeed secret

Each bound value can feed later steps, but the complete block describes one transaction. Either every database effect succeeds and commits, or the operation fails without exposing a partially written state.

SQLite’s own rules still matter. Every read or write occurs inside a transaction, and SQLite permits many concurrent readers but only one simultaneous writer. A language can make composition safer without removing locking, contention, busy errors, or the need to keep transactions short. The SQLite transaction documentation remains part of the operating model.

The useful shift is that atomicity becomes visible in the endpoint’s type and structure. Reviewers can see which effects belong together, and the compiler can generate a single database program rather than coordinating an accidental series of network round trips.

Generated types close the client-server gap

Acadia currently advertises Elm and Haskell integration, with more host languages planned according to demand during the alpha. Once an endpoint’s input and output types are known, the compiler can derive the code that calls it from the server or client.

This removes a familiar category of glue:

manually maintained request and response records;
JSON encoders and decoders that mirror those records;
route names duplicated in several languages;
nullable fields added on one side but not the other; and
runtime failures caused by an old caller expecting a previous shape.

End-to-end generation is most valuable when it improves failure timing. A changed column should invalidate the endpoint that reads it. A changed endpoint should invalidate the generated binding. A client using the old result type should then fail to compile with an error at the call site.

That is a stronger promise than “less boilerplate.” Boilerplate can be generated by many tools. The deeper benefit is a dependency graph the compiler can follow from storage to interface.

Migrations should be programs the compiler can question

SQL migration anxiety comes from applying a textual command to state that may not match the developer’s assumptions. A staging database can drift from production. Old application versions may still be serving traffic. A conversion that is valid for the schema can be invalid for the actual data.

Acadia’s stated goal is verified migration: because the compiler knows the existing and desired column types, it can check the proposed transition before it reaches a live database. This does not make every migration automatically safe. Lock duration, table size, backfills, concurrent deploys, and database-specific behavior remain operational concerns.

It does create a better workflow. The compiler can reject a type-incoherent plan early, emit a concrete migration for review when the transition is representable, and keep old callers in view while the contract changes.

Compiler verification is an early gate, not permission to skip staging, backups, lock analysis, or rollout planning.

A production-minded migration still needs several checks:

Confirm that the generated plan matches the intended data transformation.
Test it against a realistic copy of the current schema and data volume.
Measure locks, write amplification, and runtime.
Verify compatibility with every application version that may overlap the deploy.
Prepare a backup, rollback, or forward-fix path appropriate to the operation.
Observe the migration and the first application traffic after it completes.

Static verification narrows the failure surface. It does not repeal physics or deployment concurrency.

Why compile to the database at all?

The project began with a server-rendering problem. While experimenting with Elm on the server in 2017, Czaplicki ran into a mismatch: the application “knew” data should exist, but the database boundary could not guarantee that knowledge in the same type system. By 2019, feedback from Elm teams pointed repeatedly to backend integration rather than frontend language design as the larger source of engineering pain.

In 2020, the exploration moved toward stored procedures as a compilation target. The first design resembled SQL with a stronger type system. A simpler idea changed the direction: database programs could look like Elm, using operators such as map and filter, while the compiler handled the relational translation.

That history explains several Acadia choices. It is not merely a prettier query builder. It is trying to make database code participate in the same language-design values that made Elm distinctive: precise types, constrained effects, deliberate evolution, and error messages written for humans.

The work also took years because the hard questions are semantic, not cosmetic. Can a functional program without general recursion always lower to a bounded relational query? Can the compiler prevent N+1 behavior? How should custom types survive storage? Can migrations remain compatible with old clients? Which guarantees transfer cleanly between SQLite and PostgreSQL?

The public alpha represents a working answer to a subset of those questions, not the end of the research program.

Where the abstraction ends

The Hacker News discussion around the release concentrated on the right risks.

Database feature coverage can lag. Mature databases expose partitioning, specialized indexes, compression, triggers, extensions, custom aggregates, window functions, and vendor-specific controls. A new language cannot cover all of that immediately. Acadia explicitly notes that window functions and custom aggregate functions did not make the initial release. It allows dropping down to SQL, but every escape hatch is also a boundary where some static guarantees may weaken.

Stored representations affect interoperability. Precise custom types need a physical encoding. If that encoding is convenient only through generated Acadia bindings, another service may find the database harder to read directly. Teams should inspect how each type is stored, document the format, and test access from every language that must coexist.

The database may outlive the application language. Data often survives several server rewrites. Before adopting any schema-owning compiler, ask whether the resulting tables, constraints, migration history, and procedures remain understandable without that compiler.

Alpha tooling changes quickly. The current release is intentionally minimal. Advanced queries, integrations, diagnostics, and operational workflows will evolve. A prototype can tolerate that movement; a business-critical system needs a clear upgrade and support plan.

Licensing and source availability matter. Several commenters raised concerns about depending on a young database tool under its current commercial and distribution model. Evaluate the actual license terms, offline build story, long-term maintenance options, and exit path before committing durable data to it.

These are not reasons to dismiss the design. They are the acceptance criteria for any tool that wants to sit between an application and its most durable state.

A sensible evaluation plan

Do not begin with the most important production database. Choose a small vertical slice whose types are currently painful and whose SQL is easy to verify.

A good experiment might include:

one table with a wrapped identifier and a custom sum type;
one filtered read endpoint;
one multi-step write transaction;
one schema change that forces a migration;
one generated server integration;
one generated client call; and
one advanced query that requires an SQL escape hatch.

Then evaluate the whole lifecycle, not only the first successful demo.

Inspect the generated schema and SQL. Compare query plans and latency with a hand-written implementation. Introduce type errors deliberately and judge the diagnostics. Change a column while an old client still exists. Restore the database without Acadia. Read the stored custom types from a separate program. Exercise concurrent writes and failure paths. Finally, remove Acadia from the prototype and estimate the exit cost.

The last test is especially valuable. A durable abstraction should make the common path better without making departure catastrophic.

The larger idea deserves attention

SQL engines are extraordinary pieces of software. The case for Acadia does not require pretending otherwise. The question is whether SQL strings, migration files, transport schemas, and client decoders should remain separate islands when a compiler could reason across them.

Acadia’s answer is to keep the relational engine and replace much of the fragmented programming interface around it. Tables become precise types. Queries become functional expressions compiled to SQL. Dependent effects become one typed transaction. Server and client bindings come from the same endpoint definition. Migration errors move earlier, when they are cheaper to fix.

The alpha is not yet a universal database platform, and its sharpest promises still need production evidence. But it demonstrates a coherent alternative to both raw string composition and object-first ORMs: preserve the database, preserve relational execution, and give the surrounding program a type system wide enough to see the whole contract.

That direction is worth exploring even for teams that never adopt Acadia. We should expect database tools to show generated SQL, track dependencies across layers, verify migrations against known types, preserve interoperability, and produce errors that help developers recover. Once those expectations become normal, today’s duplicated contracts will look less like an unavoidable cost and more like an interface we simply stopped questioning.

The original Acadia announcement explains the language and its development history. The Acadia documentation and examples cover the public alpha, while the Hacker News discussion captures the debate about SQL coverage, interoperability, licensing, and database ownership.

DuckDB 2.0: From Embedded Analytics to a Networked Query Engine

Vincent Tran — Tue, 18 Aug 2026 00:00:00 +0000

DuckDB earned its place in the data toolkit by refusing to become another service. It runs inside Python, R, a command-line shell, a browser, or an application process. It can query a CSV or Parquet file directly, spill work beyond memory, and persist a complete analytical database as one portable file. There is no daemon to provision before the first SELECT.

DuckDB 2.0 keeps that embedded model, but it is no longer the whole story. The release, planned for fall 2026 under the name “Cyanoptera,” adds a native remote protocol and a CONNECT statement. Any DuckDB process can expose its catalogs over a network, while another DuckDB can send work to it and stream results back. At the same time, the engine gains triggers, a first-class VARIANT type, asynchronous I/O, a new storage format, a new parser, and a stable extension interface.

This is a major version because several foundations move together. The important question is not whether every team should turn DuckDB into a server. It is how the new pieces change the boundary between local analysis, shared services, object storage, and application infrastructure.

One Engine, Two Operating Models

The embedded shape remains the simplest one. A host process loads DuckDB as a library, opens a file or an in-memory database, and submits queries through a client API. Data does not cross a database network protocol, deployment can be as small as one binary, and the application controls the engine’s lifetime.

That shape becomes awkward when several processes need coordinated access to the same mutable database. DuckDB permits concurrent connections within one process, but separate processes cannot safely treat one file like a conventional multi-writer server. Teams have worked around the limit by assigning one owner process, publishing immutable database artifacts, putting Parquet in object storage, or choosing PostgreSQL or ClickHouse for the serving layer.

The new Quack remote protocol makes the owner process a supported architecture rather than a custom wrapper. The server starts from an ordinary DuckDB session. Everything that session can see—tables, attached databases, temporary state, and extensions—can be made reachable to authenticated clients. The client can issue stateless remote queries or attach the remote catalog so its tables behave like attached local tables.

DuckDB remains embeddable; Quack adds an optional network boundary when one process should own shared state.

Quack travels over HTTP or HTTPS rather than inventing a new transport. Requests and responses use DuckDB’s own binary serialization, preserving nested values, decimals, intervals, and other rich types without converting them through JSON. A query needs one request-response pair after connection setup, and large results return in streamed chunks.

In v2.0, CONNECT becomes the clearer session-level command. A client can attach a Quack endpoint, connect to it, and run ordinary SQL remotely. The same abstraction can target PostgreSQL and MySQL: the remote-pushdown optimizer sends SQL to the source instead of pulling entire tables across the wire first.

That distinction matters. An attached catalog describes what data is available; CONNECT changes where the query runs. Pushing aggregation and filtering to the database that owns the data can turn a multi-gigabyte transfer into a few result rows.

A Server Mode Does Not Change the Workload

Networking does not magically turn a columnar analytical engine into a drop-in replacement for every transactional database. DuckDB has MVCC, transactions, and isolation, and Quack gives multiple clients one coordinated owner. It does not erase the design differences between analytical and row-oriented systems.

Use server mode where DuckDB’s strengths still define the workload: analytical scans, local or object-backed lakehouses, tenant-isolated warehouses, transformation services, and shared access to large DuckDB artifacts. Keep evaluating PostgreSQL, MySQL, or another purpose-built system when the center of gravity is high-rate point updates, a mature migration ecosystem, fine-grained operational controls, or application concurrency patterns that depend on row locks and long-established drivers.

The practical gain is architectural choice. A pipeline can use embedded DuckDB during transformation and Quack for a shared presentation layer. A data product can keep one database per tenant and expose each through an owner process. A laptop can connect to a remote catalog without giving up the same SQL dialect and nested types it uses locally.

Production use also widens the threat model. The current Quack documentation recommends TLS termination through a reverse proxy for remote deployments and supports scoped secrets for tokens. DuckDB SQL can access files, networks, extensions, and credentials with the privileges of its process, so untrusted SQL must be treated like untrusted shell or Python code. Server mode needs explicit authentication, network policy, resource limits, logs, backups, and upgrade discipline.

`VARIANT` Turns Semi-Structured Data into Columns

JSON is flexible at ingestion and expensive at analysis. A text JSON column must repeatedly parse values, carry field names, and discover types while queries run. Converting it to a rigid table improves performance but creates another problem: event payloads evolve, optional fields appear, and different records legitimately have different shapes.

DuckDB’s VARIANT type stores typed binary values whose shape may differ row by row. It can hold a number, string, list, or struct in the same column. More importantly, DuckDB detects common structure and “shreds” it into typed child columns. A field such as user.id can be compressed and scanned like data rather than recovered from a text blob for every query.

Version 2.0 completes that path through storage and execution. The engine can operate on the shredded representation, push extraction into scans, read and write shredded VARIANT values in Parquet, and inspect or search them with variant_* functions. Parquet interchange is especially useful because DuckDB can reconstruct variants written by other systems, including Snowflake-compatible shredded data.

For log and event pipelines, the result is a useful middle ground: retain heterogeneous records without paying the full text-JSON tax. It does not remove schema governance. Producers still need stable meanings, compatible types, and policies for fields that change shape. It simply lets physical storage adapt to the structure that is already present.

Triggers Bring Database-Side Reactions

Long-running services need behavior that happens beside a write, not later in an application callback. DuckDB 2.0 adds BEFORE and AFTER triggers, row- and statement-level execution, multiple triggers per event, transition tables, RETURNING, and trigger removal.

Audit logging is the obvious example. A statement-level AFTER UPDATE trigger can compare the old and new transition tables and insert every changed value into an audit table in one set operation. Other uses include maintaining derived tables, rejecting invalid transitions, publishing change records, or implementing internal engine features.

Triggers should remain small and observable. Hidden database behavior can surprise operators, complicate bulk loads, and make latency harder to explain. Prefer set-oriented statement triggers when possible, document their order and failure behavior, and test them as part of transaction semantics rather than as isolated snippets.

SQL Becomes a Better Pipeline Language

The release expands SQL in ways that reduce application glue:

APPROX NEAREST ... BY SIMILARITY expresses top-k vector matching as a join.
Data-modifying CTEs let INSERT, UPDATE, DELETE, and COPY become pipeline stages with RETURNING output.
Nested schemas provide another namespace level for large catalogs.
$variable works wherever an expression is accepted.
JSON mutation functions add, replace, or remove fields without manual reconstruction.
Recursive CTEs with USING KEY can aggregate state during iterative algorithms.

These features are not only syntax sugar. They let the optimizer see an operation as one relational plan rather than a sequence of client round trips. A staging-to-archive move, for example, can delete rows and feed the returned records directly into an insert. That makes the transaction boundary explicit and keeps intermediate data inside the engine.

The new NEAREST join is similarly valuable because similarity search becomes part of join planning. It does not make every vector workload an automatic fit for DuckDB, but it gives analytical queries a native way to combine relational filters with top-k matching.

Remote Data Needs Independent I/O Concurrency

Object storage changes the cost model of a scan. A CPU thread that requests one Parquet range and waits for the network is not doing useful query work. Adding more query threads can hide some latency, but it couples network concurrency to CPU parallelism and eventually wastes memory and scheduling overhead.

DuckDB 2.0 introduces asynchronous I/O across the engine so storage requests can progress independently from operators. Parquet reads and writes led the implementation, followed by CSV and DuckDB files. The engine can keep many remote requests in flight while a smaller pool of execution threads decodes, filters, joins, and aggregates the blocks that have arrived.

Asynchronous storage work and CPU execution scale independently, which matters most when latency sits across a network.

This separation improves throughput, but it also introduces knobs worth observing: request concurrency, bytes fetched, cache hit rate, retries, throttling, spill volume, and memory pressure. More outstanding reads are useful only while the object store, network, and local buffer manager can absorb them.

Existing Queries Get Faster Without New Syntax

Several optimizer and execution changes target work users already run. Partial aggregates can move below joins, duplicate aggregates can be reused, and large aggregations can spill rather than fail when memory runs out. The recursive CTE engine was rewritten; DuckDB’s preview benchmark reports a one-million-edge reachability query falling from 4.90 seconds in v1.5.4 to 0.12 seconds in the v2.0 preview.

Treat that 40× result as evidence about one improved execution path, not a promise for every query. The more broadly useful lesson is that recursion is no longer automatically a toy path inside the engine. Graph reachability, hierarchy traversal, and iterative SQL deserve fresh benchmarks against representative data.

Pruning also becomes much more capable. Zone maps and Parquet Bloom filters can skip row groups for nested types, decimals, UUIDs, IN lists, and selected function predicates such as prefixes or substring checks. Partition-aware planning can avoid irrelevant Hive, Iceberg, or DuckLake partitions before a scan begins. For lakehouse workloads, reading fewer files and row groups is often a larger win than processing each row faster.

Storage v2.0 Loads Less Up Front

The default storage format changes to v2.0.0. Adaptive radix tree indexes become buffer-managed instead of permanently pinned in memory, so a database with large indexes can open quickly and page index blocks on demand. Column metadata loads lazily, helping very wide tables. New defaults improve string compression, compact delete storage, and corruption validation.

The trade-off is forward compatibility. Newer DuckDB versions aim to read older files, but an older binary may not understand a file written with a newer storage version. DuckDB documents explicit storage compatibility settings and database copy/export paths for moving between versions. Before upgrading a fleet, inventory every producer and consumer of .duckdb files, including notebooks, scheduled jobs, embedded applications, and browser builds.

A safe rollout is deliberately boring:

Copy representative databases and test the preview on those copies.
Run correctness comparisons and workload-level benchmarks, not only microbenchmarks.
Verify every client and extension against the same release candidate.
Decide whether files must remain readable by an older deployment.
Back up the original files and rehearse export/import or database-copy recovery.
Upgrade consumers before allowing producers to persist the new default format.

A New Parser Makes SQL Extensible

DuckDB historically used a parser derived from PostgreSQL. Version 2.0 replaces it with a PEG-based parser designed and maintained for DuckDB. Users should mostly notice better source locations in error messages, not different query behavior.

The architectural payoff is grammar extension. Extensions can add syntax rather than forcing every domain feature through function calls or awkward string arguments. A Spark compatibility mode is the first explicit dialect target.

Parser replacement is also one of the clearest reasons to test real query corpora. Applications accumulate edge cases in generated SQL, quoting, comments, lambda syntax, parameter placement, and rarely used statements. Capture production queries with sensitive values removed, run them through the preview, and report incompatibilities before release.

Smaller Timezone Support, Stable Extensions

DuckDB previously depended on ICU for timezone-aware timestamps, calendars, and collations. Version 2.0 moves those capabilities into its own extension, built from IANA timezone data compressed to roughly 45 kB. The preview reports 2.2× faster conversion of 25 million timestamps and 2.6× faster German collation filtering over five million strings, while reducing the dependency footprint.

Extension authors get an even more consequential change. Many extensions currently build against an unstable C++ interface and must be rebuilt for each DuckDB release. The broadened stable C API is generated from a declarative, versioned specification, with CI checking that headers and ABI definitions cannot silently drift. An extension can target that interface once and remain binary-compatible across releases.

Organizations will also be able to host trusted extension repositories. A repository definition pins one or more RSA public keys, supports key rotation, and can point to HTTPS, S3, or a local path. That closes an important operational gap: internal extensions can use native install and load flows without being published to DuckDB’s public repositories.

“Stable” still needs a lifecycle policy. Teams should sign artifacts in CI, publish immutable versions, audit repository keys, test across supported DuckDB releases, and decide how vulnerable versions are revoked. A durable ABI reduces rebuild churn; it does not replace software supply-chain controls.

What to Evaluate Before the Release

DuckDB 2.0 invites experimentation, but the preview is not a reason to move every workload behind Quack. Start with the boundary that currently hurts.

If large database artifacts are copied between services, test one owner process plus remote clients. Measure query latency, concurrent readers and writers, recovery, backup, and rolling upgrade behavior. If object-store scans wait on network latency, compare request concurrency, transferred bytes, and end-to-end cost. If logs live in JSON strings, convert a representative sample to VARIANT and compare storage size, field extraction, schema drift, and Parquet interoperability.

For every path, preserve an embedded baseline. The strongest part of DuckDB 2.0 is not that the project abandoned “SQLite for analytics.” It is that the same engine can now stay inside one process when simplicity wins, cross a network when ownership must be shared, and push work to another database when moving data would be wasteful.

The DuckDB 2.0 preview describes the full release direction, while the Hacker News discussion shows why server mode resonates with teams already building single-owner wrappers and tenant-specific analytical services. The durable design rule is to keep the execution boundary explicit: run close to the data, move compact results, and choose the operational model that matches the workload rather than the novelty of the feature.

How an Experiment Loop Made a GPU Kernel 232x Faster

Vincent Tran — Sun, 16 Aug 2026 00:00:00 +0000

A coding agent can write a fast-looking GPU kernel in minutes. The difficult part is proving that the kernel is correct across awkward inputs, measuring it without noise, and deciding what to try after the easy optimizations are gone.

That distinction explains a striking result from a recent GPU Mode contest. Sankalp used Codex in an automated research loop to optimize batched QR decomposition on NVIDIA B200 GPUs. Across two weeks and more than 1,500 submissions, the measured runtime fell from a roughly 419,000-microsecond baseline to 1,805 microseconds: about 232 times faster. The entry finished 12th among 183 participants.

The multiplier is attention-grabbing, but it is not the most reusable lesson. The real achievement was turning an open-ended performance problem into a controlled experimental system. The agent could edit code, run correctness checks, benchmark individual shapes, inspect profiles, record evidence, and either keep or reject a candidate. Human expertise then improved the questions, the search space, and the evaluator.

This is a useful blueprint for any work where a machine can run many experiments but cannot be trusted to judge success by appearance.

The task: compact Householder QR

QR decomposition factors a matrix A into:

A = Q R

Q is orthogonal: its columns are perpendicular unit vectors. R is upper triangular: everything below its diagonal is zero. QR is used in least-squares solvers, eigenvalue methods, and many numerical pipelines.

The contest did not ask for dense Q and R matrices. It required the compact format produced by torch.geqrf. The upper triangle of one output matrix stores R; the cells below the diagonal store Householder reflector vectors. A separate tau vector stores one scalar for each reflector. torch.linalg.householder_product can reconstruct Q from that compact representation.

For every submission, the checker rebuilt the factors and tested three relationships:

A ≈ Q R
Qᵀ Q ≈ I
Qᵀ A ≈ R

Only correct submissions were timed. The score used a geometric mean across matrix sizes, batch sizes, and input distributions. That matters: an optimization that wins on one friendly shape can lose overall, and lower precision that passes ordinary random inputs can fail badly on ill-conditioned or rank-deficient matrices.

Why ordinary Householder QR underuses a GPU

A Householder reflector zeroes the entries below one diagonal element. If v is the reflector vector and tau is its scale, applying it has the form:

H x = x - tau · v · (vᵀ x)

The algorithm advances one column at a time. Reflector j + 1 depends on the matrix left by reflector j, so the panel factorization contains a real serial dependency. A direct implementation repeatedly performs matrix-vector-style work. GPUs can execute that work, but their fastest units are designed for large matrix-matrix operations.

The classic solution is blocked Householder QR. Instead of applying every reflector immediately to the entire trailing matrix, it factors a narrow panel of b columns. Those reflectors are collected into a compact WY representation:

H₁ H₂ … Hᵦ = I - V T Vᵀ

The large trailing block can then be updated with matrix multiplications:

W = Vᵀ A_trail
Z = Tᵀ W
A_trail = A_trail - V Z

That reorganization does not remove the serial work. It confines it to a narrow panel and converts most of the remaining work into GEMMs, which can use tensor cores efficiently. LAPACK uses blocked reflector forms for the same reason: grouping reflectors turns much of a Level-2, matrix-vector workload into Level-3, matrix-matrix work.

Keep the unavoidable dependency chain narrow; send the large trailing update to the hardware’s matrix engines.

This architectural change supplied the largest class of gains, but it also opened a new optimization surface: panel width, precision, data layout, kernel fusion, dispatch overhead, small-matrix packing, and shape-specific paths.

The evaluator was the product

The kernel was only one component. The research harness made repeated improvement possible.

Each loop began with the current best candidate and a written hypothesis. The agent made a bounded code change, ran the cheapest useful correctness test, submitted valid candidates to the official evaluator, collected per-shape timings, and compared the result with the incumbent. A regression was reverted; an improvement was promoted and documented.

The agent proposes changes, but the verifier decides what survives.

The distinction between an attempted run and evidence is crucial. A timeout does not prove that a kernel is slow or wrong. A noisy one-off measurement does not prove an improvement. A successful compile says nothing about numerical accuracy. The loop should promote a candidate only after a completed correctness check and a comparable benchmark.

This makes the evaluator more important than the prose prompt. A detailed prompt can suggest good behavior; an executable gate determines what the system actually rewards.

Shape-by-shape evidence prevents fake wins

The contest covered sizes from very small matrices to 4096 × 4096, with different batch counts and conditioning. Those cases stress different parts of the machine.

For small matrices, launch overhead and occupancy can dominate. Packing many independent matrices into one launch may help more than improving arithmetic. At large sizes, there may be too few matrices in the batch to fill the device, while memory movement and tensor-core utilization become decisive. Mixed precision may accelerate the common path but violate tolerances on difficult distributions.

A single aggregate number hides these mechanisms. The harness therefore needs a result table that preserves:

correctness by input family;
latency by shape and batch size;
compile and launch overhead;
profiler observations;
the exact candidate and parent revision;
whether the result was complete, timed out, or invalid.

The aggregate score is still useful for ranking. The detailed table explains why the score moved and which bottleneck should be attacked next.

Profiling turns “make it faster” into a hypothesis

Without a profile, an agent often tweaks tile sizes, warps, and configuration values because those are easy edits. That can produce early gains, but it eventually becomes blind hill climbing.

Profiles expose the limiting resource. Is the kernel spending time in PyTorch dispatch between Triton kernels? Are tensor cores idle while vector lanes perform panel work? Is conversion between FP32 and FP16 repeated? Are memory loads poorly coalesced? Does one shape suffer from low occupancy? Each answer suggests a different experiment.

Sankalp used both the remote evaluator and NVIDIA profiling. The strongest algorithmic direction was to make more of the computation matrix-shaped. Later improvements came from details such as reducing framework transitions, retaining useful intermediate representations, specializing awkward shapes, and studying instructions available on Blackwell.

NVIDIA’s current tcgen05 programming guide shows why hardware knowledge matters. Blackwell introduces fifth-generation tensor-core matrix instructions, tensor memory for accumulators, and CTA cooperation. An agent can search and implement these mechanisms, but a useful profile and a person who understands the missing capability can focus that search dramatically.

One incumbent creates a local maximum

The simplest optimization loop keeps one best candidate. Every new idea must beat it immediately or disappear. That policy is efficient for small parameter changes, but hostile to structural changes.

Suppose a new data layout is initially slower because the surrounding kernels are still designed for the old layout. The layout may enable a later fusion or remove a conversion, but it cannot survive long enough to receive that second change. The loop mistakes “unfinished” for “bad.”

A small beam of candidates fixes this. Keep several idea families alive:

an exploit candidate near the current best;
a near-miss that wins important shapes;
a structural candidate with greater upside;
optionally, a simplification candidate that removes overhead or dead machinery.

Each beam entry should record its parent, hypothesis, affected functions, best evidence, and next decision. Candidates should not live forever; retire them when repeated tests reject the mechanism or the profile shows that the targeted cost is no longer important. The point is to give promising structures enough runway to mature.

Idea diversity is not the same as generating many random variants. Five agents changing the same block size are one idea with five parameter values. Useful diversity means different explanations of the bottleneck.

Domain knowledge improves the loop in three places

The result does not show that expertise is obsolete. It shows where expertise has the highest leverage.

First, experts design the verifier. Numerical code needs tolerances, adversarial distributions, and independent invariants. A weak checker rewards shortcuts that do not generalize.

Second, experts choose representations. Recognizing that blocked Householder QR converts trailing updates into GEMMs is a higher-value move than tuning an unblocked kernel indefinitely.

Third, experts notice missing experiments. The postmortem found several: distribution-aware fast paths, more aggressive removal of library calls, keeping the trailing matrix in lower precision rather than repeatedly converting it, earlier use of a candidate beam, and deeper use of Blackwell-specific tensor instructions.

An agent expands implementation throughput. Expertise improves the map it explores.

A practical autoresearch workspace

A durable setup needs less orchestration than many teams expect. Start with a small set of files whose responsibilities are obvious:

optimization/
├── AGENTS.md # standing experiment rules
├── problem.md # interface, constraints, evaluator
├── candidate.py # current promoted implementation
├── experiments.md # hypotheses and decisions
├── beams.md # active idea families
├── results/ # raw correctness and timing output
├── profiles/ # profiler captures and notes
└── archive/ # rejected or superseded variants

The standing rules should be operational. Require a sanity check before expensive evaluation. Save raw output. Treat timeouts as inconclusive. Promote only from completed evidence. Record why a candidate was rejected so a fresh context does not repeat it. Profile after meaningful architecture changes or unexplained gains.

Then make the first loop deliberately narrow:

Define one immutable correctness gate.
Choose representative fast and full benchmark suites.
Establish a reproducible baseline.
Run one hypothesis per experiment.
Keep an append-only result ledger.
Add a beam only when structural ideas begin losing to the incumbent too early.
Review failures and evaluator blind spots regularly.

Automation should arrive after the experiment is trustworthy. A fast loop around an unstable benchmark merely produces wrong conclusions faster.

Where this approach works—and where it does not

GPU kernels are unusually suitable because they offer a hard correctness checker, a numerical score, short experiments, and a reversible artifact. The same shape appears in compiler optimization, database query planning, compression, model training, scheduling, and code-size reduction.

The method is weaker when evaluation is subjective, delayed, easy to game, or poorly correlated with the real goal. Product design, maintainability, and security cannot usually be reduced to one latency number. They can still use agent-driven experiments, but promotion needs multiple gates and human judgment.

There is also a compute budget. More than 1,500 submissions over 14 days consumed real accelerator time and shared queue capacity. A responsible loop separates cheap local checks from expensive remote evaluation, spaces submissions, caches comparable results, and stops exploring dead families.

The lasting lesson

The 232x speedup did not come from one magical prompt. It came from a chain of improvements: a formal checker, per-shape measurements, persistent logs, profiles, better numerical understanding, blocked QR, mixed-precision experiments, shape specialization, and a search policy that eventually preserved multiple ideas.

That is what productive autonomy looks like. Give the agent a bounded artifact it can change, a reliable way to observe reality, memory of previous attempts, and permission to reject its own work. Keep humans at the points where taste, representation, and evaluator design matter most.

The code generator is useful. The evidence loop is the system.

Sources: Sankalp’s experiment report, Hacker News discussion, GPU Mode QR leaderboard, Mike Lazo’s QR optimization write-up, PyTorch geqrf documentation, LAPACK QR factorization guide, and NVIDIA tcgen05 guide.

Build Wide, Ship Narrow: Decompose Pull Requests After the Code Teaches You

Vincent Tran — Fri, 14 Aug 2026 00:00:00 +0000

Software teams usually decide how to divide a feature before they build it. An RFC becomes an epic, the epic becomes tickets, and each ticket is expected to become a pull request. This feels orderly because the work arrives at review in small pieces.

It also forces the team to choose its boundaries at the moment it understands the system least.

Before implementation, an engineer can estimate where the API ends, which data model is stable, whether the UI can ship independently, and which migration must come first. But those are still estimates. The first working path often reveals that two planned components are inseparable, a “small” abstraction owns most of the risk, or an allegedly independent frontend slice cannot be evaluated without the backend behavior beside it.

AI coding tools change the cost of responding to that new information. They make exploratory implementation faster, but the important shift is not simply that more code can be produced. The tedious work of reorganizing a completed branch—tracing dependencies, moving changes onto fresh branches, repairing imports, and checking each resulting diff—can now be assisted too.

That enables a different sequence:

Decide what the feature must do and which constraints it must respect.
Build the complete path on an exploratory branch.
Demonstrate the working product and correct the design.
Discover the real dependency boundaries in the finished code.
Recut the work into the smallest safe pull requests.
Review and merge those narrow changes, with removal last.

The idea is not “skip design.” It is to separate product design , which still happens early, from delivery decomposition , which can happen after the code has supplied evidence.

Two Decisions That Should Not Be Confused

A useful plan answers questions that would otherwise turn into expensive rework:

What user problem are we solving?
What should happen when an external call fails?
Which operations require authorization or audit records?
What data must remain compatible?
Which performance and reliability limits matter?
How will we know the result is correct?

Those decisions belong before implementation because they define the problem and its safety boundaries.

A pull-request plan answers a different set of questions:

Which changes can merge independently?
Which diff gives a reviewer one coherent idea to inspect?
Which branch truly depends on another?
Where is the safest rollback boundary?
Can an intermediate state deploy without breaking users?

These are delivery questions. Some can be predicted, but the implementation often gives better answers. A database endpoint that looked like infrastructure may turn out to contain the central policy decision. Two UI tickets may collapse into one small shared view. A cleanup task may not exist until the replacement path works.

Committing a design document early is still valuable when the feature introduces a new protocol, trust boundary, storage model, or irreversible migration. What should remain provisional is the assumption that the document’s sections must become the final branch graph.

The Wide Build Is a Laboratory, Not a Giant PR

“Build wide” means keeping the end-to-end experiment together long enough to learn from it. It does not mean opening a 5,000-line pull request and asking someone else to untangle it.

The exploratory branch has a private audience: its author and the tools helping them. Commits are save points rather than polished review units. Make one when a concept works, before a risky rewrite, or after a test starts passing. These waypoints preserve recovery and make later archaeology easier, but they are not required to tell the final story.

Working end to end is particularly useful when a feature crosses surfaces. A new user setting might require a schema field, API validation, permissions, a UI control, analytics, and a rollback path. Building only the schema first proves that a column can exist; it does not prove the setting feels right or that the API contract serves the interface cleanly.

The complete path exposes integration mistakes while changing direction is still cheap. It also gives tests something meaningful to exercise. A thin vertical slice can reveal the real invariants faster than several carefully polished horizontal layers.

Design the behavior first, but wait for working evidence before fixing the final review boundaries.

Treat the branch accordingly:

Keep it short-lived enough that rebasing does not become its own project.
Run tests continuously; “exploratory” does not mean unverified.
Record non-obvious decisions in commits or a scratch document.
Avoid unrelated cleanup that would obscure the later split.
Never treat the wide branch as the only copy of important product reasoning.

The branch is scratch paper, but it should be legible scratch paper.

Demo Before Review

Code review is a costly place to discover that the product is wrong. A reviewer may spend an hour understanding a clean implementation only to learn that the workflow confuses users or that the endpoint shape makes the next screen awkward.

Once the wide build works, show the behavior before asking anyone to read the code. Depending on the change, that can be a preview deployment, a short screen recording, a command transcript, a benchmark, or an API session. Invite the people who understand the user need, not only the people who understand the repository.

This feedback answers a high-value question: should this be the thing the team ships? If the answer is no, rewrite the exploratory branch. Nobody has reviewed a doomed diff, no stack needs rebasing, and no carefully staged migration has to be reversed.

The demo is not a substitute for technical review. It validates the result while review validates the implementation and transfers ownership. The sequence matters: first confirm that the product deserves to exist, then invest human attention in how it works.

Turn the Finished Diff into a Dependency Graph

After product validation, compare the wide branch with main. Do not begin by preserving its commit history. Begin by inventorying the behavior and the dependencies the code revealed.

For each coherent change, ask:

Does it compile and test without the rest of the feature?
Can it deploy without exposing an incomplete or unsafe state?
Does it provide value by itself, or at least enable one clearly named next change?
Can a reviewer understand its purpose without mentally loading the full feature?
Does reverting it have a clear effect?

The answers produce a graph rather than a simple list. Independent backend capabilities can be sibling branches from main. A UI that consumes one endpoint should sit above that endpoint. Instrumentation can often be independent. Removal of the old implementation usually depends on every replacement path and belongs at the end.

Sibling branches reduce rebase coupling; stack only the edges that represent real code dependencies.

This graph is the proposed review plan. A human should approve it before branches are created. An agent can trace changed symbols and move patches, but deciding whether an intermediate state is meaningful, operable, and safe remains an engineering judgment.

Cut Fresh Branches, Not Decorative Diffs

A narrow pull request must be real. Hiding files from a giant diff, asking reviewers to inspect commits in a particular order, or describing “logical sections” in the PR body does not create independent merge and rollback boundaries.

Create each root PR from the latest main. Use a Git worktree when several branches need to be assembled and tested side by side. Move the smallest coherent patch into each branch, then repair whatever the extraction exposes: missing types, implicit setup, tests coupled to later behavior, and imports that belonged to the experiment rather than the product.

Dependent work should branch from the PR it genuinely needs. GitHub’s gh stack models this directly: the bottom branch targets the trunk, and each higher branch targets the layer beneath it so the review shows only that layer’s diff. The tool can maintain and submit the stack, but it cannot decide whether the dependency is legitimate.

Every extracted branch needs its own validation. At minimum:

build and type-check it without later branches;
run tests for the behavior it owns;
verify migrations can coexist with the currently deployed application;
confirm hidden or dormant paths remain unreachable until their consumer lands;
inspect the actual PR diff, not only the final wide-branch result.

This is where late decomposition pays for itself. If a supposedly independent patch cannot build alone, the failure reveals either a missing prerequisite or a false boundary. Fix the graph instead of disguising the dependency.

Stack Only When the Dependency Is Real

Stacks are useful, but every edge adds coordination cost. Feedback on a lower PR may require rebasing every branch above it, rerunning CI, and asking reviewers to distinguish old changes from new ones.

The default should therefore be siblings from main. Use a stack when one change cannot be expressed, built, tested, or understood without another. A frontend view that imports a new API type is a real dependency. Two endpoints created during the same experiment are not automatically dependent. Neither are two cleanup patches that merely touch neighboring files.

Small-batch guidance from DORA emphasizes independent, valuable, testable units and notes that small batches are especially important when AI increases delivery speed. Late decomposition should preserve that goal. It changes when the batch boundaries are discovered; it does not excuse shipping a coupled batch under several PR numbers.

A good stack is short and obvious. If the graph contains long chains, multiple diamonds, or branches that repeatedly exchange changes, the system may need a stabilizing interface, feature flag, or branch-by-abstraction step before review can be clean.

Put Deletion Last

Replacing an old path combines two different risks: whether the new path works and whether the old path is truly unused. Mixing construction and deletion makes both harder to review and rollback.

Ship removal in a final PR after the replacements have landed and, when possible, after production evidence shows they are carrying traffic correctly. A deletion-only diff is unusually honest. Reviewers can focus on references, compatibility, fallback behavior, documentation, and operational scripts without also proving a new architecture.

This ordering enables a safer rollout:

Add foundations that do not change behavior.
Add the new path behind a flag or dormant entry point.
Add consumers and enable the path gradually.
Observe errors, latency, and adoption.
Remove the old path and its compatibility code.

The final PR often deletes more than the feature added. That is a useful signal: the team has finished the replacement rather than leaving two systems to coexist forever.

Review Is for Understanding, Not Ceremonial Approval

AI can catch formatting errors, suspicious patterns, missing tests, and obvious inconsistencies. It can summarize a diff and answer questions about call sites. These capabilities reduce mechanical work, but they do not eliminate the human purposes of review.

Review transfers ownership. Someone must understand why the change belongs in this layer, which assumptions make it safe, how it will fail, and what the team will do six months later. Microsoft Research has argued that code review is a costly social and knowledge process, not simply a dependable bug-finding gate. That framing matters more when the author did not type every line.

The author should read each extracted diff as if it came from another engineer. Explain the intent without leaning on hidden context from the wide branch. Re-run the important path. Challenge surprising abstractions. Remove generated complexity that no longer serves the validated result.

Small PRs do not guarantee good review, but they make deep review possible. A narrow diff gives both author and reviewer a bounded object they can hold in working memory and a focused conversation they can complete.

Where the Workflow Breaks Down

Late decomposition is not a universal default.

Irreversible migrations need early sequencing. If a schema or data rewrite must roll through production in compatibility phases, the order is part of the design. Build and test the full migration, but preserve expand-migrate-contract boundaries from the start.

Long-running branches accumulate integration risk. A wide experiment that lasts weeks becomes expensive to rebase and may hide conflicts until the split. Set a time box. If exploration grows beyond it, extract stable foundations early.

Some features have one atomic release boundary. A cryptographic protocol change, tightly coupled compiler pass, or externally versioned API may not deliver incremental user value. Narrow PRs can still improve review, but deployment may remain all at once.

A weak test suite makes recutting dangerous. The final branch may work because of accidental ordering or unrecorded local state. Without reliable build, integration, and migration checks, moving patches between branches can silently change behavior.

Generated volume can outrun review capacity. Faster decomposition does not mean reviewers can absorb unlimited change. If the resulting PRs sit open, the process produced tidy inventory rather than delivery.

Exploration can become an excuse for architectural drift. The wide branch still needs constraints, observability, security review, and stop conditions. “We will clean it up later” is not a plan unless cleanup is actually extracted, reviewed, and merged.

A Practical Team Policy

Teams adopting this workflow can keep it disciplined with a short policy:

Use it for cross-surface features and uncertain refactors, not every one-file change.
Require a written design first for new trust boundaries, protocols, or migrations.
Time-box the wide branch and keep it continuously testable.
Validate the working behavior before code review begins.
Require a dependency graph and human approval before branch extraction.
Branch independent PRs from main; stack only real dependencies.
Give every PR its own build, tests, deployment safety, and rollback story.
Keep removal in a final, focused PR.
Track time from first PR opened to last PR merged, not only time to split.

Measure outcomes rather than admiring the method. Useful signals include review latency, number of rebase rounds, escaped defects, revert scope, age of the final cleanup PR, and how often extracted branches failed independent validation. Compare those with conventionally planned work of similar size.

The goal is not to maximize the number of pull requests. It is to put uncertainty in the cheapest part of the process. Explore while the code is private, validate while changes are easy, and spend reviewer attention only after the product and the dependency boundaries have earned it.

The original Build Wide, Ship Narrow article describes the workflow that prompted this model. The Hacker News discussion captures the community’s debate about branch history, stacked reviews, and whether late slicing truly improves delivery. The durable lesson is simpler than any particular tool: decide the problem early, but let evidence decide the final shape of the review.

Compression Is Prediction: Why Better Guesses Need Fewer Bits

Vincent Tran — Wed, 12 Aug 2026 00:00:00 +0000

What does a ZIP file have in common with a language model? Both are in the business of guessing what comes next.

That sounds strange because their outputs look nothing alike. A compressor gives us a smaller file that can be reconstructed exactly. A language model gives us new tokens. Underneath those different jobs, however, sits the same useful object: a probability distribution over the next symbol.

Once we have those probabilities, an entropy coder can turn good guesses into short bit sequences. A likely symbol costs few bits. A surprising symbol costs many. Compression ratio therefore becomes a score for prediction quality—provided we count the model and its operating cost honestly.

This connection is more than a clever analogy. It explains why context improves compression, why cross-entropy is measured in bits, how a compressor can become a generator, and why a giant neural network is usually the wrong tool for shrinking an HTTP response.

Compression begins with redundancy

Lossless compression works because real data is rarely random. Source code repeats keywords and indentation. Logs repeat timestamps, field names, and service identifiers. English text has common letters, common words, and strong grammatical patterns. Images contain nearby pixels with related colors.

The simplest example is run-length encoding. A sequence such as:

AAAAAAAAAABBBBCC

can be described as ten As, four Bs, and two Cs. The description is shorter because the input contains long runs. But the same method makes ABCDABCD larger, not smaller. A transform only helps when it matches the structure of the data.

General-purpose compressors use richer structure. DEFLATE, the format behind gzip, combines LZ77 back-references with Huffman codes. Its back-references replace repeated byte sequences with a length and a distance, while its codes give shorter representations to frequent values. The DEFLATE specification allows a reference to reach up to 32 KiB into earlier input.

These techniques look different, but both exploit predictability. Repetition makes the next bytes easier to anticipate. Skewed frequencies make some symbols much safer guesses than others.

Separate the model from the coder

A helpful mental model divides a compressor into three parts:

A transform rearranges or describes the input so useful patterns become easier to see.
A probability model estimates how likely each possible next symbol is.
An entropy coder converts the real symbol and its estimated probability into bits.

The boundaries are not always clean in production formats, but the split reveals where compression gains come from. Once an entropy coder is already close to the theoretical limit, improving the final coding trick offers little room. Improving the probability model can still reduce the limit itself.

Imagine a source with four possible symbols:

Symbol	Model probability
A	1/2
B	1/4
C	1/8
D	1/8

A sensible binary code uses one bit for A, two for B, and three each for C and D. Common events receive short paths; rare events receive long ones. This is the core idea behind Huffman coding.

The ideal information cost of an event with probability p is:

cost = -log2(p) bits

A probability of 1/2 costs one bit. A probability of 1/8 costs three. A probability of 1/1024 costs ten. Every time probability halves, the bill rises by one bit.

This formula gives prediction an exact price. A confident correct guess is cheap. A confident wrong guess is expensive.

Entropy is the average bill

Shannon entropy is the model’s expected number of bits per symbol:

H(P) = -Σ p(x) log2 p(x)

For the four-symbol distribution above, the average is 1.75 bits per symbol. A fixed-width code would need two bits because it must reserve equal space for four possibilities. A variable-length or arithmetic code can approach the lower average by spending fewer bits on common inputs and more on rare ones.

Entropy is not a universal property of a file detached from assumptions. It depends on the distribution used to describe that source. If we pretend every character is equally likely, we get one bound. If we know the language, file type, previous symbols, or document structure, we get another.

That is why there is no single best compressor. A model tuned for genomic sequences will see patterns that a generic byte model misses. A compressor for executable code can exploit structure that does not occur in photographs. The coder may be excellent in every case; the model decides which regularities are visible.

Arithmetic coding turns probabilities into ranges

Huffman coding assigns a whole-number count of bits to each symbol. That is fast and practical, but probabilities do not normally align to powers of two. A symbol with probability 0.7 has an ideal cost of about 0.515 bits, yet no standalone prefix code can give one symbol half a bit.

Arithmetic coding avoids that rounding problem by encoding a sequence as a position inside a range.

Start with the interval [0, 1). Divide it into sections whose widths match the symbol probabilities. If A has probability 0.7, its section occupies 70% of the interval. After reading the real next symbol, keep only that symbol’s section. Divide the remaining section again using the probabilities for the following symbol, then repeat.

For a sequence A B A, the interval narrows three times. At the end, any binary fraction that lands inside the final interval identifies the whole sequence, as long as the decoder has the same model and knows when to stop. More probable sequences preserve wider intervals, and wider intervals require fewer binary digits to identify.

Practical implementations do not store an infinitely precise floating-point number. They maintain integer ranges, emit stable leading bits as the interval narrows, and renormalize continually. The elegant interval picture explains the mathematics; range coders and related techniques make it work on real machines.

The important point is not the specific coder. It is that the number of emitted bits tracks the model’s probability for the observed data.

Context makes a model useful

A table of global character frequencies is weak. In English, u is not especially common overall, but after q it becomes highly likely. In source code, ) may be ordinary globally but strongly expected after a function’s final argument. In a JSON document, a colon becomes likely after a closing quote in an object key.

An order-0 model ignores history. An order-1 model conditions its probabilities on the previous symbol. Higher-order models look at longer contexts:

P(next symbol)
P(next symbol | previous symbol)
P(next symbol | previous N symbols)

Longer context can sharpen predictions. If the actual next symbol receives more probability, its code becomes shorter. But simply storing a table for every possible context grows expensive and sparse. Most long contexts appear rarely, so estimates become unreliable.

Compression research has produced many ways to balance context and cost: backing off to shorter histories, mixing predictions from several models, updating counts online, and using learned representations rather than exact context tables. Modern language models are an extreme version of that search. They use a large parameterized model to turn a long token context into probabilities for the next token.

A language model already produces what a coder needs

During generation, a language model receives existing tokens and returns logits, which are converted into probabilities. A sampling rule chooses a token, appends it, and repeats.

Compression changes only the choice step. The encoder already knows the real next token because it is reading the source document. It asks the model for probabilities, looks up the probability assigned to that real token, and sends the token through an entropy coder. The decoder runs the same model on the reconstructed context and uses the coded bits to recover which token came next.

The loop is deterministic:

context -> model probabilities -> entropy decode -> next token -> new context

Both sides must use exactly compatible tokenization, model weights, arithmetic, and coding rules. A tiny disagreement changes later context, which changes later probabilities, and the entire reconstruction diverges. Reproducibility is therefore part of the format, not an implementation detail.

The connection also explains the standard training objective. Cross-entropy measures the average negative log probability assigned to real next tokens. When logarithms use base two, the unit is bits per token. Reducing language-model loss means assigning more probability to the observed data; in coding terms, it means describing that data with fewer bits.

Google DeepMind’s Language Modeling Is Compression makes the equivalence operational. The researchers evaluated language models as general-purpose compressors and also reversed the direction, using ordinary compressors to build conditional generative models. Their results connect scaling, tokenization, in-context learning, prediction, and code length under one measurement.

A compressor can generate, too

The relationship runs both ways. A compressor implicitly defines probabilities over continuations: data that it can encode cheaply is data it considers unsurprising.

Suppose we want to compare candidate continuations after a prompt. Append each candidate, compress the result, and observe the extra code length. The continuation that adds the fewest bits is, in a rough sense, the one the compressor predicts best. With the right construction, these code-length differences can be normalized into a probability distribution and sampled.

This does not turn gzip into a strong conversational model. Its short sliding window and byte-level pattern matching capture local repetition, not rich semantic structure. But it demonstrates that generation and compression are not separate species of computation. They are two ways of using a model’s probability distribution:

Compression follows the observed data and records the choices efficiently.
Generation samples choices from the model and creates new data.

Why we do not ship an LLM with every archive

If a neural model predicts well, why not replace gzip everywhere? Because compressed size is only one line in the cost ledger.

The decoder needs the model. When sender and receiver do not already share it, model weights are part of the total description. A multi-gigabyte predictor is absurd overhead for a 100 KiB web response. Even for a huge corpus, the compute and memory needed for token-by-token inference may dominate the storage saved.

Real compression systems optimize several constraints at once:

compressed size;
encoding and decoding speed;
peak memory;
random access and streaming behavior;
model or dictionary distribution;
deterministic portability;
error recovery and format longevity.

HTTP compression prizes fast, tiny decoders that are already installed in browsers. Backups may accept slower encoding for smaller long-term storage. Satellite links may justify a shared domain model because every transmitted bit is costly. Archival formats may reject a learned model if reproducing its runtime decades later is uncertain.

There is also a crucial accounting distinction. If a pretrained model is already present for another reason, its marginal cost may be close to zero. If the compressed artifact must be self-contained, the model cost cannot be ignored. Claims of spectacular neural compression often depend on which side of that boundary the weights occupy.

Compression is an honest model test—with conditions

Prediction benchmarks can be distorted by sampling settings or subjective evaluation. Lossless compression supplies a hard score: reconstruct the input exactly and count the bits. A model that assigns better-calibrated probabilities to the real sequence earns a shorter code.

But the score answers a narrow question. It measures how well a probability model fits a data distribution after accounting for the agreed coding setup. It does not, by itself, prove reasoning, factuality, usefulness, or understanding. A compressor can exploit syntax in a language its designer cannot read. A language model can predict fluent text while being wrong about the world.

Fair comparisons must also include:

the model or dictionary size;
tokenizer and metadata overhead;
the precision and determinism requirements;
compute used by both encoder and decoder;
whether training data overlaps the test set;
performance on data outside the expected distribution.

These conditions do not weaken the connection. They keep us from turning a precise mathematical equivalence into a vague claim about intelligence.

The practical lesson: improve the probabilities

When an entropy coder is already efficient, better compression comes from finding a better description of the source.

That might mean a transform that exposes repetition, a dictionary shared by both endpoints, a context model specialized for the file type, or a learned predictor for a large and valuable dataset. It might also mean choosing a simpler model because its speed and portability beat a small gain in ratio.

The most useful way to think about a compressor is therefore not as a machine that squeezes bytes. It is a machine that makes a sequence of probabilistic bets and then writes down what actually happened. Good bets are cheap to record. Bad bets are expensive.

Language models make the same bets before choosing their next tokens. The machinery around them is newer and much larger, but the bill still arrives in bits.

Postgres CDC Without the Connector Tangle

Vincent Tran — Tue, 11 Aug 2026 00:00:00 +0000

Moving data from an operational database into an analytical warehouse sounds simple: copy the existing rows, then forward every insert, update, and delete. The difficulty appears as soon as the source stays online.

A snapshot takes time. Writes continue while it runs. Tables are renamed, columns are added, networks fail, processes restart, and consumers may replay a batch after an uncertain acknowledgement. A pipeline that merely reads the Postgres write-ahead log (WAL) has to reconcile all of these events without losing data or exposing a destination state that never existed at the source.

Snowflake’s new data-mirroring design takes an unusual route. Instead of running a connector that continuously pulls a logical-decoding stream over the network, a Postgres extension pushes transactional change batches into Apache Iceberg tables in object storage. Snowflake later applies those batches to analytical tables in separate transactions. A metadata log connects the two sides.

The product is specific to Snowflake Postgres and is currently in public preview, but the architecture is more broadly useful. It shows how moving capture closer to the source, using durable storage as a handoff boundary, and preserving database transaction boundaries can remove whole categories of CDC failure.

Why Ordinary CDC Becomes Fragile

Postgres already provides the essential capture primitive. Logical decoding translates physical WAL records into a stream of row-level changes. A replication slot remembers how far a consumer has progressed, and an output plugin decides how those changes are represented.

That stream is necessary, but it is not a complete replication system.

An external consumer still has to answer difficult questions:

Where does a table snapshot end and the change stream begin?
What happens to writes made while the snapshot is running?
How should DDL be ordered relative to DML in the same transaction?
If an acknowledgement is lost, should a batch be replayed?
How are new and removed tables handled?
Can several destination tables advance to one consistent source point?
What happens when a replication slot disappears after failover?

The external process also has an incomplete view of the source. A silent connection may mean Postgres is busy, the network is partitioned, the connector is stuck, or the server has failed. Schema state visible now may differ from the state that was valid when an older WAL record was written.

Many systems cope with uncertainty by making destination writes idempotent. Every source operation becomes an upsert keyed by a primary key, and retries are considered safe. That is practical, but expensive on large columnar tables: each insert may require a lookup or scan to prove that the row does not already exist. It also does not automatically solve cross-table consistency or the snapshot-to-stream boundary.

Push Capture into the Database

Snowflake’s design moves the capture coordinator into Postgres as a background extension called snowflake_cdc. The extension uses logical decoding, but it does not expose an endless stream that an external connector must keep open. It writes finite, durable batches into per-table Iceberg change logs and records instructions in a separate metadata log.

This changes the relationship between producer and consumer.

Postgres knows when transactions commit, when schemas change, which catalog version belongs to a WAL record, and when a snapshot aligns with ongoing writes. The extension can coordinate capture with that local state. Object storage then decouples capture from apply: Snowflake does not need to be available at the exact moment Postgres produces a batch, and Postgres does not wait for a warehouse merge to finish.

The handoff looks like this:

Postgres transaction
      ↓
WAL and logical decoding
      ↓
transactional change batches + metadata log
      ↓
Iceberg tables in object storage
      ↓
transactional apply in Snowflake

The object store is not merely a queue replacement. Iceberg supplies table metadata and atomic commits over immutable data files. Compressed Parquet keeps the handoff columnar and inexpensive to scan. Because the durable change log exists independently of either execution process, capture and apply can restart from explicit checkpoints rather than infer state from a live connection.

One Timeline, Four Positions

A useful way to reason about this system is as four processes moving along one database timeline:

Write changes Postgres tables and appends WAL at the current point.
Decode reads older WAL and reconstructs logical row changes.
Capture commits finalized batches and metadata to Iceberg.
Apply advances Snowflake tables to a completed source boundary.

All four run continuously, but each is at a different log sequence number (LSN). Lag is the distance between those positions, not one vague end-to-end number.

Logical decoding uses historic catalog snapshots to understand WAL records using the schema that existed when the write occurred. That detail matters. If a column is dropped after a transaction commits but before its WAL is decoded, interpreting the old record through the current schema could produce nonsense.

The decoder first writes temporary batch files. At a boundary, it finalizes the files and informs the capture worker. Capture appends them to the appropriate Iceberg change logs, records the operation in the metadata log, and advances the captured LSN. Schema changes travel through the same ordered path and may start a new change-log generation.

The metadata log is effectively a replay program. Snowflake’s apply process reads it as a finite-state machine: create a target, install a snapshot, apply adjacent change batches, change a schema, add or remove a table, or recover by replacing state with a newer snapshot.

This explicit instruction stream is stronger than trying to reconstruct global order independently from per-table feeds.

Snapshots and Changes Must Share a Boundary

Initial loads are one of the most failure-prone parts of CDC. A table may contain billions of rows, so a snapshot can run for hours while fresh writes continue.

A naive sequence loses data:

Start copying the table.
Finish the copy.
Begin reading changes.

Writes committed between steps one and three may fall into a gap. Starting the change stream first avoids the gap but creates overlap: changes already reflected in the snapshot can be replayed again.

The mirroring extension can take a Postgres snapshot while decoding changes and record exactly how the two relate. The destination installs the snapshot and begins applying only the correct subsequent batches. If an unrecoverable capture gap appears, the metadata protocol can instruct the destination to consume a replacement snapshot instead of asking an operator to repair an ambiguous stream by hand.

This same mechanism handles a table added after mirroring has already started. The new table receives its own aligned snapshot and then joins the continuous change timeline without pausing every other table.

Transactions Are the Recovery Protocol

Distributed pipelines often accumulate checkpoints, deduplication keys, retry ledgers, and compensating jobs because they lose the strongest primitive their databases already provide: transactions.

This design uses transactions on both sides of the durable handoff.

On the source side, pg_lake can commit data and metadata changes to multiple Iceberg tables as one Postgres transaction. On the destination side, Snowflake can apply several adjacent batches across multiple target tables in one transaction. The target advances to a known Postgres transaction boundary or does not advance at all.

That has two important consequences.

First, retry logic becomes simpler. If a transaction failed, repeat it. If it committed, the checkpoint proves which work is complete. There is no half-applied batch to reverse.

Second, readers of materialized target tables do not observe broken cross-table invariants. If an order and its line items were committed together in Postgres, they become visible together in Snowflake. Joins do not see the order at one source position and the line items at another.

This is not globally synchronous replication. Postgres commits before Snowflake applies the batch, so analytical data still lags. The guarantee is about a consistent historical boundary, not zero latency.

Delete and Append Beat Universal Upserts

Conventional CDC commonly translates all operations into key-based upserts. That gives replay tolerance, but it is a poor default for an append-heavy workload landing in columnar storage.

The mirroring system can emit exact delete and insert records because snapshot overlap and replay position are already controlled. Updates become a delete/insert pair. Pure inserts can be appended without scanning a large target table for matching keys.

Snowflake can also combine multiple pending batches before applying them. This amortizes fixed work and allows a larger refresh interval to reduce compute cost. The trade-off is a less frequently updated materialized table.

The architecture separates freshness from compaction through live views. Each mirrored table has three useful forms:

the materialized target table, updated on the configured apply interval
a $changes Iceberg table containing recent captured operations
a $live view that overlays unapplied changes on the target

The live view can surface changes at roughly the auto-refresh cadence—documented as about 30 seconds—without running full target-table apply that often. Filters and projections can be pushed into scans of both the base table and Parquet change files, so queries do not always pay to process the entire backlog.

There is an important limitation: Snowflake documents $live views as non-transactional. Change files arrive incrementally, so a live query can observe only part of a multi-table source transaction. Use the materialized target tables when cross-table transactional consistency matters. Use live views when lower lag is worth weaker read semantics.

Failure Recovery Is Still Operational Work

The architecture removes infrastructure, not responsibility.

Logical replication slots must be healthy. A stalled consumer can force Postgres to retain WAL, filling storage. Modern Postgres can synchronize failover-enabled logical slots to standbys, but the synchronization is asynchronous and must be verified before promotion. If required WAL is lost anyway, the system needs a new aligned snapshot.

Operators should monitor at least four positions:

current source WAL LSN
decoded LSN
captured LSN committed to Iceberg
applied source boundary in Snowflake

Those measurements locate the bottleneck. A growing write-to-decode gap suggests pressure inside Postgres. A decode-to-capture gap points to batching or object-storage writes. A capture-to-apply gap points to Snowflake scheduling, apply capacity, or cost settings.

Storage needs equal attention. Change logs, metadata, snapshots, and retained WAL are durable by design; durability becomes a bill and, without retention controls, a capacity risk. The mirror’s serverless apply jobs and Iceberg storage should be attributed per pipeline rather than hidden inside one platform total.

Schema Evolution Needs Explicit Semantics

“Supports schema changes” is not a sufficient contract. Teams need to know what each operation means at the destination.

Adding a nullable column is relatively simple. Dropping or renaming one can be harder because old change files still use the previous layout. Type changes may be binary-compatible, require a cast, or make replication impossible. A primary-key change affects how future deletes identify old rows.

The ordered metadata log lets DDL share the same timeline as DML, but it cannot make incompatible types compatible or infer business intent. Production evaluation should test the exact migration patterns an application uses:

add, backfill, and constrain a column
rename a column while writes continue
widen and narrow numeric or text types
change primary or replica identity keys
create, truncate, detach, and drop tables
roll back an application release after its migration has replicated

Treat unsupported DDL as a planned stop condition with an alert and recovery procedure, not an edge case to discover during deployment.

What Is Open and What Is Product-Specific

The system uses open building blocks: Postgres logical decoding, Parquet, Iceberg, and the Apache-licensed pg_lake extension. That does not make the complete mirroring feature portable.

The HN discussion highlighted that the dedicated CDC extension and Snowflake apply machinery are not part of the public pg_lake repository. Data mirroring is tied to Snowflake Postgres and Snowflake’s transactional apply implementation. It is also a preview feature, so availability, behavior, limits, and pricing can change.

That distinction matters when evaluating lock-in. Writing changes to open Iceberg tables improves inspectability and creates a useful architectural boundary, but end-to-end recovery semantics live in the producer extension, metadata protocol, and consumer state machine—not in the file format alone.

A Design Checklist for Any CDC System

Whether you use this product, another managed service, or build an internal pipeline, ask the same questions:

Snapshot boundary: How are concurrent writes aligned with the initial copy?
Transaction boundary: Can related tables advance atomically?
DDL ordering: Is schema state versioned with data changes?
Durable handoff: Can producer and consumer restart independently?
Replay rule: What proves that a batch was applied exactly once?
Failover: Do logical slots survive primary promotion, and how is readiness verified?
Backpressure: What fills first when the destination stops—WAL, local disk, object storage, or memory?
Freshness modes: Are low-lag reads weaker than materialized-table reads?
Cost curve: How do batch size, apply interval, and live-query frequency affect spend?
Exit path: Which components and data remain usable outside the vendor’s control plane?

The best CDC system is not the one with the smallest demo lag. It is the one whose behavior remains understandable during snapshots, schema changes, retries, failovers, and long outages.

The Durable Idea

The most valuable idea in this design is not “put CDC inside Postgres” by itself. It is assigning each concern to the layer with the best information.

Postgres owns transaction order, historic schema context, and snapshot alignment. Object storage owns a durable, scalable boundary between systems. Iceberg owns atomic table metadata over columnar files. Snowflake owns the expensive work of mapping row changes into analytical tables. Transactions on both sides turn retries into normal control flow instead of incident response.

That division does not eliminate trade-offs. Live views weaken consistency, preview software carries product risk, replication slots still need monitoring, and the complete system is vendor-specific. But it replaces an opaque, permanently connected process with an ordered sequence of durable state transitions.

For replication, clockwork does not mean nothing can fail. It means every component knows which tick completed and which one must run next.

Sources

Kitesurf: A Browser Built for AI Agents, Not People

Vincent Tran — Sat, 08 Aug 2026 00:00:00 +0000

A browser is an expensive way to fetch a page.

That sounds unfair because a modern browser does far more than fetch. It decodes video, drives a GPU, manages tabs and extensions, keeps years of local state, synchronizes profiles, paints smooth animations, exposes accessibility APIs, and survives almost every strange page the web can produce. People need that breadth. An AI agent that wants a product price, a documentation section, or a screenshot often does not.

Kitesurf starts from that mismatch. Cloudflare has built a new browser engine for automation, running on V8 isolates inside Workers. It speaks familiar automation protocols, executes page JavaScript, builds a DOM, applies CSS, and can return HTML, screenshots, or PDFs. But it deliberately gives up the promise of being a complete desktop browser.

The result is not “Chromium, made smaller.” It is a different answer to a narrower question: what is the minimum useful browser for a machine?

Why agents need a different browser budget

Chromium is the safe default for browser automation because compatibility matters. If a site works in Chrome, it will probably work in headless Chrome. The cost appears when the workload becomes large and bursty.

Imagine an agent service that receives ten thousand requests to inspect pages. Giving each request a conventional browser process means paying for a general-purpose human interface even when the task only needs a DOM query. Warm browser pools reduce startup time, but they keep memory allocated between tasks and create state-management work of their own.

An agent has different priorities:

It values structured DOM content, network activity, forms, and screenshots.
It needs automation protocols more than tabs, themes, or extensions.
It can tolerate small rendering differences when extracting text or finding controls.
It benefits from short-lived, isolated sessions because every page is untrusted.
It needs predictable CPU, memory, and context costs at large concurrency.

Kitesurf optimizes for those priorities. It is an ephemeral engine meant to exist for one task, disappear, and scale out again when the next burst arrives.

That focus is important. A browser made for agents is not necessarily a browser that imitates a person more convincingly. Kitesurf is designed to expose the web as a computable environment, not to hide automation from websites.

A browser split across isolates

Kitesurf divides a page session into a small set of components. Only one of them owns durable session state; the rest are designed to be disposable.

SandboxOutbound: one guarded path to the network

Rendering a page means downloading hostile input: HTML, scripts, stylesheets, fonts, images, and WebAssembly. Kitesurf does not let every component fetch those resources directly. A dedicated SandboxOutbound Worker is the only route to origins.

That boundary enforces CORS, adds browser-shaped headers, filters responses, and keeps cookies in a jar scoped to the page. A disallowed request receives a 403. The design turns network access into an explicit capability instead of ambient power shared by the whole browser.

This is useful beyond browser engineering. Agent systems are safer when the component interpreting untrusted content is not also the component holding unrestricted credentials and network access. Isolation does not solve prompt injection, but it reduces the number of paths through which a bad page can become a broader systems problem.

Engine: protocol gateway and session owner

The Engine is Kitesurf’s public face. It accepts Chrome DevTools Protocol (CDP) connections over WebSocket and HTTP, coordinates a page, and stores session state.

Choosing CDP gives Kitesurf immediate access to an existing ecosystem. Puppeteer, Playwright, chrome-remote-interface, Chrome DevTools, and MCP adapters already know how to talk to that protocol. A developer can point an existing client at a Kitesurf endpoint rather than learning a product-specific action language.

The tradeoff is that Kitesurf implements only a subset of CDP today. The launch discussion also raised the longer-term question of WebDriver BiDi, the standards-based bidirectional automation protocol. CDP is the pragmatic compatibility layer now; broader standardized inspection coverage would be a valuable future direction.

PageScript: one clean JavaScript world per page

Each top-level page or out-of-process iframe gets a long-lived PageScript isolate created through Dynamic Workers. That isolate owns a clean globalThis, a DOM document, and the page’s JavaScript and WebAssembly execution.

Kitesurf assembles rather than reinvents every layer. It uses parts of Blitz, a modular Rust HTML and CSS engine, along with Stylo, the CSS engine also used by Firefox and Servo. Blitz’s unusually loose coupling is a good match: parsing, networking, rendering, windowing, scripting, style, layout, and text shaping can be composed without adopting one monolithic browser process.

There is an awkward exception. Workers does not natively permit eval, while real websites sometimes depend on it. Kitesurf currently runs those cases through Boa, a JavaScript engine written in Rust and compiled for Workers. Running one JavaScript runtime inside another is not elegant or cheap, but it fills a compatibility hole until native support changes.

This is a recurring browser-engine lesson: the long tail matters. A clean architecture still needs deliberate fallbacks for web behavior that cannot simply be wished away.

PageRenderer: turn a scene into pixels

PageRenderer receives the computed page scene, loads required fonts and images, rasterizes the result, and returns a PNG, JPEG, or PDF buffer. Blitz Paint handles much of the painting work, while Parley handles text shaping, font selection, glyphs, and line breaking.

The Engine calls the renderer through Workers RPC. PageRenderer keeps no important page state, only a disposable cache. If rendering stalls or fails, the Engine can kill that isolate, start another, and retry the frame without reconstructing the entire session.

That separation is the architectural center of Kitesurf. Stateful work is kept narrow; expensive stateless work can be replaced. Failures degrade into a missing element, blank frame, or retried render instead of taking the whole session down.

Tests are the map for a new engine

Building a browser in twelve weeks would be reckless without a machine-readable definition of progress. Kitesurf uses the Web Platform Tests, the shared cross-browser suite for standards behavior, as that definition.

Cloudflare reported more than 215,000 passing WPT tests at launch, with strong coverage in areas useful to agents: HTML, DOM, CSS, selection, SVG, XHR, and streams. The team used AI coding agents to accelerate implementation, but humans chose the feature order, reviewed approaches, and supplied tests as concrete acceptance criteria.

WPT is necessary, not sufficient. A standards test can prove that an API behaves correctly without proving that a complicated production site works. Kitesurf therefore also uses multi-step Puppeteer integration tests against real websites and visual regression comparisons with Chromium. That catches the gap between isolated conformance and an actual page assembled from thousands of interacting assumptions.

This combination is a practical template for AI-assisted infrastructure work:

Define behavior with a large conformance suite.
Let agents work inside bounded, testable slices.
Review architecture and sequencing with humans.
Compare the result against real workloads, not only unit tests.
Track the missing surface explicitly instead of declaring parity too early.

The novelty is not that AI “built a browser.” The useful lesson is that mature tests turned an enormous problem into many measurable loops.

For the traditional browser-engine milestone, Cloudflare also ran a WebAssembly port of Doom inside Kitesurf. It is a playful demo, but it exercises enough scripting, layout, input, and rendering behavior to expose problems that isolated conformance cases can miss.

The benchmark result is a trade, not a victory lap

Cloudflare compared median results from five Browser Run quick-action executions over a 14-URL corpus. Its launch numbers were:

Task	Kitesurf	Warm Chromium	Difference
Screenshot CPU	380 ms	1,173 ms	3.1× less CPU
HTML extraction CPU	229 ms	877 ms	3.8× less CPU
Screenshot memory	57.8 MiB	271.0 MiB	4.7× less memory
HTML extraction memory	39.4 MiB	273.7 MiB	7.0× less memory
Screenshot wall time	1,148 ms	637 ms	1.8× slower
HTML extraction wall time	820 ms	472 ms	1.7× slower

These are vendor measurements from a small corpus, not a universal browser benchmark. They still show the intended exchange clearly. Kitesurf reduces CPU and memory by several times, while a warm Chromium pool finishes the measured tasks faster.

For an interactive request, latency may matter most and Chromium can remain the better choice. For a burst of independent extraction jobs, memory density and CPU cost can dominate. A system that can run more concurrent sessions on the same budget may complete the whole queue sooner even if each individual session takes longer.

The right comparison is therefore workload-level:

How many pages are processed per dollar?
What percentage of target sites render correctly?
How often does a task fall back to Chromium?
Does the output need pixel fidelity or only structured content?
Is the session short and disposable, or logged in and long-lived?

“Lightweight” is not a sufficient buying criterion. Compatibility failures and retries can erase infrastructure savings.

Existing tools can opt in

Kitesurf is available in Cloudflare Browser Run during its beta. Browser Run supports quick, stateless actions such as screenshots and extraction, plus full sessions through Playwright, Puppeteer, and CDP.

For compatible CDP clients, the switch is deliberately small: add browser=kitesurf to the endpoint. The same idea works through chrome-devtools-mcp, allowing an MCP-capable coding agent to connect without a Kitesurf-specific tool vocabulary.

{
  "mcp": {
    "kitesurf": {
      "type": "local",
      "command": [
        "npx",
        "-y",
        "chrome-devtools-mcp@latest",
        "--wsEndpoint=wss://api.cloudflare.com/client/v4/accounts/ACCOUNT_ID/browser-run/devtools/browser?browser=kitesurf",
        "--wsHeaders={\"Authorization\":\"Bearer API_TOKEN\"}"
      ]
    }
  }
}

Quick Actions use the same selector. That makes gradual adoption possible: send known-compatible screenshot or extraction tasks to Kitesurf and preserve Chromium as the fallback for everything else.

Where Kitesurf fits today

At launch, Cloudflare says Kitesurf can render sites including Hacker News, Wikipedia, the Cloudflare Blog, TodoMVC implementations across several frameworks, and much of its own dashboard. That is enough for meaningful experiments, but not blanket production compatibility.

Kitesurf is a plausible fit for:

One-shot HTML or text extraction from compatible sites
Large batches of screenshots where small visual differences are acceptable
PDF generation for pages covered by tests
Short agent tasks using DOM and network inspection
Sandboxed browsing of unknown pages without shared session state
Workloads where CPU and memory density matter more than single-request latency

It is not the right engine today for:

Video playback
WebGL applications
Pixel-perfect screenshot testing across the entire web
Bot challenges that depend on authentic browser TLS fingerprints
Long authenticated sessions that must preserve state
Sites that depend on missing web APIs or unsupported CDP commands

Those are not minor footnotes. Authentication, anti-bot checks, and complex application state are central to many real browser agents. Cloudflare explicitly recommends Browser Run’s Chromium default for such cases.

Browser Run traffic also does not receive a secret passage through Cloudflare protection. In the HN discussion, a Cloudflare engineer said both Chromium and Kitesurf traffic is identified as bot traffic. The Browser Run changelog documents signed Web Bot Auth headers, which let origins identify the service rather than disguise it as a person.

A practical routing strategy

The strongest production design is likely hybrid, at least while Kitesurf matures.

Start with an allowlist of tasks and sites known to work. Send extraction, simple navigation, and one-shot rendering to Kitesurf. Record compatibility failures, missing protocol calls, rendering differences, and total retries. Route a task to Chromium when it requires video, WebGL, a durable login, an unsupported API, or strict visual fidelity.

Treat the fallback rate as a first-class metric. If 70 percent of jobs use one-seventh the memory but the other 30 percent retry twice before falling back, the real savings are smaller than the headline benchmark suggests. Measure the complete path.

Also keep credentials outside page execution. Use scoped bindings or a controlled proxy to inject authorization only for approved destinations. Destroy the session after the task. A fresh isolate is valuable, but it does not replace domain policy, output validation, rate limits, or human confirmation before consequential actions.

Finally, test your own corpus. Cloudflare provides a public playground with DevTools and memory inspection, but a handful of representative pages is more useful than a generic demo. Build fixtures for the sites, controls, and outputs your agents actually touch.

The bigger idea: purpose-built web runtimes

Kitesurf matters even if it never replaces Chromium for most automation.

For years, browser automation has inherited the architecture of a human browser because that was the only compatible implementation available. Agent workloads are now large enough to justify another layer: purpose-built runtimes that implement the parts of the web their tasks need, expose standard interfaces, and fall back when the long tail wins.

The modular pieces are as important as the product. Blitz supplies decomposed HTML, CSS, layout, and painting machinery. Rust and WebAssembly bring those pieces into Workers. Dynamic Workers create isolated page worlds. RPC connects disposable renderers. CDP lets existing tools arrive without a new integration.

Cloudflare plans to improve CDP coverage, rendering fidelity, WPT results, and efficiency, and says it intends to open-source Kitesurf when ready. Until that happens, the product should be evaluated as an early hosted beta with vendor-reported performance, not as an independently auditable browser engine.

Still, the direction is compelling. The web does not have to be presented to every consumer through the same heavyweight machine. People need a browser that feels like an operating system. Many agents need something closer to a safe, temporary compiler from URLs to structured state and pixels.

Kitesurf is the first serious attempt to make that distinction operational at edge scale.

Why a Smaller Coding-Agent Harness Can Do More

Vincent Tran — Thu, 06 Aug 2026 00:00:00 +0000

When a coding agent disappoints, the model usually gets blamed. When it succeeds, the model usually gets the credit. Both reactions miss a large part of the system.

The model never touches a repository by itself. A harness chooses the system prompt, describes the available tools, decides which files and earlier messages return on every turn, executes commands, stores sessions, compacts history, and determines how much context gets sent again. Two products can call the same model at the same reasoning level and still produce very different bills—and sometimes different results.

Pi makes that hidden layer unusually easy to see because it starts with almost nothing. Its default model interface contains four tools: read, write, edit, and bash. Features that many coding agents treat as mandatory—plan mode, built-in task lists, subagents, MCP, permission dialogs, and background shells—are deliberately left outside the core.

That is not minimalism as decoration. It is an argument about where complexity belongs: begin with a small, legible mechanism, then add only the behavior your work can justify.

The Harness Is Part of the Product

A coding-agent request is not just the sentence typed into the terminal. A typical model call also contains:

the harness’s system instructions;
schemas and descriptions for every active tool;
repository instructions such as AGENTS.md;
relevant conversation history;
file contents and command results gathered during the task;
summaries or bookkeeping added by the harness.

Most of that material is useful. The problem is accumulation. A tool schema added for one rare operation may be sent on every turn. A detailed planning framework may consume attention during a one-line fix. An ever-growing transcript can keep old errors alive long after they stopped helping.

This is why context should be treated as a working set, not a storage bucket. The agent needs enough information to act correctly, but every extra instruction competes with the code, the task, and the evidence produced by tests.

Pi’s default is intentionally narrow. Four primitive tools are enough to inspect a project, change it, and run the project’s own commands. A shell already composes thousands of specialized programs. A file can hold a plan. tmux can host another process. A repository instruction file can encode local rules. The core does not have to reimplement every surrounding system.

This approach also keeps the interface visible. A developer can understand the default capability boundary without reading a catalog of dozens of tool definitions. That makes failures easier to reason about: was the prompt unclear, was the wrong file read, did a command fail, or did the model make a bad edit?

Context Has a Compounding Cost

The token count of a system prompt looks small next to a large context window, but the relevant cost is not its size once. It is its size across the whole trajectory.

Suppose an agent makes 20 model calls while investigating a bug. If each request repeats a large prefix of instructions and tool definitions, the overhead is paid repeatedly. Providers can reduce that cost with prompt caching, but caching depends on an identical prefix. A small change near the beginning of the request can make the provider process everything after that point again.

Tool catalogs are especially sensitive. Tools are normally placed before the conversation. Adding, removing, reordering, or changing a schema can invalidate the reusable prefix. Pi’s own guide to prompt caching explains that newer model APIs can defer additive tools without disturbing earlier context, but support varies. Extensions that rebuild prompts or change tool sets dynamically can still destroy cache reuse.

The practical lesson is broader than Pi:

Keep the stable prefix stable.
Do not activate a tool merely because it might be useful someday.
Prefer repository-native commands over verbose wrappers when they provide the same evidence.
Measure total input per completed task, not only the price of one token or one call.

That last point matters because a cheaper model can be more expensive if it needs more attempts, reads more files, or fails often enough to require a stronger model afterward. Efficiency is the cost of reaching a verified result.

Context discipline becomes even more important with local models. Their usable context windows are often smaller, and processing a long prompt prefix can be slow enough to dominate an interactive turn. A stable, compact prefix avoids needless reprocessing and leaves more room for the repository itself. The same design that reduces API spend can therefore reduce local latency.

Databricks Put the Harness on the Benchmark

Databricks recently built an internal benchmark from real engineering work on its multi-million-line codebase. The tasks covered Python, Go, TypeScript, Scala, Rust, Java, Bazel, Protobuf, and other technologies used in production. The team derived prompts from recent human-written pull requests, held back relevant tests, reviewed tasks manually, and removed access to Git history so agents could not recover the original implementation.

That methodology is more useful than a generic leaderboard for one important reason: it tests the work the organization actually does.

The Databricks results produced several lessons:

model price per token did not reliably predict cost per completed task;
open and proprietary models occupied the quality-cost frontier together;
stronger models sometimes finished cheaply because they used fewer tokens;
the harness materially changed both quality and cost.

The most revealing comparison held the model and reasoning effort constant while changing the harness. Databricks observed task costs differing by more than 2x in some cases while quality stayed the same. Pi sent roughly one-third as much context per turn and completed the work in fewer runs.

In the published chart, Pi paired with Opus 4.8 at high reasoning effort reached the strongest overall pass rate while costing materially less per task than the same class of model used through Claude Code or Codex. The interesting result is not a permanent leaderboard position. It is the controlled difference produced by the layer around the model.

This does not prove that Pi is universally superior. Databricks explicitly warns against that conclusion. Their benchmark reflects their tasks, repositories, configuration, and evaluation process. A harness with more built-in guardrails may be worth its overhead in a regulated environment, and an integrated product may outperform on workflows its vendor optimized directly.

What the benchmark does prove is that “which model?” is an incomplete procurement question. Teams should test model-harness pairs. A benchmark that swaps models but freezes the harness measures only half the system.

Minimal Does Not Mean Fixed

A tiny core is useful only if it can grow without being forked every time a team needs a specialized workflow. Pi handles that through several layers:

AGENTS.md and related context files for project rules;
skills for instructions loaded when a task needs them;
prompt templates for repeatable requests;
TypeScript extensions for tools, commands, events, interface elements, providers, compaction, and policy;
packages that bundle extensions, skills, prompts, and themes.

This architecture replaces “batteries included” with “attachment points included.” Plan mode can be a file or extension. Subagents can be separate Pi processes or an extension. MCP can be added when a team genuinely needs it. Permission gates can match the execution environment instead of being imposed as one universal dialog.

The distinction matters. Removing features without offering a clean way to add them creates a toy. Providing a small core with stable extension surfaces creates a platform.

Shopify’s Autoresearch Shows the Payoff

Shopify provides a useful example of complexity being added at the edge rather than the center.

An engineer wanted to reduce slow and flaky build work around Polaris. A normal request to “make the build faster” produced an unsuccessful one-shot attempt. The task was too broad and lacked a mechanism for deciding whether each change helped.

The engineer instead asked Pi to create an Autoresearch extension. The resulting loop had four parts:

choose a measurable target;
record a baseline;
form a hypothesis, change the code, and measure again;
keep improvements while discarding crashes and regressions, then repeat.

Within half an hour, the first version was working. The important advance was not a smarter prompt. It was a better control loop. The metric acted as an objective function, and the repository’s build supplied evidence. The agent gained permission to explore, but every experiment had to survive measurement.

The loop did propose bad shortcuts, including changes that made the build faster by doing less legitimate work. Human review remained essential. But it also found real waste: a visual-regression build performed packaging steps that Storybook immediately repeated, and a TypeScript transform processed hundreds of files that did not need it. The accepted changes reduced that build by 65%.

The Shopify engineering account describes the tool growing from a local experiment into an open-source project used across more than 40 metrics. The broader lesson is that domain-specific orchestration can be extremely valuable. It simply does not need to be present in every agent session.

Autoresearch earned its complexity because the workflow had a repeatable target, an automated evaluator, and enough iterations for small gains to compound. That is a much stronger reason to add an agent feature than “other tools have one.”

The Cost of Moving Complexity Outward

Minimalism does not eliminate complexity. It changes who owns it.

Pi extensions execute with full system access. A package can register tools, handle credentials, run commands, and alter model behavior. Pi’s documentation tells users to review third-party packages because installing one is closer to installing software than enabling a harmless editor theme.

The core also omits permission popups. Its philosophy is that isolation should come from a container, sandbox, or operator-defined extension. That can be a cleaner security model than clicking through frequent approvals, but only when the isolation actually exists. Running an unrestricted agent in a sensitive checkout and assuming the small prompt makes it safe would be a category error.

The Hacker News discussion surfaced ordinary product trade-offs too. Some users reported slow startup with extensions, disliked the configuration directory choice, or missed familiar terminal keybindings. Others valued the ability to change those behaviors themselves. Minimal code and minimal user friction are related goals, not identical ones.

There is also a maintenance tax. A curated, integrated tool gives teams one vendor responsible for making planning, permissions, subagents, browser automation, and updates work together. A customized Pi setup may fit better, but someone must own its extensions, dependency updates, cache behavior, security review, and onboarding documentation.

The right comparison is therefore not “simple versus complex.” It is vendor-selected complexity versus user-selected complexity.

How to Evaluate a Coding Harness on Your Own Work

Teams do not need a research department to apply the Databricks lesson. A small internal evaluation can start from merged pull requests.

1. Build tasks from recent changes

Select representative fixes and features. Remove the implementation while keeping a clear description of the desired behavior. Exclude generated changes and tasks that depend on unavailable services.

2. Hold back objective checks

Use the original tests where possible, but inspect them for overfitting. A test that demands one exact implementation may reject a valid alternative. Compile checks, behavior tests, linters, and focused manual review can work together.

3. Seal accidental answer paths

Do not leave the solution in later Git history, cached build artifacts, review comments, or nearby branches. An agent that finds the original patch has tested repository archaeology, not coding ability.

4. Compare complete pairs

Run the same model and effort through different harnesses, then run different models through the same harness. Record pass rate, wall time, number of model calls, input and output tokens, cache reads and writes, and final cost.

5. Inspect traces, not just scores

A passing agent may take a dangerous shortcut. A failing agent may reveal a missing repository instruction. Trace review shows whether the harness kept a focused working set, repeated irrelevant context, selected tools poorly, or stopped without verification.

6. Price ownership honestly

Include the human cost of maintaining extensions and guardrails. A low token bill is not a bargain if every developer spends hours repairing a bespoke setup.

A Practical Minimal-Harness Checklist

If you are designing or trimming an agent workflow, these rules capture the useful part of Pi’s approach:

Start with file access, editing, and command execution.
Put durable project knowledge in versioned repository instructions.
Load specialized instructions only when relevant.
Keep tool schemas short, distinct, and stable.
Prefer measurable loops over motivational prompt language.
Add orchestration only after a repeated workflow appears.
Run risky tools inside a real security boundary.
Treat every extension as executable code.
Preserve cache-friendly prompt prefixes.
Evaluate cost per verified result on your own repository.

This does not require using Pi. The same discipline can improve a large commercial harness: disable unused integrations, shorten project instructions, remove redundant tools, isolate long-running experiments, and measure where the context goes.

Complexity Should Earn Its Keep

Modern frontier models already understand files, shells, tests, and common development conventions. A harness no longer has to explain software engineering from first principles on every request. Its job is increasingly to provide a clean interface to the environment, preserve the right context, and get out of the way when the model has enough information to act.

Pi demonstrates the upside of that restraint. Databricks shows that a lean harness can reduce end-to-end task cost without sacrificing quality on real work. Shopify shows that a minimal core can still support sophisticated autonomous loops when those loops are built for a concrete metric.

The harder lesson is that minimalism is not free. It asks teams to choose their own extensions, security boundaries, and operating practices. For developers willing to own those decisions, that is the advantage: the harness contains the complexity their workflow needs, not the complexity a product roadmap happened to collect.

Sources: Earendil — Pi’s Minimalism Is Its Advantage, Databricks — Benchmarking Coding Agents on a Multi-Million-Line Codebase, Shopify Engineering — Autoresearch Isn’t Just for Training Models, Pi documentation and source, Earendil — Prompt Caching in Agents, Hacker News discussion

Twenty Years of Pandoc: The Architecture Behind a Universal Document Converter

Vincent Tran — Wed, 05 Aug 2026 00:00:00 +0000

On August 3, 2006, philosophy professor John MacFarlane published the first version of Pandoc. It was roughly 3,000 lines of Haskell, relied only on the Glasgow Haskell Compiler’s standard library, and converted among a small set of text formats: Markdown, reStructuredText, HTML, and LaTeX, with RTF and S5 as additional outputs.

Twenty years later, Pandoc supports 51 input formats and 76 output formats. That is 3,876 possible input-output pairs before accounting for extensions and configuration. It sits underneath academic publishing systems, notebooks, static-site workflows, documentation pipelines, ebooks, slide decks, and countless one-off shell commands. More than 600 people have contributed, and the core packages contain about 85,000 lines of Haskell excluding tests.

Those numbers are impressive, but they are not the main story. Pandoc lasted because its first important design decision made future growth cheaper. Instead of writing a separate converter for every pair of formats, it parses a document into a shared representation, transforms that representation, and renders it into the target format.

That sounds obvious now. In 2006, for Markdown tooling, it was not.

The Experiment That Found a Real Problem

Pandoc did not begin with a market analysis or a grand plan for universal publishing. MacFarlane wanted to learn Haskell. He saw that the language was well suited to parsers and compilers, especially through the Parsec parser-combinator library, and chose a Markdown parser as a practical exercise.

Most early Markdown implementations worked by applying a sequence of regular-expression substitutions directly to the source. That approach can be compact, but markup is structured and context-sensitive. A heading marker inside a code block is not a heading. Emphasis delimiters interact with nesting. Lists, links, raw HTML, and escaping create cases that are difficult to model as a chain of text replacements.

Pandoc instead treated a document like a compiler treats a program:

source document -> reader -> document tree -> writer -> output document

The document tree is an abstract syntax tree, or AST. It represents meaning rather than source punctuation: this node is a paragraph, that one is emphasized text, another is a link, a table, a citation, or a code block. Once a reader has produced that structure, a writer does not need to understand the original syntax. It only needs to render the tree in its own format.

This changes the scaling problem. Pairwise converters grow roughly with every combination of source and target. With a shared AST, adding one reader makes every existing writer available to that input; adding one writer gives every existing reader a new destination. In simplified terms, N readers plus M writers enable N × M conversions.

MacFarlane soon added reStructuredText because he used it for lecture notes, LaTeX because he wanted PDFs, and a Markdown writer because he wanted to migrate those notes. The learning exercise had turned into a tool that removed friction from his own work. That tight feedback loop—build something, use it, discover the next constraint—would guide the project for years.

Distribution Mattered as Much as Code

The first public release received almost no promotion. MacFarlane emailed two friends. There was no GitHub, Hackage was not yet available, and he did not use social media.

The first big distribution event came from someone else. A Turkish developer, Recai Oktaş, contacted him about packaging Pandoc for Debian. Working through Debian’s standards improved the software and put it in front of users who would never have discovered a personal website. In 2007, Hackage and cabal-install made Haskell packages easier to publish, install, and compose. Pandoc could now take dependencies on reusable libraries instead of keeping everything inside one repository.

The pattern is worth noticing: useful software does not spread through implementation quality alone. Package managers, operating-system repositories, stable releases, documentation, and boring installation paths are product features. Pandoc’s architecture made it capable; distribution made it available.

Pandoc 1: From Converter to Publishing System

Pandoc 1.0 arrived in September 2008. It added outputs for MediaWiki, GNU Texinfo, OpenDocument, and ODT, along with fenced code blocks and syntax highlighting. Some of those features required Haskell libraries that did not exist, so MacFarlane wrote them. zip-archive enabled container formats such as ODT. highlighting-kate turned KDE syntax definitions into Haskell highlighters and gave Pandoc broad language coverage.

This period established a second theme in Pandoc’s history: supporting document formats often means building the missing infrastructure around them. A DOCX file is not merely formatted text; it is a ZIP container containing XML, relationships, media, styles, and specialized equation markup. High-fidelity conversion forces the converter to understand these surrounding systems.

The project moved from Google Code to GitHub in 2010. A template system replaced hard-coded wrappers, letting users control complete output documents. EPUB, Org mode, Textile, MathML, and other formats followed. Pandoc 1.9 added DOCX output in 2012, including Word’s Office Math Markup Language for equations.

By 2013, Pandoc was becoming a platform rather than a fixed conversion matrix. Four changes were especially important:

Markdown extensions made dialect choices explicit instead of pretending there was one universal Markdown.
YAML metadata blocks carried structured document-level information into templates.
Custom Lua writers let users produce niche output formats without modifying Pandoc itself.
JSON filters let any program transform the AST between parsing and rendering.

The filter model is a direct consequence of the shared representation:

input -> reader -> AST -> filter -> AST -> writer -> output

A filter can number sections, rewrite links, execute selected code blocks, generate diagrams, remove content, or enforce a house style without caring whether the source was Markdown and the output is HTML, DOCX, or LaTeX. It operates on document meaning, not on fragile surface syntax.

Markdown’s Ambiguity and the CommonMark Detour

While maintaining Pandoc’s Markdown parser, MacFarlane encountered a deeper problem: Markdown’s original description left many edge cases undefined. Implementations disagreed about nested emphasis, list indentation, code blocks, escaping, and how constructs interact.

Beginning in 2012, he joined a group with representatives from GitHub, Reddit, Stack Overflow, and others to create an unambiguous specification. When the group lost momentum, he wrote a formal specification and reference parsers. The effort became CommonMark after John Gruber objected to the original name, “Standard Markdown.”

CommonMark gave implementers a testable core instead of an informal description. Most modern Markdown processors now share its basic rules, even when they add different extensions.

There is an amusing wrinkle: Pandoc still keeps its older, extension-rich Markdown parser alongside CommonMark-based formats. Users can choose markdown, commonmark, gfm, or commonmark_x depending on the behavior they need. A project can help standardize its ecosystem without immediately replacing every piece of its own history.

Pandoc 2: Effects, Filters, and Rich Documents

Pandoc 2.0, released in 2017, addressed a tension in the original model. Pure readers and writers are easy to reason about, but some formats require input and output. A reStructuredText document can include another file. A writer may need to fetch an image and inspect its dimensions. DOCX and EPUB contain embedded resources.

The solution was not to let I/O spread invisibly through the codebase. Readers and writers were generalized over a PandocMonad interface. One implementation permits controlled I/O; another remains pure. The abstraction later enabled Pandoc’s --sandbox mode, which guarantees that readers and writers cannot touch the filesystem or network. A design introduced to handle richer conversions also became a security boundary.

Pandoc 2.0 also embedded Lua filters. JSON filters are language-neutral, but they serialize a whole AST and launch another process. Lua filters operate directly on the tree inside Pandoc, require no separate runtime, and are generally faster. This made custom document transformations practical for ordinary users, not just Haskell developers.

The format list continued growing: GitHub-flavored Markdown, JATS, PowerPoint, Vimwiki, TikiWiki, Creole, and more. Jupyter Notebook support arrived in 2019. Defaults files made repeatable command configurations easier to store and review.

Some work was less visible but more disruptive. Pandoc’s original table model could not express row spans or column spans. Fixing that meant redesigning the AST type and updating every reader and writer that handled tables. This is where Haskell’s algebraic data types and compiler checks paid off: when a central data structure changes, the compiler identifies code that has not been adapted.

Citation processing followed a similar path. Pandoc had long supported bibliographies through an external filter, but the implementation had become slow and difficult to maintain. MacFarlane wrote a new CSL processor from the specification and test suite. It became built-in with Pandoc 2.11, improving performance and standards fidelity. Correct sorting required yet another supporting library implementing the Unicode Collation Algorithm.

By the end of the Pandoc 2 era, users could also write custom readers in Lua, run Pandoc as an HTTP service, convert bibliography databases, and use sandboxed parsing and rendering.

Pandoc 3: Splitting the Monolith Without Breaking the Model

By 2023, Pandoc had accumulated a web server, a Lua runtime, dozens of format implementations, citation machinery, templates, highlighting, and resource handling. Not every user wanted the full binary.

Pandoc 3.0 split the system into four packages:

pandoc provides the core Haskell library.
pandoc-lua-engine contains Lua integration.
pandoc-server exposes conversion through HTTP.
pandoc-cli builds the command-line application and can omit optional components.

The architecture remained recognizable. Readers still produce the shared AST; writers still consume it. The package boundaries reduced the cost of carrying features a deployment did not need.

New formats kept testing the limits of the model. Supporting Typst input required implementing much of an interpreter because Typst is not just markup; it is a programmable typesetting language. Djot, MacFarlane’s attempt to address long-standing Markdown design problems, became both an input and output format. Later releases added terminal-oriented ANSI output, PowerPoint and Excel readers, AsciiDoc input, XML serialization of the AST, and more.

Pandoc 3.9, released in February 2026, added WebAssembly support, making a full-featured converter available in the browser. The deployment environment changed radically from a Debian package and a shell command, but the reader–AST–writer core survived.

Why Haskell Worked for a Twenty-Year Codebase

Pandoc was written in Haskell because its creator wanted to learn Haskell, not because of a formal technology selection. In retrospect, the language matched the problem unusually well.

Algebraic data types provide a precise vocabulary for structured documents. Pattern matching makes transformations over that structure direct. A strong type system helps maintainers change central types safely across a large codebase. Purity makes side effects visible and controllable, which improves testing and supports guarantees such as sandbox mode. Parser combinators make complex grammars composable without hiding their structure in regular-expression pipelines.

None of this means Haskell made maintenance automatic. MacFarlane still spends time reviewing patches, answering questions, fixing tiny incompatibilities, repairing release infrastructure, signing builds, and improving documentation. Language features reduce certain categories of risk; they do not replace stewardship.

Pandoc also demonstrates a social trade-off. A less common language may reduce the number of casual contributors, but it can attract contributors who care deeply about parsers, types, compilers, and formal document structure. For a project with limited management capacity, contributor fit can matter more than raw volume.

What Pandoc Teaches About Durable Tools

Pandoc’s history offers several practical lessons for software intended to last.

Choose a stable internal model. The shared AST allowed formats, filters, and interfaces to grow around one center. The model evolved—figures and rich tables required changes—but it was strong enough to absorb those changes.

Separate meaning from representation. A document heading is not the # character that happens to introduce it in one syntax. Systems become more reusable when they model what data means, then isolate encoding and rendering at the edges.

Create extension points before you can predict every use. JSON filters, Lua filters, templates, defaults files, and custom readers and writers let users solve specialized problems without waiting for them to become core features.

Turn constraints into explicit interfaces. PandocMonad did more than permit I/O; it defined where effects were allowed. That made both full-featured conversion and a meaningful sandbox possible.

Treat packaging and documentation as part of the product. Debian, Hackage, GitHub releases, installers, binaries, manuals, and examples converted good code into usable infrastructure.

Expect maintenance to dominate. New formats make release notes. Compatibility fixes, issue triage, tests, code signing, and careful review keep users trusting the next release. A long-lived tool is mostly a promise that yesterday’s documents will still work tomorrow.

Deterministic Conversion Still Has a Future

MacFarlane closes his retrospective by asking whether language models might eventually replace conventional document converters. A model can already infer formatting intent and translate a simple document into another syntax. It may even handle ambiguous human-authored markup more naturally than a deterministic grammar.

For production pipelines, however, Pandoc retains important advantages. It is efficient, local, testable, and deterministic. The same input, version, and options produce the same result. Failures can be reproduced. Builds can be pinned. Sensitive documents need not leave the machine. A filter can be inspected as code instead of inferred from a prompt.

The two approaches may become complementary. A model can help repair ambiguous input, design a template, or draft a Lua filter. Pandoc can then perform the repeatable conversion. Semantic inference is useful at the uncertain edges; deterministic machinery is valuable at the center of a build.

After twenty years, Pandoc is more than a universal converter. It is evidence that a small, coherent architecture can outlive several hosting platforms, packaging eras, markup fashions, and deployment targets. The formats changed. The interfaces multiplied. The central idea kept paying rent.

Sources: John MacFarlane — Twenty Years of Pandoc, Pandoc User’s Guide, Pandoc filters, Pandoc Lua filters, Pandoc release history, Pandoc on GitHub, Hacker News discussion

Go 1.27: Generic Methods, Better Leak Detection, and a Serious JSON Upgrade

Vincent Tran — Mon, 03 Aug 2026 00:00:00 +0000

Go 1.27 is expected in August 2026, and its draft release notes describe a release with an unusual amount of surface area. There is a language feature developers have requested since generics arrived, a new way to find goroutines that can never wake up, a rebuilt JSON stack, post-quantum signatures, portable SIMD, and dozens of smaller changes that affect everyday code.

The release is still being finalized, so details can move before the stable build. Even so, the direction is clear: Go is filling gaps without turning into a different language. Most existing programs should simply become faster or easier to inspect. The features that need deliberate adoption are opt-in or arrive with compatibility controls.

Here is what changes, why it matters, and how to plan an upgrade.

Generic Methods Close the Most Visible Generics Gap

Go 1.18 allowed functions and types to declare type parameters, but methods could only use type parameters already declared by their receiver type. A method could not introduce a new result type of its own.

That limitation becomes obvious when mapping a generic container. Suppose a Box[T] holds one value and needs an operation that transforms T into any other type U. Before Go 1.27, the operation had to be a package-level function:

type Box[T any] struct {
    value T
}

func MapBox[T, U any](b Box[T], fn func(T) U) Box[U] {
    return Box[U]{value: fn(b.value)}
}

name := MapBox(Box[int]{value: 42}, strconv.Itoa)

In Go 1.27, the operation can live where readers naturally look for it:

func (b Box[T]) Map[U any](fn func(T) U) Box[U] {
    return Box[U]{value: fn(b.value)}
}

name := Box[int]{value: 42}.Map(strconv.Itoa)

This is not only syntax sugar. Package-level generic functions flatten an API: every operation sits beside every type, even when the operation conceptually belongs to one receiver. Generic methods make fluent container, iterator, result, parser, and query APIs easier to discover and read.

There is an important boundary. Interface methods cannot declare their own type parameters, and a generic method cannot satisfy an interface method. Go’s interfaces still describe a fixed method set. If an abstraction must cross an interface boundary, keep the type parameter on the interface or receiver type, or use a package-level generic function.

That constraint preserves the way interface values work today. It also prevents generic methods from becoming a universal replacement for functions. Use a method when the operation belongs to the receiver; use a function when it combines unrelated values, participates in type inference more clearly, or must remain compatible with an interface-driven design.

Type Inference Reaches More Contexts

Go 1.27 also broadens function type inference. A generic function can now be inferred anywhere a matching function type is expected, including conversions and composite literals.

Consider two generic selectors:

func First[T any](values []T) T { return values[0] }
func Last[T any](values []T) T { return values[len(values)-1] }

selectors := []func([]string) string{First, Last}

The slice element type tells the compiler that T is string. Older versions required First[string] and Last[string] in this context. The change removes ceremony while keeping the destination type explicit.

Struct literals gain a smaller convenience: a keyed element may use any valid field selector, including a promoted field from an embedded struct.

type Metadata struct {
    ID string
}

type Record struct {
    Metadata
    Value string
}

r := Record{ID: "evt-42", Value: "ready"}

Previously the literal had to spell out Metadata: Metadata{ID: "evt-42"}. This makes literals shorter, but it also means adding or changing embedded fields can affect which selector a key resolves to. Treat it as a readability tool, not a reason to build deep embedding hierarchies.

Small Allocations Get a Faster Path

The compiler can now emit calls to size-specialized allocation routines. For some allocations smaller than 80 bytes, the direct cost may fall by as much as 30 percent. The Go team expects the end-to-end gain in real allocation-heavy programs to be closer to one percent, with roughly 60 KB added to the binary.

That difference between microbenchmark and application impact matters. A faster allocator does not make object churn free, and it should not replace profiling or sensible data reuse. It is an automatic improvement for many services, parsers, and request pipelines, but the correct test is still your own latency and allocation profile.

If a workload regresses, GOEXPERIMENT=nosizespecializedmalloc disables the optimization at build time for Go 1.27. The escape hatch is expected to disappear in Go 1.28, so it is intended for diagnosis and bug reports rather than permanent configuration.

Goroutine Leaks Become Directly Observable

A rising goroutine count tells you that something may be wrong, but the normal goroutine profile lists every live goroutine. A busy server can have thousands of legitimate goroutines waiting on sockets, timers, pools, or work. Finding the permanently stuck ones is often the hard part.

Go 1.27 promotes the experimental goroutine leak detector from Go 1.26 into a regular runtime/pprof profile named goroutineleak. It is also exposed through net/http/pprof at /debug/pprof/goroutineleak.

The detector uses garbage-collector reachability. If a goroutine is blocked on a channel, mutex, condition variable, or another concurrency primitive, and that primitive cannot be reached by anything capable of unblocking it, the goroutine cannot make progress. The profile reports that goroutine and its stack.

if profile := pprof.Lookup("goroutineleak"); profile != nil {
    if err := profile.WriteTo(os.Stdout, 1); err != nil {
        log.Printf("write leak profile: %v", err)
    }
}

This catches a useful class of bugs: abandoned channel sends, orphaned waits, and concurrency objects that disappeared while a goroutine remained parked on them.

It is not a proof that the program has no leaks. A blocked goroutine may not be reported if its synchronization object is reachable from a global or from a runnable goroutine’s locals. The detector answers a narrower, valuable question: which blocked goroutines can the runtime already prove will never resume?

Because requesting the profile triggers a garbage-collection cycle, do not poll it like a cheap counter. Capture it during investigation, expose the endpoint behind the same protections as other diagnostics, and compare it with ordinary goroutine, block, mutex, and execution-trace data.

JSON v2 Moves from Experiment to Platform

The biggest standard-library change is the arrival of encoding/json/v2 and the lower-level encoding/json/jsontext package.

The original encoding/json API had to preserve behavior accumulated over many years. That made it difficult to correct surprising defaults or design a cleaner streaming layer without breaking existing applications. The new stack separates the problem into two levels:

encoding/json/v2 maps Go values to and from JSON with configurable options.
encoding/json/jsontext processes JSON syntax as tokens and values while maintaining the grammar state.

The v2 API uses stricter defaults. It rejects invalid UTF-8 in strings and duplicate object member names, two behaviors that can otherwise produce inconsistent interpretations between systems. Its marshal and unmarshal functions accept options, so policy is explicit at the call site rather than hidden in package-wide behavior.

The existing encoding/json package is now backed by the v2 implementation while preserving v1 behavior. Existing applications are not required to migrate their imports. Exact error text may change, however, so tests should assert error categories or behavior instead of full prose when possible.

The implementation aims for marshal performance around the old version and substantially faster unmarshaling. If the new engine exposes a compatibility problem, building with GOEXPERIMENT=nojsonv2 restores the previous v1 implementation temporarily. As with the allocator switch, this should be used to isolate a regression and report it, not to avoid testing indefinitely.

A safe migration has two separate steps. First, build the existing application with Go 1.27 while keeping encoding/json; this tests the new engine under compatible semantics. Later, adopt encoding/json/v2 in selected boundaries where stricter input handling and explicit options are useful. Mixing those steps makes failures harder to diagnose.

Security Gains: ML-DSA Enters the Standard Library

The new crypto/mldsa package implements ML-DSA, the post-quantum signature standard defined by FIPS 204. Support extends into crypto/x509 for keys and signatures and into TLS 1.3 through the MLDSA44, MLDSA65, and MLDSA87 signature scheme identifiers.

This does not mean every Go service should immediately replace its current certificates. Post-quantum migration is an ecosystem problem: protocols, certificate authorities, hardware, peers, and operational tooling must agree. Standard-library support is important because it gives Go applications a maintained foundation for experiments and gradual integration instead of requiring every team to assemble its own cryptographic stack.

TLS also gains optional ML-KEM-1024 key exchange. It can be enabled through Config.CurvePreferences. As always, protocol configuration should follow interoperability tests and organizational cryptographic policy, not a desire to enable every new primitive at once.

Several old GODEBUG compatibility controls are removed, including switches for RSA key exchange, 3DES, TLS 1.0 server behavior, unsafe exported keying material, and older certificate-leaf handling. Search deployment manifests and startup scripts for these settings before upgrading. A forgotten switch is easier to understand in a migration checklist than in a failed production launch.

UUIDs Finally Have a Standard Home

Go 1.27 adds a top-level uuid package for generating and parsing UUIDs according to RFC 9562. It includes a UUID type, parsing helpers, random generation, and standard nil and maximum values.

That reduces the need for a third-party dependency in applications that only need conventional UUID behavior. It does not make established libraries obsolete overnight. Existing packages may provide database scanners, JSON policies, specialized versions, or compatibility guarantees that an application relies on. New projects can start with the standard package; mature projects should migrate only when the dependency reduction is worth the conversion work.

SIMD Gets a Portable Experiment

Go 1.26 began experimenting with architecture-specific SIMD. Go 1.27 adds a higher-level experimental simd package with vector-size-agnostic types such as integer and floating-point lanes. The implementation can use hardware vector instructions where available and emulate the portable operation set elsewhere.

Enable it with GOEXPERIMENT=simd. The lower-level simd/archsimd experiment remains available for architecture-specific operations and expands support across amd64, Arm Neon, and WebAssembly SIMD.

The distinction is useful:

Use portable simd when the algorithm should work across machines without maintaining several instruction-set implementations.
Use simd/archsimd when a specialized kernel needs exact control over vector width and instructions.
Keep ordinary Go as a fallback until the APIs stabilize and benchmarks prove the extra complexity is worthwhile.

SIMD is most promising for parsing, encoding, validation, compression, numerical work, and other tight loops over uniform data. It will not accelerate an application whose time is spent in network waits, database calls, allocation, or branch-heavy business logic.

Networking Behavior Changes in Useful Ways

Several HTTP changes are designed to make the correct behavior the default.

For HTTP/1, closing a response body now drains a conservative amount of unread content so the connection can be reused. This fixes a common performance trap where callers close a body early and unknowingly prevent keep-alive reuse. Code that intentionally abandons very large bodies should test the new behavior; disabling keep-alives remains available when reuse is genuinely unwanted.

HTTP/2 servers now understand client priority signals from RFC 9218. Applications can restore round-robin behavior with Server.DisableClientPriority if prioritization is undesirable.

Servers also gain a maximum header-value count, adding a direct limit for requests that split excessive values across repeated headers. On Windows and macOS, crypto/x509.SystemCertPool now respects SSL_CERT_FILE and SSL_CERT_DIR; when they are set, Go loads roots from disk and uses its native verifier instead of the platform API. Container and enterprise environments that already set those variables should verify the resulting trust store during rollout.

Tooling Catches Version Drift Earlier

Go 1.27 adds several small improvements that make builds and maintenance more predictable:

go test runs the stdversion vet analyzer by default. It flags standard-library symbols newer than the version declared by the active go directive and build tags.
go doc package@version retrieves documentation for a specific module version.
go doc -ex lists executable examples, and asking for a named example prints its source and comments.
go mod tidy consolidates duplicate require blocks for modules declaring Go 1.27 or later while preserving associated comments.
go fix adds modernizers for atomic types, embedded literals, backward slice iteration, and unsafe functions.
Compiler, linker, assembler, cgo, coverage, and packaging tools accept GCC-compatible response files, which helps build systems avoid oversized command lines.

The stdversion check may be the change teams notice first. A developer can have Go 1.27 installed while the module still promises compatibility with an older release. Using a newer API in that module is a contract violation even if local compilation succeeds. Catching it in go test turns an eventual consumer failure into an immediate development error.

Smaller Library Changes Worth Knowing

The standard library contains many targeted additions:

strings.CutLast and bytes.CutLast split around the final separator without a manual LastIndex sequence.
math/big.Int.Divide computes quotient and remainder with explicit truncation, floor, round, or ceiling behavior.
database/sql.ConvertAssign exposes conversions used by Rows.Scan, and drivers can implement direct destination scanning.
compress/flate becomes faster, which can change the exact compressed bytes produced by ZIP, gzip, zlib, and PNG writers even though decompressed data remains equivalent.
Unicode data advances from version 15 to version 17.
time package channels are now always synchronous; the old asynctimerchan fallback is gone.

The compression change is a reminder not to treat compressed output as a stable serialization. Snapshot tests should compare decoded content unless exact encoder output is truly part of a protocol.

A Low-Risk Go 1.27 Rollout

The release is broad, but the upgrade does not need to be dramatic.

Inventory compatibility switches. Search build flags, containers, and deployment settings for removed GODEBUG options.
Upgrade the toolchain without changing APIs. Run tests, race tests, static analysis, benchmarks, and representative integration workloads using existing source imports.
Exercise JSON boundaries. Test duplicate keys, invalid UTF-8, custom marshalers, unknown fields, numeric precision, error handling, and golden fixtures.
Review version promises. Decide whether go.mod should remain on an older go directive or move to 1.27. Let the new stdversion check enforce that choice.
Measure allocation-sensitive services. Record binary size, allocation profiles, throughput, and tail latency. Keep the temporary allocator opt-out available for diagnosis.
Add leak diagnostics deliberately. Protect pprof endpoints, document how to capture the new profile, and avoid running it at high frequency.
Adopt new APIs separately. Generic methods, JSON v2 semantics, UUIDs, and SIMD each deserve their own focused change and review.

This sequence separates toolchain compatibility from source modernization. When something changes, the responsible layer remains obvious.

The Shape of the Release

Go 1.27 is not built around one reinvention. Its strongest theme is closing operational gaps.

Generic methods put generic behavior back beside the types it belongs to. Better inference removes redundant type arguments. The runtime can identify a class of permanently blocked goroutines instead of making engineers search every live stack. JSON gets a modern foundation without forcing old applications into a flag-day migration. Security, networking, documentation, and module tools gain the controls production teams have been assembling around them.

That is a good kind of language release: existing code remains recognizable, while several recurring problems become easier to solve with the standard toolchain. The right response is not to rewrite working services. It is to upgrade carefully, measure the automatic improvements, and adopt the new capabilities where they remove real code or expose bugs that were previously hard to see.

Sources: Go 1.27 release notes, Go 1.27 interactive tour, Go JSON v2 experiment, Go diagnostics guide, FIPS 204: ML-DSA, RFC 9562: UUIDs, Go 1.27 discussion

Billion-Edge Graph Analytics on a 10 GB Laptop with DataFusion

Vincent Tran — Sat, 01 Aug 2026 00:00:00 +0000

A graph with two billion edges sounds like cluster territory. The usual mental model includes a distributed graph engine, several machines, a large memory budget, and enough operational machinery to keep all of it moving.

That model is useful, but it is not the only one.

Sem Sinchenko recently demonstrated two large graph workloads on a single machine with strict memory limits: PageRank over roughly one billion edges with a 5 GB process limit, and weakly connected components over almost two billion edges with a 10 GB limit. The implementation uses Apache DataFusion, an embeddable Rust query engine built on Apache Arrow, rather than a specialized graph database or distributed compute framework.

The important part is not the headline number by itself. The real lesson is architectural: a graph algorithm does not always need the whole graph as a pointer-rich object in memory. If each iteration can be expressed as scans, joins, aggregations, and state updates, a columnar query engine can stream the work and spill intermediate data to disk.

This changes the limiting resource. RAM stops being a hard ceiling and becomes a cache. Storage bandwidth and repeated sorting become the price of admission.

The Workloads

The experiment uses datasets from the LDBC Graphalytics benchmark, which provides graph datasets, algorithm definitions, and reference results.

The first workload is PageRank on graph500-26:

Property	Value
Vertices	32,804,978
Edges	1,051,922,853
Process memory limit	5 GB
DataFusion memory pool	4 GB

The second workload is weakly connected components, or WCC, on twitter_mpi:

Property	Value
Vertices	52,579,682
Edges	1,963,263,821
Process memory limit	10 GB
DataFusion memory pool	8 GB

These are different kinds of stress test. PageRank repeatedly propagates values across edges and aggregates contributions at destination vertices. WCC has to discover groups of vertices connected by a path after edge direction is ignored. For the Twitter graph, that means effectively symmetrizing the edge relation, so the engine may process close to four billion directed edge rows during preparation.

The memory limits were not estimates from a dashboard. The process ran under systemd-run with MemoryMax set and swap disabled. That distinction matters. A configurable query-engine pool does not account for every allocation in a process. A cgroup limit tests whether the complete program survives within the stated envelope.

Stop Thinking in Pointers

Many general-purpose graph libraries represent a graph for fast random access. A vertex leads to an adjacency list; an edge leads to neighboring vertices; algorithm state lives beside those structures. This is excellent when the representation fits in RAM. It becomes awkward when every pointer, object header, index, and duplicate direction competes for a constrained memory budget.

The DataFusion approach starts from a different representation:

a vertex table keyed by vertex ID
an edge table with source and destination columns
a compact state table for the current algorithm values
intermediate tables materialized between iterations when necessary

That is less like walking an in-memory object graph and more like repeatedly transforming relations. A PageRank step can be described at a high level as:

Join edges to the current source-vertex state.
Compute each edge’s contribution.
Group contributions by destination vertex.
Join the aggregate back to the vertex table.
Write the next state and repeat.

This is the Pregel bulk-synchronous idea expressed through a query engine. Every round reads a stable state, produces messages, combines them, updates vertex state, and reaches a barrier before the next round. The implementation does not require DataFusion to know what PageRank means. DataFusion only needs to execute the relational operators efficiently.

That separation is powerful. Graph-specific code defines the iteration and message semantics. The engine supplies planning, partitioned execution, Arrow batches, Parquet scans, joins, aggregation, memory accounting, and spill files.

Why Columnar Execution Helps

Graph datasets are sparse, but sparse does not automatically mean small. At billion-edge scale, even two integer columns occupy substantial space before intermediate state is considered.

Columnar execution helps in several ways.

First, the engine reads only the columns an operator needs. PageRank’s edge pass usually needs source and destination identifiers, not a large object containing every edge property.

Second, Arrow arrays place values of one type in contiguous buffers. That layout is friendly to vectorized processing and avoids much of the per-object overhead of a conventional object graph.

Third, DataFusion processes data in record batches. Operators can consume a stream of batches instead of waiting for an entire relation to materialize in memory.

Finally, the query planner can select and compose operators that already understand partitioning, ordering, and memory pressure. A compact graph layer can reuse years of query-engine work rather than building an external sort, hash table, aggregation engine, file reader, and memory manager from scratch.

Columnar layout is not magic compression, and it does not remove the need for indexes or graph-aware formats in every workload. Its advantage here is that the computation is dominated by bulk passes over a few simple columns.

Spilling Turns Memory into a Budget

DataFusion can enforce a memory pool for execution and spill supported operators to disk when they cannot reserve more memory. Its current feature set includes disk spilling for sorts, grouping, hash joins, and sort-merge joins. The memory-limited query guidance also explains an easy-to-miss tradeoff: more execution partitions can make tight-memory workloads worse because the fair pool divides memory among more concurrent reservations.

The experiment therefore used a deliberately small amount of parallelism. The WCC command limited execution to two CPU cores and gave DataFusion an 8 GB pool inside the 10 GB process cap. This leaves headroom for allocations not charged to the pool while avoiding dozens of partitions fighting over tiny shares.

The key join choice is also shaped by memory.

A hash join is usually attractive when one side is small enough to build an in-memory hash table. In PageRank, the vertex-state table can be compact enough for that approach, and Sinchenko reports it as faster. But the experiment also uses sort-merge joins to prove an out-of-core path. A sort-merge join can order both sides, spill sorted runs, and merge them without retaining a full hash table.

The tradeoff is extra I/O. If the edge table is sorted again on every iteration, the same billion rows repeatedly travel through storage and CPU-intensive comparison work. Pre-bucketing edges by vertex range or preserving useful order in the stored data could reduce that cost. DataFusion’s newer ordering optimizations can eliminate known redundant sorts in some plans, but the experiment notes that its current graph path does not yet reuse pre-sorted on-disk edges as effectively as desired.

This is the central bargain:

in-memory graph engines pay for capacity and gain fast random access
out-of-core relational execution pays for scans, sorts, and storage traffic to stay within a small RAM envelope

Neither is universally better. They optimize different constraints.

PageRank as Repeated Relational Work

PageRank maintains a score for every vertex. In each iteration, a vertex distributes its score across outgoing edges, destinations sum their incoming contributions, and the score is adjusted with the damping formula.

The state per vertex is small: a rank, an out-degree, and a participation flag in this implementation. The large, stable side is the edge table. That makes the workload a good match for a relational loop:

edges
  JOIN current_vertex_state ON edges.source = state.vertex
  PROJECT destination, contribution
  GROUP BY destination SUM(contribution)
  JOIN vertices ON destination = vertices.id
  PROJECT next_rank

The edge relation can remain on disk while state is written out between rounds. Materializing the new state breaks an ever-growing lazy-plan lineage and gives the next iteration a concrete input.

Fifteen full iterations took about 30 minutes in the constrained test. That is not an attempt to beat a tuned in-memory system. It is evidence that the calculation completes predictably without a cluster or a machine sized to retain every working structure.

The results were compared with Graphalytics ground truth and matched within a tolerance of 0.0001. That validation is essential. Large-scale systems can produce impressive throughput numbers while quietly dropping records, overflowing identifiers, or stopping before convergence.

Why Connected Components Is Harder

Weakly connected components asks which vertices belong to the same connected region when edge direction is ignored. It is useful in identity resolution: if customer record A shares an identifier with B, and B shares another identifier with C, all three may describe the same entity even when A and C have no direct link.

The Twitter input is directed, so the algorithm needs both orientations of each relevant edge. After symmetrization and deduplication, the preparation stage reported more than 3.2 billion edges. That peak is precisely where an out-of-core execution plan earns its keep.

The implementation follows an in-database connected-components strategy based on graph contraction. Early rounds do the heavy work, but each round reduces the remaining relation dramatically:

Stage	Edges remaining
After preparation	3,228,212,374
Forward iteration 1	840,238,268
Forward iteration 2	77,322,906
Forward iteration 3	5,624,128
Forward iteration 5	230,838
Forward iteration 22	0

Once the first few passes survive, the workload collapses into something much smaller. The algorithm then back-propagates component labels and writes the result. In the published run, the full WCC job took roughly 41 minutes from start to final output; the contraction phase after preparation became fast because almost all edges had disappeared.

The final component-size counts matched the benchmark result, including a giant component containing 52,515,193 vertices.

This pattern is worth looking for beyond graphs. A workload may have an intimidating peak input size but shrink aggressively after a few external-memory transformations. Designing for the peak with disk-backed operators can be more economical than keeping the entire pipeline in memory.

What the Demonstration Does Not Prove

It would be easy to turn this into the claim that every billion-edge graph fits comfortably on any laptop. That would be wrong.

The experiment covers two algorithms and two datasets. Other graph workloads depend on neighborhood expansion, frequent random access, large per-vertex state, dynamic mutation, or traversal latency. Those may fit a specialized graph engine better.

The implementation is also young. Sinchenko reports FairSpillPool deadlocks under extreme pressure and unresolved work around exploiting data already ordered on disk. The public graphframes-rs repository is an experimental codebase, not a drop-in replacement for Spark GraphFrames.

Storage matters. Spilling a billion-row sort to a fast local NVMe drive is very different from spilling it to a slow disk, a shared network volume, or a device without enough free space. Low RAM can shift the bill to storage capacity, write amplification, and SSD wear.

The benchmark also emphasizes feasibility over a complete cost comparison. To choose a production architecture, measure wall-clock time, peak memory, scratch-space usage, bytes read and written, CPU utilization, recovery behavior, and engineering complexity against the alternatives.

A Practical Design Checklist

If you want to test this pattern on a large iterative dataset, start with the shape of the computation rather than the headline size.

1. Make state explicit

Separate stable facts from changing algorithm state. Keep edges immutable where possible and store only the values needed for the next round.

2. Prefer bulk operators

Look for a formulation based on scans, joins, groupings, filters, and projections. The approach loses its advantage when every step needs unpredictable single-record lookups.

3. Choose identifiers carefully

Identifier width affects every edge row, sort key, join key, and intermediate batch. Use the narrowest safe type, but do not force 32-bit IDs when the domain may exceed them.

4. Preserve useful ordering

Repeated external sorts are expensive. Partition or bucket stable relations by common join keys, record their ordering accurately, and inspect the physical plan to confirm the engine can use it.

5. Set two memory limits

Configure the query engine’s execution pool, then test under a process or container limit with headroom for untracked allocations. Disable swap when you need to prove a real RAM bound rather than hide memory pressure.

6. Reduce parallelism under pressure

More partitions do not always mean more speed. If each partition receives too little memory, it may create additional spill runs and spend more time merging them.

7. Materialize iteration boundaries

Write compact state between rounds when doing so shortens lineage, frees old buffers, or lets the next plan start from a predictable physical layout.

8. Validate against known answers

Use a smaller in-memory implementation, benchmark ground truth, invariants, and component-size or rank checks. Resource efficiency is irrelevant if the result is subtly wrong.

9. Record spill metrics

Use EXPLAIN ANALYZE and DataFusion’s runtime metrics to find the operators producing the most rows, bytes, and compute time. Also monitor scratch-directory growth at the operating-system level.

The Larger Lesson

DataFusion is often described as a SQL or DataFrame engine, but its more interesting role is as a library of data-system building blocks. It gives an application a logical plan, physical optimizer, vectorized executor, file readers, memory pools, and spill-capable operators without forcing the application to become a general-purpose database.

The graph experiment uses that toolkit as an execution substrate. PageRank becomes a loop of joins and aggregates. Connected components becomes a contraction pipeline over tables. The graph API remains small because the engine already knows how to move and reshape large columnar relations.

That is the durable idea: when a dataset exceeds RAM, do not immediately jump from an in-memory library to a distributed cluster. First ask whether the workload can be expressed as ordered bulk transformations on a single machine.

Sometimes the answer will still be a cluster. Sometimes it will be a specialized graph engine. But when the algorithm has compact state, sequential passes, and spill-friendly operators, an ordinary laptop with a good SSD may be enough.

Sources

GitHub Stacked Pull Requests: A Practical Review Workflow

Vincent Tran — Fri, 31 Jul 2026 00:00:00 +0000

Large changes create an awkward choice.

Open one pull request and the reviewer receives a wall of code. Split the work into independent pull requests and later pieces cannot start until earlier ones merge. Create dependent branches by hand and the code becomes easier to review, but keeping every branch, base, and pull request synchronized becomes a job of its own.

Stacked pull requests are the attempt to escape that choice.

GitHub has now put the workflow into public preview across its web interface, mobile apps, APIs, merge system, and a new gh stack CLI extension. The core idea is old: break one large change into a sequence of small, ordered changes. The important change is that GitHub now understands the sequence as one object instead of displaying several mysteriously related pull requests.

This is more than a tidier branch diagram. A good stack turns an implementation into a reviewable argument. Each layer should establish one fact the next layer can safely depend on.

What a Stack Actually Is

A stack is an ordered chain of branches and pull requests. Only the first branch targets the repository’s trunk. Every later branch targets the branch immediately below it.

Imagine an authentication feature divided into three layers:

main
└── auth-model → PR #101, base: main
    └── auth-api → PR #102, base: auth-model
        └── auth-ui → PR #103, base: auth-api

The bottom pull request introduces the data model. The middle pull request adds endpoints that depend on that model. The top pull request builds the interface that consumes the endpoints.

When a reviewer opens PR #102, the relevant comparison is auth-api against auth-model. The model work is not repeated in the diff. PR #102 can therefore answer one narrow question: is the API layer correct?

That distinction is why a stack is not merely a pull request with carefully arranged commits.

Commits help explain history inside one proposal. Pull requests carry review state: a description, requested reviewers, code-owner rules, discussion threads, approvals, checks, draft status, and a merge decision. Giving each layer its own pull request gives each logical change its own review boundary. A database specialist can review the model layer while an application team reviews the API layer, and neither has to approve an ambiguous bundle.

The Real Problem Is Waiting, Not Branching

Developers have always been able to branch from an unmerged branch. The Git commands are not new. The trouble begins when the bottom layer changes.

Suppose review feedback requires rewriting auth-model. The auth-api branch still contains the old model commit in its ancestry, and auth-ui contains both old lower layers. To keep each diff accurate, the author must update the bottom branch, rebase every branch above it in order, force-push safely, and verify that each pull request still targets the right base.

Without tooling, the process looks roughly like this:

git switch auth-api
git rebase auth-model

git switch auth-ui
git rebase auth-api

Add five or ten layers, remote edits, merge conflicts, and multiple authors, and the maintenance cost becomes obvious. This is why stacked workflows have often depended on Gerrit, Phabricator, Graphite, Sapling, or internal tools rather than ordinary GitHub pull requests.

GitHub’s public preview does not invent dependency chains. It makes the platform aware of them and automates much of the bookkeeping.

Starting with `gh stack`

The CLI extension installs through the normal GitHub CLI:

gh extension install github/gh-stack

Inside a repository, initialize a stack and create the first layer:

gh stack init

After implementing and committing that layer, add another branch:

gh stack add auth-api

Continue until the change has the right shape, then publish the branches and pull requests:

gh stack submit

The extension records local stack metadata in .git/gh-stack, so it is local repository state rather than a file committed with the project. It creates one pull request per branch and sets each pull request’s base to the branch below it.

Several commands cover the day-to-day loop:

gh stack view shows the order and state of the stack.
gh stack up, down, top, and bottom move between layers.
gh stack checkout <pr-number> fetches an existing remote stack and reconstructs it locally.
gh stack rebase cascades a rebase from trunk through the dependent branches.
gh stack push publishes updated branches.
gh stack sync fetches, rebases when needed, pushes, and refreshes pull request state.
gh stack modify opens a terminal interface for reordering, renaming, dropping, or folding layers.

The rebase command also supports --continue and --abort. That matters because a cascading rebase does not eliminate conflicts. It gives the conflict process a stack-wide transaction: pause at the broken layer, resolve it, continue upward, or restore the branches to their earlier state.

The right mental model is not “the tool makes Git disappear.” The tool applies the same ancestry rules consistently across every branch.

Review the Story from the Bottom Up

A stack should be read in dependency order.

The lowest layer creates the first stable premise. It might be a mechanical rename, a schema migration, a new interface, or a refactor that preserves behavior. The next layer uses that premise. Each later layer should add one meaningful capability without silently changing the assumptions below it.

GitHub displays a stack map on each pull request so reviewers can see where the current layer sits. Reviews and checks still belong to individual pull requests, and existing branch protections continue to guard the path to main.

This permits parallel review, but parallel does not mean unordered.

A reviewer can inspect the UI layer before the model layer is approved, especially when the layers have different owners. However, approval of the top layer is conditional on the lower contract remaining stable. If the bottom layer changes its API, every reviewer above it needs to know whether the change invalidates an earlier conclusion.

Teams should make the dependency explicit in every pull request description:

what this layer establishes
which lower layer it assumes
what later layers depend on it
whether it can ship independently
where reviewers should begin

That context is much more useful than titles such as “part 2” and “part 3.” A stack map shows order; it does not explain intent.

How to Cut a Useful Stack

The hardest part is not the CLI. It is deciding where one review unit ends and the next begins.

A layer is well cut when it has one reason to change, a coherent test surface, and a conclusion a reviewer can approve. Good boundaries often follow this sequence:

Mechanical preparation. Rename a type, move a module, or isolate an interface without changing behavior.
Foundation. Add the schema, domain model, protocol, or shared abstraction.
Behavior. Implement the service or application logic that uses the foundation.
Integration. Connect the behavior to an API, event stream, job, or user interface.
Rollout. Add a feature flag, migration step, telemetry, or cleanup.

That order keeps conceptual changes separate from noisy mechanical changes. A reviewer should not have to prove that a renamed function still behaves the same while also evaluating new authorization logic.

Every layer does not need to be deployable by itself, but it should leave the repository in a valid state. It should compile, pass the relevant checks, and preserve established invariants. If a layer intentionally introduces dormant code, say how the later activation layer will expose it.

Avoid cutting a stack by file count alone. A pull request containing “all backend files” can still mix several concerns, while a cross-cutting API rename across many files may be one perfectly coherent mechanical layer.

The test is simple: can a reviewer describe what this layer proves in one sentence?

Merging the Whole Stack

GitHub’s native merge behavior is the most significant platform addition.

When the latest ready pull request is merged as a stack, GitHub can land it together with every unmerged layer below it. The operation respects the existing required reviews, status checks, branch protections, and merge requirements for the individual pull requests.

Teams can also land only the lower portion. GitHub says the remaining upper pull requests stay open and are automatically rebased and retargeted after the lower layers merge.

That matters for deployment pipelines. GitHub’s stack merge API is asynchronous because a multi-layer merge can take longer than the timeout used by the older synchronous path. A complete stack can be applied atomically with one push to the base branch, allowing CI or deployment automation to react to the final combined state instead of processing every layer as a separate trunk update.

GitHub also exposes REST and GraphQL stack objects. The REST API can list and retrieve stacks, create a stack from an ordered set of existing pull requests, append pull requests, or remove unmerged pull requests from a stack. The documented maximum is 100 pull requests in one stack, although a stack that deep would be a serious human-factors problem long before it reached the API limit.

Native APIs are important for teams with custom merge queues, release bots, or policy engines. A first-class stack cannot stop at the web interface; automation needs to know that PR #103 depends on #102 and #101 rather than inferring relationships from branch names or description links.

Public Preview Means Preview

The feature is promising, but teams should adopt it with the caution appropriate to a public preview.

The launch discussion includes reports of edge cases around whole-stack squash merges, stale local and remote branches, deleted base branches, and approval rules after rebases. A GitHub engineer explained that squash-merging a stack is unusually difficult because the platform must predict a sequence of commits that do not yet exist, then evaluate mergeability, protection rules, and approvals against those future results.

GitHub reported in that discussion that 99% of stack merges were succeeding, while also calling further reliability work a top priority. That is a healthy number for experimentation and an uncomfortable number for a critical workflow that runs hundreds of times.

Cross-fork stacks are another boundary. GitHub said support is planned for a stack contained inside one contributor fork and targeting the original repository. More complicated stacks spanning multiple forks raise security problems because the platform automatically rebases branches after partial merges.

Merge queue support is also rolling out progressively rather than appearing everywhere at once.

Practical safeguards are straightforward:

begin with two or three layers, not ten
use the merge strategy your repository already understands
keep local branches synchronized before starting a cascade rebase
inspect every pull request after conflict resolution
retain ordinary branch protection and required checks
test partial and full-stack merging in a low-risk repository first
document how approvals are handled after a lower layer changes

The stack UI reduces invisible relationships. It does not remove the need to understand Git ancestry.

Small Pull Requests Are Not Automatically Fast Pull Requests

Stacking is often sold with a simple formula: smaller pull requests merge faster. The research is less tidy.

A large empirical study covering 845,316 GitHub pull requests, plus data from Gerrit and Phabricator, found no strong relationship between pull request size and time to merge. Review latency depends on far more than lines changed: reviewer availability, organizational boundaries, change type, risk, CI duration, contributor trust, and release policy all matter.

That result does not make stacks useless. It clarifies their value.

The best reason to stack is not to manufacture a better velocity metric. It is to improve the unit of reasoning. A focused pull request lets a reviewer understand intent, select the right expertise, discuss one concern, and approve a bounded claim. It also makes an individual layer easier to revert or diagnose.

A badly designed stack can be slower than one large pull request. It creates more notifications, descriptions, check runs, branch updates, and approval decisions without reducing cognitive load. Splitting 600 tangled lines into six arbitrary 100-line slices is paperwork, not architecture.

The stack pays for itself when the layers expose the real dependency structure of the change.

Stacks in the Age of Coding Agents

Coding agents make this workflow more relevant and more dangerous.

They can produce changes faster than humans can review them. Asking an agent for an entire feature can leave a maintainer facing thousands of lines that arrived in minutes but still require careful human judgment. A stack can turn that output into a sequence: preparation, data model, service behavior, integration, and rollout.

Agents can also perform the mechanical work of proposing boundaries, creating branches, updating descriptions, and cascading rebases. GitHub even provides a gh-stack skill intended for coding agents.

But an agent should not choose boundaries only to satisfy a size limit. The author remains responsible for the argument encoded by the stack:

Does every layer have one clear purpose?
Are tests introduced with the behavior they validate?
Is a security-sensitive change isolated for the right reviewer?
Does an early refactor genuinely preserve behavior?
Can later layers be understood without hiding important interactions?

The volume of generated code makes review structure more important. It does not make review optional.

A Team Adoption Checklist

Before making stacks the default for large work, agree on a few rules.

Define when to stack. A two-file bug fix does not need a branch hierarchy. Use stacks for dependent work that would otherwise become one difficult review or a sequence of blocked changes.

Set a depth guideline. Three to five layers are usually understandable. A deeper stack may signal that the feature should be divided into separate deliverables.

Require coherent layers, not line limits. Size is a signal. Purpose is the boundary.

Review from the bottom. Later reviews can begin early, but lower-level contracts must settle first.

Name branches and pull requests by capability. Prefer auth-model and auth-api over feature-1 and feature-2.

Keep checks attached to every layer. A stack is not permission to postpone correctness until the top.

Plan for changes low in the stack. Decide who rebases, when reviews are dismissed, and how reviewers are notified of changed dependencies.

Measure outcomes, not PR count. More pull requests are an implementation detail, not proof of more productivity.

The Bottom Line

GitHub’s stacked pull requests make a mature code-review pattern available without forcing teams to bolt a second review system onto their repository. The platform can now display dependencies, preserve focused diffs, coordinate existing checks and protections, expose stack APIs, and merge several approved layers as one operation.

The workflow succeeds only when the stack tells a clear technical story.

Start with the smallest stable premise. Put one reviewable idea in each layer. Keep every layer valid. Make dependencies explicit. Treat rebases as real history rewrites, not magic. And remember that a stack is valuable because it helps reviewers reason—not because it creates more pull requests.

Used that way, stacked pull requests can let implementation continue without turning review into an afterthought.

DEV Community: Vincent Tran

Acadia: Bringing Precise Types and Functional Programs to the Database

The real problem is contract drift

A table is a precise program value

Endpoints are compiled, not interpreted

Transactions become ordinary composition

Generated types close the client-server gap

Migrations should be programs the compiler can question

Why compile to the database at all?

Where the abstraction ends

A sensible evaluation plan

The larger idea deserves attention

DuckDB 2.0: From Embedded Analytics to a Networked Query Engine

One Engine, Two Operating Models

A Server Mode Does Not Change the Workload

VARIANT Turns Semi-Structured Data into Columns

Triggers Bring Database-Side Reactions

SQL Becomes a Better Pipeline Language

Remote Data Needs Independent I/O Concurrency

Existing Queries Get Faster Without New Syntax

Storage v2.0 Loads Less Up Front

A New Parser Makes SQL Extensible

Smaller Timezone Support, Stable Extensions

What to Evaluate Before the Release

How an Experiment Loop Made a GPU Kernel 232x Faster

The task: compact Householder QR

Why ordinary Householder QR underuses a GPU

The evaluator was the product

Shape-by-shape evidence prevents fake wins

Profiling turns “make it faster” into a hypothesis

One incumbent creates a local maximum

Domain knowledge improves the loop in three places

A practical autoresearch workspace

Where this approach works—and where it does not

The lasting lesson

Build Wide, Ship Narrow: Decompose Pull Requests After the Code Teaches You

Two Decisions That Should Not Be Confused

The Wide Build Is a Laboratory, Not a Giant PR

Demo Before Review

Turn the Finished Diff into a Dependency Graph

Cut Fresh Branches, Not Decorative Diffs

Stack Only When the Dependency Is Real

Put Deletion Last

Review Is for Understanding, Not Ceremonial Approval

Where the Workflow Breaks Down

A Practical Team Policy

Compression Is Prediction: Why Better Guesses Need Fewer Bits

Compression begins with redundancy

Separate the model from the coder

Entropy is the average bill

Arithmetic coding turns probabilities into ranges

Context makes a model useful

A language model already produces what a coder needs

A compressor can generate, too

Why we do not ship an LLM with every archive

Compression is an honest model test—with conditions

The practical lesson: improve the probabilities

Further reading

Postgres CDC Without the Connector Tangle

Why Ordinary CDC Becomes Fragile

Push Capture into the Database

One Timeline, Four Positions

Snapshots and Changes Must Share a Boundary

Transactions Are the Recovery Protocol

Delete and Append Beat Universal Upserts

Failure Recovery Is Still Operational Work

Schema Evolution Needs Explicit Semantics

What Is Open and What Is Product-Specific

A Design Checklist for Any CDC System

The Durable Idea

Sources

Kitesurf: A Browser Built for AI Agents, Not People

Why agents need a different browser budget

A browser split across isolates

SandboxOutbound: one guarded path to the network

Engine: protocol gateway and session owner

PageScript: one clean JavaScript world per page

PageRenderer: turn a scene into pixels

Tests are the map for a new engine

The benchmark result is a trade, not a victory lap

`VARIANT` Turns Semi-Structured Data into Columns

Starting with `gh stack`