Why a Smaller Coding-Agent Harness Can Do More

When a coding agent disappoints, the model usually gets blamed. When it succeeds, the model usually gets the credit. Both reactions miss a large part of the system.

The model never touches a repository by itself. A harness chooses the system prompt, describes the available tools, decides which files and earlier messages return on every turn, executes commands, stores sessions, compacts history, and determines how much context gets sent again. Two products can call the same model at the same reasoning level and still produce very different bills—and sometimes different results.

Pi makes that hidden layer unusually easy to see because it starts with almost nothing. Its default model interface contains four tools: read, write, edit, and bash. Features that many coding agents treat as mandatory—plan mode, built-in task lists, subagents, MCP, permission dialogs, and background shells—are deliberately left outside the core.

That is not minimalism as decoration. It is an argument about where complexity belongs: begin with a small, legible mechanism, then add only the behavior your work can justify.

The Harness Is Part of the Product

A coding-agent request is not just the sentence typed into the terminal. A typical model call also contains:

• the harness’s system instructions;
• schemas and descriptions for every active tool;
• repository instructions such as AGENTS.md;
• relevant conversation history;
• file contents and command results gathered during the task;
• summaries or bookkeeping added by the harness.

Most of that material is useful. The problem is accumulation. A tool schema added for one rare operation may be sent on every turn. A detailed planning framework may consume attention during a one-line fix. An ever-growing transcript can keep old errors alive long after they stopped helping.

This is why context should be treated as a working set, not a storage bucket. The agent needs enough information to act correctly, but every extra instruction competes with the code, the task, and the evidence produced by tests.

Pi’s default is intentionally narrow. Four primitive tools are enough to inspect a project, change it, and run the project’s own commands. A shell already composes thousands of specialized programs. A file can hold a plan. tmux can host another process. A repository instruction file can encode local rules. The core does not have to reimplement every surrounding system.

This approach also keeps the interface visible. A developer can understand the default capability boundary without reading a catalog of dozens of tool definitions. That makes failures easier to reason about: was the prompt unclear, was the wrong file read, did a command fail, or did the model make a bad edit?

Context Has a Compounding Cost

The token count of a system prompt looks small next to a large context window, but the relevant cost is not its size once. It is its size across the whole trajectory.

Suppose an agent makes 20 model calls while investigating a bug. If each request repeats a large prefix of instructions and tool definitions, the overhead is paid repeatedly. Providers can reduce that cost with prompt caching, but caching depends on an identical prefix. A small change near the beginning of the request can make the provider process everything after that point again.

Tool catalogs are especially sensitive. Tools are normally placed before the conversation. Adding, removing, reordering, or changing a schema can invalidate the reusable prefix. Pi’s own guide to prompt caching explains that newer model APIs can defer additive tools without disturbing earlier context, but support varies. Extensions that rebuild prompts or change tool sets dynamically can still destroy cache reuse.

The practical lesson is broader than Pi:

1. Keep the stable prefix stable.
2. Do not activate a tool merely because it might be useful someday.
3. Prefer repository-native commands over verbose wrappers when they provide the same evidence.
4. Measure total input per completed task, not only the price of one token or one call.

That last point matters because a cheaper model can be more expensive if it needs more attempts, reads more files, or fails often enough to require a stronger model afterward. Efficiency is the cost of reaching a verified result.

Context discipline becomes even more important with local models. Their usable context windows are often smaller, and processing a long prompt prefix can be slow enough to dominate an interactive turn. A stable, compact prefix avoids needless reprocessing and leaves more room for the repository itself. The same design that reduces API spend can therefore reduce local latency.

Databricks Put the Harness on the Benchmark

Databricks recently built an internal benchmark from real engineering work on its multi-million-line codebase. The tasks covered Python, Go, TypeScript, Scala, Rust, Java, Bazel, Protobuf, and other technologies used in production. The team derived prompts from recent human-written pull requests, held back relevant tests, reviewed tasks manually, and removed access to Git history so agents could not recover the original implementation.

That methodology is more useful than a generic leaderboard for one important reason: it tests the work the organization actually does.

The Databricks results produced several lessons:

• model price per token did not reliably predict cost per completed task;
• open and proprietary models occupied the quality-cost frontier together;
• stronger models sometimes finished cheaply because they used fewer tokens;
• the harness materially changed both quality and cost.

The most revealing comparison held the model and reasoning effort constant while changing the harness. Databricks observed task costs differing by more than 2x in some cases while quality stayed the same. Pi sent roughly one-third as much context per turn and completed the work in fewer runs.

In the published chart, Pi paired with Opus 4.8 at high reasoning effort reached the strongest overall pass rate while costing materially less per task than the same class of model used through Claude Code or Codex. The interesting result is not a permanent leaderboard position. It is the controlled difference produced by the layer around the model.

This does not prove that Pi is universally superior. Databricks explicitly warns against that conclusion. Their benchmark reflects their tasks, repositories, configuration, and evaluation process. A harness with more built-in guardrails may be worth its overhead in a regulated environment, and an integrated product may outperform on workflows its vendor optimized directly.

What the benchmark does prove is that “which model?” is an incomplete procurement question. Teams should test model-harness pairs. A benchmark that swaps models but freezes the harness measures only half the system.

Minimal Does Not Mean Fixed

A tiny core is useful only if it can grow without being forked every time a team needs a specialized workflow. Pi handles that through several layers:

• AGENTS.md and related context files for project rules;
• skills for instructions loaded when a task needs them;
• prompt templates for repeatable requests;
• TypeScript extensions for tools, commands, events, interface elements, providers, compaction, and policy;
• packages that bundle extensions, skills, prompts, and themes.

This architecture replaces “batteries included” with “attachment points included.” Plan mode can be a file or extension. Subagents can be separate Pi processes or an extension. MCP can be added when a team genuinely needs it. Permission gates can match the execution environment instead of being imposed as one universal dialog.

The distinction matters. Removing features without offering a clean way to add them creates a toy. Providing a small core with stable extension surfaces creates a platform.

Shopify’s Autoresearch Shows the Payoff

Shopify provides a useful example of complexity being added at the edge rather than the center.

An engineer wanted to reduce slow and flaky build work around Polaris. A normal request to “make the build faster” produced an unsuccessful one-shot attempt. The task was too broad and lacked a mechanism for deciding whether each change helped.

The engineer instead asked Pi to create an Autoresearch extension. The resulting loop had four parts:

1. choose a measurable target;
2. record a baseline;
3. form a hypothesis, change the code, and measure again;
4. keep improvements while discarding crashes and regressions, then repeat.

Within half an hour, the first version was working. The important advance was not a smarter prompt. It was a better control loop. The metric acted as an objective function, and the repository’s build supplied evidence. The agent gained permission to explore, but every experiment had to survive measurement.

The loop did propose bad shortcuts, including changes that made the build faster by doing less legitimate work. Human review remained essential. But it also found real waste: a visual-regression build performed packaging steps that Storybook immediately repeated, and a TypeScript transform processed hundreds of files that did not need it. The accepted changes reduced that build by 65%.

The Shopify engineering account describes the tool growing from a local experiment into an open-source project used across more than 40 metrics. The broader lesson is that domain-specific orchestration can be extremely valuable. It simply does not need to be present in every agent session.

Autoresearch earned its complexity because the workflow had a repeatable target, an automated evaluator, and enough iterations for small gains to compound. That is a much stronger reason to add an agent feature than “other tools have one.”

The Cost of Moving Complexity Outward

Minimalism does not eliminate complexity. It changes who owns it.

Pi extensions execute with full system access. A package can register tools, handle credentials, run commands, and alter model behavior. Pi’s documentation tells users to review third-party packages because installing one is closer to installing software than enabling a harmless editor theme.

The core also omits permission popups. Its philosophy is that isolation should come from a container, sandbox, or operator-defined extension. That can be a cleaner security model than clicking through frequent approvals, but only when the isolation actually exists. Running an unrestricted agent in a sensitive checkout and assuming the small prompt makes it safe would be a category error.

The Hacker News discussion surfaced ordinary product trade-offs too. Some users reported slow startup with extensions, disliked the configuration directory choice, or missed familiar terminal keybindings. Others valued the ability to change those behaviors themselves. Minimal code and minimal user friction are related goals, not identical ones.

There is also a maintenance tax. A curated, integrated tool gives teams one vendor responsible for making planning, permissions, subagents, browser automation, and updates work together. A customized Pi setup may fit better, but someone must own its extensions, dependency updates, cache behavior, security review, and onboarding documentation.

The right comparison is therefore not “simple versus complex.” It is vendor-selected complexity versus user-selected complexity.

How to Evaluate a Coding Harness on Your Own Work

Teams do not need a research department to apply the Databricks lesson. A small internal evaluation can start from merged pull requests.

1. Build tasks from recent changes

Select representative fixes and features. Remove the implementation while keeping a clear description of the desired behavior. Exclude generated changes and tasks that depend on unavailable services.

2. Hold back objective checks

Use the original tests where possible, but inspect them for overfitting. A test that demands one exact implementation may reject a valid alternative. Compile checks, behavior tests, linters, and focused manual review can work together.

3. Seal accidental answer paths

Do not leave the solution in later Git history, cached build artifacts, review comments, or nearby branches. An agent that finds the original patch has tested repository archaeology, not coding ability.

4. Compare complete pairs

Run the same model and effort through different harnesses, then run different models through the same harness. Record pass rate, wall time, number of model calls, input and output tokens, cache reads and writes, and final cost.

5. Inspect traces, not just scores

A passing agent may take a dangerous shortcut. A failing agent may reveal a missing repository instruction. Trace review shows whether the harness kept a focused working set, repeated irrelevant context, selected tools poorly, or stopped without verification.

6. Price ownership honestly

Include the human cost of maintaining extensions and guardrails. A low token bill is not a bargain if every developer spends hours repairing a bespoke setup.

A Practical Minimal-Harness Checklist

If you are designing or trimming an agent workflow, these rules capture the useful part of Pi’s approach:

• Start with file access, editing, and command execution.
• Put durable project knowledge in versioned repository instructions.
• Load specialized instructions only when relevant.
• Keep tool schemas short, distinct, and stable.
• Prefer measurable loops over motivational prompt language.
• Add orchestration only after a repeated workflow appears.
• Run risky tools inside a real security boundary.
• Treat every extension as executable code.
• Preserve cache-friendly prompt prefixes.
• Evaluate cost per verified result on your own repository.

This does not require using Pi. The same discipline can improve a large commercial harness: disable unused integrations, shorten project instructions, remove redundant tools, isolate long-running experiments, and measure where the context goes.

Complexity Should Earn Its Keep

Modern frontier models already understand files, shells, tests, and common development conventions. A harness no longer has to explain software engineering from first principles on every request. Its job is increasingly to provide a clean interface to the environment, preserve the right context, and get out of the way when the model has enough information to act.

Pi demonstrates the upside of that restraint. Databricks shows that a lean harness can reduce end-to-end task cost without sacrificing quality on real work. Shopify shows that a minimal core can still support sophisticated autonomous loops when those loops are built for a concrete metric.

The harder lesson is that minimalism is not free. It asks teams to choose their own extensions, security boundaries, and operating practices. For developers willing to own those decisions, that is the advantage: the harness contains the complexity their workflow needs, not the complexity a product roadmap happened to collect.

---

Sources: Earendil — Pi’s Minimalism Is Its Advantage, Databricks — Benchmarking Coding Agents on a Multi-Million-Line Codebase, Shopify Engineering — Autoresearch Isn’t Just for Training Models, Pi documentation and source, Earendil — Prompt Caching in Agents, Hacker News discussion