Benchmarking Open Agents for Multi-File Refactors

TL;DR

• Skylark Creations' October 3 benchmark evaluated open coding agents against a 36-point rubric, showing they now rival proprietary solutions in multi-file reasoning and terminal execution.
• The report highlights a shift in developer workflows: engineers require tools that plan, edit across multiple files, and iterate via tests without forcing lock-in to a single ecosystem.
• MonkeysCode addresses this directly. Our recent Editor 1.2.12 (featuring the Capuchin agent) and CLI 1.0.2 releases provide a robust harness for multi-file loops, supporting both frontier and open models via BYOK.
• Effective agent execution requires strict context management; our latest updates include usage dashboards exposing cache-write tokens and architectural changes to optimize prompt caching breakpoints.

The Skylark Creations Benchmark

On October 3, 2026, Skylark Creations published an updated evaluation of open-source coding agents. The benchmark assessed tools like Aider, Cline, and Orbi against a rigorous 36-point rubric designed to measure capabilities far beyond single-file autocomplete.

The report's primary finding is clear: open agents now rival proprietary solutions when it comes to multi-file reasoning and terminal execution.

Historically, the gap between open models and proprietary frontier models was most visible in context retention and structural planning. A model might successfully write a Python script in a vacuum, but fail to correctly update an interface, modify the implementing classes across three different files, run the test suite, and parse the resulting stderr output to fix a typo. The Skylark benchmark demonstrates that this gap is closing. By effectively utilizing open models, developers can achieve complex, agentic refactoring without being tethered to a single proprietary ecosystem or incurring prohibitive API costs.

The Engineering Mechanics of Multi-File Execution

Handling multi-file refactors introduces a specific set of engineering challenges for any agent harness. It is not simply a matter of passing more tokens into the context window. The agent must successfully navigate a multi-step loop:

1. Planning: Analyzing the codebase to identify which files require modification based on a given prompt.
2. Diff Generation: Emitting precise edits (often via search/replace blocks) rather than rewriting entire files, which wastes tokens and risks introducing regressions.
3. Terminal Execution: Running compilers, linters, or test suites directly in the environment.
4. Iteration: Parsing the terminal output (stdout/stderr) and determining if a failure requires a logic change, a dependency update, or simply a retry.

When a developer asks an agent to, for example, fix a database migration error—such as changing a raw SQL query from using a ? operator to using jsonb_exists() in Postgres—the agent must locate the migration file, update the schema definition, modify the corresponding application data models, and run the test suite to verify the changes.

If the test fails, the agent must read the failure, retain the context of its previous edits, and apply a correction. This requires the agent harness to safely supervise the terminal process and manage the context window efficiently.

Running Open Agents with MonkeysCode

Developers want the flexibility to use the best model for the job, whether that is a local model, an open model, or a proprietary frontier model. We designed MonkeysCode to provide a flexible, unopinionated harness for these agentic loops.

On October 5, 2026, we shipped MonkeysCode CLI 1.0.2 and Agent Manager 1.0.16. These releases solidify our terminal execution and supervision capabilities, allowing developers to run multi-file agent loops directly from their command line.

Simultaneously, we released MonkeysCode Editor 1.2.12. This desktop IDE includes Capuchin, our built-in coding agent. Capuchin is explicitly designed to handle the workflow validated by the Skylark benchmark: it plans changes, executes diff reviews across multiple files, runs tests, and iterates based on the output.

Because MonkeysCode supports BYOK (Bring Your Own Key), developers can plug in the open models that performed well on the Skylark rubric, or utilize local models, running them through Capuchin's multi-file execution engine.

Here is an example of how you might initiate a multi-file refactor using the MonkeysCode CLI:

# Initialize an agent session targeting a specific directory
monkeyscode agent start ./src/database

# Provide the instruction for a multi-file refactor
> "Migrate all raw Postgres queries using the '?' operator to use jsonb_exists(). Update the corresponding unit tests in ./tests/database/ and run npm test to verify."

The Agent Manager will orchestrate the planning phase, apply the necessary diffs, execute the test command, and feed the terminal output back to the model for iteration.

Prompt Caching and Context Economics

As multi-file reasoning becomes standard, the economics of context management become the primary bottleneck. Running a loop that repeatedly analyzes three files, a test suite, and terminal output will rapidly consume tokens.

To make this practical, agent harnesses must leverage prompt caching. When the system prompt, the core codebase structure, and the tool definitions are cached, the agent only pays for the delta (the new terminal output or the modified file) on each iteration.

In our latest release cycle, we shipped several architectural updates to support this:

• Cache-Write Tokens: Our usage dashboards (available for both individual and Team plans) now explicitly show cache-write tokens, giving developers transparency into their context economics.
• Auto Model Routing: We shipped step 4 of our "Auto" model routing build, adding an Auto mode tab on the Models & Pricing page. This system automatically routes requests based on context requirements and displays cache ratios, warning users if they are mixing frontier providers in a way that defeats caching efficiency.
• System Block Separation: We updated our context kernel to keep system blocks separate for cache breakpoints. Whether you are using the newly integrated GPT-6 via Azure Foundry or an open model, maintaining strict cache breakpoints ensures that the static parts of your prompt do not incur redundant processing costs.
• Stream Stability: Multi-file reasoning takes time. We recently deployed a fix to keep Anthropic SSE streams alive while the model is "thinking," preventing timeouts during complex planning phases.

Takeaway

The Skylark Creations benchmark provides concrete evidence that open models are capable of executing complex, multi-file refactors. However, the model is only half of the equation. The other half is the agent harness—the system that manages the context window, applies the diffs, and supervises the terminal execution.

Developers should not have to choose between advanced multi-file capabilities and the freedom to use open models. By decoupling the agent harness from the underlying model, tools like the MonkeysCode CLI and Capuchin allow engineers to leverage the reasoning capabilities of open agents within a robust, multi-file execution environment.

You can download MonkeysCode today to test Capuchin and the CLI with your preferred models, or review our documentation for details on setting up BYOK and local model routing.