Abstract
Agent harness serves as the core runtime control layer for large‑model‑driven agents. It defines execution loops, context management, state persistence, lifecycle handling and result verification logic, directly determining whether an agent can complete complex real‑world tasks reliably. Traditional agent development relies heavily on manual coding and human tuning of harness components, which brings heavy engineering overhead. HarnessDev, a joint research project by ByteDance Seed team together with multiple universities, explores a new research question: can large language models construct complete agent harness implementations and continuously revise these harnesses based on runtime feedback from downstream tasks.
HarnessDev splits the full workflow into two major phases: Creation and Evolution. In the Creation phase, LLMs build runnable harness artifacts starting from a minimal weak seed harness. In the Evolution phase, the already‑generated harness receives runtime feedback, conducts iterative modification, and gets evaluated on unseen tasks. Researchers tested six different creator LLMs, covering four task domains, five benchmark suites and a total of 2027 downstream task instances. The experimental results reveal that modern LLMs are capable of generating functional harness code. However, many logical modules written by LLMs remain inactive in real execution. Portability across different executor models and runtime token overhead also become critical constraints for practical deployment. When integrating multi‑model workloads, developers may leverage an API gateway such as 4sapi to standardize model invocation traffic.
1. Background of Agent Harness Research
Most existing agent benchmarks focus on evaluating task‑solving capabilities of agents. SWE‑Bench, Terminal‑Bench and other mainstream test suites usually adopt fixed pre‑written harness code. The harness handles environment interaction, tool invocation and output parsing, while the LLM acts purely as the decision‑making brain inside the system. This setup conceals an important fact: harness quality itself imposes huge influence over final task success rates. Poorly‑implemented control logic will cause context corruption, checkpoint failure, incorrect termination judgment and tool parsing errors, even when the LLM has strong reasoning capacity.
Constructing a high‑quality agent harness demands systematic engineering work. Developers need to design execution flow, manage context windows, implement state snapshots, handle exception recovery and verify output validity. For custom agent scenarios across coding, data processing, writing and research domains, manual harness development consumes considerable manpower. This motivates the core research target of HarnessDev: delegate harness construction and iteration to LLMs themselves.
The starting point for all experiments is a weak seed harness. This seed implementation only provides basic infrastructure: reading task configurations, managing working directories, offering file access, search capability, process spawning and LLM invocation entry points. It records runtime logs and trajectory data, yet contains zero high‑level agent control logic. Creator LLMs must supplement six core control mechanisms based on this minimal substrate to turn the seed into a fully‑functional agent harness.
The six categories of control mechanisms are summarized as below:
- Execution: Main execution loop, planning logic, termination and stopping conditions
- Tools: Tool invocation, parameter validation, input sanitization and error handling
- Context: Task metadata, code history, log management and context assembly rules
- State: Current progress, attempt tracking, checkpoint snapshots and artifact persistence
- Lifecycle: Pre‑processing, recovery logic, post‑processing and task wrap‑up
- Verification: Output inspection, result checking, post‑condition validation and log recording
These six modules cover the full lifecycle of agent tasks, from receiving input requests to delivering final outputs. Taking coding agents as an example, the Execution module drives step‑by‑step task advancement. The Tools module standardizes tool‑call parsing. Context organizes historical messages fed to the model. State keeps runtime progress and snapshot data. Lifecycle deals with interrupt recovery. Verification conducts validation for modified code and generated artifacts. It is worth noting that these six functions do not require one‑to‑one mapping to independent source‑code files. What matters is whether these logics actually take effect during real‑world task execution.
2. Creation Phase: Generating Runnable Harness from Weak Seed
During the Creation phase, different creator models take the weak seed harness and add, modify or delete source‑code files to produce complete runnable harness artifacts. Researchers measured code‑editing scale and benchmark performance across six creator models including Opus‑4.8, GPT‑5.5, Gemini‑3.1 Pro, DeepSeek‑V4 Pro, Qwen‑3.7 Max and Seed‑2.0 Pro.
| Creator Model | Files add/chg/del | Total net LOC | Median per replica | LOC Range | SWE‑Pro | Terminal‑Bench‑2.1 |
|---|---|---|---|---|---|---|
| Opus 4.8 | 19/7/16 | 2470 | 698 | 656‑1116 | 69.3 | 64.8 |
| GPT‑5.5 | 3/10/0 | 3537 | 1231 | 1059‑2747 | 32.8 | 52.1 |
| Gemini 3.1 Pro | 4/4/0 | 1006 | 324 | 124‑412 | 43.6 | 68.8 |
| DeepSeek V4 Pro | 14/12/4 | 3562 | 1339 | 931‑1672 | 28.9 | 41.3 |
| Qwen‑3.7 Max | 12/6/1 | 3242 | 988 | 551‑1323 | 33.5 | 35.6 |
| Seed 2.0 Pro | 14/6/0 | 3294 | 1200 | 868‑2226 | 10.8 | 6.0 |
A total of 18 independent harness instances were generated in Creation experiments. Cumulative net new code reached 17 111 lines in total. There exists no direct correlation between code volume and benchmark performance. Gemini‑3.1 Pro only added 1006 lines of net new code, yet achieved 68.8 points on Terminal‑Bench‑2.1, ranking top among all tested models.
Researchers further inspected real‑execution evidence for each control module inside these generated harnesses. Execution Loop, Tools and Lifecycle obtained high activation rates. 13 out of 18 harness instances showed observable runtime evidence for State and Memory modules. However, deep inspection revealed critical defects: among 11 harnesses defining State classes, only one implemented functional checkpoint interfaces. In 26 679 real‑task execution trajectories collected, checkpoint events were never triggered even once. Although State‑relevant code existed in source files, those branches never got hit under practical workloads.
Across 108 component implementations from all harness artifacts, only 72 components were fully exercised during runtime. 18 components merely generated partial log records without actual business logic execution, and another 18 pieces of code never ran at all. The observation delivers a key insight for agent research: merely checking whether certain code segments exist is insufficient. Evaluators must verify whether these mechanisms are actually triggered and produce expected effects under real‑world task flows. Trajectory log analysis becomes an essential validation method.
The Verification module exposed another serious flaw. In 100 test runs using one Opus‑built harness, 99 instances reported self‑declared success status internally, but merely 48 cases truly passed ground‑truth evaluation. The harness tended to trust its own checking logic blindly without independent result confirmation. Similar issues appeared in Data‑domain harness implementations: among 2 325 task runs, 441 cases produced degraded outputs that were undetected by built‑in verification logic. Effective verification cannot rely solely on model self‑judgment; independent downstream validation steps are indispensable.
3. Harness Portability Across Different Executor Models
After a harness is built by one creator LLM, can it work normally when switched to a different executor model? HarnessDev designed two evaluation settings for portability testing. Under Self‑Eval, the same LLM acts both as creator and executor. Under Unified‑Eval, all harness artifacts run with Gemini‑3.1 Pro serving as the unified executor. Significant performance gaps appeared between the two groups.
Take the harness generated by Opus‑4.8 as an example. It reached 69.3 SWE‑Pro score under Self‑Eval. After switching executor to Gemini‑3.1 Pro, performance dropped sharply to 33.0. Search‑related tasks suffered even larger degradation: repeat query rate rose from 10.1 % to 88.2 %. Built‑in deduplication, retry and termination rules closely coupled with the behavioral characteristics of the original creator model. When swapping underlying executor, these preset strategies no longer match model output patterns.
Measured across multiple benchmarks including SWE‑Pro, Terminal‑Bench‑2.1, MLE‑bench, EQ‑Bench3 and BrowseComp, portability experimental data consistently proved this point. Agent harness portability issues are not limited to API calling layers or tool‑definition layers. Prompt templates, conversation protocols, execution budgets, step limits, context‑window policies and stopping criteria are all tightly coupled configuration items. Every time developers replace the underlying LLM executor, these runtime strategies require re‑validation and fine‑tuning. Multi‑model switching scenarios can bring additional complexity to operation; standardized traffic management via 4sapi helps reduce repetitive adaptation work for developers.
4. Evolution Phase: Feedback‑Driven Continuous Harness Improvement
The Creation phase outputs the initial version H₀ of each harness. The Evolution phase simulates real‑world iterative engineering: harness receives runtime feedback collected from task trajectories, creator models perform code modification, producing updated versions H₁, H₂ up to H₁₀. Researchers reserved 630 held‑out tasks which never participated in feedback datasets, to measure generalization performance of revised harness versions.
In self‑runtime feedback tracks, most creator models gained steady performance improvements. DeepSeek‑V4 rose from 47.2 to 60.6 points on held‑out sets, with a gain of 13.4 points. Qwen‑3.7 improved from 41.8 to 55.7 points. When fixing Gemini‑3.1 Pro as unified executor for all harnesses, performance trajectories diverged. Some harness versions achieved obvious gains, while other revisions brought performance regression.
Statistics from 64 official version updates show that single‑revision score fluctuation can reach ±4.75 points. In 27 out of 64 version transitions, score changes fell within this fluctuation band. This indicates that one‑shot benchmark results cannot reliably confirm whether a harness modification brings real improvement. Multiple repeated evaluations against identical harness versions are required for stable assessment.
Correlation comparison delivers another important conclusion. The correlation coefficient between creator self‑revision counts and downstream task scores is merely 0.13‑0.26. In contrast, revision counts guided by real runtime feedback achieve correlation up to 0.57. Blind modification without trajectory evidence barely helps harness quality. Only feedback‑oriented iterations based on actual execution traces produce meaningful optimization effects. When debugging agent harnesses, engineers should analyze runtime trajectories: inspect which messages the model received, which tools got invoked, and at which step failures emerged. Trajectory‑driven diagnosis reduces ineffective trial‑and‑error cycles over prompts and hyper‑parameters.
5. Execution Cost of Generated Harness
Different harness implementations bring distinct runtime overhead. Execution token volume varies greatly across harness versions, caused by context‑window management strategies, tool‑call retry logics, and checkpoint frequency. Under MLE‑bench testing, token consumption gap between different harness instances can reach nearly 19 times. Some harnesses deliver decent benchmark scores at the cost of massive token consumption.
When assessing harness quality, task success rate should never be the sole metric. Engineers must also track execution token usage, model invocation frequency, tool‑call counts and wall‑clock runtime. If a new harness version brings marginal performance gain while multiplying token overhead, cost‑benefit trade‑offs need careful consideration for production deployment. HarnessDev provides cost‑adjusted benchmark scores, combining task performance and computational expense for comprehensive comparison.
6. Key Lessons for Agent Harness Engineering
The HarnessDev research reveals multiple practical take‑aways for agent system development.
First, generating harness source‑code does not equal obtaining functional harness. Code existence cannot guarantee runtime activation. Developers must validate whether each control branch can actually be triggered in real‑task trajectories. Many state‑management and checkpoint components written by LLMs remain dead code.
Second, harness implementation carries strong executor coupling. A harness tuned for one LLM may perform poorly after switching to another model. Portability testing is essential for multi‑model agent systems.
Third, feedback‑driven iteration outperforms blind code modification. Runtime trajectories serve as the most valuable material for harness improvement. Single‑run benchmark results are susceptible to statistical noise, requiring repeated evaluation for reliable conclusions.
Fourth, cost metrics deserve equal attention alongside task accuracy. Some LLM‑generated harness implementations pursue higher success rates at the price of excessive token consumption, which creates heavy pressure for online service.
For future work, automatic harness generation shows promising potential, yet large gaps remain before full practical adoption. Researchers still need to enhance cross‑model portability, reduce unnecessary runtime overhead, and improve the hit rate of generated control logic. For practitioners building multi‑model agent platforms, unifying invocation layers can simplify maintenance complexity.
International access: https://4sapi.com
Domestic access: https://4sapi.cn
Top comments (1)
The finding that many LLM-written harness modules stay inactive during real execution matches what I've seen running a small multi-role agent fleet in production. I had fallback logic on paper — a secondary model supposed to take over when the primary hit rate limits — and it took weeks to notice it had never fired once, because the actual failure mode was silent output truncation rather than the exception the fallback was written for. The control code existed and was correct in isolation; it was just wired to a failure the runtime never produced. Diagnosing dead control paths in generated harness code seems like a hard problem in itself, since the code reads fine and the happy path passes.
The token overhead point also tracks with my experience. My API provider rate-limits on reserved output tokens, so a harness that plans with generous
max_tokensbudgets gets throttled into 429s that a leaner harness never sees. Same model, same tasks — the reliability difference came entirely from the control layer's discipline around output budgets. That suggests the Evolution phase should treat token budgeting as a first-class fitness dimension, not just task success.One question about Evolution: did you observe evolved harnesses overfitting to the feedback distribution? Iterating a harness against one task family made mine progressively better at that family and worse at recognizing when a task didn't belong in it — it kept forcing new tasks into its known loop instead of failing fast. Does the unseen-task evaluation in your benchmark suite control for that, or could the portability drop across executor models be partly that effect showing up at the model boundary?