Coding agents can write a feature, run the test suite, and report success.

Alireza Bashiri — Mon, 12 Oct 2026 14:26:37 +0000

Then you open the application and find that the feature doesn't work.

I've been thinking about this problem while studying coding-agent harnesses and building software with agents.

The problem is often in how we verify the result.

A green test suite tells us that the code passed the checks we ran. It doesn't tell us whether the agent solved the user's problem, changed unrelated code, or introduced a failure that our tests missed.

I think verification needs to become a first-class part of the coding-agent harness.

Here's how I would build it.

The research shows three different failure modes

1. The agent changes code when it shouldn't

The FixedBench research examines tasks where the reported bug is already fixed.

The benchmark contains 200 human-verified tasks. The correct solution is to inspect the repository, confirm that the issue no longer exists, and leave the code unchanged.

Across five models and four harnesses, agents still made unwanted changes in 35–65% of these tasks.

That exposes a problem with how we define success.

Many coding-agent workflows assume that every ticket needs a patch.

But a successful task can also end with no code changes.

The harness should support both outcomes:

The agent finds a real defect, fixes it, and verifies the result.
The agent confirms that the requested behavior already works and provides evidence.

A ticket is a request to investigate or change behavior. It isn't proof that the repository needs an edit.

2. The agent passes tests but violates repository rules

The SWE-CC paper studies whether coding agents follow repository policies.

The researchers extracted 823 machine-checkable policies from 12 open-source repositories and evaluated 500 software tasks.

Agents violated 43.1% of applicable policies in the reported evaluation.

Nearly half of the violations occurred during execution rather than in the final patch.

This matters because final-state tests cannot detect every action an agent took.

For example, an agent might temporarily modify a protected file, run an unsafe command, or use a forbidden dependency. It could then remove the change before submitting the patch.

A final diff would miss some of that behavior.

The harness needs to enforce important rules while the agent works, not only when it finishes.

3. The agent solves the task but adds unnecessary complexity

Developers often report another problem.

They ask for a small change, and the agent adds new abstractions, dependencies, or unrelated features.

The code might compile. The tests might pass. But the team now owns code that nobody requested.

This is different from a correctness failure.

It is a scope failure.

I would track it separately because a successful test result cannot tell us whether the agent made the smallest reasonable change.

What a verification harness should check

I would divide verification into five parts.

Layer	What it checks
Static checks	Types, lint rules, formatting, build errors
Behavioral checks	Unit tests, integration tests, browser E2E
Runtime checks	Actual application behavior, errors, logs
Policy checks	Protected files, dependencies, permissions
Scope checks	Unrelated changes, unnecessary files, extra features

Each layer catches problems that the others can miss.

For example, a TypeScript build can pass while a button does nothing.

A browser test can pass while an agent modifies a protected configuration file.

A clean diff can still contain a logic error.

The harness should collect evidence from multiple sources before it accepts a result.

Build a basic verification pipeline

Let's start with a TypeScript web application.

The project has these commands:

{
  "scripts": {
    "typecheck": "tsc --noEmit",
    "lint": "eslint .",
    "test": "vitest run",
    "build": "vite build",
    "test:e2e": "playwright test"
  }

These checks provide useful evidence, but they don't cover every failure.

We can run them from one verification script.

Create scripts/verify.sh:

#!/usr/bin/env bash
set -Eeuo pipefail

mkdir -p artifacts/verification

run_check() {
  local name="$1"
  shift

  echo "Running: $name"

  if "$@" >"artifacts/verification/$name.log" 2>&1; then
    echo "PASS: $name"
  else
    echo "FAIL: $name"
    cat "artifacts/verification/$name.log"
    return 1
  fi
}

run_check typecheck npm run typecheck
run_check lint npm run lint
run_check unit-tests npm run test
run_check build npm run build
run_check e2e npm run test:e2e

echo "All configured checks passed."

This script stops when a check fails and saves the output for inspection.

It also gives the agent a consistent command to run after making changes.

bash scripts/verify.sh

For a real harness, I would extend this to produce structured results instead of relying only on text logs.

The harness should know which check failed, which files it affected, and what evidence it produced.

A green result needs a code version

Async tool execution introduces another problem.

Imagine this sequence:

The agent starts a test run.
The agent changes the code while the test runs.
The test finishes successfully.
The agent treats the result as proof that the latest code works.

The test might have started against an older workspace state.

Even worse, a test process may read some files before the edit and others afterward.

That means a green result may not represent one consistent version of the code.

I would solve this by running verification against an immutable workspace snapshot.

For example:

BASE_SHA=$(git rev-parse HEAD)

echo "Starting verification from $BASE_SHA"

A commit SHA identifies committed code, but it does not identify uncommitted changes. For agent workspaces, we also need to account for staged, unstaged, and untracked files.

The safer approach is to create an isolated snapshot before verification.

agent workspace
      |
      v
create immutable snapshot
      |
      v
run verification in snapshot
      |
      v
save results + snapshot ID
      |
      v
compare with current workspace

If the agent changes the workspace after the snapshot, the harness should mark the result as outdated for the current state.

The agent can still use the result for debugging.

It just cannot use it as final proof.

This becomes especially important when a harness runs tests, browser checks, and code reviews in parallel.

Verify actual application behavior

For web applications, I would include browser tests that exercise real user flows.

Consider a checkout form.

A unit test might confirm that the submit function returns the expected result.

But the real application can still fail because the button is disabled, the page never updates, or the browser sends an incorrect request.

A Playwright test can check the user-visible behavior.

import { test, expect } from "@playwright/test";

test("checkout completes successfully", async ({ page }) => {
  await page.goto("/checkout");

  await page.getByLabel("Email").fill("test@example.com");

  await page.getByRole("button", {
    name: "Complete purchase"
  }).click();

  await expect(
    page.getByText("Order confirmed")
  ).toBeVisible();
});

This is an example. A production test should also control the backend, payment service, test data, and expected network responses.

I would also configure Playwright to save traces and screenshots when a test fails.

import { defineConfig } from "@playwright/test";

export default defineConfig({
  use: {
    trace: "retain-on-failure",
    screenshot: "only-on-failure"
  }
});

Now the harness has evidence it can inspect.

Instead of telling the agent only that the test failed, the harness can provide the browser trace, screenshot, console errors, and relevant logs.

That makes recovery much easier.

Enforce repository policies outside the prompt

Most coding agents support instruction files such as AGENTS.md or CLAUDE.md.

These files are useful for explaining repository conventions.

But I would not use them as the only enforcement mechanism.

If a rule is deterministic, software should check it.

For example, suppose the agent must not modify files under infra/production/.

We can add a check against the base commit.

#!/usr/bin/env bash
set -Eeuo pipefail

BASE_REF="${BASE_REF:?Set BASE_REF to the task's base commit}"

if ! git diff --quiet "$BASE_REF" -- infra/production/; then
  echo "Policy violation: protected files changed"
  exit 1
fi

This checks the current tracked-file diff against the base reference.

It does not detect every possible violation. In particular, it does not inspect untracked files or prove that the agent never modified a protected file earlier.

For stronger enforcement, the harness should also:

Restrict filesystem access where possible.
Record tool operations.
Reject writes to protected paths.
Run the agent with scoped credentials.
Review changes before merging them.

The important distinction is between detecting a violation after the fact and preventing the operation.

A sandbox or filesystem permission boundary can prevent a forbidden write.

A prompt cannot guarantee that.

Let the agent return a verified no-op

FixedBench suggests another useful feature.

A coding agent should be able to finish without changing code.

I would define an explicit completion result.

{
  "status": "no_change_required",
  "reason": "The reported behavior already works.",
  "evidence": {
    "reproduction": "passed",
    "regression_tests": "passed",
    "code_changes": 0
  }
}

The harness should not trust this object by itself.

It must verify the evidence.

For example, it should confirm that the agent ran the relevant reproduction steps and that the final workspace contains no changes.

If the evidence is incomplete, the harness should return the task for further investigation.

This prevents a model from claiming success simply because it could not reproduce the issue.

A failed reproduction attempt and a verified no-op are different outcomes.

Handle failed verification without endless retries

Another common problem is the repair loop.

An agent runs a test, gets an error, changes code, and runs the same test again.

Sometimes it repeats this process without making progress.

I would track the history of failed attempts.

For example:

{
  "task_id": "checkout-142",
  "attempt": 4,
  "check": "checkout-e2e",
  "failure_signature": "confirmation-page-timeout",
  "previous_attempts": [
    "increased-timeout",
    "added-retry",
    "changed-selector"
  ]
}

Now the harness can detect repeated failures.

It can ask the agent to inspect the underlying cause instead of applying another similar fix.

It can also switch models, request a review, or stop the task when the agent reaches a configured budget.

I would measure:

Attempts per verified solution
Repeated failure signatures
Recovery rate after runtime errors
Time spent in repair loops
Cost of unsuccessful attempts

These metrics tell us more about production reliability than a single pass rate.

Measure cost per verified solution

A cheap model can become expensive if it fails repeatedly.

A more expensive model can save money if it completes the task with fewer attempts.

The same applies to harnesses.

A harness might use more tokens but improve verification and recovery.

To compare systems, I would start with:

Cost per verified solution = Total inference and tool cost / Independently verified successes

For example:

Harness	Total cost	Verified successes	Cost per success
A	$100	40	$2.50
B	$120	60	$2.00
C	$80	20	$4.00

These numbers are illustrative, not benchmark results.

Harness B costs more overall but produces verified solutions at a lower average cost.

I would also report latency and failure rates.

Cost alone does not capture everything that matters.

A system that finishes cheaply but takes several hours may not fit an interactive development workflow.

A practical verification contract

Before an agent starts a task, the harness should know what successful completion requires.

I would represent that as a verification contract.

For example:

task: checkout-142

allowed_paths:
  - src/checkout/**
  - tests/checkout/**

required_checks:
  - typecheck
  - lint
  - unit-tests
  - build
  - checkout-e2e

evidence:
  - test-results
  - browser-trace-on-failure
  - runtime-logs
  - final-diff

limits:
  max_repair_attempts: 5
  max_runtime_minutes: 30

completion:
  allow_verified_noop: true
  require_clean_verification_snapshot: true

This is a proposed contract format, not a standard that existing coding-agent tools all support.

The harness can use it to decide which checks to run, which files the agent may change, and when to stop.

The model can still decide how to solve the problem.

But it should not decide whether to skip required verification.

What I'd build next

If I were improving a coding-agent harness today, I would prioritize these features:

Immutable verification snapshots. Tie every test result to the exact code state.
Executable repository policies. Enforce rules during execution.
Browser and runtime evidence. Test real behavior, not only compilation.
Verified no-op completion. Let the agent prove that no change is necessary.
Failure-history tracking. Detect repeated attempts and unproductive repair loops.
Independent final review. Check scope, behavior, and evidence before accepting the result.

I would also test these features across multiple models and repositories.

A harness improvement that works on one benchmark may not help everywhere.

The goal is to find which controls improve real task completion without adding unnecessary cost.

Final thoughts

Coding agents have become good at producing code.

But producing code is only part of software engineering.

We still need to confirm that the code solves the requested problem, follows repository rules, and works in the actual runtime environment.

A green build is useful evidence.

It isn't proof that the task is complete.

I think the next major improvement in coding agents will come from better verification systems around the models.

That means stronger runtime checks, clearer completion rules, and better evidence for every change.

The agent can write the code.

The harness should verify what the code actually does.

References

I write about coding agents, harness engineering, and the systems behind autonomous software development at al3rez.txt.

DEV Community: Alireza Bashiri

Coding agents can write a feature, run the test suite, and report success.

The research shows three different failure modes

1. The agent changes code when it shouldn't

2. The agent passes tests but violates repository rules

3. The agent solves the task but adds unnecessary complexity

What a verification harness should check

Build a basic verification pipeline

A green result needs a code version

Verify actual application behavior

Enforce repository policies outside the prompt

Let the agent return a verified no-op

Handle failed verification without endless retries

Measure cost per verified solution

A practical verification contract

What I'd build next

Final thoughts

References