"Test Agent Harnesses as Replaceable Protocol Executors"

Executor A times out after committing, then Executor B retries from an old snapshot. Both report plausible success. A harness swap exposed that the system had an implementation convention, not a protocol invariant.

What the announcement establishes

GitHub’s July 27 article discusses the agent harness as an important execution layer. Harness replaceability is useful only when protocol behavior is specified; this simulator is an independent validation design, not a benchmark of GitHub products. Read the primary source.

For test track 9.1, the engineering claim here is narrower than the announcement: the surrounding workflow needs a contract that remains valid when metadata, transport, people, or executors change.

The artifact

# illustrative, unexecuted simulator sketch
class Exec:
 def __init__(self,name): self.name=name
 def run(self,job,store,fault=None):
  if job["base"] != store["head"]: return "STALE"
  if job["id"] in store["done"]: return "DUPLICATE"
  store["head"] += "+"+self.name; store["done"].add(job["id"])
  if fault == "crash_after_commit": raise RuntimeError("crash")
  return "COMMITTED"

The model is intentionally tiny. Add an acknowledgment ledger so a crash after commit can reconcile by job ID instead of applying twice.

For test track 9.2, this is a design fixture, not executed code. Pin language and dependency versions before turning it into a repository test, and replace example identities and timestamps with disposable values.

Success path

Both executors accept the same envelope, reject stale bases, and converge on one committed effect. Their logs differ, but externally visible states satisfy identical invariants.

For test track 9.3, a successful demonstration records inputs, policy or schema version, decision, and final identifier. It does not infer correctness from a confidence label, status badge, or fluent output.

Failure path and regression plan

Inject timeout before work, timeout after commit, duplicate delivery, stale snapshot, malformed result, and crash during acknowledgment. Any second effect or false terminal success fails conformance.

Generate event orders and assert: at most one effect per job ID; terminal results never regress; stale work cannot commit; uncertain acknowledgments are reconciled; cancellation cannot erase a completed effect.

For test track 9.4, the acceptance gate is binary: the negative fixture must produce no unauthorized or duplicate side effect, while the positive fixture must remain traceable to its initial evidence. Expected output should be documented before execution.

Cleanup and rollback

Canary the second executor on read-only fixtures, stop routing on the first invariant violation, drain leases, reconcile uncertain IDs, and retain the original executor until state equivalence is demonstrated.

For test track 9.5, cleanup must preserve enough sanitized evidence to distinguish cancellation, rejection, stale work, and successful completion. Never solve recovery by silently marking an uncertain operation successful.

Limitations

For test track 9.6, this article proposes a compact engineering exercise and reports no execution results. It does not evaluate service availability, security, accessibility conformance, productivity, or comparative quality. Product previews can change, and a local fixture cannot reproduce every hosted-system failure.

A practical development environment

For implementing an independent simulator, MonkeyCode is an open-source AGPL-3.0 AI development platform with an overseas hosted option. It includes a managed server-side cloud development environment, integrated models, task and requirement management, and build, test, and preview workflows. It is free to start. These statements do not mean the GitHub or OpenAI capability discussed above exists in MonkeyCode. Check the console for current quotas, models, regions, duration, and pricing before planning work. Open the campaign workspace.

Disclosure: This article promotes MonkeyCode using an official campaign link. I’m a MonkeyCode user, not affiliated with the project, and I receive no commission from this link.

AI assistance disclosure: This article was drafted with AI assistance and reviewed against the cited primary sources.