Merged Isn't Delivered: Building CommitmentProof, a Two-Agent Switch Workflow

Aadi Harale — Sat, 07 Nov 2026 18:07:10 +0000

A pull request is merged. 27 of 27 checks are green. Support is ready to tell the customer "it's fixed." Can they?

The short version

CommitmentProof checks the proof behind a software promise before anyone tells a customer it is kept.
Two separate Codex agents work in one Switch-connected Discord room with a human. The Evidence Agent collects source-linked GitHub proof. The Coordinator Agent applies a deterministic completion policy and drafts the next step.
On a real merged PR, microsoft/vscode #331010, the policy returned NEEDS_EVIDENCE. The PR was merged and 27 of 27 checks passed, but nothing proved deployment or QA.
Instead of closing the promise, the Coordinator proposed an exact GitHub issue. A human replied approve, created issue #2 by hand, and the Coordinator verified it through GitHub's read-only API.
The workflow concluded with a human-approved GitHub follow-up, independently verified by the Coordinator Agent. Every decision and action remained traceable inside the Switch collaboration

Built for Flip the Switch by WeMakeDevs, Track 01: Best Business Workflow. Repo · Live dashboard · Demo video

Watch the demo: two agents and a human working in the real Switch room.

The problem: merged is not delivered

Support teams close customer promises on engineering status. A developer merges the fix, CI goes green, and someone writes "this is resolved" to the customer. That message is a claim about production. The evidence behind it is a claim about the repository.

The gap between the two is where trust breaks:

The merge may not have shipped to the customer's environment yet.
CI proves the tests passed, not that the customer's original scenario now works.
Nobody may have reproduced the bug and confirmed it is gone.

We wanted a workflow that asks one business question and refuses to guess: can we confidently tell this customer their promised fix has been delivered?

A real example

We ran the workflow on a real public pull request, microsoft/vscode #331010. We do not operate VS Code's production environment, which makes it a fair test: the code can only see what GitHub shows.

Evidence	What GitHub showed	Verdict
Pull request	Merged on 16 Aug 2026	Verified
CI checks	27 of 27 returned check runs succeeded (checks)	Verified
Deployment	No independent deployment record	Unverified
QA	No reproduction of the original scenario	Unverified

Two green rows look like success. The policy answer is NEEDS_EVIDENCE: do not tell the customer the fix is delivered until the missing proof exists. That is the system working, not failing.

The live dashboard on microsoft/vscode #331010: pull request and CI verified, deployment and QA missing, completion claim not permitted.

Why two agents and a human

We split the job so that no single participant both gathers the proof and decides what it means. The agent that collects evidence never declares a promise kept. The agent that judges the evidence never fetches it. A human makes every external decision.

Participant	Does	Never does
Human (`bhoomi_gulabani`)	Asks the question, reviews the exact proposal, replies `approve` or `reject`, performs the external action	Hands the final call to an agent
Evidence Agent (`commitmentproof-evidence-agent`)	Reads the PR and its check runs with read-only access, returns each finding with a source link, names what is missing	Infers success from missing evidence, closes tasks, messages customers
Coordinator Agent (`commitmentproof-coordinator`)	Receives the findings in the room, applies the completion policy, drafts an exact follow-up, asks for approval, verifies the result	Writes to GitHub or publishes anything without explicit human approval

We gave each agent a distinct responsibility inside our Switch-connected Discord room. The Evidence Agent focuses on collecting verifiable technical evidence, while the Coordinator evaluates those findings, prepares the next action, and involves the human before any external change.

This separation makes the workflow auditable: everyone in the room can see where the evidence came from, how the decision was reached, and who authorized the final action.

- name: "{team}-evidence"
  instructions: |
    Investigate the commitment with read-only source access.
    Return PR merged status, required test status, deployment verification, QA confirmation,
    and a link to each actual source. Never infer success from missing evidence.
    Send findings to @{team}-coordinator in the room. Do not close tasks or message customers.
- name: "{team}-coordinator"
  instructions: |
    Receive findings from the evidence agent. If any check fails or is unknown,
    keep the customer promise open and prepare an evidence-linked follow-up draft.
    Never publish customer communications or modify GitHub without explicit human approval.

The split is not decoration. Raw evidence stays separate from the decision about whether a promise is safe to close, and a person can see both in the same conversation.

The architecture

One Switch room, two agents, one human. The only write to GitHub is the human's.

Both agents only read GitHub. The single write in the whole workflow, issue #2, is a human action that comes after an explicit approve.

The dashboard's recorded Switch session: every step labelled as recorded, human-approved or manual, never as a live feed.

How we built it

Every piece of evidence carries its source

github_evidence.py is a read-only collector built on the GitHub REST API. It starts by assuming nothing. Every field begins as unknown, and deployment and QA stay unknown because GitHub cannot prove them:

unknown = {"value": None}
evidence: dict[str, Any] = {
    "pr_merged": dict(unknown),
    "tests_passed": dict(unknown),
    "deployment_verified": dict(unknown),
    "qa_confirmed": dict(unknown),
    "collection_errors": [],
}

It then reads the pull request, takes the head commit, and pages through every check run on that commit. A single failed, cancelled or timed-out run makes the test evidence negative. Anything queued, skipped or neutral leaves it unknown, never green:

statuses = [check.get("conclusion") for check in checks]
all_success = all(status == "success" for status in statuses)
any_failure = any(status in {"failure", "cancelled", "timed_out", "action_required", "startup_failure", "stale"} for status in statuses)
check_url = f"https://github.com/{encoded_repo}/commit/{head_sha}/checks"
value = False if any_failure else (True if all_success else None)
evidence["tests_passed"] = {"value": value, "source_url": check_url}

Every True comes with an https:// link a reviewer can click. That is what the agents paste into the room.

A policy that fails closed

commitment_proof.py is the completion policy. It is 58 lines of plain Python with no model inside it, so the same evidence always gets the same answer. Its docstring is the whole philosophy: "Unknown values fail closed. Review eligibility is NOT completion."

for name, description in (
    ("pr_merged", "pull request merged"),
    ("tests_passed", "required tests passed"),
    ("deployment_verified", "deployment verified"),
    ("qa_confirmed", "QA confirmation"),
):
    entry = evidence.get(name)
    if not isinstance(entry, dict):
        reasons.append(f"No evidence for {description}")
        continue
    value = entry.get("value")
    source = entry.get("source_url")
    if value is False:
        blocked = True
        reasons.append(f"Negative evidence: {description}")
    elif value is not True:
        reasons.append(f"Unresolved: {description}")
    elif not isinstance(source, str) or not source.startswith("https://"):
        reasons.append(f"Source link missing for {description}")

It can return three verdicts:

Verdict	When	Permitted action	Can claim complete?
`BLOCKED`	Any evidence is negative	`PREPARE_FOLLOWUP_DRAFT`	No
`NEEDS_EVIDENCE`	Anything is unknown or lacks a source link	`REQUEST_EVIDENCE`	No
`REVIEW_READY`	All four checks are proven with links	`REQUEST_HUMAN_APPROVAL`	No

The last column is the point. Even with perfect evidence, can_claim_complete stays false. The best the policy can do is hand the decision to a human.

The dashboard

The public dashboard is React, Vite and TypeScript on Vercel, with Python API functions for live, read-only investigations. It shows the four evidence cards, the verdict, and the recorded Switch session. It labels that session "Actual handoff – not a live event feed", because that is what it is.

The run, message by message

Everything below happened in #commitment-proof in our CommitmentProof Lab Discord server on 7 November 2026. The agents are real Switch participants with their own usernames, not characters in a dashboard.

Time (as shown in Discord)	Who	What happened
5:13 PM	Coordinator	Replies to the Evidence Agent's handoff with a follow-up task draft: "Current assessment: NEEDS_EVIDENCE / REQUEST_EVIDENCE; completion cannot be claimed."
8:52 PM	Coordinator	Posts "Proposed GitHub issue — bh00mi-11/CommitmentProof" with the exact title, body and both evidence links
8:54 PM	Human	Replies `approve`
8:54 PM	Coordinator	"Approval received for the exact proposed title and body. I have not created the issue."
8:57 PM	Human	Creates issue #2 by hand and posts the URL, asking the Coordinator to verify it
8:58 PM	Coordinator	"Verified the issue … via GitHub's read-only API. It exists and is open. The title matches exactly."

1. The Evidence Agent hands off

The Evidence Agent's handoff said the earlier check read showed 27 of 27 completed runs successful, that deployment and QA remained unknown, and that assess() returned NEEDS_EVIDENCE / REQUEST_EVIDENCE with can_claim_complete=false. Switch recorded the handoff as Evidence Agent message sw_f3edfcaadde246219504809024217a72 → Coordinator follow-up draft sw_62aebc7a04614e88bcd9d06065b6ec04.

The handoff in #commitment-proof: the evidence summary, then the Coordinator's follow-up draft with NEEDS_EVIDENCE.

2. The Coordinator proposes, it does not act

The Coordinator turned the gap into a concrete request. This is the exact proposal it posted:

Title: Verify deployment and QA for merged VS Code fix #331010

The fix in microsoft/vscode PR #331010 is merged, and all 27 completed CI checks succeeded. Deployment to the affected customer environment and QA reproduction of the original scenario are still unverified. Please attach deployment evidence identifying the deployed build/environment and record QA reproduction results before the customer is told the fix is delivered.

Reply APPROVE to approve this exact title and body for you to create manually, or REJECT to decline it. No issue has been created. The automated Discord approval backend is not deployed.

Notice the last two sentences. The agent states what it has not done and why.

3. A human decides

The Coordinator acknowledged that the human had approved the exact proposed title and body this exact title and body, and again said it had not created the issue. Switch recorded the decision as approval sw_353065a05331d2295d29b05961ea6c17.

The exact proposal, the human's "approve" at 8:54 PM, and the Coordinator acting only after it.

4. The human acts, the agent verifies

The Coordinator confirms it has not created the issue; the human creates issue #2 and asks for verification.

The human created issue #2 and asked the Coordinator to check it, adding: "Clearly state that issue creation was manual, not automated." The Coordinator read the issue back through GitHub's read-only API and reported:

Verified the issue at https://github.com/bh00mi-11/CommitmentProof/issues/2 via GitHub's read-only API. It exists and is open. The title matches exactly: "Verify deployment and QA for merged VS Code fix #331010". The body matches the approved proposal, including the unresolved deployment/QA request and both evidence links (the merged PR and 27/27 successful checks). Issue creation was performed manually by you after approval; it was not automated by CommitmentProof.

GitHub's preview of issue #2 and the Coordinator's read-only check.

The business task moved from a question to a traceable outcome: an open, evidence-linked follow-up that a person approved and an agent checked.

Why this matters to real teams

Engineering, QA, and customer support often have different definitions of "done."

CommitmentProof connects those perspectives before a customer-facing promise is closed. Instead of simply reporting that code was merged, the team receives an evidence-backed assessment, a clear explanation, and an actionable next step.

Switch makes this collaboration visible. Specialized agents investigate and coordinate while humans retain final authority.

This creates a practical foundation for reducing premature delivery claims, improving accountability, and connecting engineering progress with customer confidence.

The end state on the dashboard: human approved, issue created manually, verified by the Coordinator.

What the demonstration proves

Our working demonstration brings together real GitHub evidence, two collaborating Switch agents, an explicit human decision, and an independently verified GitHub follow-up.

The human manually created the approved issue, while the Coordinator confirmed the result. The public dashboard provides live GitHub investigations alongside a clearly labeled recorded Switch session.

The example uses a public VS Code pull request to demonstrate the process without claiming access to VS Code's production environment.

Together, these components demonstrate an evidence-backed business workflow with transparent decisions and meaningful human oversight.

Why it fits Track 01

What judges look for	Where it shows up
Agents working in a real chat with teammates	Two named Codex agents and a human in `#commitment-proof`, through Switch
A real business task reaching an outcome	"Can we tell the customer it's fixed?" ends in an open, evidence-linked GitHub issue
Working technical integration	GitHub REST collector, deterministic policy, read-only verification of issue #2
Distinct agent responsibilities	Evidence Agent gathers proof; Coordinator judges it and drafts the next step
Human oversight	Nothing external happens without an explicit `approve`, and the human performs the write

What we learned

"Unknown" needs its own state. Most status tools have two colours. The most useful thing our policy does is keep a third one, unknown, and refuse to round it up to green.
Let the model talk, let the code decide. The agents read, summarise and draft well. The verdict comes from 58 lines of deterministic Python, so a reviewer can predict it and test it.
Make agents say what they did not do. "I have not created the issue" turned out to be the most trust-building sentence in the whole conversation.
Separate roles make the chat readable. With one agent gathering and one judging, anyone scrolling the room can see where a claim came from and who decided what.

Where this goes

CommitmentProof is designed to extend across the complete software delivery lifecycle.

By incorporating deployment records and QA confirmation directly into the evidence workflow, teams could move from verifying engineering progress to validating customer outcomes.

The longer-term vision is a reusable accountability layer for engineering, QA, and support teams, powered by collaborative agents through Switch.

Try it

Open the live dashboard, switch to Live investigation, and enter microsoft/vscode and 331010. When we ran it, it returned the same four cards and the same verdict: merged, green, and still not delivered.

Where would you draw the line on what an agent may decide on its own? Tell us in the comments, and if the idea is useful, a star on the repo helps.

Prove it before you close it.

DEV Community: Aadi Harale