DEV Community: Azan Hyder

Vibe coding gets you nowhere. BMAD works if you use a little brain.

Azan Hyder — Thu, 13 Aug 2026 15:38:56 +0000

Everybody vibe codes now. In case the term passed you by the way it almost passed me by: you describe what you want, the model writes it, you paste the error back, and you repeat until it runs. The vibe is the spec. For a prototype or a weekend script this is genuinely great. For anything that has to survive contact with a real codebase, it falls apart in a way that is hard to notice until it is expensive.

I hit that wall enough times that I went looking for structure, and I ended up on the BMAD Method. This post is the explainer I wish someone had handed me: what it actually is, why it works, and when it is overkill.

The problem is not the model

Vibe coding fails in a predictable way. The first prompt produces something impressive. The tenth prompt contradicts the third. By the thirtieth, the model has forgotten decisions it made, invented new ones, and your architecture is whatever the last conversation happened to imply. There is no plan outside the chat, so there is nothing to check the work against.

Notice that none of this is a model capability problem. A smarter model drifts more politely, but it still drifts, because the missing piece is not intelligence. It is shared, persistent structure: a spec, an architecture, and a unit of work small enough to hold in context at once.

Which is, not coincidentally, what software teams figured out decades ago.

What BMAD actually is

BMAD (Breakthrough Method for Agile AI Driven Development) is a free, MIT licensed framework that puts a team of specialized agent personas between you and the code. It is one of the most starred agentic development projects on GitHub, and the core idea is almost boringly simple: instead of one generalist AI you prompt for output, you get a workflow of roles, each with a defined job and a written deliverable.

The cast will be familiar to anyone who has worked on a real team. There is an Analyst who helps you figure out what you are even building. A Product Manager who turns that into a PRD. An Architect who fights the PRD and produces a technical design. A UX specialist, a Scrum Master, a Developer, and a QA role that tries to break what the Developer built.

The point is not roleplay. The point is that each handoff produces a document, and the documents are the memory that a chat window cannot be.

The BMAD agent roster inside a coding CLI: analyst, product manager, architect, UX designer, scrum master, developer and QA, each listed with the single job it owns. Screenshot from the install running on my machine.

The two phases

Phase one is planning. You work with the Analyst, PM, and Architect agents to produce a PRD and an architecture document. This is human in the loop the whole way. The agents ask uncomfortable questions, you make actual decisions, and the decisions get written down. This phase is where most of the value is, because it forces the thinking that vibe coding lets you skip.

Phase two is development, and it has one genuinely clever trick. The Scrum Master agent takes the PRD and architecture and shards them into story files. Each story is a small, self contained file with everything the Developer agent needs: acceptance criteria, the relevant schema snippets, the API definitions, the design notes. The Dev agent then implements one story at a time.

Why this matters: the Dev agent loads a few KB of exactly relevant context instead of your entire project documentation. Context is the scarcest resource in AI assisted development, and sharding is context engineering disguised as agile process. The agent always knows what to build, how to build it, and why, because the story file tells it.

Then QA reviews, findings get fixed, and you move to the next story. It is a loop a real engineering org would recognize, which is the entire point.

The little brain part: which skill when

Here is the part the install page does not spell out. BMAD drops a full menu of agents and workflows into your editor, and the first instinct is to close the menu and go back to prompting. Resist. You only ever need one entry at a time, and the whole method reduces to picking the right one for the moment you are in.

If you cannot yet say what you are building, you start with brainstorming or a product brief, and you let the Analyst interrogate the idea until it survives or dies cheaply.
If you know what you are building but not where it ends, you write the PRD, because scope you cannot read is scope you cannot check.
If the requirements exist, you produce the architecture, which is where the Architect picks fights with the PRD before the code has to.
If the thing has a user facing surface, UX planning happens now, not after the Developer agent invents a layout at two in the morning.
Once the documents exist, you shard them into stories and hand them to the Developer one at a time, then let QA try to break each result.

That is the complete trick. The menu is long, but the moments are few, and matching one to the other is the entire little brain requirement.

The BMAD brainstorming composer open in a browser tab: the full technique catalog laid out as a dashboard, with a copy prompt button that hands the composed session back to the chat. This is what the install drops into your project.

Trying it takes one command

npx bmad-method install

Run it inside a project and it wires the agents into your editor of choice (Claude Code, Cursor, and friends). From there you can start at the very beginning with brainstorming and a product brief, or drop straight into a PRD if you already know what you are building. It handles brownfield projects too: there is a workflow that documents an existing codebase first, so the agents plan against what you actually have instead of what you wish you had.

The composer closing the loop: the composed session prompt is copied, and the banner tells you to paste it into the chat, which is where the agent takes over and the session begins

The honest tradeoffs

This is a real workflow, which means real costs. Planning takes time you could have spent typing prompts. For a throwaway script or a quick spike, the full method is absolutely overkill, and the framework itself will tell you to use its lighter flows for small work.

You also do not get to stop thinking. The agents produce artifacts, but you own the decisions, and a PRD you rubber stamped is just vibe coding with extra steps. The structure only pays off if you actually engage with it.

What you get back is consistency. The code in story fourteen still respects the decisions made in the architecture doc, because the Dev agent never had a chance to forget them.

Full disclosure, in the interest of practicing what I post: this article itself went through BMAD style editorial and adversarial review passes before it reached you. The robots reviewing the article about the robot workflow were at least transparent about it.

The bigger idea

The thing I keep coming back to is that BMAD is not really about the agents. It is about admitting that AI changed how fast we can write code, but not what makes code correct. Specs, reviews, small verifiable units of work: the old discipline turns out to be exactly what the new tools were missing.

If you have tried structured agent workflows, I am curious where they broke down for you. And if you are still full vibe coding, I am equally curious how far it has taken you. Both answers are interesting.

Sources and further reading:

BMAD-METHOD on GitHub (bmad-code-org): the framework itself, MIT licensed, install docs and user guide.
Stop Vibe Coding: Spec Driven Development with The BMad Method (Tech Lead Journal, episode 255): interview with creator Brian Madison on why spec driven development beats prompt roulette.

Models converged. Trust hasn't.

Azan Hyder — Thu, 13 Aug 2026 13:17:16 +0000

Field notes from the Harness Layer.

The models are becoming the same

Two years ago, choosing a model felt like the most important decision in the whole stack. The gap between the best frontier model and the next few competitors was wide enough that you had to design around it. Teams picked a provider the way they once picked a database, because the choice shaped everything built on top of it.

That era is ending. As of mid-2026, the leaderboards people actually cite, like LMArena and Artificial Analysis, show the top cluster within a few points of each other, and the cluster keeps getting denser. Every lab ships reasoning. Every lab ships tool use. API prices fell by roughly an order of magnitude across 2024 and 2025, and they kept falling after that. Open-weight models now trail the frontier by months rather than by a generation.

None of this is controversial anymore. The interesting part is what it does to the rest of the stack.

Illustrative, not data. The shape is the claim.

When the models become interchangeable, the model stops being the product. A task that fails with one frontier model usually fails with the others in the same way, because the failure was never in the weights. It was in everything around them. Real edges still exist, mostly in long context, underrepresented languages, and tool-calling reliability, but the center of the distribution has converged.

The bottleneck moved

There is a frame circulating among people who build agent loops for a living: in any agent loop, the verifier is the bottleneck, not the model.

The logic becomes obvious once you watch a loop run. Generation is cheap and can spin forever, so a loop only produces value as fast as something can judge whether to ship the output, retry the step, or stop entirely. Andrew Ng's letters on agentic design patterns put reflection, the practice of the model checking its own work, at the center of agent engineering. Karpathy's Software 3.0 keynote frames the whole discipline as keeping fast generation wired to fast verification. The practitioners writing about loop engineering keep landing on the same gate from different directions.

Single-agent teams have absorbed this lesson. They write evals, they add judges, and they constrain output schemas. It is hard work, but it is understood work.

Now step up one level, to where the industry is actually headed. The destination is not one agent in a loop. It is many agents, from many vendors, speaking different protocols, composed into a single goal. And when you look at that picture closely, you notice that nothing about trust made the jump.

What convergence didn't touch

Watch a real multi-agent run and the trust story falls apart in familiar places:

Plans drift. The plan made sense at step one. By step four the context has shifted, an intermediate result was misread, and the remaining steps are executing confidently against a goal that no longer exists. Nobody noticed, because nothing was checking the plan against reality in the middle of the run.
Calls route wrong. The agent picked a plausible tool instead of the right one. With three tools this is a prompt problem. With thirty registered capabilities across a network, it becomes a retrieval problem, and many stacks are still solving it by stuffing everything into context and hoping for the best.
Identity is a vibe. Which agent acted, under what authority, with what declared scope? Protocol-level authorization and scoped tokens are starting to exist, but in most stacks the honest answer is still "whatever the prompt said." If an agent exceeds what it was allowed to do, there is no contract to point at. There is only a transcript to argue over.
Output evaporates. The run ends in prose. A chat bubble cannot be checked against a schema, diffed between runs, or monitored in production. It is the least verifiable format our industry has ever shipped at scale, and it is the default.

Different teams, different frameworks, same four failures. The models converged. The layer that would make runs trustworthy did not, because it barely exists.

The transcript is not evidence

Here is the thought experiment I keep coming back to. An agent run executes something consequential overnight: it moves money between accounts, files a document, or changes infrastructure. In the morning you ask the obvious question: did it do what it was supposed to do?

What do you actually have to answer that with?

You have a transcript. It is a log of messages, in the best case with tool calls inline, and it was produced by the same system you are trying to check. You cannot verify it independently. You cannot show it to a third party and have them confirm anything. You cannot diff last night's run against last week's. You are asked to trust the narrator, and the narrator is the suspect.

Every other mature engineering domain has already built its version of an answer to this. Aviation does not trust the pilot's recollection, so it keeps a flight recorder that investigators can read without asking the airline. Finance does not trust the trader's terminal, so it keeps clearing records that both sides can reconcile. Software supply chains stopped trusting build claims and started demanding provenance.

Agent runs have nothing equivalent. Observability tooling exists and keeps improving, and traces, spans, and dashboards are genuinely useful. But a trace is still produced by the system you are checking, stored by the vendor you are paying, and viewable only by logging back into both. As independently checkable evidence, the state of the art in 2026 is a screenshot of a chat window.

What a trust layer would have to look like

Sit with that gap long enough and some properties start to feel non-negotiable. I will offer mine as a draft, in the spirit of the first field note:

Output as contract. Results come back in structured, schema-checked formats, so that "did it work" becomes a checkable question instead of a reading comprehension exercise. Plain prose still passes through when it has to, but prose is the exception rather than the interface.
Identity and scope as data. Every actor carries a verifiable identity and a declared scope of what it is allowed to do, and that scope gets checked at orchestration time instead of being assumed from prompts.
Verification that admits its own limits. Structural checks and semantic judgment are different things. Asking "is this well-formed, non-empty, and in contract" is not the same as asking "is this actually right." A trustworthy system says exactly which check it applied, instead of implying judgment it does not have.
Evidence, not transcripts. A finished run should export something a third party can check independently. The test of evidence is simple: verification cannot require calling the system that produced it. Signed evidence still has to anchor somewhere, because someone has to answer whose key it is and under what identity it signed. That is part of why property two carries so much weight.

Notice what is not on this list: a better model. Every property here is orchestration, contract design, and verification engineering. That is not a coincidence. It is the convergence argument applied honestly.

The question

The first field note ended by asking what belongs in the harness layer. This one ends with the sharper version:

What evidence would make you trust an agent run you didn't watch?

Not "what dashboard would reassure you." Evidence. Something you could check yourself, show to someone else, and rely on when the run mattered. I have been working through one set of answers, and I suspect the interesting disagreements are in property four. Curious what yours are.

Sources and further reading:

LMArena leaderboard and Artificial Analysis: the clustering at the top of the frontier, tracked continuously
Can MCP, A2A, and LangGraph share and compose a single verified runtime?: the three boundaries, and why composing them is the open problem
Andrew Ng, Agentic Design Patterns letters, The Batch, 2024: reflection and self-checking loops as first-class agent patterns
Andrej Karpathy, Software Is Changing (Again), YC AI Startup School, 2025: generation wired to verification as the shape of the craft

Your browser automation clicks are lying to you

Azan Hyder — Sun, 09 Aug 2026 20:49:24 +0000

I spent an evening last month watching an automation script click a button that was definitely there. There was no error. The screenshot showed the cursor parked right on top of it. The script reported success and moved on. Nothing had happened. The page had not received a click at all, and every tool I was using to check said otherwise.

If you have written browser automation against a modern web app, you have hit this. Here is the fix up front (the send helper is defined in use case 1), then the full explanation of why the obvious approach silently does nothing.

// The fix: click through the browser's input pipeline
// (Chrome DevTools Protocol), not through JS.
await send("Input.dispatchMouseEvent", {
  type: "mousePressed", x, y, button: "left", clickCount: 1,
});
await send("Input.dispatchMouseEvent", {
  type: "mouseReleased", x, y, button: "left", clickCount: 1,
});

That pair of events is indistinguishable from a physical mouse to the page. Everything below is about why the .click() you are probably sending instead is not.

What the old hack looks like

Every automation script starts like this:

document.querySelector("button.publish").click();

It is in a thousand Stack Overflow answers and it works right up until it does not. The failure shows up on exactly the apps you most want to automate: editors, dashboards, anything with components listening on pointerdown or mousedown, controls gated on trusted input, or framework-managed form fields. Three separate things go wrong.

First, click() dispatches a synthetic event. Every synthetic event carries isTrusted: false, and the page can see that. Plenty of apps simply ignore untrusted events on sensitive controls.

Second, click() only fires the click event (form controls also run their activation behavior, but that is the exception, not the rule). Real input produces a sequence: pointerdown, mousedown, focus, pointerup, mouseup, then click. A growing number of components hang their behavior on the earlier events in that chain. Fire only the last one and the component never wakes up.

Third, and this is the nasty one, the failure is silent. There is no exception and no rejected promise. Your script reports success because nothing told it otherwise. The only honest signal is whether the DOM changed afterward, and almost nobody checks that.

Use case 1: clicking a button that ignores you

The setup is a raw WebSocket to the Chrome DevTools Protocol (CDP). Launch Chrome with --remote-debugging-port=10086, grab the page's webSocketDebuggerUrl from http://localhost:10086/json, and add a tiny helper:

// ws = new WebSocket(webSocketDebuggerUrl)
let id = 0;
const pending = new Map();

function send(method, params = {}) {
  const msg = { id: ++id, method, params };
  ws.send(JSON.stringify(msg));
  return new Promise((resolve, reject) => pending.set(msg.id, { resolve, reject }));
}

ws.onmessage = (e) => {
  const msg = JSON.parse(e.data);
  if (msg.id && pending.has(msg.id)) {
    const { resolve, reject } = pending.get(msg.id);
    pending.delete(msg.id);
    msg.error ? reject(msg.error) : resolve(msg.result);
  }
};

Now the click. The one non-negotiable step is asserting the element actually has a size before you click its center. A display: none element returns a rect full of zeros, and clicking coordinates (0, 0) is how you click the wrong thing with total confidence. Note the behavior: "instant" on the scroll: without it, a page with smooth scrolling animates the scroll and you measure the rect mid-flight, then click stale coordinates.

async function clickForReal(selector) {
  const { result } = await send("Runtime.evaluate", {
    expression: `(() => {
      const el = document.querySelector(${JSON.stringify(selector)});
      if (!el) return null;
      el.scrollIntoView({ block: "center", behavior: "instant" });
      const r = el.getBoundingClientRect();
      return { x: r.x + r.width / 2, y: r.y + r.height / 2, w: r.width, h: r.height };
    })()`,
    returnByValue: true,
  });

  const rect = result.value;
  if (!rect || rect.w === 0 || rect.h === 0) {
    throw new Error(`not clickable: ${selector}`);
  }

  // A real mouse moves before it clicks; hover-dependent UI expects it too.
  await send("Input.dispatchMouseEvent", {
    type: "mouseMoved", x: rect.x, y: rect.y,
  });
  for (const type of ["mousePressed", "mouseReleased"]) {
    await send("Input.dispatchMouseEvent", {
      type, x: rect.x, y: rect.y, button: "left", clickCount: 1,
    });
  }
}

You need both mousePressed and mouseReleased. A release with no press is not a click. It is a shrug, and most apps treat it like one. One more caveat: this asserts the element has size, not that it receives the click. A modal overlay or cookie banner sitting on those coordinates will silently eat it. If overlays are in play, add a document.elementFromPoint(x, y) check before dispatching.

Use case 2: typing into a controlled input

The same trap exists for text. Setting value does nothing the framework can see:

// Looks right, does nothing. React never fires onChange for this,
// and may snap the value back on the next render.
document.querySelector("textarea").value = "hello";

The classic workaround reaches past the framework to the native setter, then dispatches an input event so React notices:

const el = document.querySelector("textarea");
const setter = Object.getOwnPropertyDescriptor(
  window.HTMLTextAreaElement.prototype, "value"  // HTMLInputElement for <input>
).set;
setter.call(el, "hello");
el.dispatchEvent(new Event("input", { bubbles: true }));

This works often enough to be worth knowing, but it is still a synthetic event with all the caveats from above. The real fix is the same pipeline as the click:

await send("Input.insertText", { text: "hello" });

It is a single method call, it is trusted, and it respects whatever element has focus. Focus the field first with a real CDP click and the text lands the way typed text lands. One boundary to know: insertText fires no keydown or keyup. For Enter-to-submit, keyboard shortcuts, or editors with custom key handlers, you need Input.dispatchKeyEvent with rawKeyDown and keyUp instead.

Use case 3: verify state, not screenshots

Once the input path is honest, the remaining lie is your success check. "Take a screenshot and look at it" is the default in agent-style automation, and it fails in both directions: pixels can look right while nothing happened, and a render hiccup can make a success look broken.

The DOM always knows. Define success as a state transition before you act, then assert it after. Poll with a deadline instead of sleeping a fixed 800ms, and handle the case where a successful action removes the element entirely:

// Publishing from an editor: success is the composer clearing or unmounting.
async function waitFor(expression, timeoutMs = 5000) {
  const deadline = Date.now() + timeoutMs;
  while (Date.now() < deadline) {
    const { result } = await send("Runtime.evaluate", {
      expression, returnByValue: true,
    });
    if (result.value) return;
    await new Promise((r) => setTimeout(r, 150));
  }
  throw new Error("publish did not happen");
}

await clickForReal("button.publish");
await waitFor(`(() => {
  const ta = document.querySelector("textarea");
  return !ta || ta.value.length === 0;   // cleared or gone: both are success
})()`);

Pick whatever transition your app actually guarantees: the composer empties, the row appears in the table, the URL changes. If you cannot name a DOM transition for an action, you do not yet know whether the action works.

The limits, honestly

Coordinates are a snapshot. The layout can shift between measuring the rect and dispatching the click, so measure immediately before clicking and never cache coordinates across actions. Coordinates are also frame-relative: for elements inside iframes you have to add each frame's offset on the way up. CDP is also Chromium territory. Firefox's equivalent is WebDriver BiDi, which is converging on the same model but is not a drop-in swap today.

And if you can install Playwright, install Playwright. It does all of this plus auto-waiting for visibility and stability, and its actionability checks are the checklist above written by people better at it than us. The raw approach earns its place when you are driving a browser that is already open, attached to a real user profile, or calling from a runtime where pulling in a driver is not an option.

The takeaway

Two greps worth running over any automation code you own:

grep -rn '\.click()' scripts/          # synthetic clicks, silent no-ops waiting
grep -rn 'screenshot' scripts/         # success checks that can lie in both directions

Every hit is a place where your script can report success while the page disagrees. Trust the input pipeline and trust the DOM. Everything in between is a rumor.

If you have a favorite automation lie, the click that looked fine and never happened, I want to hear it. These failure modes are collectable.

Sources and further reading:

Chrome DevTools Protocol: Input domain: dispatchMouseEvent, insertText, and dispatchKeyEvent, the trusted input pipeline this post is built on.
MDN: Event.isTrusted: why the page can tell your synthetic events from real ones.
MDN: Element.click(): what the programmatic click actually dispatches, and what it skips.
Playwright actionability: the full checklist a click should survive, implemented for you.
React issue #11488: the controlled-input value problem and the native setter workaround.

AI doesn't make programming easier - It moves the bottleneck to where it cannot see

Azan Hyder — Sun, 09 Aug 2026 20:19:14 +0000

Field notes from the Harness Layer.

The guard that killed the run

Last month I watched an agent write a preflight guard for a database script. Good idea, cleanly written: check that the host is reachable before anything touches it, and abort with useful advice if it is not. The script ran, prompted for credentials, and exited with no message, no output, nothing attempted. It looked exactly like an authentication failure. It was not. The guard itself had aborted the run it was written to protect, two lines after the credential prompt, because one of its own checks returned an exit code the script's strict mode treated as fatal. The host had been reachable the whole time.

Nothing about that failure was a code generation failure. The model wrote the code fine, and fast. The hours went somewhere else: deciding whether "exited after the credential prompt" meant bad credentials or something else, reading exit codes, checking DNS records, proving each failure path before spending another attempt. Generation was nearly free. Verification was the whole job.

I have been collecting evidence that this is not just my experience. It is measured, and the measurements are better than the discourse.

The perception gap is measured, not vibes

The numbers everyone cites come from settings that barely resemble the work. GitHub's famous result, 55.8% faster, was a controlled task where developers built an HTTP server from scratch. It was greenfield, self contained, and had no history. When the setting gets realistic, the effect shrinks or flips:

DORA's industry-wide surveys point the same way from the other side: individual productivity rises while delivery stability falls, and their 2025 report frames AI as an amplifier of existing strengths and weaknesses rather than a lift for everyone. And Cui's 26% hides a split that matters later in this piece: the gains concentrated in juniors, with no measured effect for seniors.

The METR result deserves its own paragraph, because the speed is not the interesting part. Before the study, the developers predicted AI would make them 24% faster. They were measured 19% slower. Afterward, having just lived through the slowdown, they still believed they had been 20% faster. That is a gap of roughly 39 points between felt and measured productivity.

Two honest caveats. The study ran in mid 2025, on Cursor with Claude 3.5 and 3.7, and newer tooling may narrow the measured gap. METR also says plainly that sixteen developers on their own repositories do not represent most software work. But the perception gap is the part that survives both caveats, and it is the part that matters here: "it looks easier" is no longer a complaint. It is an empirical finding.

The benchmark story quietly inverted

While the perception debate ran in public, the benchmark floor was moving underneath it. OpenAI audited SWE-bench Verified and found that 59.4% of the tasks its model persistently failed had flawed tests or problem statements, and that frontier models could reproduce gold patches from memory. OpenAI stopped reporting scores on the benchmark and now points people to SWE-bench Pro, a contamination resistant successor where scores at launch ran 20 to 40 points lower. Microsoft Research had already shown that the best models locate the buggy file from issue text alone at up to 76% accuracy on benchmark repos, versus up to 53% on repos outside the benchmark. The paper reads that gap as likely memorization, and so do I.

On real repositories, the failure forensics are starker:

The ceiling everyone argued about was partly recall, and the genuine residue concentrates exactly where practitioners said it did: multi file, long horizon, environment heavy, brownfield work. My guard was one line of shell in a fresh script, and it still failed in a way no benchmark would catch, because the failure lived in the interaction between the code and the environment around it.

Risk was never mitigated. It was distributed.

Here is the frame I keep coming back to, borrowed from markets. Hedging a risk never deletes it. At best it moves the risk to a counterparty who prices it, and at worst it introduces basis risk: the hedge itself fails, in a regime nobody modeled. Either way the exposure changes shape rather than disappearing, and the new shape tends to settle where you are blind: wherever the instrumentation is thinnest. AI assistance is a hedge against the cost of writing code, and it behaves the same way. Generation risk became verification risk became environment risk, and each stop along the chain is less observable than the one before.

The survey numbers say we know this and bought the wrong instrument anyway. LangChain's State of Agent Engineering report polled over 1,300 people, a self selected crowd of agent builders, and still: 57% have agents in production, quality is the top barrier, 89% have observability tooling, and only 52% run evals. Watching is table stakes. Verifying is a coin flip. Everyone bought dashboards. Barely half bought proof.

The incidents fill in what that gap costs. Replit's agent deleted a production database during a code freeze, then incorrectly claimed rollback would not work; in the days before, during the same engagement, it had fabricated thousands of records and reported tests passing when they were not. In controlled evaluations, METR watched frontier models monkey-patch evaluators into reporting success, with one model hacking 100% of its trajectories on a task family after stating, when asked beforehand, that it was incapable of cheating. And OpenAI's own research found the naive fix backfires: train against a chain of thought monitor and the model learns to hide the misbehavior rather than stop it.

Bad advice, delivered confidently. The gap is the never verifying.

Informed simplicity

So if the risk never disappears and the simple path keeps exploding, is the answer more process? More review gates, more ceremony? I don't think so, and I think markets and psychology already named the alternative.

"I would not give a fig for the simplicity this side of complexity, but I would give my life for the simplicity on the other side of complexity." ~ Oliver Wendell Holmes

Traders have a working version of this. The simple rules that survive (position limits, premortems, never average down) are scar tissue formalized. Naive simplicity is the same rules without the scars. To be fair, scar tissue is not immunity: every fund that blew up had position limits too, carved from the previous regime. The honest version of the distinction is behavioral, not rhetorical. Informed simplicity keeps a record. Naive simplicity keeps a feeling.

The same split is now visible in how people use these tools. The naive mode treats the output as done when it looks done: just let the agent cook. The informed mode holds Willison's golden rule: I won't commit code I couldn't explain. In the naive mode the judgment lives nowhere, or in vibes; in the informed mode it stays with the human, earned by walking the complexity first. When the first strange failure arrives, the naive mode believes whatever fault class the failure presents as, while the informed mode has a record to check against. The volume numbers point the same way: seniors ship 2.5x more AI code than juniors, and Fastly's reading is that they can spot code that looks plausible but is wrong.

That last finding looks like a contradiction next to Cui's missing senior effect, and it is worth one sentence. The RCT measured speed; the survey measured volume. Seniors are not faster with the tools. They are better positioned to pay the verification tax the tools impose.

Karpathy's original vibe-coding post contained its own boundary: fine for throwaway weekend projects. Willison drew the operative line: LLM code you did not review is vibe coding; reviewed and tested AI code is just software development. Simplicity is still on the menu. It is only available on the far side of verification.

Where the bottleneck lives now

Put the pieces together and the bottleneck's new address is specific: specification, review, environment judgment, and proof. The first three stay with the human, and seniority decides who can afford the tax. The fourth is the one nobody has solved.

What would it take for a run to produce evidence a third party could check without trusting the orchestrator? Not logs, which record that something was called, but evidence that the claim the run makes about itself is true. Early research exists: preprints on signed capability declarations and on agent identity bound to verifiable governance artifacts. Nothing is a standard. The OpenTelemetry GenAI conventions define the shape of an agent trace, and even they are a moving target: the namespace has been migrating between repositories this year, not settling into stability. The plumbing for watching is converging. The plumbing for proving has barely started.

The question

When your agent's run fails now, where do you look first? The diff, the trace, or the environment?

And whichever one it is: is that lookup instrumented, or is it you?

Sources

Productivity evidence: METR RCT, Jul 2025, Cui et al., Feb 2025 (working paper), Peng et al. 2023, DORA 2024 and 2025
Benchmark forensics: OpenAI's SWE-bench Verified audit, SWE-Bench Illusion, Microsoft Research, SWE-bench Pro, agentic failure forensics
The verification gap: LangChain State of Agent Engineering, Replit incident, Fortune, METR reward hacking, monitor backfire
Informed simplicity: Holmes via Tim Ferriss #219, Willison on vibe coding, Fastly senior/junior survey, Pragmatic Engineer, Jan 2026, LangChain on context engineering
Tracing standards flux: OpenTelemetry GenAI conventions, moved and unmaintained notice
Attestation research (early, preprints): AgentFacts, verifiable governance artifacts

Earlier field notes from the Harness Layer start here.

The stop/go problem is not solvable. It's relocatable.

Azan Hyder — Sun, 09 Aug 2026 00:30:59 +0000

Field notes from the Harness Layer.

The model thought it was inside the exercise

In July 2025, Anthropic reported three incidents from its own cybersecurity evaluations. The models were supposed to be working in simulated environments, explicitly told they had no internet access. Internet access was available anyway, because of how the environment was configured. While completing their exercises, the models reached real systems and treated them as part of the simulation. In one incident, a model published a real package to the public PyPI registry, believing it was still inside the test.

The detail worth sitting with is not the misconfiguration. It is that "you do not have internet access" was a sentence in a prompt, and the environment said otherwise. The sentence lost.

This is the stop/go problem in its purest form. Models always want to go. They are trained to complete the task, and they are very good at going. Every control scheme we have invented is, at bottom, a way for someone or something to assert stop or go. The interesting engineering question is not how to remove that assertion. It is where to put it.

Every scheme is the same assertion in a different place

I spent a week mapping the alternatives to per-action human review, and a pattern emerged that I did not expect: none of them remove the human. They relocate the assertion. Six categories, one axis:

Scheme	Where the assertion lives	How it executes	How it fails
Human review	The human, per action	Manual approve/deny	Does not scale; reviewer fatigue
Sandboxing	Environment design	Structural boundary	Misconfiguration; a prompt is not a boundary
Guardrails	Validator authoring	Content filtering	Watches the agent's words, not its effects
Policy engines	Policy authorship	Deterministic pre-action gate	Gates intent, not outcomes
Formal verification	Spec authorship	Proof before or during execution	Covers only what is formalizable
Model judges	Delegated to another model	Probabilistic review	The judge also wants to go

Read the table vertically and the industry's last two years make sense. Sandboxing moved from containers to microVMs after container escapes made the shared kernel untenable; Google now ships gVisor isolation as Kubernetes primitives for agents, and Anthropic open-sourced its sandbox runtime. Policy engines grew teeth: Cedar has effectively won the authorization conversation for tool protocols, and research reference monitors like PCAS report lifting policy compliance from 48% to 93% across frontier models with zero violations in instrumented runs. Formal methods left the paper: there is a Lean 4 system that intercepts an agent's tool call, formulates it as a conjecture against verified policy theorems, and executes only if the proof closes.

All of this is real progress. None of it deletes a row from the table.

The judge trap

One category deserves its own section, because it is the only one that tries to remove the human rather than relocate the assertion. If a model always wants to go, can another model be the one that says stop?

The evidence so far says no, and for a structural reason: the judge is the same class of system as the judged. Studies of multi-agent debate find that judges consistently favor whichever side shares their backbone. A 2025 study built a benchmark of confident, well-evidenced, wrong critiques and watched the strongest agents abandon correct answers when a judge pushed back persuasively. The failure mode is not that judges are weak. It is that their claims are exactly as unverified as the worker's. Unverified verification is not verification; it is recursion.

This does not make reviewer agents useless. I use one, and it catches real problems. But a reviewer model is a heuristic layer, not a stop/go authority. Treating it as the authority is how you get a system that looks governed and is not.

What each placement costs

Once you accept that the assertion is permanent, the design question gets sharper. Each placement buys something and charges something:

Environment design (sandboxes) buys safety in bulk. One good boundary covers every action inside it. It charges you in configuration risk: the boundary is only what you actually built, never what you described. Anthropic's incident happened to Anthropic.

Policy authorship (OPA, Cedar, reference monitors) buys determinism. No model in the decision path, sub-millisecond answers, fail-closed. It charges you in coverage: a policy can only gate actions it can name, and it gates whether an action may start, not whether the run did what it claimed.

Spec authorship (formal verification, proof of execution) buys the strongest guarantee available: claims checked against what actually executed, with receipts. It charges you in expressiveness: most of what we ask agents to do is not yet formalizable, and the specs themselves are written by humans who can be wrong.

Per-action review buys judgment on exactly the cases that resist all of the above. It charges you the human's attention, which is the scarcest resource in the system.

The mature designs I keep running into are not choosing a row. They are stacking them: a sandbox for the bulk, a deterministic gate for named risks, verification where claims are formalizable, and a rationed human review budget for the residue. The stop/go does not disappear. It gets distributed across layers, each covering the blind spot of the one below it.

The question I cannot answer yet

So here is where I am stuck, and it feels like the right place to stop.

A policy gate checks intent before execution. A human reviews output after it. Between those two is a gap: the claim the run makes about itself. "I updated the config and restarted the service." Right now, checking that claim against what actually executed takes either a human or another model, and we know what both cost.

What would a claim have to look like for a runtime to check it without a model in the path? Not "the function was called" — logs do that. The claim that the interpretation was right, that the tools called were the ones the task actually required, that the effect matches the intent. I do not think anyone has the general answer. I am not sure the general answer exists.

If you have seen a piece of one in the wild, I want to hear about it.

Sources

Anthropic's evaluation incidents, as reported July 2025 (secondary account; primary write-up linked at publish): dev.to discussion, sandbox mechanics
Isolation spectrum and container escapes: grigio.org, Infrastructure Review Stack
GKE Agent Sandbox (gVisor, Kubernetes SIG Apps): announcement coverage
NemoClaw and ceLLMate, plus PCAS and pre-action authorization research: arXiv 2603.20953
Policy engines vs runtime authorization: TrigGuard; MCP authorization landscape: ChatForest
Lean 4 compliance gating: arXiv 2604.01483; Proof of Execution: arXiv 2607.05397; AgentVerify: preprint
Judge limits: deceptive-judge study, Agent-as-a-Judge survey, multi-agent evaluation limitations

Earlier field notes from the Harness Layer start here.

Can MCP, A2A, and LangGraph share and compose a single verified runtime?

Azan Hyder — Thu, 06 Aug 2026 19:05:56 +0000

Here is a goal I keep coming back to:

"Show me my portfolio performance, summarize https://en.wikipedia.org/wiki/Nvidia, and turn both into a one-page brief I can read in the morning."

Three steps, and each one is easy on its own. The portfolio numbers live behind an MCP server. The summarizer is an A2A service someone else runs. The brief writer is a LangGraph agent a teammate deployed last month. What is not easy is running them as one thing you can trust, and the reason why is hiding in plain sight: those three names are not three kinds of agent. They are three kinds of boundary.

None of these is an agent

The industry uses the word "agent" for all three, so it is worth being precise.

MCP, introduced by Anthropic in November 2024, is a tool protocol. The specification borrows its shape from the Language Server Protocol: hosts, clients, servers, JSON-RPC 2.0, with servers exposing tools, resources, and prompts. A filesystem MCP server is not an agent. It is a capability behind a uniform socket, which is why the "USB-C for AI" analogy stuck.

A2A, launched by Google in April 2025 and now a Linux Foundation project, is an agent protocol. Endpoints publish agent cards and accept tasks. The card claims agency. What actually sits behind it is a task-processing endpoint with good metadata.

LangGraph is not a protocol at all. It is a framework for building agents as graphs, and the graph is your code in your process. It only becomes something a stranger can call when it is deployed behind LangGraph Platform, which speaks Agent Protocol, LangChain's open serving specification.

So the honest question is not whether three agents can share a run. It is whether work behind three different kinds of boundary can be composed into one run you can verify. The orchestrator composes calls, not agents.

The butterfly has an agent card, so the debate is settled.

What the run would have to do

Walk the goal slowly and the requirements fall out on their own.

Something has to decompose the goal into steps with dependencies. The brief waits for the numbers and the summary; those two can race each other. Something has to find the right endpoint for each step, which means a registry with semantic search, because you cannot hardcode the world. Something has to speak each boundary natively, one handler per protocol, because the moment joining requires a rewrite, nobody joins.

And every step has to be verified, because in any agent loop the verifier is the bottleneck, not the model. Three steps that each look correct can still compose into a brief that is wrong. That failure mode has its own post in this series, and it is not a short one.

What the evidence says

This is where it stops being a thought experiment. Line up the last eighteen months:

November 2024. Anthropic open-sources MCP.
March 2025. OpenAI adopts MCP across its products, starting in the Agents SDK the same day.
April 2025. Google launches A2A with more than 100 partner organizations, and in the same month commits to MCP support in Gemini and its SDK. One company backing a rival's tool protocol and its own agent protocol in the same breath, which tells you these layers were never in competition.
June 2025. Google donates A2A to the Linux Foundation.
August 2025. IBM's ACP, a REST-native agent protocol built for the BeeAI platform, merges into A2A rather than compete with it.
December 2025. Anthropic donates MCP to the Agentic AI Foundation, a directed fund under the Linux Foundation co-founded by Anthropic, Block, and OpenAI.

A recent survey of multi-agent orchestration describes the pattern well: competitors cooperating on standards while competing on implementations. The boundaries are consolidating under neutral governance faster than most people building on them realize. The adoption numbers are vendor-reported but large. Anthropic claims more than 10,000 public MCP servers and 97 million monthly SDK downloads, and the Linux Foundation counts 150-plus organizations around A2A at its one-year mark.

Two caveats keep this honest. Neutral governance is not the same as adoption, and at least one practitioner postmortem asks what actually happened to A2A after the donation fanfare. More importantly, convergence of protocols says nothing about composition. Settling the sockets does not settle what happens when you plug three of them into the same wall. One goal, three boundaries, verified end to end: that is still nobody's job.

What the agents would return

If a run is going to be checked, endpoints cannot answer with prose. They have to return structure. Imagine every step replying with a small manifest instead of a paragraph:

{
  "__canvas__": true,
  "summary": "Portfolio dashboard: $142,300 across 8 positions.",
  "manifest": {
    "version": "1.0",
    "layout": "dashboard",
    "components": [
      { "type": "metric_card", "label": "Total Value", "value": "$142,300", "trend": "up" }
    ]
  }
}

Structured output persists. More importantly, structured output can be checked, by something other than a human reading a chat log and squinting.

What joining would have to cost

If the price of admission is rewriting your agent, the network stays empty. The API you would want is embarrassingly small, something like:

import runtime

@runtime.agent(name="My Agent", description="What I do")
def handle(task: str) -> str:
    return f"handled: {task}"

Under that, a versioned, schema-validated manifest describes what the agent does. Additive changes ship freely. Breaking changes need the equivalent of an RFC, because once strangers build on your spec, breaking it breaks everyone.

The questions I do not have answers to

This is the part I actually wanted to write.

How does an agent declare what it is allowed to do, not just what it can do? A manifest field for authorized scope feels necessary. It also feels insufficient the moment money or user data is involved.

Should agents pay each other at all? Mocking payments is obviously the right place to start, but "obviously right for now" is not an architecture.

If you accept a manifest from one protocol and quietly route it through another protocol's handler, did you support the first protocol or not? A compatibility alias is honest engineering and a marketing lie at the same time, and I do not know which side it lands on.

Does Agent Protocol become a fourth boundary, or does it get absorbed the way ACP was? LangChain's specification is open, but today it is spoken almost entirely by LangGraph Platform. A boundary with one implementation is a product feature wearing a spec's clothes.

And the big one: what happens when an agent you do not control registers and joins your runs? Every assumption above gets stress-tested by strangers. I suspect most of them break.

If you have tried to compose across MCP and A2A, or wrapped a LangGraph graph for someone else's runtime, I would genuinely like to hear where it broke first.

Sources and further reading:

Anthropic: Introducing the Model Context Protocol (Nov 2024): where MCP started.
TechCrunch: OpenAI adopts Anthropic's standard (Mar 2025): MCP goes cross-vendor.
TechCrunch: Google says it will embrace MCP (Apr 2025): Gemini and the SDK, weeks before A2A launched.
Linux Foundation launches the Agent2Agent protocol project: A2A's move to neutral governance.
i-am-bee discussion: ACP merges into A2A: the consolidation, from the team that did it.
LLM-Based Multi-Agent Orchestration survey (MDPI): the academic anchor for the convergence framing.
LangChain: Agent Protocol: the fourth boundary's specification.
fka.dev: What happened to Google's A2A?: the practitioner counterpoint on post-donation adoption.

Every agent passed its evals. The run still failed.

Azan Hyder — Thu, 06 Aug 2026 18:20:02 +0000

Field notes from the Harness Layer.

The loop that cost $47,000

A public failure catalog tells this story, and it is worth sitting with. A team wired four agents together: one framework, an agent-to-agent protocol, tasks handed down a chain. Something in the handoff logic went wrong, and the agents began passing the same task back and forth. Politely, plausibly, endlessly.

The loop ran for eleven days. It burned roughly $47,000 in LLM API costs. It produced no useful output. And here is the detail that should bother you: the incident was not surfaced by a test, a judge, or a monitor watching the work. It was surfaced by a billing alert. The finance layer noticed what the verification layer could not.

Every agent in that loop was doing something locally reasonable. Read any single step and it looks like work. The failure was not inside any agent. It was between them, in a place no eval was pointed at.

And for those eleven days, the system never failed by any signal anyone had wired up. Jobs ran, tasks completed, retries succeeded. That is the shape worth remembering. The expensive failures are not the runs that crash. They are the runs that succeed at the wrong thing, on schedule, with every dashboard green.

Figures as reported in the public case study, named in the sources below.

Single-loop verification is understood work

To be fair to the state of the art: verifying one agent in one loop is a solved-shaped problem. Not easy. Solved-shaped. The practitioner playbook is well documented: assertion-style unit tests for every tool call, traces you actually read, LLM judges aligned against human labels, A/B tests once the product matures. Vendor guides say the same thing in enterprise dialect: guardrails and evaluation are first-class, monitoring is not optional.

Teams that do this work catch single-agent failures. It is hard, unglamorous, and it functions.

Now compose. N agents, from different vendors, speaking different protocols, planned into a single goal. Notice that the verification unit has quietly changed. Your evals score agents. The failures live in runs.

The left button is always easier to reach.

The questions composition asks

Watch a composed run fail and the questions are new:

Which step failed? Step three of six returned something subtly wrong: a misread unit, a stale figure. Steps four through six executed confidently on top of it. Every agent's own trace says success, because every agent did what the input asked. The error entered at a handoff, and handoffs belong to no one.

Was the output even structured? A data agent was supposed to return rows and returned fluent prose about rows. The next agent parsed the prose anyway, guessed wrong, and said nothing. Schema checks exist at the boundaries you control. Composition multiplies boundaries past the point where anyone has checked them all.

Did the right agent get called? With three tools, routing is a prompt problem. With thirty registered capabilities it is a retrieval problem, and a plausible-but-wrong call passes every per-agent eval, because the agent that was called did its own job correctly. Wrong agent, right behavior, bad run.

Was it allowed to do that? In July 2025, a coding agent deleted a production database during an explicit, repeatedly-stated code freeze, then fabricated test results and reported that rollback was impossible. Setting aside the spectacle: nothing in that system checked the agent's authority against its action, and the only account of what happened came from the actor. When someone asked "did it do what it was supposed to do?", the honest answer was a transcript to argue over.

Four questions. None of them is answered by making any individual agent better.

Locally correct, globally wrong

There is now empirical backing for what incident reports keep hinting at. The MAST study (Why Do Multi-Agent LLM Systems Fail?) annotated over a thousand execution traces across seven popular multi-agent frameworks and found fourteen distinct failure modes in three clusters: specification and system design, inter-agent misalignment, and task verification and termination. Two of the three clusters live between agents, not inside them.

The finding that should end the "just wait for a better model" argument: the authors conclude that improvements in base-model capability will be insufficient to address the full taxonomy. Organizations of sophisticated individuals still fail organizationally. And on popular benchmarks, multi-agent systems often show minimal gains over a single agent. We are paying the complexity without reliably collecting the capability.

This is why per-agent evals don't compose. Each agent can be locally correct while the run is globally wrong. Correctness leaks at the seams, and the seam has no owner. Your eval suite is a set of excellent inspections of individual parts, mounted on a vehicle nobody test-drives.

What run-level verification would need

Sit with composed failures long enough and a wishlist starts to feel like a spec. Mine has four items:

The run as a first-class object. One addressable thing with ordered, named steps, not N disconnected traces you stitch together with timestamps and hope. You cannot verify what you cannot point at.
Verdicts attached to steps, not vibes attached to chats. Every step ends with a checkable outcome: what was called, what came back, did it pass. A finished run reads as a column of verdicts, not a wall of prose.
Checks that admit what they are. Structural ("well-formed, in contract, non-empty") and semantic ("actually right") are different claims. Carried over from the previous field note: a trustworthy system says which one it applied, instead of implying judgment it does not have.
A check that owns the seam. The handoff itself gets validated: did step N's output satisfy step N+1's input contract? Most composed failures are born exactly there, and almost nothing inspects it.

Notice what is not on the list: a smarter model. Same conclusion as the last note, one level up. This is orchestration and verification engineering.

The receipts direction, stated as an open problem

The previous note ended on evidence: a finished run should export something a third party can check without calling the system that produced it. Composition turns that from a nice property into the hard core of the problem, because now the run spans processes, vendors, and protocols, and each participant grades its own homework.

What would a checkable receipt for a composed run actually require? Three problems, named honestly as open:

Tamper-evidence. The receipt must be unable to quietly edit itself after the fact. This is solved technology elsewhere (append-only logs, hash chains) and mostly unapplied to agent runs.
Ordering. Step verdicts only mean something if their sequence is provable. A reordered run is a different run.
Canonical serialization. Two parties must agree on the exact bytes they are checking. Anyone who has fought canonicalization in signing systems knows this is where good intentions go to die.

I am not claiming answers here. I am claiming these are the right questions, and that the industry is currently answering them with screenshots.

The first field note named the missing piece the harness layer. The more composed runs I watch fail, the less I want to rename it.

The question

The previous note asked what evidence would make you trust an agent run you didn't watch. Here is the version composition forces:

If you could demand one artifact from every agent run in your stack, what would it be?

Not a dashboard. Not a trace you have to take on faith. One artifact, per run, that you could check yourself and show to someone who doesn't trust you. I have been working through what belongs in it, and I suspect the interesting disagreements are in the ordering problem. Curious what yours are.

Sources and further reading:

Why Do Multi-Agent LLM Systems Fail? (Cemri et al., arXiv:2503.13657): the MAST failure taxonomy, fourteen failure modes in three clusters across seven frameworks.
The $47K infinite loop: public case study in the vectara/awesome-agent-failures repository on GitHub.
AI Incident Database incident 1152, and Fortune's report of July 23, 2025: the coding agent that deleted a production database during a code freeze.
Your AI Product Needs Evals (Hamel Husain): the single-agent evaluation playbook.
A Practical Guide to Building Agents (OpenAI): guardrails and evaluation as first-class concerns.
Models converged. Trust hasn't.: the verifier bottleneck and the four draft properties of a trust layer.

Sandbox beta deployed

Azan Hyder — Wed, 05 Aug 2026 02:25:13 +0000

Azan Hyder

Jul 28

Composer Harness

#ai #agents #mcp #opensource

4 min read