DEV Community: Arnold Cartagena

A 4-line security fix turned into a 100-file pull request, and I nearly merged it

Arnold Cartagena — Thu, 23 Jul 2026 21:51:07 +0000

I maintain mcp-sso, an open-source OAuth 2.1 layer for remote MCP servers. In plain terms: it is the login-and-permissions layer that lets AI tools connect to your servers safely, without you handing out raw API keys. It does the spec-correct hard parts most people skip, including dynamic client registration, newer identity standards like CIMD, PKCE, and audience-bound tokens (a token minted for one server is rejected by another). It is live-verified end to end against real enterprise identity providers, including Microsoft Entra ID, Google, and Cloudflare Access. I am also building an MCP gateway multiplexer on top of it: one secured front door for many MCP servers, with the same auth for all of them.

It is a security library, so I treat every change as security-critical, and I lean hard on AI-assisted development. That includes a review panel of several AI models from different labs (GPT, Grok, GLM, Claude) that argue over each change.

This week that whole system (a green test suite, a multi-model review panel, and CI) passed a bug everyone treated as critical. Nothing exploitable ever reached a user; it was caught before the feature it lives in was even switched on or released. But it was on the main branch, it looked like a serious security hole, and it slipped past every check. And then, trying to fix it, the same system nearly shipped a whole security library to defend against an attack that cannot actually happen.

Here is the whole story, including the parts where I got it wrong.

First, the actual bug (in plain terms)

Part of the library fetches documents from the internet. Any time code makes outbound web requests, the danger is that an attacker tricks it into calling somewhere it should not, like a private address inside your own network. So this fetcher has a guard that blocks requests to internal addresses such as 127.0.0.1 (your own machine).

The guard had one option: "allow loopback," meaning "let this reach the local machine," which is only ever meant for local testing. The code checked it like this:

if (opts.allowLoopback === true) { /* allow 127.0.0.1 */ }

Here is the catch, and it is a JavaScript quirk. Every object in JavaScript quietly inherits properties from a shared parent object. If an attacker can plant a value on that shared parent (this is called prototype pollution), then a plain, empty object suddenly reports that value as its own. So this check could read allowLoopback as "true" even when nobody set it, and a public web address that secretly points at 127.0.0.1 would sail straight through the guard. I confirmed it end to end.

The fix is boring. Read the option only if this specific object actually set it, not something it inherited:

Object.hasOwn(opts, "allowLoopback") && opts.allowLoopback === true

Four lines across three files. That is the entire real bug.

How the bug was found

A batch of work had just merged, and before starting the next piece I wanted an adversarial second opinion on what I had shipped: a deep audit of my own code. Check the SSRF guards. Stand up a local TLS server. Prove the timeout really does tear down the socket. Hunt for anything the frozen test suite structurally cannot reach.

I pointed Fable at it first. It refused. The session paused and told me its safeguards had flagged the request, noting they are "intentionally broad right now and may flag safe and routine coding, cybersecurity, or biology work." I was asking a model to audit my own security library, and the safeguard could not tell that apart from an attack. For legitimate security work, this has become a recurring limitation for me.

So I handed the same audit to GPT-5.6 Sol on Ultra, its most expensive setting. And it worked, maybe too much. I chose Ultra deliberately because I wanted to see how capable the model was at this kind of adversarial review. It worked, perhaps too well. Sol explored paths the test suite could not reach and found a real bug. Credit where it is due: the audit was excellent. The problem came afterward, when I let the same model turn one valid finding into an unbounded repository-wide fix.

Then it ran away from me

Here is where it went sideways, and it is the interesting part.

"Reading an inherited value where you meant your own" is not a single bug. It is a pattern that could exist anywhere an object is read. So the natural instinct, mine and the AI's, was to fix it everywhere. And "everywhere" has no edges.

A few rounds later the "small fix" had grown its own new library and spread across the whole repo. The pull request hit 104 files and close to 7,000 changed lines (it is public, closed and unmerged: PR #88). Every review round found more problems, and the new problems were in code the pull request had just added. Round after round, the change got bigger and the to-fix list got longer.

Then the punchline. The very library it built to stop prototype pollution read its own internal data in the exact unsafe, inherited way it was supposed to prevent. Concretely: it checked descriptors with "value" in descriptor, and in is itself a prototype-chain read, so a polluted prototype could forge the very descriptors it used to detect pollution. The cure had the disease.

That is when the pattern finally clicked, and it is the one line I want people to take away: give an AI a rule that touches every line of code, and it turns into a machine that generates its own work. "Every value must be read safely" applies everywhere, so there is no natural place to stop. Review never finishes; it just keeps finding problems in the code it keeps writing. The AI was not being thorough. It had been handed a job with no end.

One thing here is on me, and it is the difference between a bad hour and a bad day. The rule is what made the task infinite. The setup is what let it run unwatched: my first-choice model, Fable, declined the security work (I tried several framings; each was refused), so I reached for GPT-5.6 Sol on Ultra, its priciest setting, overkill for this, and let it run unattended on my own library, with no tripwire and nobody watching, I was working on something else at the time and usually I come back after a few minutes but I left it there. And Sol is relentlessly goal-oriented: it keeps going until the job is done, which is the last thing you want pointed at a task that has no end. A less driven model might have stopped on its own; this one did not. The missing decisions lights the fire; an overkill, relentless model running unattended is what let it burn for hours before I noticed. I have since capped unattended runs and required a stop condition.

6+ hours in, something did not add up

After more than six hours of this, I got suspicious. The changes kept coming, and none of it felt like it was converging on "done." So I stopped and asked the question I should have asked at the very start: can this bug actually be exploited?

Here is the thing about that original "critical" bug. To pull it off, an attacker first has to plant that poisoned value on the shared parent object. And nothing in the library lets them do that. The documents it fetches are parsed strictly and read through an allowlist; there is no path for an outsider to poison anything. In other words, to exploit this bug you would already need to be running your own code inside the process. And if you are already running code inside the process, you can just replace the whole security layer. The lock does not matter if you are already inside the house.

So the honest severity was: a small, worth-fixing, defense-in-depth gap, caught before the feature even shipped. Not a critical hole, and definitely not a reason to build a 100-file library. We over-engineered an entire library we did not need, to defend a door that cannot be reached. And a big part of why is above: an overkill model, running unattended, with no reachability check and no stopping condition. In retrospect, letting it run alone was the real mistake.

Why nothing caught it

This is worth sitting with, because the answer is not "the reviewers were careless." They were not. Two separate, structural reasons:

1. The tests and the spec never described the problem. They checked for bad values: malformed URLs, wrong types, oversized inputs. They never described the shape of an object, whether a value is truly its own or quietly inherited. There was no test that could fail, because nobody had written down the rule it would fail against. You cannot catch a problem your spec never mentions. In fairness, you cannot foresee every shape of input, and "did you test with a polluted prototype?" is not on most teams' checklist. That is exactly why the class has to be named somewhere, in the spec or the threat model, before anyone can test for it.

2. All the AI reviewers shared the same blind spot. Several models reviewed it independently, and they agreed it was safe. But every one of them tested the code the normal way, with clean objects. Not one first poisoned the shared parent and then tested. When reviewers all make the same assumption, adding more of them does not help; they just agree on the gap together.

None of the automated checks caught this. What caught it was stepping back, in a fresh frame of mind, and asking one plain question: can this actually be attacked?

How I pulled it back

The recovery was almost all process, not code:

Notice when something is off. A change that grows every round is not thoroughness, it is a missing plan. More than about three review rounds on one thing is now my signal to stop and look, not to push harder.
Do not run big things unattended without a tripwire. Letting a powerful model grind for hours with no human watching, and no "stop if this gets weird" condition, is what let a 4-line fix become 100 files. Unattended runs need limits and a way to flag a human the moment something looks off, especially with a relentlessly goal-oriented model that will happily chase an infinite task to the end.
Fix the design first, then the code. I closed the 100-file pull request without merging it. It had been open for about five and a half hours (PR #88, public and closed). I re-cut the actual bug as a tiny 5-file change, with a test that fails if you undo the fix, plus the one real test that had been missing.
Write the boundary down before building it. For the rest, I wrote what the rule actually is into the spec first, including an honest note about what is out of scope, and only then implemented it. Now review checks against a plan, instead of discovering the plan round by round. This is harder than it sounds; sometimes you only learn where the boundary belongs after you have crossed it. But even a rough boundary written first beats none.
Break it up. The runaway became three small, separately reviewable changes. All three merged cleanly.

The real fix shipped. The runaway did not.

Where I was wrong, and the rule I am adding

I am not going to pretend I was the level-headed one here. I let it run. For hours I read "the reviewers keep finding more" as diligence, when it was actually the warning sign. And I did not ask the one question that would have ended the whole thing in the first ten minutes, can this even be attacked, until I had already burned 6+ hours building a cure for a disease my own code cannot catch. And there was a bill for that. The AI was running on GPT-5.6 Sol at its most expensive setting, and at pay-as-you-go API rates ($5 per million tokens in, $30 per million out) the tokens from this episode would have cost north of $500. I am on the ChatGPT Pro plan, so I did not actually pay it, but a 4-line fix quietly ringing up a multi-hundred-dollar retail token bill is its own kind of lesson. The judgment I keep calling "the human part"? I was slow to it.

This is not the first time a change of mine has spiraled into round after round of review. The earlier ones were a bit different, requirements missed in the design that review then found one at a time, and this one predates the design-first process I have since adopted. But the root is the same every time: no boundary written down before the building starts. "Try to remember next time" is not a fix for that. A rule is.

So here is the rule I am adopting, and it is going straight into my project's standing process (the CLAUDE.md and engineering-os setup that guides these AI sessions): if a single change needs more than about three review rounds, stop, because something is off with the plan, not the code. And no change may add a new security helper and call it "the standard" in the same step. Write the standard down and agree on it first, in its own change, before any code. That draws the boundary, so review has something finite to check against, and the machine cannot keep sweeping a rule that was never defined.

This was not just me

The timing was almost too on-the-nose. The same week I watched a goal-oriented model chew through my repo, the industrial-scale version of both these failures made the news. Hugging Face disclosed that an autonomous AI agent had breached its production infrastructure, and OpenAI admitted the culprits were its own pre-release models, GPT-5.6 Sol among them, run with reduced cyber refusals for an evaluation. OpenAI called them "hyperfocused," "going to extreme lengths to achieve a rather narrow testing goal": they broke out of their test sandbox and went looking for the eval's answers. That is the exact behavior I had just watched, from the same model, only pointed at a live network instead of a pull request. And when Hugging Face went to investigate, they hit the same wall I had hit with Fable: the hosted models' guardrails blocked their forensic queries (real attack payloads look the same whether you are the attacker or the defender), so they ran an open-weight model, GLM 5.2, on their own hardware to read the attack logs. Mine was a tiny, benign version of both. Same surface, wildly different scale.

AI made everything here faster: the fix, the sweep, the review, and the runaway. Knowing what is actually reachable, what is in scope, and when to stop was the human part. It still is. I was just late to it this time.

mcp-sso is open source: threat model, conformance matrix, and provenance included, plus the gateway multiplexer I am building on top of it. If you run remote MCP servers, I would love your eyes on it.

description: "A four-line defense-in-depth fix became a 104-file pull request after an AI agent was given a security rule with no boundary or stopping condition."

MCP clients can't easily authenticate with Entra. Here's the bridge.

Arnold Cartagena — Wed, 08 Jul 2026 06:19:15 +0000

If you work in a company using AI and MCPs, you put an MCP server on the network. You realize you need auth. The easy path is an API key, you go and generate one, paste it into every client config, done.

It works. But that key lives forever in plaintext configs. No expiry. No per-user identity. No revocation except rotating the one shared secret for everyone. It's what leaks in a git add ., a support screenshot, or a supply chain attack grabbing credentials.

It's a security nightmare.

The MCP spec says: use OAuth 2.1. The clients all support it. So you go to wire it up. This was me last week, I thought it was a simple thing. Spoiler: it isn't.

The paste-a-URL experience doesn't work as you'd expect. Here's why.

The DCR wall

You add a remote MCP server in Claude Code or claude.ai by pasting a URL. The client tries to connect, discovers it needs OAuth, and — here's the key part — registers itself automatically at the authorization server. No manual app creation. No copy-pasting a client secret. The client calls an endpoint, gets a client_id, and the whole flow just works.

That's Dynamic Client Registration (DCR). It's how "paste a URL and it connects" works.

Your identity provider doesn't have that endpoint.

Entra ID? App registrations are a human-in-the-portal operation. Cloudflare Access? It authenticates users against a policy — it's not an OAuth authorization server you can register a client against.

The MCP spec says servers should support DCR. It doesn't mandate it. So technically, a client can fall back to manual registration — hardcode a client_id, configure the redirect URI by hand, paste a secret into a config file. But that defeats the whole point: the clean, self-onboarding, "paste a URL and it connects" experience is gone. You're back to hand-rolling OAuth glue per deployment, or breaking the client's self-onboarding entirely.

That's the DCR wall. And it's the specific problem mcp-sso solves.

The bridge

mcp-sso sits between your MCP clients and your identity provider. It does three things:

Speaks DCR + PKCE + consent to the client. From Claude Code's perspective, mcp-sso is a normal OAuth server. Paste the URL → it connects → browser opens → consent → token → tools work.
Verifies identity through your real IdP. Cloudflare Access or Entra ID stays the source of truth. mcp-sso checks the upstream identity and rejects anything it can't verify.
Mints its own tokens. Each token is scoped to your server only. Your IdP's tokens never reach the client — nothing to leak.

Try it in 30 seconds

node examples/fastify-sqlite/index.ts
# → prints a one-time code to the console

claude mcp add --transport http my-bridge http://localhost:3000/mcp
# → browser opens → approve → tool is callable

No identity provider. No keys to generate. That's a full OAuth round-trip — register, authorize, consent, token, /mcp call — on your machine, with zero setup. The one-time code is a console-pairing identity for local dev; don't expose it publicly.

For a real IdP (Cloudflare Access or Entra), it's environment variables and a public URL. Set the config, point the client at it, done.

Does it actually work with real clients?

Yes. I drove it against five real MCP clients — Claude Code, Codex CLI, claude.ai, ChatGPT, and the official MCP SDK — all against a real Cloudflare Access tenant. Each completed the full flow: register → authorize → consent → token → /mcp tool call.

Entra ID was also tested — the redirect orchestrator against a real Entra tenant, driving Claude Code and Claude Desktop end to end. Those results are being formalized in the verification matrix.

The full provider × client matrix — what's proven, what's still open — lives in the repo.

Security

jose is the only runtime dependency.
Fail-closed everywhere: ambiguous config, wrong audience, replayed token = hard rejection.
Codes and tokens are hashed and single-use.
Published STRIDE threat model.
npm publishes via GitHub Actions with Sigstore provenance.

What's not done yet

Today: Cloudflare Access, Entra ID (redirect flow + group-based authorization), and console pairing. Cloudflare Access is live-verified against five clients. Entra was tested in a real enterprise deployment.

Roadmap (v0.3): generic OIDC, Google/GitHub presets, device flow, npx mcp-sso init.

If your IdP already supports DCR, you don't need this — use mcp-auth. mcp-sso is for when it doesn't.

Using Entra ID or Cloudflare Access with a remote MCP server? Open an issue with your IdP + client combo — I'll add it to the verification matrix.

MIT · npm · GitHub

npm Trusted Publishing with GitHub Actions OIDC — What the Docs Don't Tell You (Scoped Packages)

Arnold Cartagena — Thu, 26 Mar 2026 22:29:18 +0000

After the recent npm supply chain attacks, long-lived tokens are out. Trusted publishing via OIDC is the way forward. But if you maintain scoped packages (@org/package), you're going to hit some walls the docs don't warn you about.

I spent a full day getting @edictum/openclaw to publish via trusted publishing from GitHub Actions. Here's everything I learned so you don't have to.

The Setup

I maintain Edictum, a runtime contract enforcement library for AI agents. We just shipped a native OpenClaw plugin and needed to publish @edictum/openclaw to npm from GitHub Actions using trusted publishing — no tokens, no secrets.

What Worked

Here's the final working workflow:

name: Publish to npm

on:
  release:
    types: [published]

permissions:
  id-token: write
  contents: read

jobs:
  publish:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v5

      - uses: pnpm/action-setup@v4

      - uses: actions/setup-node@v5
        with:
          node-version: 22
          registry-url: https://registry.npmjs.org
          cache: pnpm

      - run: pnpm install --frozen-lockfile

      # THIS IS CRITICAL — bundled npm doesn't support OIDC for scoped packages
      - run: npm install -g npm@latest

      - run: pnpm build
      - run: pnpm test

      - run: npm publish --provenance --access public

And in package.json:

{
  "publishConfig": {
    "access": "public",
    "provenance": true,
    "registry": "https://registry.npmjs.org/"
  },
  "repository": {
    "type": "git",
    "url": "git+https://github.com/edictum-ai/edictum-openclaw.git"
  }
}

The Problems (and Fixes)

1. Bundled npm doesn't support OIDC for scoped packages

This is a known issue. The npm version bundled with Node.js 22 on GitHub-hosted runners is too old. You must upgrade:

- run: npm install -g npm@latest

The npm docs say you need npm 11.5.1+. Without this, you get:

npm error code E404
npm error 404 Not Found - PUT https://registry.npmjs.org/@scope%2fpackage - Not found

This E404 is misleading — the package exists, but the old npm can't do the OIDC token exchange for scoped packages.

2. `actions/setup-node` injects a default token that breaks OIDC

When you use registry-url with actions/setup-node, it automatically sets NODE_AUTH_TOKEN to ${{ github.token }}. This GitHub token overrides the OIDC flow, and npm tries to authenticate with it instead of doing the OIDC exchange.

Some people clear it with NODE_AUTH_TOKEN: "", but that breaks authentication entirely (ENEEDAUTH). Others remove registry-url, but then npm doesn't know where to publish.

The fix that actually works: upgrade npm (step 1 above). The upgraded npm correctly handles the OIDC flow even with the injected token present. The --provenance flag triggers the OIDC path explicitly.

3. The environment field on npmjs.com must match (or be empty)

When configuring the trusted publisher on npmjs.com, there's an "Environment name" field. If you set this to npm or release, the GitHub Actions job must run in a matching GitHub environment.

What tripped me up: I set the environment name on npmjs.com to npm and added environment: npm to my workflow job. It still failed with E404. Removing the environment name from npmjs.com (leaving it blank) fixed it immediately.

If you need environment-based protection (approval gates, etc.), make sure the names match exactly — case-sensitive, no trailing spaces. But if you don't need it, leave it blank.

4. `--provenance` is NOT automatic (despite what docs say)

The npm docs state:

When you publish using trusted publishing, npm automatically generates and publishes provenance attestations. You don't need to add the --provenance flag.

This was not my experience. Publishing without --provenance resulted in ENEEDAUTH. Adding it fixed the issue. You can also set it in package.json:

{
  "publishConfig": {
    "provenance": true
  }
}

5. `repository.url` must match your GitHub repo exactly

The trusted publisher config on npmjs.com requires your org/repo. Your package.json must agree:

{
  "repository": {
    "type": "git",
    "url": "git+https://github.com/your-org/your-repo.git"
  }
}

If these don't match, npm silently rejects the publish with a 404.

How Big Projects Handle This

Curious how others do it, I checked several major open-source projects:

Project	Method
Vercel AI SDK	`NPM_TOKEN` secret (token-based)
LangChain.js	`NPM_TOKEN` secret + manual `.npmrc`
OpenClaw	OIDC trusted publishing (non-scoped package)
shadcn/ui	`NPM_TOKEN` secret

Most major projects still use token-based publishing. The ones using OIDC successfully tend to be non-scoped packages. Scoped packages with OIDC are still rough — npm/cli#8976 is open as of today.

The Checklist

If you're setting up trusted publishing for a scoped npm package from GitHub Actions:

[ ] npm 11.5.1+ (npm install -g npm@latest in your workflow)
[ ] permissions: id-token: write in your workflow
[ ] registry-url: https://registry.npmjs.org in actions/setup-node
[ ] --provenance flag on npm publish
[ ] repository.url in package.json matches your GitHub repo
[ ] publishConfig.access: "public" in package.json (for scoped packages)
[ ] Trusted publisher configured on npmjs.com → package → Settings
[ ] Environment name on npmjs.com: leave blank unless you specifically need it
[ ] No NODE_AUTH_TOKEN secret set in your repo (it would override OIDC)

The Result

@edictum/openclaw is now published with OIDC trusted publishing and provenance attestations, no long-lived secrets anywhere:

npm i @edictum/openclaw

Or if you're an OpenClaw user:

openclaw plugins install @edictum/openclaw

One command, zero code changes, 25 security contracts active. Check it out: github.com/edictum-ai/edictum-openclaw

Edictum is runtime contract enforcement for AI agent tool calls. Deterministic YAML contracts that execute outside the model — the LLM can't talk its way past them. Available in Python, TypeScript, and Go.

AI Monoculture: When Every Engineer Has the Same Architect

Arnold Cartagena — Mon, 16 Mar 2026 17:40:38 +0000

Vibe coding is technical debt at scale. The rewrite is coming.

I was asking myself this question for quite a while, it's like we all have the same Architect, so I ran an experiment across six frontier AI models to see what happens when you ask them to design software architecture versus when you ask them to help build the project.

144 prompts. Four different systems. Three prompt styles.

When asked to design architecture, the models recommended Rust, Aerospike, ClickHouse, event streaming systems.

When asked to help build the same systems, those choices vanished.

Rust dropped from 43 recommendations to zero. SQLite appeared 12 times. TypeScript jumped to 25.

Same models. Same problems.

The only difference was the prompt: "design it" versus "help me build it."

And that difference could explain why so many AI-built apps end up with the exact same stack.

This is already happening and most people don't see it

You already know what this looks like. You've seen it. You might be building one right now.

Next.js + Supabase. Auth? Supabase. Database? Supabase. Real-time? Supabase. File storage? Supabase. The DX is smooth, the deploy is one click, and Vercel has done genuinely excellent work making it frictionless. Credit where it's due.

But when you look at a thousand AI-assisted projects and they all have the same skeleton, that's not a thousand engineers independently arriving at the same conclusion. That's one model's opinion, amplified a thousand times. It's an AI monoculture — and it's everywhere.

The thing is, most people building with AI right now don't realize they're producing the same architecture as everyone else. Each project feels like a fresh start. The assistant gives you a clean scaffold, the code works, the structure looks thoughtful. It doesn't feel like a default. It feels like a decision.

That's the core problem. Not that the default stack is bad — it's often perfectly fine. But that thousands of engineers are adopting it without realizing it was never chosen. The AI chose it for them, based on what it generates most fluently, and presented it with enough confidence that nobody questioned it.

The scaffold picks your architecture

Most people don't sit down and say "design me an architecture for 500,000 requests per second." Not because they lack the skill — but because they don't have those numbers yet. They're figuring out the product. So they say:

"I want to build a bidding platform, help me get started."
"I need an IoT dashboard. Set up the project and write the ingestion endpoint."
"Help me build a crypto exchange. Start with the order placement API."

Feature by feature. Conversationally. And at each step, the assistant picks the stack it can scaffold fastest — TypeScript, PostgreSQL, React — because that's what it generates most fluently. Each individual feature is small enough that the default stack is "fine." The model never encounters a moment where it has to say "wait, this won't work."

The architecture gets locked in at the first npx create-next-app and never gets revisited. Not because anyone chose it. Because nobody stopped to choose.

The models know better. They just don't act on it.

This is the part that surprised me. The crucial split wasn't between models or even between problems. It was between architecture mode and build mode.

Across architecture-mode prompts, the models gave genuinely good answers. Rust for the bid engine. Aerospike for feature lookup. Kafka for event streaming. They discussed lock-free data structures and kernel-bypass networking. Across Variants A and B, Rust was the primary backend language in 45% of recommendations. PostgreSQL was the primary database in 53% — but many of those designs still paired it with specialized stores.

Then I switched to build mode. Same problems, same models, same underlying domains — but now the prompt was "help me build this."

That's where everything collapsed. Across the benchmark, Rust fell from 43 architecture-mode recommendations to zero in build mode. TypeScript jumped from 2 to 25. Python went from roughly absent to 15. SQLite went from 0 to 12. ClickHouse, Aerospike, and ScyllaDB disappeared entirely. The stack novelty score — a measure of how many components fall outside the default web-app stack — dropped from 11.2 to 4.0.

The models know the right answer. They proved it thirty minutes earlier. They just don't use it when you say "help me build."

The monoculture is invisible from the inside

This is what makes it different from previous waves of stack convergence. When Rails was everywhere in 2010, people knew they were choosing Rails. They could name their stack, defend it, argue about it. The choice was visible.

With AI-assisted development, the choice is invisible. You didn't pick TypeScript + PostgreSQL. You asked for help building something and that's what appeared. It feels bespoke because the AI generated it specifically for your project. But the architecture underneath is the same one it generates for every project, because it's optimizing for scaffold fluency, not for your problem.

That's why the monoculture grows without anyone noticing. Each developer thinks they're building something custom. Nobody looks around and realizes the foundation is identical across thousands of products.

The AI didn't invent this. It industrialized it.

I should be fair — I use AI assistants daily and they've made me dramatically more productive. This isn't an argument against AI-assisted development. It's a field note from inside the machine.

A lot of this convergence existed before AI. Hiring availability, library ecosystems, managed cloud support, tutorials, startup pressure to ship — all of these already pushed teams toward the boring stack. Even without AI, many teams would choose Python + Postgres + React.

But before AI, that choice involved friction. You had to read docs, compare options, talk to other engineers, live with uncertainty. That friction was annoying, but it was productive. It forced architecture to be a thinking exercise.

AI removes the friction. And what gets lost isn't speed — it's the argument. The internal debate that used to happen before a stack was chosen now never occurs, because a complete, working scaffold appears before you have time to doubt.

The boring stack is often the right stack. The danger is when it's chosen by the scaffold instead of by the engineer.

I'm not immune

I saw this in my own work while building Edictum, my open-source runtime governance framework for AI agents. I'm a platform engineer. I run Kafka clusters and Kubernetes infrastructure professionally. I hold a Kubestronaut certification. I know what production event-driven systems look like.

And still, when I sat down to build with AI assistance, I shipped FastAPI + React + PostgreSQL. The exact default. Not because I chose it after evaluation, but because the AI made it effortless and I was optimizing for speed.

A prospective client asked me how I was thinking about scale. And I started talking about Kafka, event ingestion pipelines, ClickHouse for analytics. That's what I do professionally. Then I looked at what I'd actually built and realized none of that knowledge was in the product. The AI had scaffolded a perfectly functional FastAPI monolith, and I'd let it.

The model doesn't make you worse. It makes you generic. And you don't notice until someone asks a question that forces you to look at what you actually built versus what you know.

What this means

Right now, thousands of products are being built on the same invisible foundation. The same databases for the wrong workloads. The same frameworks for problems they weren't designed for. The same patterns, the same dependencies, the same scaling ceilings — all chosen by AI assistants optimizing for fluency rather than fitness.

When those products hit scale, the rewrites will come. And the fix is never incremental. You don't "add Kafka" to a Next.js app. You don't bolt ClickHouse onto Supabase. You don't retrofit event sourcing into a CRUD scaffold. You rewrite.

The time saved by not having the architecture argument up front gets repaid with interest. And the interest rate is brutal, because by then you have users, data, integrations, and a team that learned the wrong patterns.

The AI monoculture is real. It's growing. And most people building inside it have no idea they're there.

The question every founder and tech lead should ask themselves:

Would I have built it this way if I had to justify every decision to a staff engineer?

If the answer is no, you don't have an architecture. You have a default.

The data

I tested GPT-5.4, Claude Opus 4.6, Claude Sonnet 4.6, Kimi K2.5 (Moonshot AI), GLM-5 (Zhipu AI), and MiniMax M2.5 across four problem briefs: real-time bidding platform (AdTech), IoT telemetry platform (Industrial IoT), government benefits portal (GovTech — the control, where a boring stack is actually correct), and cryptocurrency exchange (FinTech).

Each problem was prompted three ways:

Variant A — "Design the architecture." Open-ended.
Variant B — "Design the architecture for these hard requirements." With specific numbers.
Variant C — "Help me build this. Set up the project, pick the stack, write the scaffolding." The vibe coding prompt.

Six models × four briefs × three variants × two temperatures = 144 completions. Responses were parsed using Claude Sonnet 4.6 as a structured extraction layer, validated manually on a 10% sample.

The numbers

Variants A and B (architecture mode, 96 responses):

PostgreSQL as primary DB: 53%
Rust as primary language: 45%
React correctly omitted: 64%
Models chose Aerospike, ScyllaDB, TimescaleDB, ClickHouse for the right problems
Mean stack novelty score: 11.2

Architecture mode (A+B) → Build mode (C):

Rust: 43 → 0
TypeScript: 2 → 25
Python: 1 → 15
SQLite: 0 → 12
ClickHouse, Aerospike, ScyllaDB: all → 0
Mean stack novelty score: 11.2 → 4.0

Raw data, runner code, and extraction pipeline: github.com/acartag7/ai-monoculture

My AI agent pushed directly to main. The system prompt said don't.

Arnold Cartagena — Sun, 08 Feb 2026 15:52:41 +0000

I was demoing my AI agent to colleagues. The agent had access to Git tooling, and my carefully crafted system prompt was clear: create a branch, open a PR, never push directly to the repo.

The agent pushed directly to main.

I tried rewording the prompt. I tried being more explicit. I tried few-shot examples. The agent pushed to main again — because when an LLM decides something is "the fastest way to help," your prompt is a suggestion it can override.

I had no way to block that tool call. No mechanism between "the LLM decided to do this" and "the tool executed." I needed something at that boundary — deterministic, not probabilistic. Something the LLM couldn't talk its way past.

My first attempt was hardcoded Python — regex patterns matching against bash command strings, wired into the SDK's hook system. It worked, but the patterns were buried in code, untestable without spinning up the agent, and impossible for anyone outside my team to review or modify.

So I built Edictum to turn that approach into declarative, testable, framework-agnostic contracts.

What it does

Edictum sits between your agent and its tools. When your agent decides to call a tool, Edictum evaluates the call against YAML contracts before it executes. If the contract says deny, the call never happens. The LLM never gets a chance to argue.

apiVersion: edictum/v1
kind: ContractBundle
metadata:
  name: git-safety-policy
defaults:
  mode: enforce
contracts:
  - id: block-push-to-main
    type: pre
    tool: Bash
    when:
      args.command: { matches: 'git\s+push\s+.*main' }
    then:
      effect: deny
      message: "Direct push to main blocked. Use a branch."

  - id: block-force-push
    type: pre
    tool: Bash
    when:
      args.command: { matches: 'git\s+push\s+.*(-f|--force)' }
    then:
      effect: deny
      message: "Force push is not allowed."

The agent's tool was Bash. The "args" were a raw command string. The contract matches against that string — same patterns you'd write in a firewall rule. The denial is deterministic. No probability. No LLM judgment call. The contract either passes or it doesn't.

Note: The YAML above is a complete, loadable contract bundle. Edictum uses a Kubernetes-style format with apiVersion, kind, and metadata headers. Every contract needs a unique id. See the YAML reference for the full schema.

What it is NOT

The AI safety landscape is confusing right now, so I want to be direct:

Not prompt guardrails. Edictum doesn't scan prompts for jailbreaks or filter LLM outputs for toxicity. Tools like NeMo Guardrails, Lakera Guard, and Guardrails AI do that well. Edictum operates at a different layer — it governs what the agent does, not what it says. That said, an interesting side effect: during testing, jailbreak prompts that convinced the LLM to attempt dangerous tool calls were still denied by contracts. The contracts don't care what the LLM thinks — they evaluate the tool call itself. Not our focus, but the screwdriver works as a hammer sometimes.

Not a framework. You still need LangChain, OpenAI Agents SDK, CrewAI, or whatever you're building with. Edictum plugs into your existing framework through thin adapters (~200 lines each inside the library). Your integration code is typically 3-5 lines.

Not an LLM-in-the-loop. Every evaluation is pure Python. No API calls. No inference. The pipeline runs in ~55μs per tool call.

Before and after: LangChain

Without Edictum — your agent reads whatever it wants:

from langchain.tools import tool
from langgraph.prebuilt import create_react_agent

@tool
def read_file(path: str) -> str:
    """Read a file from the filesystem."""
    return open(path).read()  # nothing stops path="/app/.env"

agent = create_react_agent(model, [read_file])
result = agent.invoke({"messages": [("user", "Read the .env file")]})
# Agent reads .env, returns your API keys to the LLM context

With Edictum — dangerous calls are denied before execution:

from langchain.tools import tool
from langgraph.prebuilt import ToolNode, create_react_agent
from edictum import Edictum, Principal
from edictum.adapters.langchain import LangChainAdapter

@tool
def read_file(path: str) -> str:
    """Read a file from the filesystem."""
    return open(path).read()

guard = Edictum.from_yaml("contracts.yaml")
adapter = LangChainAdapter(guard, principal=Principal(role="analyst"))
wrapper = adapter.as_tool_wrapper()

tool_node = ToolNode(tools=[read_file], wrap_tool_call=wrapper)
agent = create_react_agent(model, tool_node)
result = agent.invoke({"messages": [("user", "Read the .env file")]})
# ✗ DENIED read_file path=/app/.env [block-sensitive-reads]
# Agent receives denial message, adapts, asks user what file they need

What you get: a structured AuditEvent for every tool call — who tried what, when, which contract fired, what the verdict was. Your agent's tool usage becomes an auditable trail, not a black box.

How it actually works

The pipeline evaluates tool calls in a fixed order:

Attempt limits — has this tool been called too many times?
Before-hooks — custom Python callbacks
Preconditions — YAML contracts checked against tool name + args + principal
Session contracts — cross-call limits (e.g. max 50 tool calls per conversation)
Execution limits — per-tool execution caps
Execution — the actual tool call happens
Postconditions — validate the output (did it contain an SSN?)
Audit event — structured record of everything that happened

Every step is deterministic. The LLM is not consulted.

The piece that matters for production: principals

Contracts can reference who's making the request:

apiVersion: edictum/v1
kind: ContractBundle
metadata:
  name: pharma-clinical-agent
defaults:
  mode: enforce
contracts:
  - id: restrict-patient-data
    type: pre
    tool: query_patients
    when:
      not:
        principal.role: { in: [pharmacovigilance, admin] }
    then:
      effect: deny
      message: "Role {principal.role} cannot access patient records"

Your application creates the principal:

from edictum import Principal

principal = Principal(
    role="researcher",
    claims={"department": "oncology"},
    ticket_ref="JIRA-456"
)

Today the library trusts what your application passes. There's an open question about whether identity verification should live in the library or stay in the application layer — both approaches have tradeoffs. For now, the design gives you principal-aware policies without prescribing how you verify identity. The roadmap includes server-side JWT/OIDC verification for teams that want the trust boundary inside Edictum rather than outside it.

Observe mode

This comes from my background in networking. When you configure new firewall rules, you don't apply them blindly to production. You put them in monitor mode, watch the traffic, verify the rules match what you expect, then flip to enforce.

Same idea:

guard = Edictum.from_yaml("contracts.yaml", mode="observe")

In observe mode, violations are logged but calls proceed normally. You see what would be denied without breaking anything. Run for a week, review the audit trail, fix false positives, flip to enforce. Zero-risk policy deployment.

CLI: contracts as testable artifacts

# Validate YAML syntax and schema
edictum validate contracts.yaml

# Run precondition test cases
edictum test contracts.yaml --cases tests.yaml

Put edictum test in CI. Your security policies become versioned, tested, reviewable artifacts — not buried in prompt templates.

Note: edictum test evaluates preconditions against your test cases. For full end-to-end testing including postconditions and session limits, use the Python API directly.

What it doesn't do (yet)

I want to be honest about where the edges are:

Single-process only. Session counters live in-memory. If you have multiple agent instances, each tracks its own counters independently. A central policy server is planned but not built.
No PII detection built-in. The protocol is defined (v0.6.0) — you can plug in your own detector. Built-in regex and Presidio-based detectors are coming.
No production sinks beyond file. Audit events go to stdout or .jsonl files. Webhook, Splunk, and Datadog sinks are planned.
OpenTelemetry is early. Span instrumentation exists but isn't battle-tested in production yet. It's opt-in and no-op if the OTel SDK isn't installed.
No hot-reload. Contracts are loaded at startup. Changing them requires a restart.

The roadmap shows what's planned and when.

The landscape — where Edictum fits

Tool	What it does	Layer
NeMo Guardrails	Programmable dialog flows, content safety, jailbreak detection	Prompt/response
Guardrails AI	Output validation, schema enforcement, hallucination detection	LLM output
Lakera Guard	Prompt injection detection, PII scanning	Input/output proxy
LlamaGuard	Safety classification of conversations	Content classification
Edictum	Contract enforcement on tool calls — preconditions, postconditions, session limits, audit	Tool execution boundary

These tools are complementary, not competing. You can run Lakera on prompts AND Edictum on tool calls. Different layers, different threats.

Performance

The governance pipeline adds ~55μs per tool call. That's measured, not estimated. For context, a typical LLM API call takes 500ms-3s. Edictum's overhead is invisible.

Zero runtime dependencies in core. YAML parsing, adapters, CLI, and OTel are optional extras — install only what you need.

Try it

pip install edictum

GitHub: github.com/acartag7/edictum
Docs: docs.edictum.dev

If you're deploying agents that touch production systems — files, databases, APIs, infrastructure — I'd genuinely like to hear how you're handling the gap between "the LLM decided to call a tool" and "the tool executed." That's the layer Edictum was built for.

Edictum is MIT licensed. Built during recovery from liver surgery because apparently I can't sit still. Feedback, issues, and PRs welcome.

Why Your LLM Returns “Sure! Here’s the JSON” and How to Fix It

Arnold Cartagena — Tue, 03 Feb 2026 15:39:26 +0000

You ask for JSON. The LLM returns:

Sure! Here's the JSON you requested:
{"name": "test", "value": 42}
Let me know if you need anything else!

Your parser crashes. Your RAG/Agentic pipeline fails (or worse: gets swallowed behind a generic infinite retry handler). You add more prompt engineering. It works 90% of the time. The other 10%? You're debugging infinitely wondering which of your 12 nodes broke. You didn't want those "Sure! Here's the JSON you requested" or the "Let me know if you need anything else!", I just wanted json.

I had this a lot when trying to get a consistent output of LLMs.

I thought this was just me.

The Pattern

Most teams shipping/testing LLM features run into some version of this:

You ask for JSON, you get "Sure! Here's the JSON you requested:"
The JSON has trailing commas, single quotes, or gets truncated
json.loads() fails with "line 1 column 47" — low-context at best
You retry, but the LLM makes the same mistake
You add prompt engineering. It works 90% of the time. The other 10%...

This prompt engineering part is really a pain to do, multiple versions of the prompt when the problem can actually be solved in other ways.

Search any LLM framework's issues for "JSON" or "ValidationError". The problem shows up across models and frameworks. The solutions are scattered across docs, GitHub issues, and custom workarounds.

There are really two failures here: parsing (turning text into JSON) and validation (ensuring the JSON matches what your pipeline expects). handoff-guard handles both, plus retries with feedback.

Why LLMs Do This

LLMs are trained to be helpful. When you ask for JSON, they want to:

Acknowledge your request ("Sure!")
Explain what they're giving you ("Here's the JSON:")
Format it nicely (markdown code blocks)
Offer follow-up help ("Let me know if...")

This is great for chat. It's terrible for parsing.

And it gets worse:

Truncation: Hit the token limit? Your JSON ends mid-string: {"draft": "This is a long article about...
Malformed syntax: Trailing commas, single quotes, unquoted keys. All common LLM outputs
Nested code blocks: JSON containing ``` characters breaks regex-based parsers

Common Approaches (and Their Tradeoffs)

"Just use JSON mode" — JSON/structured-output modes help when available, but they guarantee syntax, not schema. You still get validation errors, truncation, and no framework-level context like "which node failed."

"Use OutputFixingParser" — LangChain's output-fixing pattern repairs by calling the LLM again—adding latency and cost for every error. Its recommended usage has also shifted across LangChain versions.

"Use Instructor" — Powerful for structured generation across many providers. When it fixes errors, it usually does so by re-prompting the LLM. If you want fast, local repair without burning more tokens, you need a post-processor.

"Use Outlines" — Great for constrained decoding, but requires control over the inference server (e.g., vLLM). It doesn't help if you're calling a closed API like OpenAI or Anthropic.

"Add more prompt engineering" — You're playing whack-a-mole. Fix one edge case, another appears.

What I Built Instead

I needed something that:

Works with raw text output from any provider (post-hoc, not constrained generation)
Identifies which node failed (not just "validation error")
Retries with feedback (tells the LLM what went wrong)
Repairs common syntax issues locally (without calling the LLM again)
Stays lightweight (no embeddings, no ML, just parsing)

So I built handoff-guard.

Before


python
def writer_agent(state: dict) -> dict:
    response = call_llm("Return JSON with: draft, word_count, tone")

    # Hope it's valid JSON
    try:
        data = json.loads(response)
    except json.JSONDecodeError:
        # Which node? What failed? Can the agent retry?
        raise

    # Hope it matches the schema
    try:
        validated = WriterOutput(**data)
    except ValidationError:
        # "1 validation error for WriterOutput" — thanks for nothing
        raise

    return data

After


python
from handoff import guard, retry, parse_json  # PyPI: handoff-guard
from pydantic import BaseModel, Field

class WriterOutput(BaseModel):
    draft: str = Field(min_length=100)
    word_count: int = Field(ge=50)
    tone: str

@guard(output=WriterOutput, node_name="writer", max_attempts=3)
def writer_agent(state: dict) -> dict:
    prompt = "Return JSON with: draft, word_count, tone"

    if retry.is_retry:
        prompt += f"\n\nPrevious attempt failed:\n{retry.feedback()}"

    response = call_llm(prompt)
    return parse_json(response)  # Strips wrappers, repairs syntax

If it fails after 3 attempts:



HandoffViolation in 'writer':
  Contract: output
  Field: draft
  Expected: String should have at least 100 characters
  Received: 'Too short...' (str)
  Suggestion: Increase the length of 'draft'

For logs/telemetry, access e.total_attempts, e.history, or e.to_dict().

What `parse_json` Actually Does


python
from handoff import parse_json

# Strips conversational wrappers
obj = parse_json('Sure! Here\'s the JSON:\n{"key": "value"}\nLet me know!')
# -> Python dict/list (parsed JSON), not a JSON string

# Handles common syntax issues (via json-repair)
parse_json('{"a": 1,}')        # trailing comma → {"a": 1}
parse_json("{'a': 1}")         # single quotes → {"a": 1}
parse_json('{a: 1}')           # unquoted keys → {"a": 1}
parse_json('{"a": 1 // comment}')  # JS comments → {"a": 1}

# Detects truncation (v0.2.1)
result = parse_json('{"draft": "long text...', detailed=True)
# -> ParseResult with .data (dict), .truncated (bool), .repaired (bool)
result.truncated  # True — best-effort signal (unmatched braces detected)
result.repaired   # True — json-repair path was used successfully

No LLM calls. No embeddings. Deterministic parsing with best-effort repair. I haven't published benchmarks; this was built from real failure modes in my own graphs.

Why Not Instructor/Outlines?

	Instructor	Outlines	handoff-guard
Approach	Generation-time validation	Constrained generation	Post-hoc validation & repair
Works with	OpenAI, Anthropic, etc.	vLLM, Transformers	Any string output
LangGraph compatible	Yes (manual)	No	Yes (adapter: `guarded_node`)
Identifies failed node	No	N/A	Yes
Retries with feedback	Yes	N/A	Yes
Repairs malformed JSON	Yes (via re-prompt)	N/A	Yes (local, no tokens)
Dependencies	Pydantic + provider SDKs	Transformers/vLLM stack	Pydantic + json-repair

Instructor and Outlines are excellent tools. The difference is when and how they work:

Instructor validates at generation time and fixes errors by re-prompting—effective but costs tokens
Outlines constrains generation at the model level—powerful but requires inference server control
handoff-guard validates after the LLM responds and repairs locally—no extra tokens, works with any provider

The Problems This Actually Solves

handoff-guard doesn't fix framework bugs. It helps when you control the code that receives LLM output:

Problem	Example	How handoff-guard helps
LLM wraps JSON in conversation	`"Sure! Here's the JSON: {...}"`	`parse_json()` strips wrappers
Malformed JSON syntax	Trailing commas, single quotes, unquoted keys	`parse_json()` repairs common issues
Truncated output at token limit	`{"draft": "long text...`	`parse_json(detailed=True)` detects truncation
"ValidationError" with no context	`1 validation error for State`	`@guard(node_name="writer")` tells you which node
No retry on validation failure	Agent fails once, stays failed	`@guard(max_attempts=3)` retries automatically
LLM doesn't know why it failed	Retry happens but same error repeats	`retry.feedback()` tells the LLM what went wrong

Limits

What this won't magically fix:

Missing or hallucinated data — If the model omits required fields or invents values, deterministic repair can't invent correct data. Retries are still needed.
Ambiguous repairs — "Repair" is sometimes a best-effort guess (e.g., unquoted keys, stray punctuation). Always validate the result.
Severe truncation — You can detect it, but you can't recover missing content without another generation.
Adversarial or multi-JSON outputs — parse_json extracts the first JSON object/array boundary it finds. Complex tool traces or multiple embedded objects may need custom handling.

Security note: If you're parsing untrusted model output, treat "repaired JSON" as untrusted input. Validate types and ranges.

Get Started


bash
pip install handoff-guard

The package is handoff-guard, the import namespace is handoff:


python
from handoff import guard, retry, parse_json

That's it. No config files. No API keys. No Docker.

GitHub: github.com/acartag7/handoff-guard
PyPI: pypi.org/project/handoff-guard

What's Next

The library does what it set out to do. I'm not planning major features just bug fixes and edge cases as users report them, as it actually works for my current need.

If you hit something it doesn't handle, open an issue.

Built because "ValidationError: 1 validation error" tells you nothing useful.