Skill frameworks for AI coding agents are exploding in popularity. As of July 2026, Superpowers has roughly 256,000 GitHub stars, Matt Pocock's skills have roughly 176,000, and Agent Skills has roughly 79,000. All three promise to make AI agents write better code by feeding them structured Markdown instructions.
But there is a fundamental problem that deserves more attention than it gets: the behavioral control layer in these frameworks is plain English written in Markdown files (what literary types call "prose"). Not code. Not configuration. Not compiled rules. Just natural-language sentences that the agent reads and interprets. Ambiguous, probabilistically interpreted, context-sensitive sentences. And the industry is calling that engineering.
This article is about why that matters, what the real failure modes are, and what actual guardrails would look like.
What Is the Control Plane?
In traditional software, the control plane is the layer that governs how a system behaves: configuration files, type systems, access control policies, deployment pipelines, CI/CD gates. These mechanisms share a common property. They are deterministic. A type checker either passes or fails. A CI pipeline either succeeds or errors. The behavior is reproducible and independently verifiable.
AI agent frameworks introduce a new kind of control plane. Instead of configuration files or type systems, they use prose instructions: SKILL.md files, system prompts, AGENTS.md files, CLAUDE.md files. The agent reads these instructions and decides what to do.
This is not a minor implementation detail. It is a fundamental shift in how behavioral control works. And it has consequences that the framework creators have not fully addressed.
The Core Problem: Prose Is Probabilistic
When you write an instruction like You MUST write tests before implementation in a skill file (for example), the agent does not execute that instruction the way a compiler executes a type annotation. It interprets the instruction probabilistically. The same instruction might produce different behavior on different runs, in different contexts, or after the agent has read additional context that reframes its understanding of what "tests" means or what "implementation" means.
This is not speculation. Anthropic's own SWE-bench documentation acknowledges that agent performance "can vary significantly based on this scaffolding, even when using the same underlying AI model." The scaffolding is the prose. The variation is the probabilistic interpretation.
Three specific failure modes follow from this.
1. Semantic Drift
Over a long-running session, the agent accumulates context. Earlier instructions get pushed further back. New context reframes old instructions. The agent's interpretation of "write tests first" in step 1 might differ from its interpretation in step 47, not because the instruction changed, but because the surrounding context changed.
No current framework addresses this at the instruction level. The instruction is prose. There is no mechanism that detects when the agent's interpretation has drifted from the original intent.
2. Goal Reinterpretation
Agents expand scope. You ask for a bug fix. The agent decides the surrounding code also needs refactoring. You ask for a new endpoint. The agent decides the existing API contract should be updated. These are not hallucinations in the traditional sense. They are rational reinterpretations of an ambiguous goal, executed confidently without confirmation.
Anti-rationalization tables (used in Agent Skills) attempt to address this by pre-rebutting common excuses. But the rebuttals are also prose, interpreted by the same probabilistic system. You have not removed the drift. You have added more prose to argue with it.
3. Correlated Verifier Failure
This is the sharpest problem, and the hardest to dismiss.
Most agent frameworks include a verification step. The agent runs tests, reviews code, or checks output against criteria. But if the verification step is also an LLM reading prose instructions, your verifier is subject to the same drift as the actor. You have not built a guardrail. You have built two drifters checking each other.
"Correlated" here means the actor and verifier share the same failure modes: the same model, the same prompt context, or the same misinterpretation of the goal. When that happens, they will agree on a wrong answer. The verifier fails in the same direction as the actor, so it never catches the error. This is a known problem in LLM-as-judge evaluation, which has documented reliability issues when the judge and the judged share biases.
Does This Mean Prose Control Is Useless?
No. And this is where it is important to reject both extremes.
Deterministic enforcement is a purity test that traditional engineering does not meet either. Code review is a human-interpreted process. QA testing is fallible. CI pipelines catch what they are configured to catch, not what they should catch. Feature flags can be misconfigured. None of these are formally verified. We use them because they shift the probability of catching errors in the right direction, not because they provide mathematical guarantees.
Skill frameworks do the same thing, at a different layer. If a TDD skill shifts the probability of the agent writing tests from low to high, that is a meaningful improvement. It is not a guarantee. But neither is human code review compliance. The question is whether the probability shift justifies the token overhead, and that question has not been answered empirically.
The honest claim is narrower: prose-based control is useful but insufficient for production systems. It shifts probability distributions in useful directions. It does not provide guarantees.
What Anthropic Actually Recommends
Anthropic's own Building Effective Agents guide is instructive here. Their core recommendation is the opposite of what most skill frameworks do:
"We recommend finding the simplest solution possible, and only increasing complexity when needed. This might mean not building agentic systems at all."
They describe workflows (predictable, deterministic pipelines) as preferable to agents (autonomous, model-driven loops) for well-defined tasks. They specifically warn: "Agentic systems often trade latency and cost for better task performance, and you should consider when this tradeoff makes sense."
Their SWE-bench agent, which achieved state-of-the-art performance, deliberately kept scaffolding minimal. The prompt was short. The model was given tools and freedom to choose its own approach. They did not pile on prose instructions. They gave the model a clear goal, clear tools, and let it work.
This is the opposite of the "encode everything in Markdown" philosophy. Anthropic's own evidence suggests that less prose, not more, produces better agent behavior.
What Real Guardrails Look Like
The critique exposes a genuine gap. Current frameworks rely on prose for behavioral control. If you want actual guardrails against drift and reinterpretation, you need deterministic mechanisms layered on top of the probabilistic layer. Here is what those look like in practice.
Compiled schema validation. Instead of telling the agent "return a valid JSON object with these fields," use JSON Schema enforcement at the API boundary. The agent either returns valid JSON or gets an error. No prose interpretation involved. Spring AI's validateSchema() does exactly this: it validates the response against a compiled schema, feeds the error back to the model, and retries.
Type checking at boundaries. If the agent writes code, the type checker is a deterministic gate. If types do not pass, the code does not merge. This is independent of the agent's interpretation. TypeScript, Java's compiler, Rust's borrow checker. These are real guardrails because they are not probabilistic.
Test suites as independent verifiers. A well-written test suite is a deterministic specification. If the agent's code passes the tests, it meets the spec. If it fails, it does not. The key is that the tests must be written independently of the agent's implementation, not generated by the same agent that wrote the code. Agent Skills' eval framework, which runs in CI and checks that skills route correctly, is a step in this direction. But it tests the skills, not the agent's output.
Immutable audit logs. Instead of relying on the agent to report what it did, record every tool call, every file change, every command execution to an append-only log. This does not prevent drift, but it makes drift detectable after the fact. You can diff what the agent said it did against what it actually did.
Non-LLM verifiers for critical paths. The most pragmatic approach for production systems: use LLMs for pattern matching and research, never let them execute critical actions autonomously. The LLM proposes. A deterministic system disposes. For a trade execution, a code merge, or a production deployment, the final decision should not be LLM-driven.
The Honest Assessment
The debate around AI agent frameworks has two extremes, and both are wrong.
Extreme 1: "Prose control is engineering." It is not, by any traditional definition. Engineering requires reproducibility, verifiability, and independent enforcement. Prose instructions provide none of these. Calling a collection of Markdown files a "software development methodology" stretches the word "methodology" past its breaking point.
Extreme 2: "Prose control is useless." It is not. It is an incomplete but useful approach to steering probabilistic systems. The same way configuration management is an incomplete but useful approach to steering deterministic systems. The gap between "useful" and "sufficient" is where the real work needs to happen.
The honest position: skill frameworks as they exist today are R&D tooling, not production engineering. They shift probability distributions in useful directions. They do not provide guarantees. They are worth using for tasks where human review catches the remaining failures. They are dangerous for tasks where they run unattended.
The frameworks that survive will be the ones that integrate deterministic guardrails (type checking, compiled schemas, CI gates, independent test suites, immutable logs) as first-class citizens alongside the prose layer. The prose tells the agent what to try. The deterministic layer enforces what is acceptable. Until that integration exists, calling this "engineering" is aspirational.
Sources:
Top comments (0)