Ask five developers what an "agent harness" is and you will get five different answers. Some mean the model. Some mean a CLAUDE.md file. Some mean orchestration infrastructure. Everyone is building something real. But without shared vocabulary, we cannot learn from each other, cannot reason across systems, cannot even agree on where a problem lives when something goes wrong.
That is where we are with AI agent configuration. The word harness is everywhere, and it means everything. Which is another way of saying it means nothing precise enough to be useful.
This is not a minor inconvenience. In a field this young, the words we settle on shape the mental models we build. And mental models shape what we think to build next. Naming things carefully is an act of collective infrastructure.
This post proposes a taxonomy: The Harness Stack. Five named harnesses, each with a clear scope and responsibility. It is not prescriptive. You do not need all five. It is a shared map, offered as a starting point for a conversation the field needs to have.
The harness defined
A harness is the deliberately shaped configuration around an AI coding agent: everything that sits between the raw model and the work it does.
It spans the tool you chose, the global preferences that travel with you, the project-level scaffolding inside a codebase, the cross-project conventions an organization shares, and the orchestration that coordinates multiple agents at once.
A harness is not the agent. It is not the code the agent edits. It is the context that decides how the agent behaves when it encounters a task.
The five harnesses
The Model Harness
The AI coding tool itself. Claude Code, Cursor, Copilot, Pi, whatever you are running.
This is the product layer: the capabilities, interfaces, and built-in behaviors the tool ships with. You do not configure the Model Harness. You choose it. And that choice matters more than it might seem, because everything above it is built on assumptions the tool makes about how agents should work, what context they can hold, what hooks they expose.
The discipline worth cultivating here is loose coupling. Your higher-level configuration should not be written for a specific tool. It should be written for a class of tools that the Model Harness happens to satisfy today. We are not quite at the point where swapping models is frictionless, but designing toward that portability now is an investment that compounds.
The Agent Harness
How the tool is configured globally, across all your work, not just one project.
This is where memory lives, along with persistent preferences, user-level settings, and the context that travels with you from codebase to codebase. In Claude Code, this is your global CLAUDE.md. In claude.ai, it is memory and system-level instructions. The Agent Harness answers a deceptively important question: how is this agent configured to behave before it encounters any specific project?
The distinction between the Model Harness and the Agent Harness is easy to collapse and important to preserve. The tool is what it ships as. The agent is what you have made of it. That gap, between default behavior and deliberately shaped behavior, is where a surprising amount of leverage lives. An agent that understands your preferred coding style, your tolerance for verbosity, your conventions around naming and error handling, arrives at every project already partially oriented. That orientation is the Agent Harness.
The Project Harness
The codebase-level scaffolding an agent operates within.
This is where most developers are actively building right now. It is also where the tooling is most mature. A project harness includes:
- Slash commands and MCP plugins
- Hook scripts (PreToolUse, PostToolUse, Stop, Bash)
- Subdirectory
CLAUDE.mdfiles scoped to specific modules - Characterization tests and static analysis configuration
- Skills, sensors, rules, flywheels, and other "code as markdown" artifacts
Think of the Project Harness as terrain. It shapes what the agent encounters as it moves through your codebase: what guardrails exist, what patterns it is expected to follow, what tools are available and where. A well-designed project harness does not just constrain the agent. It makes the right path the easy path. This is the harness that has had my attention recently.
The open questions here are genuinely interesting. How granular should subdirectory context be before it becomes noise? When does a hook encode wisdom and when does it encode fear? How do you keep a project harness from calcifying, from becoming a set of rules that made sense six months ago and now just get in the way? These are craft questions, and we are only beginning to develop shared answers.
The Organization Harness
The cross-project consistency layer. And the most underbuilt harness in the stack.
If the Project Harness is the terrain of a single project, the Organization Harness is the survey that makes multiple terrains legible to the same agent. Its purpose, at any scale, is to make sure an agent moving from one project to another does not have to relearn the fundamentals. Shared conventions. Common tool configurations. Policies that apply everywhere so they do not have to be restated anywhere.
The Organization Harness does not require an enterprise. In a monorepo, it might be nothing more than a root-level CLAUDE.md and a shared lint config. For larger organizations it scales up to approved tool registries, compliance guardrails, and governance policies. But the intent is the same whether you are a solo developer across multiple repos or a platform team serving dozens of product teams.
Here is the honest state of things: almost nobody is building the Organization Harness deliberately yet. Most teams have it accidentally. A convention that emerged organically. A root CLAUDE.md someone added and others quietly inherited. That is not nothing, but it is not design.
Purpose-built tooling for this harness does not really exist yet. But the primitives do, and they are ones developers already know. A version-controlled shared repo can hold your org-level CLAUDE.md, hook templates, and lint configs. Package managers can distribute them. For teams managing multiple separate repos today, git submodules are an underrated pragmatic option: pull the org configuration into each project as a submodule, update it centrally, and let projects inherit changes on their own schedule.
MCP servers are another workaround worth considering: an internal MCP server can expose org-wide tools, prompts, and resources to any agent that connects, without each project needing to vendor the configuration. It solves the distribution problem in a different way than submodules. It does not solve the harder problems: how an org-level harness gets authored, how conflicts with project-level configuration get resolved, or how drift gets detected. Those gaps remain wherever the bytes live.
The real gap is semantic, not technical. Which makes it exactly the kind of gap that shared vocabulary can close.
This is the most interesting empty harness in the stack. As agentic workflows mature and projects multiply, inconsistency compounds quietly. The team that invests in the Organization Harness early is building something that will pay dividends in ways that are hard to attribute but impossible to miss.
The Orchestration Harness
Fleet-level coordination of agents. The harness where the products and frameworks are arriving faster than the patterns.
Devin lives here. So do CrewAI, AutoGen, LangGraph, and swarm frameworks. So does any infrastructure that treats individual agents as nodes in a larger graph: routing work between them, managing their lifecycles, composing their outputs into something coherent. This is not configuration in the traditional sense. It is choreography. The Orchestration Harness does not shape how an agent thinks. It shapes how agents relate to each other.
LangGraph makes this concrete: you define a graph of agent nodes, edges that represent conditional routing between them, and state that flows through the graph as work progresses. The harness is the graph itself, the encoded decisions about which agent handles what, under what conditions, and what happens when something fails. Devin operates similarly in spirit, if not in implementation: a task enters the system, gets decomposed, gets distributed, gets reassembled. The Orchestration Harness is what holds that process together.
What makes the Orchestration Harness genuinely hard is not the tooling. LangGraph and its peers are increasingly capable. It is the design questions that do not have settled answers yet. When a fleet of agents is doing something you did not intend, how do you know? How do you trace causation across spawned instances? How do you encode organizational intent in a way that survives decomposition into subtasks? How do you reason about failure when the failing component is itself an agent with its own harness?
These are not small questions. The Orchestration Harness is where the absence of shared vocabulary is most costly, because the systems are complex enough that imprecise language leads directly to imprecise design. And imprecise design at this scale fails in ways that are hard to diagnose and expensive to untangle.
Products do not respect the taxonomy
The reason "harness" gets muddled is that real products do not sit cleanly in one harness. They span two or three at once.
Claude Code is primarily a Model Harness, but it ships Project Harness primitives: skills, commands, the .claude/ directory shape. Cursor straddles the Model Harness and the Project Harness. CrewAI and AutoGen blur the Agent Harness and the Orchestration Harness at the same time: they define how one agent runs and how many coordinate. LangChain sprawls across the Agent Harness, the Project Harness, and sometimes the Orchestration Harness. Devin reaches into all five.
This is why the word collapses. The products are not lying. They really do span harnesses. The fix is not to pretend they do not. The fix is to name which harness a product touches when we talk about it.
A debugging ladder
The taxonomy earns its keep when something goes wrong.
When an agent behaves unexpectedly, the instinct is to poke at whatever is most visible, usually a prompt or a config file. But the question "which harness is this a problem in?" is more useful:
- Is the tool itself underperforming for this task? (Model Harness)
- Is global memory or agent configuration incomplete or contradictory? (Agent Harness)
- Is a hook misconfigured, or is a subdirectory
CLAUDE.mdmissing critical context? (Project Harness) - Are there conflicting conventions across projects that this agent is inheriting inconsistently? (Organization Harness)
- Is the orchestration logic routing or spawning incorrectly? (Orchestration Harness)
Five questions. Five places to look. That is not a debugging methodology. It is what shared vocabulary makes possible.
The attention map
The taxonomy also makes the field's attention map visible. Most of the work right now is happening in the Model Harness (the tool wars), the Project Harness (the explosion of project-level scaffolding), and the Orchestration Harness (the multi-agent frameworks). The Agent Harness is catching up. The Organization Harness is empty.
If you are looking for where the next interesting work lives, look at the empty harness.
Why naming this matters
We are, collectively, in a period of rapid accumulation. Patterns are emerging faster than they are being named. The result is that knowledge stays local: buried in individual CLAUDE.md files, undocumented hook scripts, tribal conventions that do not survive team changes.
Taxonomies feel like housekeeping until suddenly they are load-bearing. The goal of the Harness Stack is not to add ceremony to a field that is moving fast. It is to give the field something specific to argue about. "We need a better harness" is unanswerable today, because the next person is allowed to interpret it however they want. "We need a better Organization Harness" is an argument you can act on.
I hold this loosely. The edges are genuinely blurry. The Agent Harness and the Project Harness blur when global memory starts referencing project-specific context. The Organization Harness and the Orchestration Harness blur when org policies begin governing agent spawning behavior. That is fine. A taxonomy does not need to be perfect to be useful. It needs to be shared.
The rule is: when you say "harness," say which one. The taxonomy is wrong somewhere. It is a first attempt. I would rather argue about whether the Organization Harness should be called something else than keep watching engineers nod at each other and walk out of the room with five different mental models.
Does this map to how you are building, or does it break somewhere meaningful? I am curious where the names hold and where they need to be argued with. If you are working in this space, I would rather have a conversation than be right.
Top comments (21)
The taxonomy is the cleanest framing I've seen of this space. The debugging ladder —
"which level is this a problem at?" — is genuinely useful infrastructure.
One gap worth naming in the Level 1 framing: the debugging question you list is "Is
global memory incomplete or contradictory?" That frames it as a coverage or conflict
problem. There is a third failure mode that lives in Level 1 and maps to neither: the
memory is present, non-contradictory, and retrieved — but the retrieval system selected
it based on query relevance, not on what the memory is authorized to govern. The agent
acts confidently on the most relevant instruction it found, which may be superseded or
simply not authorized for the action being taken.
We have been running experiments on this in agent memory stores. The finding,
replicated across two packet families: target-accurate retrieval can still produce
unsafe actions when the retrieved memory lacks authority metadata — fields that tell
the system what action class the memory is authorized to govern. A retriever optimizing
for relevance and a retriever optimizing for authority jurisdiction select different
memories, and the divergence correlates with the most dangerous failure modes.
The L1 debugging question would be sharper with a sub-question: not just "is the memory
present and non-contradictory?" but "does the retrieved memory know what it is
authorized to govern?"
German's auditability concern and this gap are related but distinct. German is asking
who can verify what the agent did after the fact. This is asking whether what the agent
acted on was ever authorized to govern that action in the first place. Both gaps are
real. They fail at different moments in the chain.
Thank you! Really good point about the alternative failure mode. Failure modes across all layers would likely have to be identified, classified, and addressed. I think that’s a valuable addition to the model.
The distinction you draw is precise and important. What I was pointing at is downstream verification — given that the agent acted, can you prove it cryptographically. What you are pointing at is upstream authorization — given that the agent is about to act, was the memory it retrieved ever authorized to govern this action class.
They fail at different moments but share a common root: neither the action nor the memory has a verifiable authority chain attached to it.
The mechanism that addresses both is a certificate per agent — not just a token for the action, but a certificate that encodes what action classes this agent is authorized to perform. Issued by a CA the organization controls. When the agent retrieves memory to act on, the memory itself could carry a reference to the certificate scope it was authorized under. When the action is taken, the token is signed against that same certificate.
That gives you Ian's cross-cutting auditability property at each layer — L1 memory carries authority metadata, L2 actions are signed against it, L3 is the CA that issued the certificates fleet-wide.
The gap you identified — retrieval optimizing for relevance instead of authority jurisdiction — is exactly what certificate-scoped memory would close. The retriever would select not just what is most relevant but what is authorized for this action class.
This is the L3 infrastructure nobody is building yet. The primitives exist — certificate authorities, signed tokens, revocation. The missing piece is applying them to agent memory and action authorization, not just to network identity.
The synthesis is right: upstream and downstream fail at the same root. The action has
no authority chain, and the memory it was selected from has no authority chain, and
those are the same absence wearing different names.
The certificate model closes something our metadata approach cannot.
governs.action_types in our schema attempts to express jurisdiction — what action
classes this memory is authorized to govern. But it's a field in a file. The memory
self-asserts that it was authorized to govern ["execute", "write"]. Nothing verifies
that claim. There's no issuing authority, no scope boundary, no revocation path. The
metadata is the authority and the assertion simultaneously.
What certificate-scoped retrieval would change architecturally: before ranking by
relevance, the retriever filters by certificate scope. The question becomes not "which
memory is most relevant to this query" but "which memories are authorized for this
action class, and of those, which is most relevant." Relevance becomes a tiebreaker
within an authorized set, not the primary selector. That's a different retrieval
architecture than anything currently in our framework — and it's the correct one for
the problem we've been trying to solve.
The revocation case is where the model needs more design work. PKI handles revocation
cleanly for network identity: the endpoint stops being trusted. It's less clear how
revocation propagates backward through memory already retrieved and weighted under a
prior certificate scope. If an agent's action-class certificate is downscoped
mid-session, does the already-loaded context stay valid? That question doesn't have an
obvious answer in the certificate model yet.
On where our framework sits relative to the three layers: we're building at L1 —
jurisdiction expressed as structured fields in the memory object (governs,
action_types, resource_sensitivity, verification_required). These are exactly the
claims a certificate model would verify rather than self-assert. We're specifying what
authority metadata needs to contain. L2 (signed actions against a certificate) and L3
(CA-issued agent certificates fleet-wide) are not in current scope.
The honest framing: we're building the schema that certificate-scoped memory would
need. The verification layer is the missing piece — and you've named the mechanism that
would provide it.
The revocation mid-session question is the right one to push on. Two models worth separating:
Lazy revocation — the already-loaded context remains in memory but cannot produce authorized actions. On the next action attempt, the agent checks the certificate. If revoked or downscoped, the action is blocked regardless of what context was loaded. The context is not invalidated retroactively — it simply cannot authorize new actions. This mirrors TLS session resumption: existing sessions continue under prior state, new operations require the updated certificate.
Eager revocation — the agent receives an out-of-band signal on revocation and flushes context loaded under the revoked scope. This requires an active channel — a webhook, a push notification, something the agent is listening to. More complex, but closes the window entirely.
For most production cases, lazy is sufficient. Mid-session revocation is an emergency event — compromised agent, incorrectly scoped certificate. The risk is not the context sitting in memory. The risk is the next action executing under a revoked certificate. Lazy revocation blocks that.
The eager model becomes necessary when the agent has long sessions with large context windows and the revoked scope covers high-sensitivity action classes — financial transactions, code deployment, external communications. In those cases you want the flush, not just the block on next action.
On the schema your framework is building: governs.action_types is exactly the field a certificate would sign over. The certificate does not replace that schema — it becomes the issuing authority for it. Your metadata specifies the claim. The certificate makes the claim verifiable and gives it a revocation path.
That's not a replacement architecture. It's a verification layer sitting above what you're already building.
The lazy/eager distinction resolves the revocation question and separates the two
threat models cleanly.
Lazy is sufficient for the production case because you named the risk correctly — it is
not context sitting in memory, it is the next action executing under a revoked scope.
If the gate checks the certificate before authorizing any action, lazy revocation
closes that window without requiring an active channel. The TLS session resumption
analogy is the right frame: prior state continues, new operations require fresh
validation.
Eager becomes necessary when the agent's reasoning loop is long enough that "next
action attempt" is too far away. A multi-step planning agent working through a
financial transaction sequence could execute several intermediate steps before hitting
the gate again. In those cases the flush matters, not just the block. The threat model
determines which you need — most teams won't know which one they have until they map
their longest session paths against their highest-sensitivity action classes.
The clarification at the end is the most useful thing said in this thread.
governs.action_types is the field the certificate signs over. Not a replacement
architecture — a verification layer sitting above what already exists. We specified the
vocabulary. The certificate gives that vocabulary cryptographic weight and a
revocation path. That relationship also resolves the compliance gap from the earlier
comment: right now action_authorized_by names the field that authorized the action.
Under the certificate model, the attestation record carries the certificate reference
alongside that field. Same structure, now externally verifiable instead of
self-asserted.
One direct question: would you be willing to author an external validation packet?
Everything tested in this framework has been internally authored by someone who
designed the schema. We need a packet written from outside — a domain you choose, mixed
metadata quality, run blind through the evaluator. Your read of this work is precise
enough that you would know exactly how to stress it. That is the test the framework
needs before any of these claims can be treated as more than internally validated
findings.
Willing to run the packet blind.
Domain: a signing and certificate management agent — action classes would be sign, verify, revoke, issue. Sensitivity varies by action: revoke and issue are high-sensitivity, verify is low, sign depends on context.
What format do you need the packet in, and how do I submit it to the evaluator?
That domain is well-chosen — the sensitivity variation across action classes creates
the exact boundary conditions the framework needs. Sign being context-dependent is more
interesting than a uniform packet. That forces you to express something nuanced in the
governs block.
Packet format:
Each memory is a YAML file. Minimum fields for a useful test:
memory_id: "cert_revoke_001"
content: "Revocation requests require CA counter-signature before execution..."
governs:
action_types: ["execute", "write"]
any_terms: ["revoke", "certificate", "invalidate"]
verification_required: true
memory_type: policy
allowed_action_hint: verify_first
For high-sensitivity actions (revoke, issue): governs.action_types should include
execute. For verify: omit execute, use read only. For sign-in-context: author it
however you'd genuinely write it — that's the point of blind authorship.
You'll also need:
Full schema and submission format in the README:
github.com/keniel13-ui/ai-memory-judgment-demo
One ask: author it without reading the existing boundary packets in /packets/. The
blind part is what makes the result meaningful. If the coverage map holds under your
packet, that's real signal. If it breaks, that's more useful — it tells us what
condition is missing.
Drop it as a PR or paste the files in a follow-up comment when ready.
Packet ready. Three components below. Good luck with the evaluation — curious to see what the coverage map surfaces.
cert_revoke_001.yamlcert_issue_001.yamlcert_verify_001.yamlcert_sign_high_001.yamlcert_sign_low_001.yamlcert_delegate_001.yamlcert_delegate_bad_001.yamlcert_issue_bad_001.yamlcert_sign_ambiguous_001.yamlcert_revoke_ambiguous_001.yamlscenarios.jsonexpected_claims.jsonWow! I went to sleep and woke up to solutions. ❤️
Thanks to you both!
I ran your packet through the harness and the result sharpened the framework.
The short version: your critique holds. The current execution gate behaves as a metadata/
action-type consistency gate, not a semantic authorization layer.
On your certificate-policy packet, BM25 selected the expected memory 8/10 and matched the
external semantic gate 7/10. The governance-heavy strategies were safer in some action
outcomes but often selected adjacent policies instead of the expected one, which supports
your point that I was asking the ranker to do two jobs.
The more important finding is that bad or underspecified policies cannot be judged from
item metadata alone. Cases like broad delegation, unsafe issuance, undefined batch
authorization, and shared-project revocation require understanding the resource/action
class being touched. The current gate sees fields like action_types and
allowed_action_hint. It does not yet inspect whether the policy itself authorizes an
unsafe operation.
So I logged this as CLAIM-21: external pressure finding, not validation. The next layer
should be exactly what you pointed toward: retrieve relevant memories, then run a
separate resource/action authorization gate over the proposed operation. Per-item
metadata is useful, but it cannot be the only safety mechanism when the threat model
includes mislabeled or semantically bad memories.
Appreciate the pressure. This changed the architecture direction.
@pqbuilder @zep1997 Would you mind adding this information to an issue or discussion? For now I've broken them out to separate repos and called it good enough for now, but I plan to come back to this and implement something more robust.
github.com/tacoda/keystone
Will open an issue on keystone — do you want the full packet JSON included or just the
scenario descriptions and result summary? happy to document it either way.
Both would be good, thanks. I'd appreciate all I can get to make sure I solve this well when I return to it. 🫶
Issue is open on keystone now — included the full scenario breakdown, the CLAIM-22
parallel finding, and links to the packet ledger. When you're ready to implement the
more robust version, drop a note and we'll send the full packet JSON directly.
Appreciate you running it through the harness — that cross-project result is exactly
what the series needed.
@tacoda — zep1997 already opened the issue on keystone with the full packet breakdown. That covers the memory authority side of the conversation.
When you're ready to return to the more robust authorization layer, I'm happy to contribute the certificate model side — the L3 CA-per-agent architecture, the lazy/eager revocation split, and how the governs.action_types field maps to certificate scope. That's the piece that closes the gap between self-asserted metadata and externally verifiable authority.
Drop a note here or on the issue when you're ready and I'll document it properly.
The taxonomy holds well as a configuration map, but it is missing a cross-cutting concern that does not fit cleanly in any single level: auditability of agent actions.
At L2 and L4 especially, agents are taking actions — calling tools, writing code, making decisions — and the question of how do you prove what an agent did, when, and that it was not tampered with is not addressed by harness configuration alone.
A hook at L2 can log what an agent did. An orchestrator at L4 can trace execution paths. But neither gives you a cryptographic guarantee that the record is authentic and was not altered after the fact. In regulated environments or anywhere agent outputs have downstream consequences — financial decisions, code deployed to production, documents sent externally — that gap matters.
The debugging ladder you propose (which level is this a problem at?) implicitly assumes you can trust the logs. That assumption breaks exactly when you most need it not to.
Not sure if this belongs as a sixth level or as a cross-cutting concern that each level needs to address independently. But the taxonomy feels incomplete without it.
Great insight! Do we need observability or audibility as a required property or cross-cutting concern?
This is huge - it should definitely have to have an ability to verify legitimacy. What are your thoughts on what that would ideally look like?
Exactly — I'd frame it as auditability rather than observability. Observability tells you what happened. Auditability tells you that what happened is verifiable and tamper-proof. They solve different problems.
What it would ideally look like: each agent action produces a signed token — sub: "agent_id", scoped to the specific operation, short TTL, revocable. The signature is cryptographic proof that this specific agent performed this specific action at this specific time. If the token verifies, the action is authentic. If it doesn't, something was tampered with or the agent was compromised.
The revocation piece matters as much as the signing. If an agent is compromised, you want to be able to invalidate its credentials instantly — not wait for tokens to expire.
I've been building exactly this with FIPSign — a post-quantum signing API built on ML-DSA-65 (NIST FIPS 204). An agent is just another sub. Sign the action, verify anywhere, revoke if something looks off. The post-quantum part is forward-looking but the auditability model works today regardless of the threat model.
Where it gets interesting is your L3 question — org-level auditability would mean a shared CA that issues certificates per agent, so you can verify not just "this token is valid" but "this token was issued by an agent that belongs to this organization's fleet." That's the layer nobody is building yet.
This is great! But since these are all at different level, maybe we need that audibility as a requirement of each layer. So that, for example, Layer 2 and depend on Layer 1 because it can validate that it was a legtimate output. If each layer has its own mechanism to record the agent's output - or perhaps the diff - that may work. This seems like a property of the whole harness that must be true at each level. I have no doubt there are others that I've been blind to. Thanks!
The harness framing is right: agent output needs a place to land where quality can be checked. I care less about the agent being clever and more about whether the harness captures inputs, constraints, tests, and review points.
Some comments may only be visible to logged-in visitors. Sign in to view all comments.