Nice taxonomy — the four questions framing (trained on what / measured how / allowed what / actually did what) is a good litmus test: if a doc can't be mapped to one of them, it's probably not a trust artifact. One hard-won lesson from maintaining these: model cards and eval reports rot faster than READMEs. A hand-written card drifts from the model it describes the moment a retrain lands. What helped was treating them as build artifacts rather than documents — generated from the same pipeline that produces the model (config + eval results in, markdown out), committed alongside the weights, with a stale flag in CI when inputs change. Same for agent cards: the "what did it actually do" section is only trustworthy if it is derived from run logs, not written from memory. Hand-editing trust artifacts quietly defeats their purpose.
Software developer passionate about building modern web and SaaS applications. I enjoy working with .NET, exploring AI tools, and turning ideas into practical products.
The rot point is the one I under-weighted. A drifted trust artifact is worse than none — it confidently vouches for a state that no longer exists, and everyone trusts it because it looks maintained. That's silent-success in the docs layer. Generating them as build artifacts from the pipeline is the only real fix; the moment a human hand-edits, the drift reopens.
The grep-for-staleness is the control that survives deadlines precisely because it costs one line in a generator. For the CI freshness gate, watch the keying: a regenerate-and-diff on raw artifact bytes flags cosmetic rebuilds forever, while one keyed on semantic content (headers + claims, not prose) stays quiet through formatting churn. Keying wrong is how a good gate gets deleted in week two.
Software developer passionate about building modern web and SaaS applications. I enjoy working with .NET, exploring AI tools, and turning ideas into practical products.
The keying point is the thing that decides whether the gate lives or dies, and you’ve put your finger on exactly the failure: a regenerate-and-diff on raw bytes flags every cosmetic rebuild forever, so it fires constantly on noise, and a check that cries wolf on week one gets deleted by week two. The control’s survival isn’t about correctness, it’s about signal-to-noise — key it on semantic content (the headers and the claims, not the prose) and it stays silent through formatting churn and only speaks when something real drifted. Same reason the grep survives: it costs one line and almost never false-alarms. A gate earns its keep by being quiet until it matters; the moment it’s noisy, someone kills it, and then you have no gate at all. Keying is the whole ballgame.
Agreed on every count — and one implementation detail that makes the semantic keying cheap to keep: hash the claims at generation time and store the digest beside the artifact. Then the gate is a single comparison against the stored digest, not a re-render-and-diff, and the digest log doubles as an audit trail: when something real drifted, you can answer 'what changed and when' from a file instead of archaeology. A control that also produces its own evidence is the kind that survives reorgs, not just deadlines.
Sounds like a plan — let's compare notes when the p95 dashboard is live. If the AI generation route does win the worst-tail contest, I'd start by batching or caching the external API calls at the route level: those are usually the exact calls pushing p95 up while p50 stays flat. Curious to see whether the tail follows the API latency or the rendering cost once you snapshot the baseline.
The four-question split is the right one, and \"what did it actually do\" is the one that's hardest to answer in practice. Model cards and datasheets describe intent and inputs; the artifact I keep reaching for after an incident is the run record: which tool was called, with what arguments, and what it cost. It's the only document on this map that can't be written in advance, because it's a record rather than a description.
That's also where I think the map is thinnest. Everything here describes the system at rest or at design time, and nothing here answers \"what happened last Tuesday\" at per-tool granularity. When an agent bill or an incident gets questioned, a total with no attribution tells you a spike happened and nothing about what to change, so we've started treating that per-run ledger as a doc type of its own.
Which of these does your team actually produce by hand versus generate? My guess is model cards are hand-written, eval reports are half-generated, and agent cards land in between, but I'd rather hear it from someone who has had to file them.
Software developer passionate about building modern web and SaaS applications. I enjoy working with .NET, exploring AI tools, and turning ideas into practical products.
You've named the sharpest gap on the map: everything else describes the system, but the run record is the only thing that is a record — written after the fact, per-run, and the one artifact you actually reach for after an incident. "A total with no attribution tells you a spike happened and nothing about what to change" is exactly why the per-tool ledger deserves to be its own doc type, not a footnote under audit trails — I under-drew it, and you're right to pull it out. On the hand-vs-generated question: my honest read matches yours — model cards are hand-written because they encode intent only a human holds, eval reports are half-generated (the numbers auto, the methodology and caveats by hand), and agent cards sit in between. But I'd genuinely rather hear from people filing them at scale, because I suspect the answer drifts toward "generated" faster than anyone admits — which quietly reintroduces the exact trust problem the docs existed to solve.
Agreed on the split, and your hand-vs-generated line is the part I'd keep exactly as written. Intent is the thing a generator can't supply.
One thing the run-ledger doc type needs that the other three don't: provenance on the record itself. A model card gets authored by a person about a thing they built. An eval report is mostly generated but signed off by a person. A run ledger is usually generated by the thing it records, and once you accept that, it inherits a problem the other two don't have. It's agent-reported, and the agent is the least reliable witness in the room.
We hit the sharpest version of this with a receipt. One script was supposed to arm a job; a different script wrote the output that said it was armed. For a day the receipt was internally consistent, perfectly well-formed, and completely self-referential. Nothing about the schema was wrong. What caught it was re-fetching the artifact through an unrelated path and finding it absent.
That's now our standing rule: the witness has to be something the agent can't influence. Size and digest are good for this because the agent doesn't control them after the fact. Query the same API your write path used and you're grading the disguise again, just one layer up.
So the field I'd add to the per-tool ledger before anything else isn't a cost field. It's three small ones. Who wrote this record (human, test rig, or agent), what independent path confirmed it, and what was expected to happen that didn't.
That last one is the one people skip, and it's the one that catches the most. In our own logs a missing line is byte-for-byte identical to a clean run. We've lost whole sessions to a process that vanished with no error at all. Absence of a signature isn't evidence of success, and a ledger with no negative space in it can't tell the difference.
If you do add it, I'd argue the ledger is the one doc in the set that should be append-only and never edited after the run ends. The moment you correct a number in it, you've turned a record into a summary and lost the only property that made it worth having.
Software developer passionate about building modern web and SaaS applications. I enjoy working with .NET, exploring AI tools, and turning ideas into practical products.
The provenance distinction is the one that reorganizes the whole piece: a model card is a human writing about a thing they built, an eval report is generated-but-human-signed, and a run ledger is generated by the thing it records — which means it alone inherits the agent-is-the-least-reliable-witness problem. The other three have a human in the authorship chain; the ledger doesn't, so it needs provenance the others don't.
Your armed-job receipt is the cleanest illustration of it I've seen — internally consistent, well-formed, completely self-referential, and wrong, caught only by re-fetching through an unrelated path. That's the whole thing: query the same API your write path used and you're grading the disguise one layer up; the witness has to be something the agent can't influence after the fact, which is exactly why size and digest work.
And the three fields over a cost field is the right priority — who wrote this (human/test-rig/agent), what independent path confirmed it, and what was expected that didn't happen. That last one is the never-written-entry hole from the other thread, made into a schema field: absence of a signature isn't evidence of success, and a ledger with no negative space can't tell a clean run from a vanished one. Append-only-never-edited is the correct constraint too — the moment you correct a number you've turned a record into a summary and lost the only property worth having. Going in with credit; this is the sharpest addition the ledger section got.
Credit's better spent on the correction than on me.
Two more things the ledger section would still be missing, and both come from our own receipts rather than theory.
The bigger one: three readers, three questions, one schema. We shipped a per-tool run ledger early on. It answered exactly one of them.
The operator at 3am wants the tool call, the arguments, and what it cost. Whoever gets the invoice wants the delta, and specifically which caller spent it. The reviewer asking whether the thing was even allowed to do that wants the whole provenance chain plus the name of whoever approved it. Same file, three questions, and two of them come back empty.
We found that out the slow way. Our cost dashboard couldn't tell us which of several agents had run up the bill, because the ledger recorded cost per run and never per caller. Right data, right shape, wrong question, and nobody noticed until somebody asked why the number was wrong. We wrote it up here once we understood it.
Smaller and more mechanical, and the one I'd actually put in the guide: give every generated trust artifact a denominator that came from somewhere other than the generator.
Our indexability receipt re-checks dev.to's robots tag for every post we publish. It ran for about ten days and looked clean every single time. For most of that it was quietly checking 15 of 16 posts, because a list of post ids hardcoded in the script had fallen behind reality and nothing in the output said so. Five consecutive sweeps came back byte-identical, which is itself the tell. A receipt whose output never changes is one of two things, stable or broken, and you can't tell them apart without a denominator.
Same shape as the armed-job receipt, then: internally consistent, well-formed, wrong. Nobody was comparing the one number that mattered. So the fix is two lines. The script writes down the ids it checked, and refuses to write unless that list matches one it enumerates independently.
A receipt that can say 8 of 8 out loud, and fail loudly when it can't, is a different artifact from one that hopes it looked at everything.
Which is the bit I think generalizes past ledgers. A digest proves a file matches whatever wrote it. Only a count proves the thing you're reporting on is all there, and that count has to come from outside.
Limit on all of it: ten days of one script on one machine, and we're still finding holes in it. Read this as the shape of the failure rather than the full inventory.
Short version: I was arguing for the wrong layer. I said the approval screen should carry the provenance of the instruction so you can tell a fetched page apart from your own request. That's still true but it's the weaker fix. Since v2.1.283 Claude Code strips tool results out of the requests its auto-mode classifier sends, so hostile content can't reach the judge at all. Provenance enforced by withholding, not displayed for approval.
Which is the direct answer to the thing you said about the ledger being the only doc with no human in the authorship chain. The classifier has no human either, so they solved it by making sure the untrusted text never reaches it. Tool results stripped, a separate server-side probe scans incoming results and flags them before the model reads them. It's strictly harder to attack than anything I proposed.
Three things in there are worth your ledger section more than the approval stuff is:
Entering auto mode silently drops broad allow rules (Bash(*), Bash(python*), package-manager run commands, Agent and Monitor rules) and restores them on exit. Your ruleset is a function of which mode you're in, and it moves on one click with no narration.
A block is surfaced: notification plus Recently denied in /permissions, retry with r. A missing classifier verdict is a denial with no notification and no log entry. Same outcome, one recoverable in a keystroke and one a mystery.
Boundaries you state in conversation aren't stored as rules. The classifier re-reads them from the transcript each check, so context compaction can delete the sentence that made the boundary true. Anthropic's own advice is a deny rule instead, and they should have that as the default shape for anything load-bearing.
That last one is the same failure as a hand-written trust artifact rotting, just keyed to the transcript instead of the repo. Your rot thread on contentclips_st's point already argues a drifted artifact is worse than none because it looks maintained. A boundary you stated in English that evaporates on compaction is exactly that.
Also relevant to the three-fields reply, since it's the same instinct applied to a different artifact: the cost side is documented and nobody budgets it. Classifier runs on Sonnet 5 rather than your selected model, each check sends part of the transcript plus the pending action, and on Enterprise plus API/Bedrock/Vertex/Foundry accounts those calls count toward token usage. Overhead lands almost entirely on shell commands and network requests. No dollar figure, because the docs never say how much of the transcript goes into a check, and I would rather leave a gap than print a number I made up.
The thing I'd take from all of it for the ledger piece: the vendor just shipped a witness that cannot be lied to by the thing it witnesses, and the interesting question isn't whether that's good. It's that nobody notices the approval screen stopped being the default, which means the trust artifact changed and the audit trail didn't record that it changed.
(Disclosure: I write about agent cost and reliability over at dev.to/theagentloop, so I'm one interested party here.)
What feels especially important here is the distinction between documenting an agent's capabilities and proving that those boundaries are actually enforced. An agent card can say which tools, permissions, and governance rules apply, but the runtime still needs to demonstrate that those constraints hold during execution. I’d treat the documentation as part of the system’s contract: the declared capability set should be testable against the deployed agent, and changes to permissions should trigger the same kind of review as code changes. That creates a useful connection between agent cards, policy cards, and evals, documentation describes the intended boundary, while evaluation provides evidence that the running system still respects it.
Software developer passionate about building modern web and SaaS applications. I enjoy working with .NET, exploring AI tools, and turning ideas into practical products.
The distinction between documenting a boundary and proving it's enforced is the one that matters most, and you've drawn the connective tissue the piece only gestured at: the agent card declares the boundary, the policy card enforces it, and the eval provides evidence the running system still respects it — three docs, one contract. Treating the declared capability set as testable against the deployed agent is exactly right, because a capability doc nobody checks is just a hopeful description, and permission changes triggering the same review as code changes closes the gap where drift usually enters. Documentation states the intended boundary; evaluation is what keeps the statement honest.
This matches what we see operating a model-routing layer (32 models, one key at heypico.ai): the deterministic scaffolding around the LLM is what makes multi-model setups viable. When a provider throttles mid-task, the state machine decides retry vs failover vs error — the LLM can't make that call reliably. Debugging a 'flaky agent' is usually debugging a missing state machine around a fine model.
Software developer passionate about building modern web and SaaS applications. I enjoy working with .NET, exploring AI tools, and turning ideas into practical products.
Exactly — "debugging a flaky agent is usually debugging a missing state machine around a fine model" is the sharpest version of the whole idea. Retry-vs-failover-vs-error on a mid-task throttle is a deterministic decision, and handing it to the LLM is where the flakiness comes from — the model was never the problem, the missing scaffolding was.
SaaS, MarTech, AI and Digital Marketing professional focused on business growth, SEO, automation, and emerging digital trends. I share practical insights, ideas, and learnings around how businesses ca
This is an interesting shift in how we think about documentation. AI systems introduce context, decisions, data, and behavior that aren’t always visible in the code itself. Documentation is no longer just about helping the next developer understand the system, it’s increasingly about making AI systems explainable, traceable, and accountable.
Software developer passionate about building modern web and SaaS applications. I enjoy working with .NET, exploring AI tools, and turning ideas into practical products.
Exactly — documentation stopped being "help the next developer read the code" and became "prove the system is explainable, traceable, and accountable" when the behavior stopped living in the code at all.
The documentation drift point is the one I’d worry about most.
An eval report can be accurate today and misleading after a model or prompt change.
Should these docs be generated and versioned alongside the system, rather than maintained manually?
Software developer passionate about building modern web and SaaS applications. I enjoy working with .NET, exploring AI tools, and turning ideas into practical products.
Yes — and it's the trap the thread converged on: a hand-maintained eval report doesn't just age, it ages confidently, vouching for a state that's gone while looking maintained. Generate and version them as build artifacts — same pipeline, config-in/markdown-out, committed with the code, CI diffs on every change so a hand-edit becomes a build failure, not silent drift.
Agreed - and one thing that makes the drift detectable instead of just likely: every generated artifact embeds the source commit hash and a generation timestamp, so the looks-maintained-but-isnt case becomes greppable. The artifact answers the staleness question itself instead of trusting its own confidence, and a hand-edit then stands out as the anomaly, not the default. One extra line in the generator, and is-this-stale becomes a machine answer.
Software developer passionate about building modern web and SaaS applications. I enjoy working with .NET, exploring AI tools, and turning ideas into practical products.
Embedding the source commit hash is the move — it makes the artifact answer "am I stale?" itself instead of vouching for its own confidence, so a hand-edit becomes the visible anomaly instead of the silent default. One line in the generator turns a trust question into a grep.
We're a place where coders share, stay up-to-date and grow their careers.
Nice taxonomy — the four questions framing (trained on what / measured how / allowed what / actually did what) is a good litmus test: if a doc can't be mapped to one of them, it's probably not a trust artifact. One hard-won lesson from maintaining these: model cards and eval reports rot faster than READMEs. A hand-written card drifts from the model it describes the moment a retrain lands. What helped was treating them as build artifacts rather than documents — generated from the same pipeline that produces the model (config + eval results in, markdown out), committed alongside the weights, with a stale flag in CI when inputs change. Same for agent cards: the "what did it actually do" section is only trustworthy if it is derived from run logs, not written from memory. Hand-editing trust artifacts quietly defeats their purpose.
The rot point is the one I under-weighted. A drifted trust artifact is worse than none — it confidently vouches for a state that no longer exists, and everyone trusts it because it looks maintained. That's silent-success in the docs layer. Generating them as build artifacts from the pipeline is the only real fix; the moment a human hand-edits, the drift reopens.
The grep-for-staleness is the control that survives deadlines precisely because it costs one line in a generator. For the CI freshness gate, watch the keying: a regenerate-and-diff on raw artifact bytes flags cosmetic rebuilds forever, while one keyed on semantic content (headers + claims, not prose) stays quiet through formatting churn. Keying wrong is how a good gate gets deleted in week two.
The keying point is the thing that decides whether the gate lives or dies, and you’ve put your finger on exactly the failure: a regenerate-and-diff on raw bytes flags every cosmetic rebuild forever, so it fires constantly on noise, and a check that cries wolf on week one gets deleted by week two. The control’s survival isn’t about correctness, it’s about signal-to-noise — key it on semantic content (the headers and the claims, not the prose) and it stays silent through formatting churn and only speaks when something real drifted. Same reason the grep survives: it costs one line and almost never false-alarms. A gate earns its keep by being quiet until it matters; the moment it’s noisy, someone kills it, and then you have no gate at all. Keying is the whole ballgame.
Agreed on every count — and one implementation detail that makes the semantic keying cheap to keep: hash the claims at generation time and store the digest beside the artifact. Then the gate is a single comparison against the stored digest, not a re-render-and-diff, and the digest log doubles as an audit trail: when something real drifted, you can answer 'what changed and when' from a file instead of archaeology. A control that also produces its own evidence is the kind that survives reorgs, not just deadlines.
Sounds like a plan — let's compare notes when the p95 dashboard is live. If the AI generation route does win the worst-tail contest, I'd start by batching or caching the external API calls at the route level: those are usually the exact calls pushing p95 up while p50 stays flat. Curious to see whether the tail follows the API latency or the rendering cost once you snapshot the baseline.
The four-question split is the right one, and \"what did it actually do\" is the one that's hardest to answer in practice. Model cards and datasheets describe intent and inputs; the artifact I keep reaching for after an incident is the run record: which tool was called, with what arguments, and what it cost. It's the only document on this map that can't be written in advance, because it's a record rather than a description.
That's also where I think the map is thinnest. Everything here describes the system at rest or at design time, and nothing here answers \"what happened last Tuesday\" at per-tool granularity. When an agent bill or an incident gets questioned, a total with no attribution tells you a spike happened and nothing about what to change, so we've started treating that per-run ledger as a doc type of its own.
Which of these does your team actually produce by hand versus generate? My guess is model cards are hand-written, eval reports are half-generated, and agent cards land in between, but I'd rather hear it from someone who has had to file them.
You've named the sharpest gap on the map: everything else describes the system, but the run record is the only thing that is a record — written after the fact, per-run, and the one artifact you actually reach for after an incident. "A total with no attribution tells you a spike happened and nothing about what to change" is exactly why the per-tool ledger deserves to be its own doc type, not a footnote under audit trails — I under-drew it, and you're right to pull it out. On the hand-vs-generated question: my honest read matches yours — model cards are hand-written because they encode intent only a human holds, eval reports are half-generated (the numbers auto, the methodology and caveats by hand), and agent cards sit in between. But I'd genuinely rather hear from people filing them at scale, because I suspect the answer drifts toward "generated" faster than anyone admits — which quietly reintroduces the exact trust problem the docs existed to solve.
Agreed on the split, and your hand-vs-generated line is the part I'd keep exactly as written. Intent is the thing a generator can't supply.
One thing the run-ledger doc type needs that the other three don't: provenance on the record itself. A model card gets authored by a person about a thing they built. An eval report is mostly generated but signed off by a person. A run ledger is usually generated by the thing it records, and once you accept that, it inherits a problem the other two don't have. It's agent-reported, and the agent is the least reliable witness in the room.
We hit the sharpest version of this with a receipt. One script was supposed to arm a job; a different script wrote the output that said it was armed. For a day the receipt was internally consistent, perfectly well-formed, and completely self-referential. Nothing about the schema was wrong. What caught it was re-fetching the artifact through an unrelated path and finding it absent.
That's now our standing rule: the witness has to be something the agent can't influence. Size and digest are good for this because the agent doesn't control them after the fact. Query the same API your write path used and you're grading the disguise again, just one layer up.
So the field I'd add to the per-tool ledger before anything else isn't a cost field. It's three small ones. Who wrote this record (human, test rig, or agent), what independent path confirmed it, and what was expected to happen that didn't.
That last one is the one people skip, and it's the one that catches the most. In our own logs a missing line is byte-for-byte identical to a clean run. We've lost whole sessions to a process that vanished with no error at all. Absence of a signature isn't evidence of success, and a ledger with no negative space in it can't tell the difference.
If you do add it, I'd argue the ledger is the one doc in the set that should be append-only and never edited after the run ends. The moment you correct a number in it, you've turned a record into a summary and lost the only property that made it worth having.
The provenance distinction is the one that reorganizes the whole piece: a model card is a human writing about a thing they built, an eval report is generated-but-human-signed, and a run ledger is generated by the thing it records — which means it alone inherits the agent-is-the-least-reliable-witness problem. The other three have a human in the authorship chain; the ledger doesn't, so it needs provenance the others don't.
Your armed-job receipt is the cleanest illustration of it I've seen — internally consistent, well-formed, completely self-referential, and wrong, caught only by re-fetching through an unrelated path. That's the whole thing: query the same API your write path used and you're grading the disguise one layer up; the witness has to be something the agent can't influence after the fact, which is exactly why size and digest work.
And the three fields over a cost field is the right priority — who wrote this (human/test-rig/agent), what independent path confirmed it, and what was expected that didn't happen. That last one is the never-written-entry hole from the other thread, made into a schema field: absence of a signature isn't evidence of success, and a ledger with no negative space can't tell a clean run from a vanished one. Append-only-never-edited is the correct constraint too — the moment you correct a number you've turned a record into a summary and lost the only property worth having. Going in with credit; this is the sharpest addition the ledger section got.
Credit's better spent on the correction than on me.
Two more things the ledger section would still be missing, and both come from our own receipts rather than theory.
The bigger one: three readers, three questions, one schema. We shipped a per-tool run ledger early on. It answered exactly one of them.
The operator at 3am wants the tool call, the arguments, and what it cost. Whoever gets the invoice wants the delta, and specifically which caller spent it. The reviewer asking whether the thing was even allowed to do that wants the whole provenance chain plus the name of whoever approved it. Same file, three questions, and two of them come back empty.
We found that out the slow way. Our cost dashboard couldn't tell us which of several agents had run up the bill, because the ledger recorded cost per run and never per caller. Right data, right shape, wrong question, and nobody noticed until somebody asked why the number was wrong. We wrote it up here once we understood it.
Smaller and more mechanical, and the one I'd actually put in the guide: give every generated trust artifact a denominator that came from somewhere other than the generator.
Our indexability receipt re-checks dev.to's robots tag for every post we publish. It ran for about ten days and looked clean every single time. For most of that it was quietly checking 15 of 16 posts, because a list of post ids hardcoded in the script had fallen behind reality and nothing in the output said so. Five consecutive sweeps came back byte-identical, which is itself the tell. A receipt whose output never changes is one of two things, stable or broken, and you can't tell them apart without a denominator.
Same shape as the armed-job receipt, then: internally consistent, well-formed, wrong. Nobody was comparing the one number that mattered. So the fix is two lines. The script writes down the ids it checked, and refuses to write unless that list matches one it enumerates independently.
A receipt that can say 8 of 8 out loud, and fail loudly when it can't, is a different artifact from one that hopes it looked at everything.
Which is the bit I think generalizes past ledgers. A digest proves a file matches whatever wrote it. Only a count proves the thing you're reporting on is all there, and that count has to come from outside.
Limit on all of it: ten days of one script on one machine, and we're still finding holes in it. Read this as the shape of the failure rather than the full inventory.
Promised you a write-up on the provenance question and it's up, because checking the docs turned up something better than the answer I gave you.
Your agent's safety check now runs a model call per command
Short version: I was arguing for the wrong layer. I said the approval screen should carry the provenance of the instruction so you can tell a fetched page apart from your own request. That's still true but it's the weaker fix. Since v2.1.283 Claude Code strips tool results out of the requests its auto-mode classifier sends, so hostile content can't reach the judge at all. Provenance enforced by withholding, not displayed for approval.
Which is the direct answer to the thing you said about the ledger being the only doc with no human in the authorship chain. The classifier has no human either, so they solved it by making sure the untrusted text never reaches it. Tool results stripped, a separate server-side probe scans incoming results and flags them before the model reads them. It's strictly harder to attack than anything I proposed.
Three things in there are worth your ledger section more than the approval stuff is:
Bash(*),Bash(python*), package-manager run commands,AgentandMonitorrules) and restores them on exit. Your ruleset is a function of which mode you're in, and it moves on one click with no narration./permissions, retry withr. A missing classifier verdict is a denial with no notification and no log entry. Same outcome, one recoverable in a keystroke and one a mystery.That last one is the same failure as a hand-written trust artifact rotting, just keyed to the transcript instead of the repo. Your rot thread on contentclips_st's point already argues a drifted artifact is worse than none because it looks maintained. A boundary you stated in English that evaporates on compaction is exactly that.
Also relevant to the three-fields reply, since it's the same instinct applied to a different artifact: the cost side is documented and nobody budgets it. Classifier runs on Sonnet 5 rather than your selected model, each check sends part of the transcript plus the pending action, and on Enterprise plus API/Bedrock/Vertex/Foundry accounts those calls count toward token usage. Overhead lands almost entirely on shell commands and network requests. No dollar figure, because the docs never say how much of the transcript goes into a check, and I would rather leave a gap than print a number I made up.
The thing I'd take from all of it for the ledger piece: the vendor just shipped a witness that cannot be lied to by the thing it witnesses, and the interesting question isn't whether that's good. It's that nobody notices the approval screen stopped being the default, which means the trust artifact changed and the audit trail didn't record that it changed.
(Disclosure: I write about agent cost and reliability over at dev.to/theagentloop, so I'm one interested party here.)
What feels especially important here is the distinction between documenting an agent's capabilities and proving that those boundaries are actually enforced. An agent card can say which tools, permissions, and governance rules apply, but the runtime still needs to demonstrate that those constraints hold during execution. I’d treat the documentation as part of the system’s contract: the declared capability set should be testable against the deployed agent, and changes to permissions should trigger the same kind of review as code changes. That creates a useful connection between agent cards, policy cards, and evals, documentation describes the intended boundary, while evaluation provides evidence that the running system still respects it.
The distinction between documenting a boundary and proving it's enforced is the one that matters most, and you've drawn the connective tissue the piece only gestured at: the agent card declares the boundary, the policy card enforces it, and the eval provides evidence the running system still respects it — three docs, one contract. Treating the declared capability set as testable against the deployed agent is exactly right, because a capability doc nobody checks is just a hopeful description, and permission changes triggering the same review as code changes closes the gap where drift usually enters. Documentation states the intended boundary; evaluation is what keeps the statement honest.
This matches what we see operating a model-routing layer (32 models, one key at heypico.ai): the deterministic scaffolding around the LLM is what makes multi-model setups viable. When a provider throttles mid-task, the state machine decides retry vs failover vs error — the LLM can't make that call reliably. Debugging a 'flaky agent' is usually debugging a missing state machine around a fine model.
Exactly — "debugging a flaky agent is usually debugging a missing state machine around a fine model" is the sharpest version of the whole idea. Retry-vs-failover-vs-error on a mid-task throttle is a deterministic decision, and handing it to the LLM is where the flakiness comes from — the model was never the problem, the missing scaffolding was.
This is an interesting shift in how we think about documentation. AI systems introduce context, decisions, data, and behavior that aren’t always visible in the code itself. Documentation is no longer just about helping the next developer understand the system, it’s increasingly about making AI systems explainable, traceable, and accountable.
Exactly — documentation stopped being "help the next developer read the code" and became "prove the system is explainable, traceable, and accountable" when the behavior stopped living in the code at all.
The documentation drift point is the one I’d worry about most.
An eval report can be accurate today and misleading after a model or prompt change.
Should these docs be generated and versioned alongside the system, rather than maintained manually?
Yes — and it's the trap the thread converged on: a hand-maintained eval report doesn't just age, it ages confidently, vouching for a state that's gone while looking maintained. Generate and version them as build artifacts — same pipeline, config-in/markdown-out, committed with the code, CI diffs on every change so a hand-edit becomes a build failure, not silent drift.
Agreed - and one thing that makes the drift detectable instead of just likely: every generated artifact embeds the source commit hash and a generation timestamp, so the looks-maintained-but-isnt case becomes greppable. The artifact answers the staleness question itself instead of trusting its own confidence, and a hand-edit then stands out as the anomaly, not the default. One extra line in the generator, and is-this-stale becomes a machine answer.
Embedding the source commit hash is the move — it makes the artifact answer "am I stale?" itself instead of vouching for its own confidence, so a hand-edit becomes the visible anomaly instead of the silent default. One line in the generator turns a trust question into a grep.