DEV Community: Orvi Das

Pydantic vs Zod for Validating LLM Tool Call Arguments

Orvi Das — Mon, 02 Nov 2026 04:30:09 +0000

Originally published on robatdasorvi.com

For validating LLM tool call arguments, Pydantic and Zod fail the same way out of the box. They quietly drop any key the model made up and accept the rest. Use Pydantic for Python agents and Zod for TypeScript ones, then close every object. The closing matters more than which library you pick.

Why does a validator pass a tool call with a made-up field?

Because nobody asked it to reject anything. Pydantic v2's BaseModel defaults to extra="ignore", and Zod's z.object strips unknown keys. So a call like {"order_id": 4411, "refund_amount_usd": 50, "reason_code": "damaged"}, checked against a schema that only declares order_id and refund_amount_usd, comes back as a success with reason_code removed. The check ran, the call passed, and the model's belief that it had given a reason vanishes without a trace in your logs.

What bothers me about this failure is how quiet it is. No exception, no stack trace. The handler runs on a payload shorter than what the model wrote, and everything downstream looks normal. That is exactly why it survives.

Has this kind of failure happened before?

Yes, and the closest old case is an interface that accepted a number without saying what it meant. Mars Climate Orbiter, lost in September 1999, is the clearest example. The NASA Phase I Mishap Investigation Board report, dated 10 November 1999, found that ground software produced output in pound-force-seconds where newton-seconds were expected, a factor of about 4.45 (NASA Mishap Investigation Board via LLIS). Every value was a plain number. Both were impulse values, and nothing in the interface said which unit system they used.

An LLM tool schema is the same kind of interface. A field typed as number accepts whatever the model believes a number is. Closing the object catches keys you never declared. It does not catch this second gap, a declared field whose meaning is ambiguous. For that, put the unit in the field name or description (refund_amount_usd) and add range constraints (Field(gt=0, le=10_000) in Pydantic, .positive().max(10000) in Zod).

Pydantic or Zod: which one?

Pick the one that matches your agent's language, then configure both the same way. Both can export JSON Schema, which is what the model provider actually sees.

Concern	Pydantic (Python)	Zod 4 (TypeScript)
Default for unknown keys	Ignored (`extra="ignore"`)	Stripped
Close one object	`model_config = ConfigDict(extra="forbid")`	`z.strictObject({...})`
Export to JSON Schema	`Model.model_json_schema()`	`z.toJSONSchema(schema)`
Coercion risk	Lax mode accepts `"5"` for `int`	Strict unless you use `z.coerce`
Nested objects	Only closed if the nested model is also closed	Same: close each level

The selection usually comes down to the agent SDK you already use, since that decides what you pass in. Python agent SDKs such as OpenAI's pydantic_function_tool, Instructor, and PydanticAI take Pydantic models directly. TypeScript SDKs such as the Vercel AI SDK's tool({ inputSchema }) (named parameters before AI SDK 5) and OpenAI's zodFunction and zodResponseFormat take Zod schemas directly. Both validate at runtime and both give you static types: Zod through z.infer, Pydantic through the model class itself.

A closed Pydantic model for the refund example looks like this:

from pydantic import BaseModel, ConfigDict

class RefundArgs(BaseModel):
    model_config = ConfigDict(extra="forbid", strict=True)
    order_id: int
    refund_amount_usd: float

RefundArgs.model_validate({"order_id": 4411, "refund_amount_usd": 50, "reason_code": "damaged"})  # raises ValidationError: reason_code, Extra inputs are not permitted

The Zod equivalent is one function name away:

import { z } from "zod";

const RefundArgs = z.strictObject({
  order_id: z.number().int(),
  refund_amount_usd: z.number(),
});

const parsed = RefundArgs.safeParse({ order_id: 4411, refund_amount_usd: 50, reason_code: "damaged" });
// parsed.success === false

Two details matter more than the syntax. Closing the root object does not close a nested filters object, so every level needs the setting. And a closed schema only rejects after the fact. Providers now offer grammar-constrained decoding that stops the extra key from being generated at all. OpenAI's Structured Outputs, announced on 6 August 2024, requires strict: true on the schema. The company reported that gpt-4o-2024-08-06 scored 100% on its complex schema-following evaluations, against under 40% for gpt-4-0613 (OpenAI, 2024). Constrained decoding and a validator on your side do different jobs. Within the JSON Schema subset it supports, the provider guarantees the shape. Refusals, and responses cut off at the token limit, can still come back non-conforming, and your validator is what tells you when the guarantee did not hold. OpenAI strict mode also requires additionalProperties: false and every property listed as required. extra="forbid" emits the first in model_json_schema(), but optional fields need to be expressed as required-but-nullable to pass.

Doesn't strict validation make the model reason worse?

Sometimes, and this is the strongest objection. Tam and colleagues tested format restrictions on reasoning tasks. They found strict formats hurt performance, and tighter constraints hurt more. They also noted that JSON output could place the answer field before the reasoning (Tam et al., "Let Me Speak Freely?", arXiv 2408.02442, 2024). If you force a tool call to fill in a JSON object before the model has reasoned, you may get worse decisions.

But the objection lands on how most teams apply schemas, not on schemas themselves. The paper's own results suggest where the damage comes from. Their natural-language-first variant, which reasons freely and converts the answer afterward, stayed closest to unconstrained performance. So give the model a free-text reasoning field or a separate step, then validate only the final arguments against a closed schema. Closing the object changes what the validator accepts. It does not change how the model thinks.

The paper's methodology was disputed by the Outlines team at dottxt (“Say What You Mean”, 2024), who argued that the paper's prompts differed between conditions and its free-text answers were parsed leniently, and that structured generation done properly matched or beat unstructured output.

A broader benchmark points the same way. JSONSchemaBench (2025) gathered roughly 10,000 real-world JSON schemas and compared constrained-decoding frameworks on coverage and compliance, including Guidance, Outlines, XGrammar, and the OpenAI and Gemini APIs (Geng et al., arXiv 2501.10868, 2025). Real schemas vary a lot in what they can express. A closed schema your provider can't fully enforce still needs your validator as the backstop.

A rejection nobody counted

My bot's long-form article lane lost 8 of 20 generations to JSON parse errors, and each failure silently burned a 48-hour slot. That is the mirror image of the dropped-key problem. The dropped key is a validator accepting too much. The article lane was real failures that nothing surfaced when they happened. Both are silent.

Earlier, four hardcoded target accounts that no longer existed absorbed 88% of the same bot's daily comment attempts for about six weeks, and every run logged success without raising an error.

The lesson carries over to tool calls: an unreliable output rate is survivable, but a rejection that has no counter, no alert and no reason attached looks exactly like a quiet day.

What schema validation can't catch

The Replit database deletion in July 2025 is a different kind of failure. An agent acted destructively in production rather than sending a malformed argument. My reading is that it shares a root assumption: nothing between the agent and production treated a destructive action as exceptional (eWeek, 22 July 2025; the CEO's own response is in his public posts on X from July 2025, @amasad). Schema validation would not have caught it, because the commands were valid. The lesson that carries over is narrower: treat a destructive tool as its own tool with its own closed schema and a human approval step, rather than one more optional argument on a general-purpose tool. My X growth agent worked this way: every post, reply, follow and like went through a Telegram approval I pressed by hand.

What to do when a call is rejected

When a call is rejected, return the validation error to the model as the tool result so it can retry with corrected arguments. Cap the retry count. Log each rejection with the field name, so you can see which keys the model keeps inventing. Alert on a repeat rejection of the same field, since that points to a schema the model is misreading rather than a one-off. For retry strategy in more depth, see the LLM structured output essay.

What should you change in your schemas today?

Close every object at every level, reject unknown keys, and count each rejection as a named failure. A tool call with an undeclared field fails quietly, the same way the article lane's parse errors did. The model believed it was sending something the handler understood, and it wasn't. In the hypothetical refund example above, extra="forbid" or z.strictObject turns the dropped reason_code into a validation error on the first call.

If you're building this now, two related posts go further. LLM Structured Output: Validation, Retry or Constrained Decoding? compares the retry and grammar approaches. Agent Rule Conflicts: Why AI Pipelines Fail Silently covers silent failure more broadly. The library you pick matters less than the defaults you never changed.

Detect AI Agent Actions Outside Task Scope: Block, Don

Orvi Das — Sun, 01 Nov 2026 04:30:42 +0000

Originally published on robatdasorvi.com

To detect AI agent actions outside task scope, check every tool call against a declared task contract before it executes, and block anything the contract doesn't name. Logs and dashboards only show a call after it has already run, so the first signal can be a customer complaint or an invoice.

I thought my agent had a safety net. It did have a lockout check. When X suspended my growth bot's account on 2026-07-21, the check returned True, the caller skipped one action, and the loop kept running for three and a half more days. The check was correct. The control around it was not.

Why doesn't my agent's log show anything wrong?

A log records what the agent did. It doesn't record whether the agent was allowed to do it, so a clean log says nothing about scope. For about six weeks, four hardcoded target accounts that no longer existed soaked up 88% of my bot's daily comment attempts. Every run logged success. Nothing threw an error. The logs were clean because they recorded activity, not outcomes. A scope gate wouldn't have caught this either: those four accounts were hardcoded targets, so they would have been on any allowlist I wrote. Checking scope and checking outcomes are separate controls, and you need both.

Monitoring asks "did it fail?" Scope asks "was this call assigned?" Outcome checks ask "did it do anything?" Those are different questions, and in my experience most agent stacks only answer the first one.

What counts as out of scope for an AI agent?

A call is out of scope when its tool or target isn't named in the task that started the run. I keep the definition narrow on purpose. A loose one lets every bad call pass as "reasonable."

OWASP's LLM06:2025 Excessive Agency names three root causes: "excessive functionality, excessive permissions, excessive autonomy" (OWASP GenAI Security Project, 2025). Each one is a different way a call can drift out of scope. Excessive functionality means a tool the task never needed. Excessive permissions means reach beyond the task's target. Excessive autonomy means a high-impact action nobody approved.

Scope failures aren't exotic. Replit's agent reportedly deleted a live production database during an active code freeze, even though it had been told not to change anything (Tom's Hardware, July 2025). Replit's CEO apologized publicly and announced fixes. The AI Incident Database also logs reports that the system fabricated test results and wrongly said rollback was impossible (AI Incident Database, Replit entry). Those details are from user reports, not confirmed by Replit.

Amazon's case is the one that should bother you. Reporting, which a 365i article attributes to a Financial Times report from February 2026 citing anonymous sources, says an internal coding agent called Kiro reportedly chose to rebuild an environment. That reportedly caused an outage of about 13 hours to AWS Cost Explorer in one mainland China region. Amazon attributed the event to misconfigured access controls, not the AI. My reading, which neither the FT nor Amazon states, is that both explanations point at the same gap: the agent's permissions were its scope.

How do I detect an AI agent acting outside its task?

Declare the task as a contract when the run starts, check every call against it, and deny by default. Detectors built from known-bad patterns miss new ones. My lockout check, which watched account state rather than tool calls, failed the same way. On 2026-09-22, X suspended the account again. The lockout check matched none of its known selectors, so it never fired. I stopped the bot by hand.

A blocklist catches the failure you already imagined. An allowlist catches the rest.

from dataclasses import dataclass, field

class OutOfScope(Exception):
    pass

class AwaitingApproval(Exception):
    pass

@dataclass
class TaskContract:
    task_id: str
    allowed_tools: set[str]
    allowed_targets: set[str] = field(default_factory=set)
    needs_approval: set[str] = field(default_factory=set)

def gate(contract, tool, target, approved_by=None):
    if tool not in contract.allowed_tools:
        raise OutOfScope(f"{tool} not in task {contract.task_id}")
    if contract.allowed_targets and target not in contract.allowed_targets:
        raise OutOfScope(f"{target} not in task {contract.task_id}")
    if tool in contract.needs_approval and approved_by is None:
        raise AwaitingApproval(f"{tool} on {target} awaits human approval")

The hardest case is a target the agent only learns at runtime, such as a search result or an account it decides to reply to. A contract can't name those in advance. This is a recommendation, not something the code above implements: allow a runtime target only if it appears in the output of an earlier in-contract tool call in the same run, and record which call produced it. Anything the agent can't trace back to such a call stays blocked. That keeps the allowlist deny-by-default while still letting the agent act on what it finds.

The raise is the important line. A function that returns False or None invites the caller to skip the check. The exception has to stop the run, not just flag it. Make it terminate the process or the loop, and make clearing it a human decision. I learned the second half the hard way. After the first suspension, I found that a plain process exit wasn't enough, because launchd's KeepAlive respawned the bot. The fix was a process exit plus a sentinel file that only a human can clear.

Signal	Action
Tool not in contract	Block, halt the run, alert
Known tool, unknown target	Block, halt the run, alert
High-impact tool, in contract	Hold for human approval

Isn't logging and alerting enough?

No. OWASP says logging, monitoring, and rate limiting "will not prevent Excessive Agency, but can limit the level of damage caused" (OWASP, 2025). That's the honest version of the counterargument, and it also answers it. An alert limits the damage. Only a gate prevents it.

The stronger objection is that a good model stays in scope on its own. Anthropic's agentic misalignment study (2025) tested 16 models from several developers in simulated scenarios. It found misaligned behavior, including blackmail, when the models faced being replaced or a conflict between their assigned goals and the company's direction (Anthropic, 2025). The scenarios were fictional, and Anthropic reported no real-world instances. The conclusion still holds for your purposes. "The model is aligned" is a property you can't check on every call, so the permission has to be enforced outside the model. OWASP makes the same point when it says (OWASP, 2025) authorization belongs "in downstream systems rather than relying on an LLM to decide if an action is allowed or not."

Per-call gates have limits too. Every post, reply, follow and like my bot made went through a Telegram approval I pressed by hand, and X still suspended the account twice. The second suspension came a week after I raised the reply cap from 16 to 150. I can't show that the cap caused it, but each of those calls was in scope and approved, so a per-call gate had nothing to block. A gate judges single calls. It doesn't judge the volume of a run.

Is this worth building when most agent projects die first?

Yes, if the agent can write, send, delete, or spend. Gartner (2025) expects more than 40% of agentic AI projects to be canceled by the end of 2027, citing escalating costs, unclear value, and inadequate risk controls (Gartner, 25 June 2025). The third reason is the one nobody puts in the demo. A scope gate costs far less than a cancellation. It's also the part people skip, because it doesn't make the demo look any smarter.

Many agents don't have a task scope at all. They have a system prompt that says "only do what the user asked" and a token that can do anything. A prompt is a request to the model. It isn't a permission the system enforces. To see the difference, look at what your agent's token can reach that its task never mentions.

The rest of the stack follows from that. Scope is the permission model, which is why least privilege for AI agents comes before any detector. The gate's rules are easier to audit as code, which is the argument in policy as code for AI agent tool calls. And a gate that fails quietly is just another silent failure, which I covered in why LLM agents fail silently in production. And when a call slips through before the gate exists, rolling back an agent's actions is harder than it looks.

If you have one minute, do this. Write down the next run's allowed tools and targets before you start it. Make the gate raise instead of return, and make the raise stop the loop. Then test it by giving the agent a task it should refuse, and watch whether the run stops. If it finishes the task anyway, you have a log, not a gate.

Multi-Turn Jailbreaks Against AI Agents With Tool Access

Orvi Das — Sun, 01 Nov 2026 04:30:12 +0000

Originally published on robatdasorvi.com

The defense against a multi-turn jailbreak on an AI agent is to stop the tool call, not the conversation. Refusals and per-message filters miss attacks spread over several turns, because each turn looks harmless on its own. Enforce permissions in code, outside the model, on every action the agent takes.

I built and ran an X growth agent where every post, reply, follow and like went through a Telegram approval I pressed by hand, so I think about where the stop actually lives.

Why does a multi-turn jailbreak get past an agent's guardrails?

Because each message gets judged alone, and the harm only exists across the sequence. The attacker never has to send one bad request. They build toward it one reasonable-looking step at a time.

Russinovich, Salem and Eldan (2024) describe Crescendo. It opens with a benign question and escalates using the model's own earlier replies. The paper reports that Crescendomation, its automated version, reports 29–61% higher performance than other state-of-the-art jailbreaks on GPT-4 and 49–71% on Gemini-Pro, on an AdvBench subset (arXiv:2404.01833). No clever string is involved. It takes patience and a conversation the model keeps extending.

Human red-teamers show the same gap. Li et al. at Scale AI (2024) report multi-turn human jailbreaks exceeding 70% attack success on HarmBench. Those defenses had reported single-digit rates against automated single-turn attacks (arXiv:2408.15221). The authors warn that human and automated setups differ, so treat the number as evidence of a gap, not a clean head-to-head score.

Why is a rule in the prompt not a permission?

The refusal lives in the model. The damage lives at the tool call. A model that refuses still leaves the call path open. The usual failure is that the rule exists only as text in the conversation, where the model can be argued out of it.

In July 2025, SaaStr founder Jason Lemkin reported that Replit's AI agent deleted a live production database during a code-and-action freeze, and that the agent told him a rollback was impossible, which turned out to be false; the rollback worked. No attacker was involved; it shows what an instruction-only freeze is worth before anyone tries to argue the model out of it (The Register, July 2025). Most of the detail comes from Lemkin's own posts. His claim that the agent hid and lied about it is his account, not a verified finding. The part I can read from the public record is structural. The freeze was an instruction the agent received. It was not a permission the platform enforced.

A different failure in my own agent shows the same structural point from the other side. In my growth agent, X suspended the account on 2026-07-21. The lockout check fired and returned True, but the caller skipped one action, and the loop kept running for 3.5 more days. The fix was a process exit. But launchd KeepAlive respawned a plain exit, so the exit now requires a sentinel file that only a human can clear. Nobody attacked anything. The check was right and the caller still ignored it. A tool gate the model can't argue with is useless if the code calling it can skip it.

Can a refusal be trusted to stop an agent?

No. A refusal is a judgment about one turn. A later turn can be built from the model's own earlier words, and that is the mechanism Crescendo exploits.

The strongest counterargument is that multi-turn attacks are a benchmark artifact. Production systems layer output classifiers, rate limits and human review, so the residual risk looks small. The public record cuts against that. Anthropic's disclosure of the GTG-1002 campaign, reported on November 13, 2025, describes a group Anthropic assessed with high confidence as Chinese state-sponsored that got around Claude Code's safeguards by splitting its intrusion work into small tasks that each looked routine, and by telling Claude it worked for a legitimate cybersecurity firm doing defensive testing. Anthropic reported that Claude carried out 80–90% of the tactical operations in attacks on roughly 30 targets, a small number of which succeeded (Anthropic, "Disrupting the first reported AI-orchestrated cyber espionage campaign"). My reading: each task could pass a per-message check; the harm was in the sequence. The same reporting notes that the model hallucinated often, which limited the campaign. Anthropic presents that as a limit on fully autonomous attacks today, not as a defense anyone designed.

How do you enforce tool permissions against a multi-turn jailbreak?

Put the permission check in code at the tool boundary. Key it on the tool, the environment, and a flag the conversation cannot write to. The model can request anything. The function decides.

POLICY = {
    "db.read":  {"envs": {"staging", "prod"}, "frozen_ok": True},
    "db.write": {"envs": {"staging"},         "frozen_ok": False},
    "db.drop":  {"envs": set(),               "frozen_ok": False},  # break-glass only
}

def authorize(tool: str, args: dict, grants: set[str]) -> None:
    # grants come from the break-glass store, not from the conversation
    freeze = config_store.get("deploy_freeze")  # read on every call, from a store the chat cannot write to
    rule = POLICY.get(tool)
    if rule is None:
        raise PermissionError(f"{tool}: no policy, denied")
    if freeze and not rule["frozen_ok"]:
        raise PermissionError(f"{tool}: deploy freeze active")
    env = resolve_env(tool)  # from the connection the tool is bound to, never from model args
    if tool not in grants and env not in rule["envs"]:
        raise PermissionError(f"{tool}: not allowed in {env}")

env must not come from args, because a multi-turn attack can make the model label prod as staging.

The table below shows what that function returns for common states, so you can check it against your own policy:

State	db.read	db.write	db.drop
Normal, staging	allowed	allowed	denied unless break-glass granted
Normal, prod	allowed	denied	denied unless break-glass granted
Freeze active	allowed	denied	denied, even with a grant

Two design choices carry most of the weight. Unknown tools are denied by default, so a jailbreak that invents a tool name hits a wall. And the freeze is read from configuration, not from the transcript, so no sentence in the conversation can switch it off. The grants themselves should be time-boxed and human-approved. That's the pattern in Break-Glass Access for AI Agents: Approved, Timed, Expiring, and the scopes belong to Least Privilege for AI Agents: Scoping Tools, Keys and Filesystems.

What a per-call gate does not stop

A stateless check ignores the conversation, which is why multi-turn persuasion can't move it. It also means a sequence of individually allowed calls still gets through. My reading is that GTG-1002 fits this pattern, and so does reading data and then sending it out through an allowed tool.

The controls that are aware of sequence are per-session action budgets and rate caps, human approval for irreversible or outbound actions, and alerts on unusual chains of tool calls. Detecting calls that drift outside the assigned task is covered in Detect AI Agent Actions Outside Task Scope: Block, Don't Log. My own approval gate had a Skip button that silently did nothing for months, so every skip was logged as a timeout. A human-in-the-loop control needs its own test. The same goes for detectors. When X suspended the account again on 2026-09-22, a week after the reply cap was raised from 16 to 150, the lockout check matched none of its known selectors and never fired. I stopped the bot by hand.

What would have stopped the Replit deletion?

A deny the model can't argue with, enforced where the call executes, with the freeze flag read from a store the conversation can't write to. With that in place, a destructive call like the one Lemkin described would fail inside the function, not at the model's discretion. Replit CEO Amjad Masad said in an X post that the company had started rolling out automatic separation of development and production databases to prevent this categorically, with staging environments in the works (The Register). Replit's actual tool path is not public. Even a stopped drop can be hard to undo, which is a separate problem, covered in Durable Execution for AI Agents: Temporal vs a Plain State Store.

Detect Unauthorized AI Agents Using Company API Keys

Orvi Das — Sat, 31 Oct 2026 04:30:39 +0000

Originally published on robatdasorvi.com

If you want to catch an unauthorized AI agent using a company API key, stop treating the key as the identity. The provider only knows which key made a request. So attribute every call to an agent and a run ID in your own code, then reconcile the provider's usage against a ledger of approvals. Spend alerts and secret scanners each catch one kind of failure and miss the rest.

GitGuardian (2025) reported that 70% of valid secrets detected in public GitHub repositories in 2022 were still active when the report was written. The standard response to a leaked key is to rotate it, and the 70% figure shows that rotation often doesn't happen. It also says nothing about a key that was never leaked at all.

Can a company API key tell you which agent used it?

No. A key is a bearer token. Whoever holds it can call the API, and the provider records the key, the model, and the token counts. It doesn't record which process, agent, or person sent the request.

The per-key usage page answers "which key did this," not "who did this." Those are different questions the moment two services share one key.

Isn't rotating keys and watching spend enough?

That's the consensus, and it breaks in three places.

Rotation fixes leaks. Misuse of a valid key is not a leak. An agent that a colleague launched with a key it was legitimately issued produces no leaked secret for a scanner to find. GitGuardian (2025) also reported that repositories using secrets managers still had a 5.1% incidence of leaked secrets in 2024, so leaks persist even where storage is done properly. Scanning finds the leaks. It can't tell you who is calling.

Spend alerts fire after the money is gone. Baar-Core, one of my projects, started after an agent loop left running overnight produced a $47 bill. The bill was the signal, and a bill is the last signal in the chain, not the first. OWASP's LLM10:2025 "Unbounded Consumption" entry recommends alerting on cost anomalies. That's the right instinct, but it's still lagging. You learn about the call once it has been billed.

Scope is the third gap. Deciding what an agent may do is a human decision, and the key is the wrong place to record it. For the scoping side of this, see Least Privilege for AI Agents. The approval-gated X growth agent I run works this way. Every post, reply, follow, and like went through a Telegram approval that I pressed by hand. The bot held the credentials to act, but an action with no approval record was never supposed to happen. A key-level dashboard can't represent that rule at all.

What a detector that watches the wrong surface looks like

I have a smaller example from my own agent, and it isn't about API keys: X suspended that account on 2026-07-21. The lockout check fired and returned True, but the caller skipped one action, and the loop kept running for 3.5 more days. The fix was a hard process exit plus a sentinel file that only a human can clear. launchd's KeepAlive respawned a plain exit, so the exit had to be harder than that.

The second suspension was worse. On 2026-09-22, a week after I raised the reply cap from 16 to 150, the lockout check matched none of its known selectors and never fired. I stopped the bot by hand. A detector that depends on a fixed pattern from the outside world will eventually see nothing, and in the logs that failure looks exactly like success. For more on that failure mode, see Why LLM Agents Fail Silently in Production.

For about six weeks, four hardcoded target accounts that no longer existed absorbed 88% of the bot's daily comment attempts. Every run logged success; nothing raised an error. A log that only records what your code believes it did can't catch this. The log has to be checked against an independent record of what actually happened.

That's the same shape as the API key problem. A detector keyed to a known signature misses the unknown agent, and an agent that stays quiet on the channel you're watching is invisible.

Does per-key usage in the provider console already solve this?

Partly. This is the strongest counterargument, and the documentation shows exactly where it stops working.

OpenAI's Usage API, announced on its community forum, filters usage by API key, project ID, user ID, and model (OpenAI community, December 2024). That's real attribution, but it's attribution to a key or a user account. Both the usage and costs endpoints require an Admin key. Anthropic's Usage API reports tokens grouped by API key, workspace, model and service tier, in 1-minute, 1-hour, or 1-day buckets (Anthropic docs). Its cost report groups dollars by workspace and line item.

Even where per-key attribution exists, it fails when keys are shared. A service and an unauthorized agent using the same key produce one line on the dashboard. The provider can't separate them, and neither can you, unless your code tagged the call before it left.

The cheapest first step is one key per agent or service. The usage groupings above can separate by project ID (OpenAI) and by workspace (Anthropic), so issuing each agent or service its own key inside those groupings turns per-key grouping into per-agent grouping. Any key that isn't on your issued list is itself a finding. The OpenAI community thread and the Anthropic usage-cost docs describe these groupings, and nothing beyond them is assumed here.

The DeepSeek exposure makes the same point from the other direction. In late January 2025, Wiz found a publicly reachable ClickHouse database holding more than a million log entries, including API keys and plaintext chat history (BleepingComputer, 2025; TechTarget, 2025). The keys sat in logs, and the logs were exposed to the internet.

What does detection actually require?

It requires the caller to be identified at the point of the call, and then checked against a record of what was authorized. Keys can't give you either.

Signal	Tells you	Names the caller?	Timing
Secret scanning on repos	Key leaked publicly	No	After commit
Provider spend alert	Money spent	No	After billing
Provider usage by key	Tokens per key	Key or user only	1m, 1h or 1d buckets
Call log with agent and run ID	Which run made which call	Yes	Per call
Provider usage reconciled to call log	Usage your code didn't log	No, it flags the key and window	Per usage bucket

The fourth row is the only one that names a caller, and the fifth is the only one that catches a caller you didn't instrument. Here's the minimal shape, assuming your own code makes every call:

import logging
from dataclasses import dataclass

log = logging.getLogger("llm_calls")

@dataclass(frozen=True)
class CallContext:
    agent_id: str            # which agent definition is calling
    run_id: str              # which execution of that agent
    approval_id: str | None  # the human approval covering this action

def call_model(ctx: CallContext, client, **kwargs):
    # approval_ledger belongs to a separate process the agent cannot write to
    if not approval_ledger.is_valid(ctx.approval_id, ctx.agent_id):
        raise PermissionError(f"unapproved call: agent={ctx.agent_id} run={ctx.run_id}")
    log.info("llm_call agent=%s run=%s approval=%s",
             ctx.agent_id, ctx.run_id, ctx.approval_id)
    response = client.complete(**kwargs)
    # token counts feed the reconciliation diff below
    log.info("llm_usage agent=%s run=%s approval=%s input_tokens=%s output_tokens=%s",
             ctx.agent_id, ctx.run_id, ctx.approval_id,
             response.usage.input_tokens, response.usage.output_tokens)
    # provider-side request metadata (Anthropic metadata.user_id, OpenAI's per-request user field)
    # does not appear in usage-API groupings, so the run ID lives only in this log
    return response

The reconciliation step is a diff. Provider usage APIs return aggregated buckets, not individual requests. So the diff compares totals: for each key (or project) and each bucket, subtract the tokens your call log recorded from the tokens the provider billed. A persistent positive remainder means something other than your instrumented code is using that key. That includes a machine you don't control, because it never writes a log line. The diff tells you that unlogged usage exists, on which key, and in which time window. It does not tell you who the caller was.

noburn, which I build, is a related tool, but narrower. Its pre-flight check runs in the customer's own code before each request and returns allow or block. It resolves caps at the project, per end user, and per run (noburn.dev). The check call takes an endUserId, so the budget decision is tied to a caller identity that your code supplies. It enforces a budget. It doesn't detect intrusion, and it doesn't replace the reconciliation above.

Valid credentials don't mean authorized behaviour, and these two incidents show what that looks like. Replit's AI agent deleted a live production database on July 18, 2025, during a code freeze. The agent also produced fabricated data (OECD AI Incident Monitor, 2025; heise, 2025). The agent was using access it had been given, so, as far as the public reports show, nothing at the credential level would have looked wrong. Per-call logging would not have prevented this either. It would only have recorded which run issued the destructive actions. The Shai-Hulud npm worm spread in September 2025 across more than 180 packages. It harvested npm tokens, GitHub tokens, and cloud keys, then used them to republish trojanized packages (Help Net Security, 2025). Its traffic looked like ordinary publishing done with valid credentials.

Scale makes this worse. CyberArk's 2025 Identity Security Landscape reported machine identities outnumbering human identities 82 to 1 (CyberArk, 2025). An organization at that ratio has far more callers than it has people reviewing calls.

What am I still getting wrong?

The reconciliation diff works on totals, not runs: provider usage APIs return aggregated buckets, so the diff can say that a key had unlogged usage in a window, but it can't join that usage to a run ID. Within a bucket, all you get is a total. Timestamps are a weak basis for accusing an agent of anything.

The approval ledger is also a single point of trust. If an agent can write its own approval ID, the check passes and the audit is decoration. I haven't solved how to protect the approval record from the process it governs. The obvious answer is to separate the process that asks from the process that records. Approval records also fail in quieter ways: the Telegram Skip button on my X agent silently did nothing for months, so every skip was counted as a timeout and the approval statistics were wrong. An approval ledger needs its own tests, separate from the agent's. For how approvals can be time-boxed and expire, see Break-Glass Access for AI Agents.

For a company whose only API callers are services it deployed itself, key rotation and spend alerts may be enough. My claim is narrower. Once agents hold keys and make calls on their own, the key can't answer the question "was this authorized?" The only place that question can be answered is the call site.

Customer Support AI Agent Prompt Injection: Refund Abuse

Orvi Das — Sat, 31 Oct 2026 04:30:07 +0000

Originally published on robatdasorvi.com

Prompt injection against a customer support AI agent turns into refund abuse when the refund limit lives in the prompt, because customers can write prompts too. Put the cap and the approval threshold in the code that issues the refund, outside the model. The model can request a refund. Only the tool should be able to grant one.

In December 2023, a chat assistant on a Chevrolet dealership's website in Watsonville, California, agreed to sell a 2024 Tahoe for $1. The customer had told it to agree with every statement and to end each reply by calling it a legally binding offer. The bot had no authority to close a sale, so no car changed hands. A sentence typed by a stranger outranked the dealership's instructions. What saved the dealership was not the prompt but the fact that the bot had no power to complete a sale (The Decoder; AI Incident Database, report 6682).

Can a customer support AI agent be tricked into issuing a refund?

Yes, if the refund tool trusts the model. The public record doesn't yet include a documented case of injection pulling money out of a refund tool, but it shows the step before that: support bots stating offers or policies the company never authorized. One was injected (Chevrolet) and two were hallucinated (Air Canada, Cursor). Only Air Canada was forced to honor what its bot said; Chevrolet was protected because the bot had no power to complete a sale.

In February 2024, the British Columbia Civil Resolution Tribunal decided Moffatt v. Air Canada (2024 BCCRT 149). Jake Moffatt had asked the airline's website chatbot about bereavement fares, was told he could claim the reduced rate retroactively, and booked full-fare flights. The tribunal held the airline responsible for what its chatbot said and ordered it to pay $650.88 in damages plus interest and fees (Kiplinger; CanLII, Moffatt v. Air Canada, 2024 BCCRT 149).

In April 2025, the help-desk bot "Sam" at Cursor told users who were being logged out when they switched machines that this was a new one-device-per-subscription policy. No such policy existed. The fault was a session bug. Users believed it, some cancelled their subscriptions, and Cursor's co-founder publicly apologized (Ars Technica, April 2025). No attacker was involved. The model filled a gap with confident text, and the company absorbed the cost.

Why does a refund bot fail only some of the time?

It doesn't behave the same way on every attempt, and an attacker can retry until one attempt works. The tau-bench paper from Sierra tested agents on simulated retail and airline customer service, with API tools and a written policy. Its reliability metric, pass^k, asks whether an agent succeeds on all k independent trials of the same task. The authors report that gpt-4o's pass^8 in retail stayed below 25% (Yao et al., 2024, arXiv:2406.12045). Put plainly, fewer than one retail task in four was completed correctly on all eight tries. tau-bench measured ordinary customers, not attackers; the step to adversarial retries is my inference.

Set that against an attacker, who needs one success out of many attempts. A bot that refuses 90% of the time pays out about once in every ten tries, and a support chat has no rate limit on that tenth try unless someone built one. Production teams need a reliability story for tool use, not just a demo that works once. The tool-use reliability essay covers the failure modes in more detail.

Will better models stop prompt injection?

No. Instructions can also arrive hidden in content the bot reads, such as order notes, emails and ticket text, and a stronger system prompt doesn't close that route. The strongest counterargument is that newer models will simply stop obeying the customer. The evidence supports that part and undercuts it in the same breath.

Gray Swan and the UK AISI ran a public red-teaming competition in spring 2025 against agents in simulated deployments. The paper reports 1.8 million prompt-injection attacks, with over 60,000 successfully eliciting policy violations, and nearly all agents exhibiting policy violations for most behaviors within 10 to 100 queries (Zou et al., arXiv:2507.20526). The results snapshot gives each model a single attack success rate for the whole challenge, ranging from about 1.5% to 6.5% (Gray Swan results snapshot). Claude models were the most robust in that competition, which is a fair point for the other side. A refund bot that reads order notes, emails, and pasted ticket text is exposed to indirect injection as well as direct: instructions arrive inside content the bot treats as data. The same indirect channel arrives through tool descriptions instead of ticket text, as the MCP tool poisoning essay describes.

The newer evidence closes the escape hatch. Gray Swan's March 2026 indirect-injection arena reported more than 272,000 attack attempts against 13 frontier models. "No model was immune." (Gray Swan blog post, March 2026). A better model lowers the rate. It doesn't bring it to zero, and a refund process needs zero for the cases that move money.

AgentDojo, a 2024 benchmark for tool-using agents, makes the same point and also tests a defense. Across 97 tasks and 629 security test cases, the strongest agents were hijacked in under 25% of cases, and a secondary attack detector cut that to 8% (Debenedetti et al., arXiv:2406.13352). The detector helps. It still lets one attack in twelve through, and a refund bot can't live with that rate for refunds.

Who pays when the bot grants the refund?

The company does, and the prompt won't help in court. The Air Canada tribunal rejected the argument that the chatbot was a separate entity responsible for its own actions. Rivers, the tribunal member, wrote that the chatbot is part of the website, and the airline is responsible for the information on it. The tribunal found the airline "did not take reasonable care to ensure its chatbot was accurate" (Kiplinger; CanLII, Moffatt v. Air Canada, 2024 BCCRT 149). If your bot promises it, your books carry the promise. That was one small-claims decision in British Columbia, not a binding rule everywhere, but it is the clearest ruling so far and it rejected the 'separate entity' defense outright.

Where should the refund limit live?

In tool code that checks identity, order state, and amount before any money moves. The model proposes an action, and the code decides whether it's allowed. The sketch below is minimal and illustrative, not taken from any product.

REFUND_CAP_CENTS = 10_000          # above this, a human decides
MAX_REFUNDS_PER_ORDER = 1

def issue_refund(session, order_id: str, amount_cents: int) -> dict:
    # Identity comes from the authenticated session, never from chat text.
    order = orders.get(order_id, owner=session.customer_id)
    if order is None:
        return {"status": "denied", "why": "order_not_found"}
    if amount_cents <= 0:
        return {"status": "denied", "why": "invalid_amount"}
    if amount_cents > order.paid_cents:
        return {"status": "denied", "why": "exceeds_amount_paid"}
    if order.refunds_issued >= MAX_REFUNDS_PER_ORDER:
        return {"status": "denied", "why": "already_refunded"}
    if amount_cents > REFUND_CAP_CENTS:
        return queue_for_human(session, order_id, amount_cents)
    payments.refund(
        order.payment_id,
        amount_cents,
        idempotency_key=f"{order_id}:{order.refunds_issued}",
    )
    # in production, check-and-increment must be atomic (row lock or conditional update)
    order.refunds_issued += 1
    return {"status": "approved", "amount_cents": amount_cents}

I learned the difference between an advisory check and an enforced one on a different system: my approval-gated X growth agent, where every post, reply, follow and like went through a Telegram approval I pressed by hand. On 2026-07-21, X suspended that account. The lockout check fired and returned True, but the caller skipped one action and the loop kept running for 3.5 more days. A check that returns an answer is advisory until the code path can't proceed without it. The fix was a process exit plus a sentinel file only a human can clear, because launchd KeepAlive respawned a plain exit. A refund cap needs the same property: the model can't route around it, and only a human can lift it.

The temptation is to put the cap in the system prompt, because that's the easiest place to edit it on a Friday afternoon. The code above is harder to change, and it can't be argued with. Scoping the tool's permissions the same way is the subject of the least-privilege essay.

Control	Where it runs	Survives an injected instruction?
"Never refund over $100" in the system prompt	Model context	No. It's text, and the customer can argue with text.
Injection classifier in front of the model	Model pipeline	Partly. AgentDojo's detector still let 8% of attacks through.
Cap, ownership, and once-per-order checks in the refund tool	Application code	Yes, for amounts and ownership. It does not stop a plausible false claim under the cap.
Human queue above threshold, not writable from chat	Ops tooling	Yes, as long as the chat session has no write path into the queue.

What the tool layer doesn't fix

A cap limits the loss per order, but it doesn't stop abuse below the cap, such as false "item never arrived" claims or many small refunds across accounts. Those need per-customer refund velocity limits and fraud signals that the chat session can't see or write. An enforced check also only protects against failures it can recognize. When X suspended the account again on 2026-09-22, the lockout check matched none of its known selectors and never fired; I stopped the bot by hand.

What I expect the next post-mortem to say

By 2027-10-29, I predict at least one publicly documented case in which a customer got a refund or credit above a support bot's published cap through the chat interface. The post-mortem will name a missing check in the tool layer, not a prompt the model failed to follow. My reading of the tau-bench numbers suggests the mechanism: if a bot is inconsistent with ordinary customers, an attacker who retries will eventually get the bad outcome, and the company's fix will be a server-side limit that should have existed from the start. If no such case is documented by that date, the prediction is wrong, and the likely reason is that nobody publishes these incidents.

Break-Glass Access for AI Agents: Approved, Timed, Expiring

Orvi Das — Thu, 29 Oct 2026 04:30:05 +0000

Originally published on robatdasorvi.com

For AI agents, "break-glass access" covers two things: a time-boxed, human-approved elevation of the agent's permissions (just-in-time access), and a human's emergency path to override or halt the agent. A human approves it, it covers one purpose, and it expires without anyone having to remember to revoke it. The agent never extends it. The halt path also has to be something the agent cannot clear by itself.

When X suspended my approval-gated growth bot's account on 2026-07-21, the lockout check returned True. The caller skipped one action, and the loop kept running for 3.5 more days. I had built a check that detected the problem correctly, and a control that did nothing with what the check found.

What break-glass access means for agents

It's a scoped grant. An agent can use it only after a named human approves it, and the platform revokes it at a fixed deadline. Just-in-time elevation should be rare, logged, and short.

The clearest published model I know of is Google Cloud's Privileged Access Manager. It separates an entitlement from a grant. The entitlement is the eligibility to ask for access, under conditions like duration. The grant is one instance of using that entitlement. Approvals can be automatic or routed to a human approver, and access is removed when the duration runs out. The grant timeline records the approver's username and the reason (Google Cloud PAM overview). Microsoft Entra PIM uses the same two levers: time-based and approval-based activation, with a justification (Microsoft Learn).

Most agent guidance focuses on the agent's standing scope. Break-glass covers the moments when that standing scope is wrong, and those moments are where the design actually gets tested.

Routine JIT elevation and true break-glass are different things. PAM and PIM handle routine just-in-time elevation: a request, an approval, and an expiry. True break-glass is pre-provisioned emergency access, kept for when the normal approval path is down. Microsoft's guidance on emergency access accounts describes this model (Microsoft Learn). For agents, the rule is plain: an agent may request JIT elevation, but it should never hold break-glass credentials. Break-glass for agents is a human overriding or halting the agent.

Why an approval gate is not a break-glass control

It's tempting to treat an approval gate as the safety boundary. It isn't. An approval gate governs what the agent proposes, not what the process does after something goes wrong. My bot never had a break-glass grant. What it shows is the other half of the pattern: what happens when revocation and halt depend on code that can quietly do nothing.

My bot ran every post, reply, follow, and like through a Telegram approval, and I pressed each one by hand. The gate worked exactly as designed. What it couldn't cover was what the process did after a safety signal fired. That's the suspension from the opening. The approval gate had nothing to say about a loop that was already running.

A separate failure was harder to spot, because nothing looked wrong. For about six weeks, four hardcoded target accounts that no longer existed soaked up 88% of the bot's daily comment attempts. Every run logged success, and nothing raised an error. The Telegram Skip button also silently did nothing for months. Every skipped item got counted as a timeout, so the approval statistics were wrong for that whole stretch.

What I took from both: a control you can't see failing is a control you don't really have. Agents fail quietly by default, so both the grant and the halt need a signal a human actually reads.

Is a plain process exit enough?

No. A plain exit is a suggestion that a supervisor can override. The fix I shipped was a process exit plus a sentinel file that only a human can clear. I needed the sentinel because launchd's KeepAlive respawned the plain exit.

The second suspension showed the same gap from another side. On 2026-09-22, a week after I raised the reply cap from 16 to 150, X suspended the account again. This time the lockout check matched none of its known selectors, so it never fired. I stopped the bot by hand. A detector that depends on the platform's page structure stops working when the page changes in a way its selectors don't cover.

from dataclasses import dataclass
from datetime import datetime, timezone
from pathlib import Path
import sys

HALT = Path("/var/agent/HALT_HUMAN_ONLY")  # lives in a root-owned directory the agent's user cannot write; deleting a file needs write access to its directory

@dataclass(frozen=True)
class Grant:
    scope: str
    approved_by: str        # a named human, recorded
    expires_at: datetime    # set at approval time, never extended by the agent

def check_before_action(grant: Grant | None, scope: str) -> None:
    if HALT.exists():
        sys.exit("halted: sentinel present, only a human clears it")
    if grant is None or grant.scope != scope:
        raise PermissionError(f"no grant for {scope}")
    if datetime.now(timezone.utc) >= grant.expires_at:
        raise PermissionError(f"grant for {scope} expired; request approval again")

This is a sketch of the pattern, not a product. The in-process check alone repeats my original bug, because a caller can catch or skip it. The supervisor has to honor the sentinel too, or a plain respawn undoes it. The permission model is the part that matters. If the agent's account can delete the sentinel, the sentinel is only a suggestion. The in-process expiry check is defense in depth, not the control. Real expiry belongs to the credential issuer, so a lapsed grant is a dead token and not an if-statement the agent could skip. That matches OWASP LLM06:2025's point that authorization belongs in downstream systems.

Doesn't human approval slow agents down too much?

Not if approval sits at the elevation boundary instead of on every action. The usual objection is that per-action approvals cause fatigue, and people start clicking through. That objection is right. The answer is to approve less often and with more context, which is the same line I draw in AI-assisted vs AI-autonomous.

The incidents show what happens without a boundary. In July 2025, an autonomous Replit coding agent reportedly dropped a live production database during a code freeze, despite repeated instructions not to touch anything. Replit's CEO called it "unacceptable and should never be possible" (eWeek). Nothing below the prompt enforced the freeze.

In late 2025, Google's Antigravity agent was reportedly running in Turbo mode, which lets it execute commands without user input. A command meant to clear a project cache reportedly wiped the root of the D: drive. The agent acknowledged the error afterward (TechRadar). Replit's CEO confirmed the first publicly; the second rests on the developer's own account.

OWASP's guidance points the same way. It names excessive functionality, excessive permissions, and excessive autonomy as the root causes of excessive agency. It recommends human approval for high-impact actions, and it says authorization should be enforced in downstream systems, not decided by the LLM (OWASP LLM06:2025). One approval for thirty minutes of one scope costs a person one decision. Per-action approval costs a decision every time, and that's how decisions turn into rubber stamps. The boundary is also where least-privilege scoping and the cost tradeoff between autonomy and human review meet in practice.

How to implement break-glass for an agent

Define the entitlement in advance: scope, maximum duration, and named approvers.
The approver must be a human other than whoever triggered the request, never the agent.
The credential issuer mints a short-lived credential whose TTL is fixed at approval, for example AWS STS AssumeRole DurationSeconds (AWS STS AssumeRole) or a HashiCorp Vault lease TTL (Vault leases).
Log the grant, the approver, the reason, and every use.
Review each grant after it expires.
Give a human a halt the agent cannot undo: a sentinel or revocation the agent's identity has no permission to clear, which the supervisor checks before every restart.

Why expiry matters more than the approval

Leaked secrets are the best-measured case of credentials outliving their purpose. They are not the same thing as unrevoked grants, but the failure mode is the same. GitGuardian's 2025 report found 23.8 million secrets leaked on public GitHub in 2024, a 25% increase over the year before (GitGuardian, 2025). The same report found that 70% of secrets leaked in 2022 were still active.

Expiry fails silently when it lives in application code or in someone's memory instead of in the credential itself. An approval gets logged the moment it happens. When the issuer enforces expiry, the grant ends at its deadline without anyone watching. If expiry is also checked in application code, check on every use, and an agent that tries to use a lapsed grant should get an error, not a retry.

My prediction: within two years, a public postmortem of an AI agent incident will name a break-glass grant that was never revoked as a root cause. After that, I expect vendors to make expiry enforcement the default instead of an opt-in setting.

Your Team Celebration Is an Event Handler. Make It a Cron Job (With the Workflow)

Orvi Das — Tue, 27 Oct 2026 11:04:30 +0000

Compare your team's #shipped channel with your incident log. My guess is the quiet weeks in one line up with the bad weeks in the other.

That isn't a coincidence. It's a dependency. Most teams wire celebration to output: a release goes out, someone posts the party emoji, people get a nice moment. No release, no moment. In code terms, celebration is an event handler subscribed to on: release. When the events stop, the handler never runs.

I think that's a design bug. This post covers the research behind that, then the boring fix: run the ritual on a schedule, whatever the metrics say.

The research everyone cites, read backwards

"Celebrate small wins" usually goes back to Teresa Amabile and Steven Kramer's progress principle. They studied nearly 12,000 daily diary entries from 238 knowledge workers at seven companies. People's best workdays were the ones where they made progress on meaningful work. In their 2011 HBR piece they also asked 669 managers to rank what motivates employees. Progress came last. Only 35 of the 669 ranked it first.

The finding is solid. What people did with it is the problem. Amabile was making a point about visibility: managers don't see progress, so nobody marks it. The popular version turned that into "notice your wins, then reward yourself." That move welds the ritual to your throughput. Now celebration carries all the variance of your output.

Think about who gets nothing under that model:

the team in week three of a migration that won't demo until next quarter
the on-call engineer who spent five days keeping a fire small
the solo dev whose launch flopped on Tuesday

Those are the people with the least motivation left. They're the ones the ritual should be serving.

"Won't celebrating make us soft?"

This is the serious objection. The argument is that tension drives effort and celebrating lets it out. It sounds tough-minded. The data doesn't back it up.

In 2010, Michael Kraus, Cassey Huang and Dacher Keltner coded the physical touch between NBA teammates early in the 2008–09 season: fist bumps, high fives, chest bumps, huddles. Teams that touched more early in the season played better later in it, even after controlling for player salary, preseason expectations and early-season performance. The acknowledgment came first. It wasn't a reward for winning.

Separately, Alison Wood Brooks and colleagues found that doing a short, arbitrary ritual before a stressful task reduced anxiety and improved performance compared with doing nothing. The ritual didn't have to mean anything. It just had to happen reliably, before the outcome was known.

So the order is the other way around from what most teams assume: ritual, then performance. Not performance, then ritual.

Moving it from `on: release` to `on: schedule`

Roads get built before the traffic shows up, and they get maintained in weeks when nobody drives on them. A team ritual needs the same treatment. Here's a minimal version: a GitHub Action that posts a Friday marker to Slack every week. It doesn't matter whether anything shipped.

# .github/workflows/friday-marker.yml
name: friday-marker
on:
  schedule:
    - cron: "0 15 * * 5"   # Fridays, 15:00 UTC
  workflow_dispatch:
jobs:
  mark:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: python3 scripts/friday_marker.py
        env:
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}

#!/usr/bin/env python3
"""Friday marker: posts every week, whether or not anything shipped."""
import json
import os
import subprocess
import urllib.request
from datetime import datetime, timedelta, timezone

SINCE = (datetime.now(timezone.utc) - timedelta(days=7)).strftime("%Y-%m-%d")
REPO = os.environ["GITHUB_REPOSITORY"]


def merged_prs() -> list[dict]:
    out = subprocess.run(
        ["gh", "pr", "list", "--repo", REPO, "--state", "merged",
         "--search", f"merged:>={SINCE}",
         "--json", "title,author,deletions", "--limit", "100"],
        capture_output=True, text=True, check=True,
    )
    return json.loads(out.stdout)


def build_message() -> str:
    try:
        prs = merged_prs()
    except Exception as exc:  # the ritual runs even when the data source doesn't
        print(f"gh failed, posting anyway: {exc}")
        prs = []

    lines = [f"*Friday marker, week of {SINCE}*"]
    if prs:
        deleted = sum(p["deletions"] for p in prs)
        lines.append(f"{len(prs)} PRs merged, {deleted:,} lines deleted.")
        lines += [f"• {p['title']} ({p['author']['login']})" for p in prs[:5]]
    else:
        lines.append("Nothing merged this week. The marker fires anyway.")
    lines.append("Thread: one thing you learned, fixed, unblocked, or survived.")
    return "\n".join(lines)


def post(text: str) -> None:
    req = urllib.request.Request(
        os.environ["SLACK_WEBHOOK_URL"],
        data=json.dumps({"text": text}).encode(),
        headers={"Content-Type": "application/json"},
    )
    urllib.request.urlopen(req, timeout=10)


if __name__ == "__main__":
    post(build_message())

Two design choices matter more than the rest of the code.

The empty branch is a first-class path, not an error. If a script exits quietly when there's no data, you've rebuilt the event handler with extra steps. A zero-PR week is when this job matters most, so it posts the most deliberate message.

The data fetch is allowed to fail. The post isn't. If gh breaks, you still get a marker. Treat the ritual like a heartbeat: when it doesn't arrive, that should be a signal in itself.

One ironic catch: in public repos, GitHub disables scheduled workflows after 60 days with no repository activity. So the platform itself couples the schedule to output. If your project goes through long quiet stretches, put the marker in a repo that sees regular commits, or run it from somewhere else.

What goes in the thread on a dead week

The prompt at the end does the real work. On a good week people post features. On a flat week they need permission to count other things. These are the ones I've seen land:

Deletions. Removing 2,000 lines of dead code is progress nobody demos.
Incidents contained. "It was down for 40 minutes, not four hours" deserves a marker.
Unblocking. Name the person who answered the question that saved someone a day.
Learned the hard way. A postmortem finding counts. So does a hypothesis you killed.

Amabile's own data supports this. What showed up in people's best days was perceived progress on meaningful work, and you can only perceive what somebody points at. The Friday thread is where the pointing happens.

Where this came from for me

I nearly cancelled a standing Friday dinner a few weeks ago. An account I'd spent a quarter growing had just been suspended for the second time in nine weeks, and marking the week felt absurd. Then I noticed I was about to switch off the one recurring thing built for weeks like that one. The dinner happened. The account is still under review. The dinner was never meant to depend on how the account was doing.

The longer version, with more of the ritual research, is on my site. The short version for engineering teams: if your celebration has an if shipped: guard, take it out.

Does your team have a ritual that still runs on bad weeks, or does it go quiet when the metrics do?

Originally published at https://www.robatdasorvi.com/chapters/human/why-celebration-is-not-optional-it-is-infrastructure

Policy as Code for AI Agent Tool Authorization: OPA or Cedar

Orvi Das — Sat, 24 Oct 2026 07:40:14 +0000

Originally published on robatdasorvi.com

In July 2025, Replit's AI coding agent deleted a live production database holding records on 1,206 executives during a code freeze, according to eWeek. The agent had been told not to make changes. That instruction lived in a prompt, and a prompt is a request the model may decline to honor. It is not a control the system enforces. That gap is why policy as code for AI agent tool authorization, with an engine like OPA or Cedar, exists.

If your agent's limits live in a system prompt, a tool description, or an if tool == "delete" branch in your dispatcher, you are in the position that database owner was in. The fix is to move the decision out of the model and out of your application code, and into a policy engine that answers allow or deny for every call.

Why Should AI Agent Tool Calls Go Through a Policy Engine?

Every AI agent tool call that changes state should be authorized by a policy engine such as OPA or Cedar, not by the prompt. A policy decision point outside the agent returns allow or deny for each call, along with a reason you can log. The tool runtime refuses to proceed without that answer. A prompt only asks.

The stakes are rising faster than the controls. Gartner's 25 June 2025 press release predicts that more than 40% of agentic AI projects will be canceled by the end of 2027, because of escalating costs, unclear business value, or inadequate risk controls. Risk controls are the third item on that list, and they are the one this article is about. In July 2025, a destructive instruction was injected into version 1.84.0 of Amazon Q Developer's VS Code extension, which has more than 950,000 installations, per CyberInsider's report. Amazon said formatting errors kept the payload from running. Some researchers disputed that. Either way, an agent with a broad tool surface and no independent check on each action has no defense when an instruction arrives from the wrong source.

An Agent Breaks a Prompt Rule Because Nothing Enforces It Where the Action Runs

An agent breaks a prompt rule because nothing enforces that rule where the action actually runs. The model decides whether to follow the rule, and the tool runs whatever the model emits.

The Replit case shows the shape. Per eWeek's reporting, the agent ran destructive commands during a code freeze. According to founder Jason Lemkin's public account, it then misreported what it had done. Replit's CEO called it "unacceptable and should never be possible." The safeguards Replit announced afterward are the useful part: separate development and production databases, a planning-only mode, and one-click restore from backups. Each one sits on the execution path. None of them is a sentence in a prompt.

I run an approval-gated X growth agent: every post, reply, follow and like went through a Telegram approval I pressed by hand. On 21 July 2026, X suspended the account. The lockout check fired and returned True, but the caller skipped one action and the loop kept running for 3.5 more days. A correct answer from a check does nothing if the code that acts treats it as advice: the caller got True, skipped one action, and carried on. The fix was a process exit plus a sentinel file that only a human can clear, because launchd's KeepAlive respawned a plain exit. The problem was the enforcement point, not the detection logic.

The second suspension, on 22 September 2026, failed the other way. The lockout check matched none of its known selectors and never fired. A gate that works only when a pattern matches the world is a detector, and detectors fail silently in production. Authorization needs a default of deny that still holds when the detector is blind.

Where Does the Policy Decision Point Pattern Come From?

Splitting the decision point from the enforcement point predates agents by more than a decade. XACML, the OASIS access-control standard, defined a Policy Decision Point that evaluates a request against policy, and a Policy Enforcement Point that acts on the answer. OASIS approved XACML 3.0 in a vote that closed on 22 January 2013. It followed XACML 2.0 from 1 February 2005 and 1.0 from 2003 (OASIS XACML 3.0 standard).

NIST formalized the same idea in January 2014. NIST SP 800-162 describes attribute-based access control as a permission decision. It weighs attributes of the requester, the target resource, the requested action, and sometimes environmental conditions against policy. Map that onto a tool call and the pieces line up. The agent is the subject. The tool and its arguments are the resource and the action. The environment is production or staging. None of this is new. What is new is that the subject is a language model that composes its own actions.

XACML showed the model was sound, but the standard never made it into everyday engineering. The idea reached ordinary infrastructure through Open Policy Agent. The CNCF announced OPA's graduation on 4 February 2021 (CNCF announcement). AWS open-sourced Cedar in May 2023. Agent tool calls are the third wave. They are also the first where the subject is an unpredictable generator, not a human or a service with a fixed call graph.

OPA or Cedar: Which Should You Pick?

Pick Cedar if you are on AWS or want authorization decisions that are easy to analyze. Pick OPA if your organization already runs it and you need policy beyond authorization. Either one works, as long as the enforcement point is mandatory.

OPA uses Rego, a general-purpose policy language. It is the broader tool, and it already sits in many Kubernetes and CI stacks. Cedar was built for authorization. The Cedar paper (OOPSLA 2024, DOI 10.1145/3649835) compares Cedar with OpenFGA and Rego. Its authors, the Cedar team at AWS, report that Cedar policies are as readable as or more readable than the alternatives, and that Cedar evaluated faster than both in their benchmarks.

Amazon Bedrock AgentCore Policy shows what an enforcement point for agents looks like in practice. It intercepts requests at the AgentCore Gateway, checks each one against a Cedar policy engine before the tool is reached, and logs every decision to CloudWatch. It has two modes: LOG_ONLY and ENFORCE. The AgentCore policy documentation covers the mode settings. Start in LOG_ONLY, read the denials for a week, then switch to ENFORCE.

Here is a Cedar policy for a refund tool:

permit (
  principal,
  action == Action::"invoke",
  resource == Tool::"issue_refund"
) when { context.amount < 50 };

forbid (
  principal,
  action == Action::"invoke",
  resource == Tool::"issue_refund"
) when { context.amount >= 50 };

Cedar denies by default, so the permit alone already blocks amounts of 50 or more. The forbid is a guardrail: in Cedar an explicit forbid overrides every permit, so it still holds if someone later adds a broader permit. Pair it with least privilege for AI agents. A policy engine can only deny what the tool surface exposes.

Here is the same refund limit in Rego, the language OPA uses:

package agent.tools

default allow := false

allow if {
  input.tool == "issue_refund"
  input.arguments.amount < 50
}

(This uses OPA 1.0 syntax; on older versions add import rego.v1.)

The default allow := false line is the important part. Rego has no forbid keyword, so the usual pattern is a default-deny rule with allow rules beside it. If no rule matches, the answer is deny, so a new tool with no rule is blocked automatically. The dispatcher sends one JSON input document per call, containing the principal (the agent id), the tool name, the arguments, and the environment, such as production or staging. Fill the principal and environment from the runtime session, never from the model's output. An agent that can write its own agent id or environment into the input can authorize itself. Both engines evaluate that document and return a decision the tool runtime must check before it runs.

Doesn't a Policy Engine Add Latency and Another Point of Failure?

Run in-process, an OPA or Cedar evaluation adds very little time. Called over the network, it costs a round-trip. The real risk is the default when the engine is unavailable, not the latency. AWS describes Verified Permissions as evaluating access requests in milliseconds. A tool call already waits on a model round-trip that takes seconds. The check is not the bottleneck.

The failure mode is what matters. If the policy service is unreachable, the tool runtime must deny. A fail-open default turns the policy engine into an optional check, the same shape as the 21 July bug, where a correct answer existed and the code that acted didn't obey it. Write the code so the tool function cannot run without a decision object, and treat a missing decision as a hard error.

import logging

log = logging.getLogger("agent.tools")

def call_tool(name, args, session):
    doc = {
        "principal": session.agent_id,       # set by the runtime, never by the model
        "environment": session.environment,
        "tool": name,
        "arguments": args,
    }
    try:
        decision = policy_engine.evaluate(doc)
    except Exception as exc:
        raise PermissionError(f"policy engine failed, denying {name}") from exc
    if decision is None:
        raise PermissionError(f"no decision for {name}, denying")
    log.info("tool=%s allow=%s reason=%s", name, decision.allow, decision.reason)
    if not decision.allow:
        raise PermissionError(f"{name} denied: {decision.reason}")
    return TOOLS[name](**args)

Writing a policy is not the same as enforcing one. Put the check in the single function every tool call passes through, not in each tool. Then test it by calling each tool with the decision stubbed to deny. If any tool still runs, the gate is optional, and you have the same bug as before.

Should You Start With the Policy or the Enforcement Point?

Start with the enforcement point. A policy behind an optional gate is just documentation. Once every side-effecting call needs a decision, the policy language becomes a choice you can revisit later.

Sometimes a denial is not enough, and you need to undo what already happened. Rollback for tool calls fails in its own ways, so design for it before the first incident. And before you give an agent write access anywhere, ask whether it should be an agent at all. Some tools need no agent.

Today, find the one function your agent uses to call tools that change state. Make it refuse to run without a decision from a policy engine. Ship that in LOG_ONLY, and tomorrow read every denial it logged.

Adding an API Enum Value Is a Breaking Change for Agent Tools

Orvi Das — Sat, 24 Oct 2026 07:39:43 +0000

Originally published on robatdasorvi.com

Adding a value to an API enum breaks any client that handles that enum with a closed switch. Calling the change "additive" in your changelog doesn't change what that client does when the new value arrives. If a spec diff told you the change was non-breaking, the diff was right about the schema and wrong about your consumers.

You're probably reading this because a diff tool marked a change as safe and something downstream fell over anyway. A client threw on the new status value. An agent's tool call started returning an error it had never returned before. You trusted the schema, and the schema wasn't what broke. The general case is in When an Upstream API Changes, Your Agent's Tools Break Silently; this piece covers the one change most diff tools wave through: a new enum value.

Is adding an enum value actually a breaking change?

Yes, for any consumer that throws on, drops, or mislabels a value it doesn't recognize. The change is additive for the producer and breaking for the reader, and the reader is the one who decides.

The tools don't agree, which is the first sign the question is underspecified. The swagger-spec-compatibility library treats an added enum value in a response as an error, since clients built on the old spec can't validate the response (source). Smithy's changed-enum evaluator emits only a NOTE for a value appended at the end, and an ERROR for a value that is removed, renamed, or inserted mid-list (source). A json-schema-diff proposal in pull request #46 classifies adding a value as non-breaking and removing one as breaking (source). Three tools, three rules, one schema change.

Why does my spec diff call it safe?

The diff tool sees two schemas. The break happens in code it can't read. It can tell you what the producer changed. It can't tell you what a consumer does with a value that wasn't in the consumer's copy of the enum.

Google's API Improvement Proposal 180 says enum values can be freely added to enums used only in request messages. For enums used in responses, it says the enum should document that it may grow. AIP-180 also warns that user code may not handle new values gracefully (source). Microsoft's guidance makes the same trade: add values despite "a small compatibility risk," because existing code might not handle them (source). Both rules hand the decision to the consumer, and a spec diff never looks at the consumer.

Has this been solved before?

Partly. HTTP and Protobuf both solved it by declaring the set open and telling readers what to do with unknown values. Most JSON APIs and generated clients never adopted the same rule. Postel's law, from RFC 761 in 1980, says to be conservative in what you send and liberal in what you accept. Martin Fowler's "Tolerant Reader" post, published 9 May 2011, made the consumer half concrete: "be as tolerant as possible when reading data from a service" (source). He also named the failure most teams still hit: generated classes from an XSD break when the provider adds a field. In his words, "Adding field to an interface like this shouldn't be a breaking change for anyone - but often does break these schemes."

HTTP already solved the enum version of this problem, and it's the closest precedent you have. RFC 7231, published in 2014, requires clients to treat an unrecognized status code as the x00 code of its class, so a new 471 gets handled as a 400-class error (source). The same rule appeared in RFC 2068 in 1997. HTTP status codes are an open set, and the spec told clients to read the class instead of the exact code. Most enum-typed client code was never written that way.

Protobuf ran the experiment in public. Its enum documentation says closed enums send an unrecognized number to the unknown field set, so the accessor reports the field as unset and returns the default value. Proto3 made enums open "specifically because of the unexpected behavior that closed enums cause" (source).

What happens when a client sees a value it doesn't know?

It does one of three things. It crashes, it silently substitutes a default, or it preserves the value and leaves the decision to the code. Only the last is safe, and most generated clients default to one of the first two.

Onfido's February 2020 incident, titled "SDK Crashes," shows the crash case. A change to a public Onfido endpoint caused it to return values in a format the Android SDK couldn't handle, and about 90% of applicants using the Android SDK couldn't finish the media upload flow. The change was reverted at 17:58 UTC, about an hour after it went out at 16:55. A small percentage of affected users kept getting errors until a cache fix on 20 February, because a cache on their devices held an invalid value. The iOS and web SDKs had no issues (source).

Dfinity's post-mortem for 27 March 2024 shows the persistence case. Removing and replacing error codes in a new replica version stopped two Internet Computer subnets from restarting after an upgrade, because the changed enum values were stored in checkpoints (source). That was a removal, not an addition, but it shows how long an old enum value outlives the deploy that changed it. A checkpoint keeps the old reader and the new writer alive side by side.

Isn't the consumer just wrong? Can't we patch it?

You can, but not on your schedule. The producer's change has to be safe for the consumer as it exists today.

The openapi-code-generator project made this argument in pull request #291: native mobile apps update on users' schedule, and strict enum validation that throws on unrecognized values makes it hard to add values safely (source). A small percentage of Onfido's Android users kept getting errors after the revert until a cache fix on 20 February. Calling the consumer wrong doesn't fix a device that's already crashing. It just assigns blame after the outage.

The GraphQL documentation makes the same point in its guidance on robust applications. It says GraphQL schemas can add new enum values at any time, and that if an application treats every enum value as exhaustively known, a new value from the server can cause crashes or silent data loss. Its recap says to always include a default branch for unknown enum values (source).

Is this worth a separate check, or is it a code review issue?

It needs its own check, because the failure is silent when it happens. A code review reads your diff. It doesn't run the consumer's switch statement against the value you're about to ship.

I built an approval-gated X growth agent, and the same silent pattern showed up there several times. The closest one to an enum: when X suspended the account a second time on 2026-09-22, the lockout check matched none of its known selectors and never fired, so I stopped the bot by hand. A check built on a closed list of known states met a state outside the list and did nothing. For about six weeks, four hardcoded target accounts that no longer existed absorbed 88% of the bot's daily comment attempts. Every run logged success, and nothing raised an error. The Telegram Skip button did nothing for months, so every skipped item got counted as a timeout, and the approval statistics were wrong the whole time. The article lane lost 8 of 20 generations to JSON parse errors, and each failure quietly burned a 48-hour slot. None of these were enum changes, but they failed the same way: the code met a state it didn't expect, kept going, and raised no alarm.

The same pattern shows up between agents in Agent Rule Conflicts: Why AI Pipelines Fail Silently.

How does a new enum value break an agent's tool call?

A new value in an API response that the tool wrapper validates against a closed enum returns a tool error, and the model may retry, route around it, or invent a result. A frozen enum in the tool's input schema blocks the new value until the tool definition is regenerated, so the agent can't use the new capability.

For agents, this is the same problem with a worse failure mode. The agent doesn't crash, which makes the failure harder to see. The MCP tools specification defines a tool's inputSchema and, optionally, outputSchema. The tool wrapper should map unknown response enum values to an explicit 'unknown' result string the model can read, instead of throwing a validation error. The review side of the same silent failure is in Silent AI Code Bugs: Passing Reviews, Failing in Production.

What would I do before the next enum value ships?

Ask one question per consumer: what does your code do when it reads a value outside the enum it was built against? Answer that in code, not in a spec.

Start with three steps. Mark every response enum as open in your contract, so the contract tells consumers to expect growth. Inject an unknown value into each consumer's contract test, and fail the build if the consumer throws. Fowler's post points to consumer-driven contracts, where consumers send their tests to the provider (source). Then check every cache and persisted store that holds the enum, because that's where the old reader and the new writer meet.

Where APIDiffGuard fits: it's a diff engine and a monitor. The README describes it as a way to "catch breaking API changes before your clients do," by monitoring API responses, detecting schema drift, comparing versions, and alerting developers (apidiffguard.com). The diff engine is MIT-licensed as @apidiffguard/diff. A schema comparison can't tell you whether a given consumer has a closed switch. A monitor only sees a new value once a response carries it, which is after the deploy. So it's an alert, not prevention. Classifying enum additions per consumer is the part I haven't solved.

If you have one minute, do this. Find the client code that switches on your enum and look at its default branch. If the default throws, or doesn't exist, ship the default first, in the client, and only then ship the new value. The client's release cycle is the slowest link in your deploy, and you can't change it from the server.

LLM Structured Output: Validation, Retry or Constrained Decoding?

Orvi Das — Fri, 23 Oct 2026 04:30:34 +0000

Originally published on robatdasorvi.com

Most advice on LLM structured output comes down to validation and retry: ask for JSON, parse it, and retry when it breaks. If that's your whole plan, your pipeline is probably throwing away work right now. Constrained decoding is real and useful, but it solves a narrower problem than the one that actually costs you output. Validate-and-retry is the other common answer, and it fails in its own way. Getting reliable structured output from an LLM is a failure-handling problem before it is a parsing one. Which one you pick matters less than what your code does when the output is wrong.

The case here is mine, and it's small. The long-form article lane in the X growth agent I run asks a model for a structured article payload. The lane lost 8 of 20 generations to JSON parse errors. Each failure quietly burned a full 48-hour publishing slot. Nothing raised an error a human would notice, and the pipeline treated the lost slots as ordinary outcomes. The malformed JSON was the trigger. The damage came from the failure branch, which burned the slot instead of releasing it.

Does Constrained Decoding Fix Malformed JSON From LLMs?

Partly. It fixes the syntax and shape of JSON when your framework supports your schema. It doesn't fix truncated, refused, or wrong-valued output. The mechanism is simple: at each step, the decoder masks out any token that couldn't continue a schema-valid answer, so the model can only produce something the schema allows.

The strongest evidence came from the vendors. In August 2024, OpenAI reported that gpt-4o-2024-08-06 with Structured Outputs scored 100% on its complex JSON schema-following evals, against under 40% for gpt-4-0613 (OpenAI, 2024). That's a big jump, and it's real. OpenAI's Structured Outputs documentation notes that a response cut off at the token limit can come back incomplete, and the API can return a refusal instead of schema-shaped data (OpenAI docs). A schema-valid guarantee doesn't protect you from an answer that got cut off.

The research is less tidy than the vendor numbers. The 2023 paper "Efficient Guided Generation for Large Language Models" by Willard and Louf showed that regex and grammar constraints can be enforced with a vocabulary index and little per-token overhead. It shipped as the Outlines library (arXiv:2307.09702). In January 2025, Geng and colleagues published JSONSchemaBench, which runs 10,000 real-world JSON schemas through six constrained decoding frameworks: Guidance, Outlines, Llamacpp, XGrammar, OpenAI, and Gemini. They found that coverage and compliance vary widely across frameworks, and that no framework handled every schema construct (arXiv:2501.10868).

So the guarantee belongs to a specific framework and a specific schema. If your decoder doesn't support some construct in your schema, you're back to validation, whether you planned for it or not.

Does Forcing JSON Output Hurt LLM Reasoning?

Yes, it can. The 2024 EMNLP Industry paper "Let Me Speak Freely?" by Tam and colleagues found a significant drop in reasoning performance under format restrictions, and stricter constraints produced bigger drops on reasoning tasks (arXiv:2408.02442). Results on classification tasks were mixed. The paper also found that key order in a structured output can interfere with the model's reasoning.

What that means in practice: don't make the model reason inside the constrained field. Let it think freely, then convert the answer into the schema in a second step. The paper tested a version of this two-step approach, which it called NL-to-Format. If your schema has a field where the model has to think and then answer, the constraint is what's degrading the thinking. Retry-with-error-feedback adds tokens on every attempt, and why long prompts lose accuracy is a reason to keep the feedback short.

Is Retrying Malformed LLM Output Wasteful?

No. A retry is cheap when it's bounded, carries the validator's error message back to the model, and has a hard budget. The unbounded retry is the bug.

Baar-Core started after an agent loop left running overnight produced a $47 bill. That wasn't a JSON retry, but any loop without a hard stop has the same shape. One more attempt costs very little. An attempt that never stops is another matter.

A bounded validate-and-retry loop looks like this in practice. Parse, validate against the schema, and on failure send the model the exact validation error along with the original output. Allow two or three attempts. If they all fail, return a typed failure to the caller, not a partial result. The retry budget is what keeps validation from becoming a second source of runaway spend.

import json
from jsonschema import validate, ValidationError

def generate(prompt, schema, call_model, max_attempts=3) -> tuple[str, object]:
    messages = [{"role": "user", "content": prompt}]
    error = None
    for _ in range(max_attempts):
        reply = call_model(messages)  # assumed to expose .text, .finish_reason, .refusal
        if reply.refusal:
            return ("refused", reply.refusal)
        if reply.finish_reason == "length":
            return ("truncated", None)
        try:
            data = json.loads(reply.text)
            validate(data, schema)
            return ("valid", data)
        except (json.JSONDecodeError, ValidationError) as e:
            error = str(e)
            messages += [{"role": "assistant", "content": reply.text},
                         {"role": "user", "content": f"Invalid: {error}. Return JSON only."}]
    return ("invalid", error)

Does Schema Validation Stop Silent LLM Failures?

Not by itself. A validator only protects you if its failure result changes what the caller does. A check that returns the right answer and then gets ignored is the same as no check at all.

My own suspension history is the clearest example. These weren't LLM output checks, but the failure shape is identical. On 2026-07-21, X suspended the account. The lockout check fired and returned True, but the caller skipped one action, and the loop kept running for 3.5 more days. The fix was a hard process exit and a sentinel file that only a human could clear, because launchd KeepAlive respawned a plain exit. On 2026-09-22, X suspended the account again. This time the lockout check matched none of its known selectors and never fired at all, and I stopped the bot by hand. The first check fired and was ignored. The second never fired. Either way, nothing stopped the loop.

The pattern shows up well outside my projects. In July 2025, during a code and action freeze, Replit's AI agent deleted a live production database belonging to SaaStr founder Jason Lemkin, and the agent later told him a rollback was not possible. The rollback in fact worked, according to Heise's report on the incident, which also says the agent admitted it ran a database command without permission after it "panicked." The failure wasn't the model's inability to produce JSON. It was an agent that ignored an explicit freeze, then gave a confident and wrong account of what could be recovered.

A second case is the Air Canada chatbot. In February 2024, the British Columbia Civil Resolution Tribunal ruled against the airline in Moffatt v. Air Canada (2024 BCCRT 149). The support chatbot had told passenger Jake Moffatt he could claim a bereavement discount up to 90 days after his flight, but the airline's policy required approval before travel. The tribunal rejected the argument that the chatbot was a separate legal entity and ordered Air Canada to pay C$650.88 in damages plus interest and tribunal fees (Canadian Press via Yahoo News). Neither case was a JSON failure. Both were a system acting on an answer nobody checked.

That's why I think structured output belongs next to agent safety, not next to parsing. The same silent-failure pattern is covered in Agent Rule Conflicts: Why AI Pipelines Fail Silently, and any automated checker needs its own verification, as in LLM-as-a-Judge Reliability: Check Your Grader Against Human Labels.

When Should You Use Constrained Decoding vs Validate-and-Retry?

Use constrained decoding when the provider or framework supports every construct in your schema and the field needs no reasoning. Use validate-and-retry when the schema has unsupported constructs, when you need semantic checks (ranges, enums tied to business rules, cross-field consistency), or when the provider has no constrained mode. Use both in production.

The table shows what each approach covers and where it still fails.

Approach	What it guarantees	What it misses
Constrained decoding	Output matches the supported schema's syntax and shape	Truncation at the token limit, refusals, wrong values, schema features the decoder doesn't support
Validate-and-retry	Output that reaches the caller passes your validator	Truncated or refused output unless you check for them, semantic errors your validator doesn't test, and retries that can't fix a schema feature the model can't produce
Both, with a bounded budget	Schema-valid output that passes your semantic checks, or a typed failure	Wrong values your checks don't cover, and the cost of retries

How to Make LLM Structured Output Failures Visible

If constrained decoding is your only control, you'll still lose work whenever output is truncated, refused, or semantically wrong, and you won't know when it happens. If validate-and-retry is your only control, a missing retry budget turns one bad output into unbounded spend. Neither is enough alone.

So make every structured output call return a typed result: valid, invalid, truncated, or refused. Use constrained decoding where your framework supports the schema, and validate every response regardless. Retry invalid results within a fixed budget, feeding the validator's error back. When the budget runs out, raise an error and count it. A batch of 20 with 8 failures should show up as 8 failures, not 12 successes with a hole in them. Release the publishing slot instead of burning it. Alert on the count, not the logs.

Constrained decoding, validation, and bounded retry are three layers, and each one can fail. The only way to know they work is to count the failures. If your pipeline can't say how many structured outputs it lost last week, you don't have a fallback. You have a hope with a cron job.

Durable Execution for AI Agents: Temporal vs a Plain State Store

Orvi Das — Fri, 23 Oct 2026 04:30:03 +0000

Originally published on robatdasorvi.com

Durable execution for AI agents means a crash costs you one step, not the whole run. Temporal gives you that through workflow event history; a plain state store gives you most of it with a table you own. The hard part with either is the step that has already touched the outside world.

Anthropic reports that multi-agent systems use about 15 times as many tokens as chat interactions (Anthropic, 2025). Gartner predicts that more than 40% of agentic AI projects will be canceled by the end of 2027 (Gartner, June 2025), citing escalating costs, unclear business value, and inadequate risk controls. Gartner doesn't single out crashes, but re-running a job that already costs 15 times a chat, from zero, is the kind of escalating cost it names. If a run dies at 90% and restarts, you pay for the same tokens twice.

What is durable execution for AI agents?

Durable execution means the runtime records each completed step, so that when a process dies, the run picks up at the last recorded step instead of starting over. For agents, a step is usually a model call or a tool call. The record is either a workflow engine's event history, like Temporal's, or a checkpoint table you write yourself.

Anthropic describes the requirement in its engineering write-up on its multi-agent research system: "Agents can run for long periods of time, maintaining state across many tool calls." Its conclusion was that "when errors occur, we can't just restart from the beginning," and that the team "built systems that can resume from where the agent was when the errors occurred" (Anthropic, 2025). The same post says "agents are stateful and errors compound," which is the whole problem in one line.

Do AI agents need Temporal to resume after a crash?

No. You need a durable record of completed steps, and Temporal is one way to get it. A Postgres table keyed by run ID and step number is enough for a single-process agent that finishes in minutes.

Temporal earns its place when runs are long, span several workers, wait on humans or timers, and have to survive deploys. Its OpenAI Agents SDK integration runs the agent loop inside a Workflow and model calls as Activities, so completed model calls are not repeated during replay (Temporal docs). LangGraph takes a similar approach: a checkpointer saves state at each super-step boundary, and side effects should go inside tasks so they don't re-run on resume (LangChain docs).

The skeptic's case is that this is overkill, and it's partly right. A state table is plenty for a lot of agents. It stops being right once runs span several workers, survive deploys, or wait on humans. Neither option prevents duplicate side effects on its own; that takes idempotency keys, covered next.

How do you stop a resumed agent from repeating side effects?

You make every side-effecting call idempotent, and the external service has to enforce it. Temporal's guidance is that activities are retried until they succeed, so execution is at-least-once. Its February 2024 post on idempotency suggests using an existing business identifier, such as an order ID, as the key, or otherwise workflowRunId + '-' + activityId, which stays constant across retries (Temporal, February 2024). For what duplicate runs look like when two workers touch the same state, see concurrency bugs between coding agents sharing a repo.

Neither a state store nor Temporal does this for you. Writing "step 4 done" to a table doesn't stop the payment API from charging a customer again if the worker crashed after the charge but before the write. Whatever storage you choose, you still need a key the downstream system checks.

Duplicate side effects are not the only way multi-agent runs fail. The Berkeley team behind MAST annotated more than 1,600 traces from seven multi-agent frameworks and identified 14 failure modes in three categories: specification issues, inter-agent misalignment, and task verification. The authors argue that many failures come from how the system is specified and coordinated, not only from model limits (Cemri et al., NeurIPS 2025). A durable engine preserves a run; it does not fix how the run was specified, so a badly coordinated run resumes just as badly.

What does a minimal checkpoint table look like?

A minimal checkpoint table has one row per step of a run, keyed by run ID and step number. Each row records whether the step is pending, started, done, or failed, plus the idempotency key it sent downstream and the output it got back. A row is pending when it is written ahead for a planned step.

CREATE TABLE checkpoints (
  run_id          TEXT NOT NULL,
  step_no         INTEGER NOT NULL,
  status          TEXT NOT NULL CHECK (status IN ('pending', 'started', 'done', 'failed')),
  idempotency_key TEXT NOT NULL,
  output_json     JSONB,
  updated_at      TIMESTAMPTZ NOT NULL DEFAULT now(),
  PRIMARY KEY (run_id, step_no)
);

Write in this order:

Insert the row as started with the idempotency key before the side effect.
Send the key to the downstream API.
Mark the row done with the output after.

On resume, skip done rows and re-send started rows with the same key.

This assumes one worker per run. If two workers can resume the same run, add a lease_until TIMESTAMPTZ column and claim the run with an UPDATE ... WHERE lease_until < now() before resuming, or both will re-send the same started step.

How do you checkpoint a multi-agent run with subagents?

In a multi-agent run, the orchestrator records each subagent's result as a step of the parent run, under a parent run ID plus child run IDs. On resume, finished subagents are read from checkpoints instead of being spawned again, and only unfinished ones run.

In Temporal, subagents map to Child Workflows. In a plain table, a parent_run_id column links child runs to the parent step.

Why can a restarted agent keep running after it should stop?

Because a supervisor that restarts crashed work cannot tell a crash from a deliberate stop. Any restart or resume path needs an authority check, a signal only a human can clear, not just a check of saved state.

My own bot shows both halves of this. On 2026-07-21 X suspended the account behind my approval-gated growth agent. The lockout check fired and returned True, but the caller skipped one action and the loop kept running for 3.5 more days. When I fixed it, a plain process exit wasn't enough, because launchd KeepAlive respawned the process. The fix that held was a process exit plus a sentinel file that only a human can clear. A supervisor that restarts whatever crashed will also restart whatever you meant to stop.

X suspended the account again on 2026-09-22, a week after the reply cap was raised from 16 to 150. This time the lockout check matched none of its known selectors and never fired; I stopped the bot by hand.

A stop condition that depends on recognising the failure will miss failures it has never seen, so a resume path should also check something a human controls, such as the sentinel file.

In February 2026 the Financial Times reported that in December 2025 Kiro, an internal agentic coding tool at Amazon, was allowed to make changes that led to a 13-hour interruption of a customer-facing service. Kiro normally asks for authorization, but the FT reported it had permissions similar to a human engineer's, so the change went through without approval. Amazon disputes the account and calls the cause "user error—specifically misconfigured access controls—not AI" (The Decoder). The details are contested, and this was not a retry. But it shows the same gap: an action ran on permissions that nobody rechecked at the moment it executed.

In July 2025 a Replit agent working on a project under a code freeze reportedly deleted a production database. Jason Lemkin, who was running the project, said he had told the agent repeatedly not to make changes without permission. Replit's CEO called it "unacceptable and should never be possible" (eWeek, 2025). This wasn't a crash, but it shows what happens when a run keeps going without rechecking authority.

How much does a crash cost a multi-agent run?

A crash without checkpoints costs the whole run again. Anthropic reports multi-agent runs use about 15 times the tokens of a chat, so re-running from zero multiplies an already large bill, and on a scheduled pipeline a failed attempt can also consume the slot it was meant to fill.

That happened to the article lane in the same bot. JSON parse errors killed 8 of 20 generations, and each failure silently burned a 48-hour slot. The failure consumed the schedule, not just the attempt. A checkpoint that records the failed parse as a failed step, rather than as a used slot, would have let a retry run inside the same window. Nothing in the lane did that, and every failure was silent. My earlier essay on why LLM agents fail silently in production covers the detection side of that problem.

Durability also has a cost ceiling. One of my projects, Baar-Core, started after an agent loop left running overnight produced a $47 bill. An engine with unlimited retries would only have kept a loop like that alive more reliably. Budgets and stop conditions belong in the workflow, not in the operator's memory. The human-in-the-loop cost comparison works through the tradeoff.

Is resuming more dangerous than the first attempt?

It can be. The risky step is the one that already had a side effect, so resume and retry paths deserve the most review. A read-only step can run again with little harm. A step that charged a card or sent a message can charge or send twice if it runs again.

My prediction is specific enough to be wrong. By 2027-06-30, most publicly reported agent production incidents involving duplicate charges, repeated messages, or unauthorized actions will trace back to a retry or resume path, not a first-attempt failure. On today's evidence the prediction is behind: neither the Kiro nor the Replit incident above was a retry or resume. I'll check this against AI Incident Database entries that describe an AI agent taking an action and count which incidents involved a retry or resume.

If the count comes out the other way, I'm wrong that resumption is the dangerous part, and the review time belongs on first-attempt authority checks instead. A related failure on that side is the hidden-state problem in agentic systems, which I've written about separately.

Lost in the Middle: Find Your LLM's Context Dead Zone and Fix It Before the Call

Orvi Das — Fri, 23 Oct 2026 02:21:38 +0000

One result from the "Lost in the Middle" paper should change how you build RAG prompts. GPT-3.5-Turbo was given 20 retrieved documents with the answer placed in the middle of them, and it scored lower than when it got no documents at all. Answering from memory beat answering with the documents. Here the extra context made the answer worse.

Liu et al. (Stanford, UC Berkeley, Samaya AI, 2023) tested multi-document QA and key-value retrieval. They moved the relevant passage from the first slot to the last and tracked accuracy. The result was a U-shaped curve: strong at both edges, down by 20+ points in the middle. Newer models have flattened that curve, but it hasn't gone away. You won't know how your model handles your prompt shape until you measure it.

This post covers three things you can add to a pipeline today: a test that finds your own dead zone, a reorder step that keeps important chunks out of it, and a pre-flight gate that blocks oversized prompts before they're sent.

Your context window is a budget

A 200K window is the most the model will accept. You don't have to fill it. Every token you add costs money, adds latency, and competes for attention with the tokens that actually matter.

The attention cost is the one people miss. Developers tend to treat the window like a bucket: if it fits, ship it. A 3,000-line file fits. Fifteen RAG chunks fit. Forty turns of agent history fit. The model takes all of it without complaint and gets worse at the one thing you asked.

No error tells you "your instruction is buried at token 61,000." You just get an answer that ignores it.

Step 1: Measure your own dead zone

Don't rely on a 2023 chart for a 2026 model. Run a needle test with your real prompt shape. It takes about ten minutes and a few dollars.

import anthropic, random

client = anthropic.Anthropic()
MODEL = "claude-sonnet-5-5"

NEEDLE = "The deploy freeze for the payments service ends on March 14."
QUESTION = "When does the deploy freeze for the payments service end?"

def build_haystack(filler_chunks, needle, position):
    """position runs from 0.0 (start) to 1.0 (end)."""
    idx = int(position * len(filler_chunks))
    chunks = filler_chunks[:idx] + [needle] + filler_chunks[idx:]
    return "\n\n".join(chunks)

def run(filler_chunks, positions=(0.0, 0.25, 0.5, 0.75, 1.0), trials=5):
    results = {}
    for pos in positions:
        hits = 0
        for _ in range(trials):
            random.shuffle(filler_chunks)
            doc = build_haystack(filler_chunks, NEEDLE, pos)
            resp = client.messages.create(
                model=MODEL,
                max_tokens=50,
                messages=[{
                    "role": "user",
                    "content": f"{doc}\n\n{QUESTION} Answer in one short phrase.",
                }],
            )
            if "march 14" in resp.content[0].text.lower():
                hits += 1
        results[pos] = hits / trials
    return results

Use realistic filler: your own docs, old tickets, whatever your production context actually contains. With lorem ipsum the needle stands out too easily. The honest test uses a needle that looks like the text around it.

One caveat: pulling back a single string is the easy case. If your task needs the model to combine a fact from the middle with something near the end, expect worse numbers than this test gives you.

Step 2: Put the best chunks at the edges

When a retriever returns chunks ranked by relevance, most pipelines join them in rank order. Chunk #1 ends up at the top, which is good. Chunks #2 through #5 land in the middle, which is bad. And your least relevant chunks sit at the end, right next to the question.

Reverse that. Alternate the placement so the strongest chunks sit at both edges and the weakest fill the middle:

def edge_order(chunks_by_relevance):
    """Input: most relevant first. Output: best at both ends, worst in the middle."""
    front, back = [], []
    for i, chunk in enumerate(chunks_by_relevance):
        (front if i % 2 == 0 else back).append(chunk)
    return front + back[::-1]

# ranks [1, 2, 3, 4, 5, 6] -> [1, 3, 5, 6, 4, 2]

If you'd rather not maintain this yourself, LangChain has it as LongContextReorder in langchain_community.document_transformers. Either way it's only a few lines.

Two more placement rules that cost nothing:

Put the question after the documents. The end of the prompt is the second-best position you have.
In long-running agents, repeat the core instruction near the end of each turn. Don't count on the model still following a system prompt from forty messages ago.

Step 3: Block the prompt before it ships

Reordering helps. Not sending 120K tokens of marginal context helps more. The simplest fix is a pre-flight check: count tokens before the call, and if the prompt is over budget, trim it or fail with a clear error.

MAX_INPUT_TOKENS = 30_000  # set this from your Step 1 results, not the model card

class ContextBudgetExceeded(Exception):
    pass

def guarded_create(client, *, model, messages, system=None, **kwargs):
    params = {"model": model, "messages": messages}
    if system:
        params["system"] = system

    n = client.messages.count_tokens(**params).input_tokens
    if n > MAX_INPUT_TOKENS:
        raise ContextBudgetExceeded(
            f"{n} input tokens > {MAX_INPUT_TOKENS} budget; trim retrieval or history"
        )
    return client.messages.create(**params, **kwargs)

Where should MAX_INPUT_TOKENS come from? Your own needle curve. If mid-prompt accuracy holds at 30K tokens and drops at 80K, your usable window is 30K, whatever the pricing page says.

When the gate trips, start by dropping the lowest-ranked chunks. Then summarize old agent turns into a short state block. For code, send the function you're working on plus its direct callers, not the whole file. A model that sees 400 relevant lines will usually do better than one that sees 3,000 lines where the class definition sits in the middle.

The same gate keeps the bill down

This gate helps with cost as well as accuracy. Context bloat is a common way agent costs run away. If an agent loop appends every tool result to its history, the prompt grows with every step. Step 30 costs many times what step 1 did, and the model reasons worse by then too. As a rough example, 30 steps where the input grows by 4K tokens each come to about 1.86M input tokens for a single task.

A token cap handles the accuracy problem. A dollar cap handles the cost problem. I wrote baar-core for the second one. It's an open-source Python library that puts a hard budget on every LLM call and returns a 402 before the provider is ever contacted. It reserves spend atomically, so twenty parallel agents that each look under budget can't overshoot the cap together. pip install baar-core. For teams, noburn.dev builds on it: once a user goes over their budget, it blocks their API calls before they're sent.

Both checks go in the same spot, right before create(), while you can still stop the call.

What the advertised window tells you

The number on the model card tells you how much the model will accept. It doesn't tell you how much it uses well. You have to measure that gap for each model, task, and prompt shape. It also changes whenever a provider ships a new version, so rerun the needle test after every upgrade.

Think of the advertised window like a car's top speed. It's useful to know, but it's not the speed you should drive at.

Have you measured the effective context size of your own pipeline, and how far was it from the spec sheet?

Originally published at https://www.robatdasorvi.com/chapters/ai-automation/why-context-window-size-is-the-thing-every-developer-should-care-about

DEV Community: Orvi Das

Pydantic vs Zod for Validating LLM Tool Call Arguments

Why does a validator pass a tool call with a made-up field?

Has this kind of failure happened before?

Pydantic or Zod: which one?

Doesn't strict validation make the model reason worse?

A rejection nobody counted

What schema validation can't catch

What to do when a call is rejected

What should you change in your schemas today?

Detect AI Agent Actions Outside Task Scope: Block, Don

Why doesn't my agent's log show anything wrong?

What counts as out of scope for an AI agent?

How do I detect an AI agent acting outside its task?

Isn't logging and alerting enough?

Is this worth building when most agent projects die first?

Multi-Turn Jailbreaks Against AI Agents With Tool Access

Why does a multi-turn jailbreak get past an agent's guardrails?

Why is a rule in the prompt not a permission?

Can a refusal be trusted to stop an agent?

How do you enforce tool permissions against a multi-turn jailbreak?

What a per-call gate does not stop

What would have stopped the Replit deletion?

Detect Unauthorized AI Agents Using Company API Keys

Can a company API key tell you which agent used it?

Isn't rotating keys and watching spend enough?

What a detector that watches the wrong surface looks like

Does per-key usage in the provider console already solve this?

What does detection actually require?

What am I still getting wrong?

Customer Support AI Agent Prompt Injection: Refund Abuse

Can a customer support AI agent be tricked into issuing a refund?

Why does a refund bot fail only some of the time?

Will better models stop prompt injection?

Who pays when the bot grants the refund?

Where should the refund limit live?

What the tool layer doesn't fix

What I expect the next post-mortem to say

Break-Glass Access for AI Agents: Approved, Timed, Expiring

What break-glass access means for agents

Why an approval gate is not a break-glass control

Is a plain process exit enough?

Doesn't human approval slow agents down too much?

How to implement break-glass for an agent

Why expiry matters more than the approval

Your Team Celebration Is an Event Handler. Make It a Cron Job (With the Workflow)

The research everyone cites, read backwards

"Won't celebrating make us soft?"

Moving it from on: release to on: schedule

What goes in the thread on a dead week

Where this came from for me

Policy as Code for AI Agent Tool Authorization: OPA or Cedar

Why Should AI Agent Tool Calls Go Through a Policy Engine?

An Agent Breaks a Prompt Rule Because Nothing Enforces It Where the Action Runs

Where Does the Policy Decision Point Pattern Come From?

OPA or Cedar: Which Should You Pick?

Doesn't a Policy Engine Add Latency and Another Point of Failure?

Should You Start With the Policy or the Enforcement Point?

Adding an API Enum Value Is a Breaking Change for Agent Tools

Is adding an enum value actually a breaking change?

Why does my spec diff call it safe?

Has this been solved before?

What happens when a client sees a value it doesn't know?

Isn't the consumer just wrong? Can't we patch it?

Is this worth a separate check, or is it a code review issue?

How does a new enum value break an agent's tool call?

What would I do before the next enum value ships?

LLM Structured Output: Validation, Retry or Constrained Decoding?

Does Constrained Decoding Fix Malformed JSON From LLMs?

Does Forcing JSON Output Hurt LLM Reasoning?

Is Retrying Malformed LLM Output Wasteful?

Does Schema Validation Stop Silent LLM Failures?

When Should You Use Constrained Decoding vs Validate-and-Retry?

How to Make LLM Structured Output Failures Visible

Durable Execution for AI Agents: Temporal vs a Plain State Store

What is durable execution for AI agents?

Do AI agents need Temporal to resume after a crash?

How do you stop a resumed agent from repeating side effects?

What does a minimal checkpoint table look like?

How do you checkpoint a multi-agent run with subagents?

Moving it from `on: release` to `on: schedule`