DEV Community: vsangaraju

The gate that stops an agent's memory from re-firing an attack

vsangaraju — Sat, 07 Nov 2026 04:37:47 +0000

The carryover problem

Most prompt-injection work treats the attack as a one-turn event. Agents with long-lived
memory and scheduled tasks don't work like that. If the injection gets written into memory, or turns into a recurring
task, it comes back. The attacker plants it once and leaves. The next session recalls it, the
heartbeat fires it, and it runs again with no attacker present and no user in the loop.
"After the Attacker Leaves" (CarryoverBench, arXiv:2610.28425) measures this. Planted payloads
keep re-firing across rounds without decaying. That paper measures the attack. It doesn't
build a defence. So I built one and piloted it.

What the gate does

The idea is simple. Before a recalled memory entry or a scheduled task is allowed to execute,
check where it came from. Not what it says. Where it came from.

Every entry gets a derivation record when it's stored. The store writes it. The agent never
does, because you can't trust an agent's later account of where something came from. The
record holds the source channel, the author, parent links, a policy version, and a hash over
the content and its parents' hashes. From that, the store assigns an authority class:

User-authored.
From a source the user approved.
Agent-derived, but only from class 1–3 parents with the chain intact.
Untrusted or unknown: web and tool content, pasted third-party text, unknown imports, broken chains, and anything the agent derived from class-4 material.

At re-fire time the gate checks the record again and applies a fixed table:

Classes 1–2 run.
Class 3 can read freely. A side effect has to be inside a pre-approved mandate, otherwise the gate asks the user.
Class 4 can be read, with a label telling the agent not to follow instructions in it. If the entry looks like an instruction, or it's about to cause a side effect, it's blocked and quarantined.
If the hash doesn't match or a parent is missing, it's quarantined as broken lineage, whatever its class.

The class-4 side-effect rule is block, not warn. That was a deliberate default.

The gate is stdlib Python, deterministic, with no model calls. A check took tens of
microseconds in the pilot (p50 37 µs in the harness, 53 µs on the benchmark).

How I tested it

There were two parts, and they measure different things.

A synthetic carryover harness. This is a small agent loop with a persistent memory file and
a scheduled-task list. I planted five latent payloads in memory and five autonomous payloads as
scheduled tasks. The autonomous ones include tasks created by an unknown plugin, tasks tampered
on disk after storage, and tasks the agent built from pasted third-party text. Alongside them
are legitimate controls (user reminders, agent summaries with intact chains, scheduled tasks
the user created), some benign untrusted entries, and a no-injection scenario. Each scenario
ran five rounds, gate off and gate on. I used two agents: a deterministic stub that obeys
anything it can see, and qwen3-4b-instruct through Ollama.

Amazon's Task-Tampering benchmark (arXiv:2611.07640). I plugged the gate in as a defense
condition in the benchmark's own runner. With the gate off, it's their runner untouched. The
pilot used 30 of the 90 cases: the original injection for each, plus a clean run, gate off and
on. That's 120 samples, and all 120 were scored.

I wrote the hypotheses down and committed them before the pilot ran. Nothing in the gate was
re-tuned after I saw numbers.

Results

Derivation-authority gating eliminates scheduled re-fires: on the synthetic harness, autonomous
attacks went from 60% to 0% (ollama agent) and 100% to 0% (deterministic stub), with zero false
positives on legitimate controls (0/605). On the single-session Task-Tampering benchmark the
drop was 20% to 13.3% (6 to 4 of 30 cases), within noise at this size. The gate does not
adjudicate truth: it withheld only 5 of 24 injection-bearing leaves, because most of this
benchmark's injections read as plain facts.

A bit more detail on each side.

Harness. With the real model, the autonomous channel went from 15 of 25 attack sessions to
0 of 25. The planted tasks that fired every round with the gate off (the unknown plugin's task,
the tampered task, the one built from pasted text) never ran with it on. Legitimate scheduled
tasks finished at the same rate, 97%, in both conditions.

The latent channel was flat at 20% for the real model, and that number needs explaining. The
small model ignored four of the five planted memory notes even with the gate off. The one that
worked was L5, a false claim that a landlord's account number had changed. It isn't phrased as
an instruction, and I said before the run that the gate would miss it. It did, 5 of 5 rounds
in both conditions. The stub, which obeys everything it sees, shows the mechanism more
clearly: latent went from 100% to 20%, and L5 is the 20%.

Benchmark. Overall attack success went from 6 of 30 to 4 of 30. By environment, devops went
from 3 of 10 to 1 of 10, marketplace was flat at 3 of 10, and productivity was 0 in both
conditions. That's a two-sample swing on 30 cases, so I'm not claiming much from it.

What it doesn't do

The biggest limit is in the title of that last result. The gate doesn't judge truth. It
judges where something came from and whether it's trying to act. Most task-tampering injections
in this benchmark don't say "ignore your instructions." They say things like "the policy was
updated" or "this key needs write access." Those read as facts, so the gate shows them to the
agent with an untrusted-origin label and moves on. Whether the agent believes them is up to the
agent. That's a design limit, not something tuning would fix.

It also isn't free on unlabelled content. In the clean benchmark runs, 13 of 30 samples (43.3%)
had some gate intervention, either a withheld leaf or a call held for approval. In the
harness, a third of the benign untrusted entries were quarantined. That third is one benign
note written like a request, blocked every round. The 0% false-positive rate is on the
legitimate controls as I defined them. It doesn't cover everything a real deployment would
read.

Two of my four hypotheses didn't come out cleanly:

H3, that the gate adds the most where the benchmark's own defenses are weakest, wasn't tested. I ran the gate alongside the none defense only. Composing it with spotlight, sandwich, trust-hierarchy and DRIFT is the next run.
H4, the no-injection negative control, didn't hold. In the harness, the real model's actions were the same with the gate on and off, but the email text wasn't. The gate had withheld a benign note and added labels, and the model wrote the agenda differently. On the benchmark, 20 of 30 clean cases matched. With one run per condition I can't separate gate effects from run-to-run variance there.

I also hit a bug: the chain check treated a shared ancestor as a cycle, which mislabelled four
blocked calls in the logs. None of them executed either way, so no outcome changed. It's fixed,
with a regression test.

Limits on the numbers

30 of 90 benchmark cases, one injection variant each. A difference of a few samples is noise.
The benchmark's LLM judge was mistral-nemo-12b running locally, not the model the paper used. Its accuracy on my preflight probes was 2 of 3. The agent was a 4B open-weight model. So none of my attack success rates compare to the paper's published numbers, and I'm not making that comparison.
The benchmark is single-session. I applied the gate where tools read content and where they get called. Real cross-session re-firing is only exercised by the synthetic harness.
In the benchmark adapter, the pre-approved mandate for agent-derived side effects is each case's list of required tools. That stands in for a user who approved exactly what their task needs. A real deployment would have to get that from the user.
Running the benchmark locally took three logged deviations: a bigger context window, a different judge (the first returned empty answers), and a request timeout. None touched the gate.

So what does it show?

Narrowly: if an attack only works by re-firing something the user never authorized, a scheduled
task or a tampered entry or a summary laundered from web content, a deterministic provenance
check at re-fire time shuts it down. In this pilot it did that with no false positives on
controls and microsecond overhead. It is not a general defence against task-tampering, which
in this benchmark mostly shows up as false facts the gate can't judge.

The code, the preregistration, the full report and every run log are here:
https://github.com/sangaraju1988/carryovergate

If you work on agent memory or scheduled agents, it's worth a look.

BI MCP or Warehouse MCP? A 7-toggle rubric for picking the right doorway

vsangaraju — Wed, 04 Nov 2026 16:04:37 +0000

One question I keep getting as a BI MCP builder serving a large enterprise user base: which MCP should I use for this use case?

Not "which one is better." Which doorway for this job.

The question usually arrives in a stronger form: "Why do we need a BI MCP when the warehouse MCP does the job?" The lazy answer is "because BI has a semantic layer." That's true and useless. It doesn't tell a team which doorway to give their agent on Monday.

I got tired of answering it from scratch every time, so I turned the answer into seven toggles. Here they are — and there's an interactive version at the end that gives you the verdict.

The 7 questions

1. Who's asking?
Others — stakeholders, customers, anyone non-technical → BI MCP. They can't validate raw SQL; they need answers built from vetted definitions.
You, and you know the data → warehouse MCP. A data engineer asking for themselves can read the SQL, spot a bad join, re-ask. No curation needed.

2. Does it need a defined business metric?
Yes — "revenue," "churn," "active user" → BI MCP. If two people can define the term differently, the definition must come from the model, not the model's imagination.
No — row counts, null rates, "what's in this table" → warehouse MCP. No business definition at stake; raw SQL is the honest tool.

3. One-off or repeated?
Repeated — dashboards, weekly reports → BI MCP. Repeated questions deserve a curated path: same definition, same shape, every time.
One-off — debugging, "let me check something" → warehouse MCP. Ad-hoc work dies in curation queues.

4. Do the consumers have warehouse access?
No → BI MCP, and this one is decisive. It's the only doorway that works. The BI tool queries under its own service identity while the end user has zero warehouse access; authorization happens at the BI project level. This is not a security hole — it's how BI works. The semantic layer is the authorization boundary.
Yes → either. Through the warehouse MCP the agent acts with the requester's own grants.

5. Exploring or asking a known question?
Exploring a new table, "what's possible here" → warehouse MCP. The curated model can't answer questions it wasn't built for.
Known business question → BI MCP.

6. Does the agent need to write or create?
Yes — DDL, staging tables, "build me a mart" → warehouse MCP. Decisive. The BI doorway is read-only by design.
No → either.

7. Does the semantic model cover this?
No → warehouse MCP, or go model it first. The honest caveat the semantic-layer pitch skips: BI MCP is only as good as the model behind it. An unmodeled question through the BI doorway gets a confident wrong answer or no answer.

The decision rule

Two of these are decisive: no warehouse access → BI MCP; needs writes → warehouse MCP. Otherwise, majority wins. A tie means hybrid — explore in the warehouse MCP, operationalize the metric in the semantic layer, serve it through BI MCP.

"Warehouse MCP" covers BigQuery, Snowflake, Databricks, Fabric. "BI MCP" covers Looker, dbt, Cube — any semantic-layer doorway.

Try it

I built the rubric as an interactive page — seven toggles, live verdict:
https://sangaraju1988.github.io/mcp-doorway-rubric/

And the repo (CC BY 4.0 — use it, credit it):
https://github.com/sangaraju1988/mcp-doorway-rubric

Two caveats

Managed warehouse MCP endpoints ship with guardrails — row caps, runtime caps. They rarely bind for aggregated Q&A, but bulk extraction belongs to the warehouse API, not any MCP doorway.

And unguarded agent + raw warehouse = surprise bills. Cost guardrails (quotas, dry runs, billed-bytes limits) belong in the warehouse MCP setup.

Most mature teams end up running both doorways. The rubric just tells you which one for this job.

Data apps, defined: levels, principles, and a scorecard builders can actually use

vsangaraju — Tue, 27 Oct 2026 13:06:17 +0000

The Data App Framework, v2.0 · October 2026
Venkata M Sangaraju, Sr. Database Engineer

On October 8, Anthropic put Claude Dashboards into beta. You connect Redshift, BigQuery, ClickHouse, Databricks or Snowflake, ask a question in plain English, and Claude writes the queries, picks the charts, and keeps the dashboard current. Four days later, Google announced Looker Data Apps in preview: an AI coding agent generates the app, and you publish it into Looker, where every query runs under the viewer's Looker permissions. Three weeks before either, Tableau ran its Dreamforce keynote under the title "Beyond BI" and introduced Tableau Data Apps, which you vibe-code with an outside agent and publish onto Tableau's governed data. Sigma's homepage now calls it a platform for "analytics, apps, and agents." Databricks has Apps. Snowflake has Streamlit and Native Apps. Domo has been selling "apps" for a decade and must be enjoying this.

So everyone has data apps, and nobody agrees on what one is.

I work on governed conversational analytics, and I keep having the same argument in design reviews. Someone shows an impressive demo, and the questions that decide whether it survives production get pushed to phase two: whose permissions does it run under, where do its metric definitions live, what happens when it writes something back. Those questions are the product, not a later phase.

Platform teams are hearing a version of this a lot right now. Someone starts building in Claude Dashboards and likes what they get. Then they notice that their row-level and column-level security, and the role mappings behind it, already live in Looker or Tableau, and none of it carries over to the new tool. Nobody wants to rebuild a security model for every new tool. They want the new app to inherit the one they already trust. That's a reasonable ask, and today it's harder than it should be.

This framework is my attempt to write down the vocabulary those reviews need: a definition, levels, principles, the failure modes I keep seeing, and a scorecard. Map your product against it and tell me where it's wrong.

What a data app is

A data app is a software application whose primary interface is live data, built on governed definitions, whose output is a decision or an action and not only a display.

Live data rules out the static extract dressed up as an app. Governed definitions rules out the app that quietly re-implements "revenue" in its own SQL and drifts from the official number for a year. And decision or action rules out most of what got announced this year. If nothing can change because the app exists, it's a report with better marketing.

Under this definition a dashboard isn't a different species from a data app. It's the simplest kind. What separates products is how much they're trusted to do, and in production systems trust should come in levels.

The autonomy levels

Classify a data app by the most consequential thing it is allowed to do, the way self-driving got classified by how much of the driving the machine does. Governance follows the level, not the vendor or the roadmap.

Level	What the app may do	Governance floor
L0 View	Shows governed metrics; filtering is the whole interaction	Certified definitions, viewer-level access control
L1 Explore	Drill-downs, follow-up questions, conversational querying	Row/column security enforced when the query runs, not in the UI; queries logged
L2 Simulate	What-if inputs and scenarios; nothing is written anywhere	Assumptions stored with every output, so a scenario can be re-run and audited
L3 Act	Writes back to the warehouse or triggers a workflow; a human approves each action	Runs as the end user; every write approved, logged and reversible
L4 Delegate	An agent acts on its own inside a pre-approved mandate	Machine-readable mandate with scope, limits and expiry; a kill switch; lineage on every action

Two clarifications, because this is where marketing will try to blur things. Level is per capability, not per app: an app that views at L0 and submits at L3 is an L3 app for governance purposes. And levels can't be skipped. An app that writes back while its row-level security lives only in the UI is missing a level, and that gap is where incidents come from.

Now place the market. The BI vendors are climbing from the bottom. They're strong on L0–L1 governance and are adding write-back and actions: Sigma writes back to the warehouse and runs actions inside warehouse permissions, and Tableau's MCP servers push governed answers into Slack, Teams, Claude and ChatGPT, with published Data Apps inheriting Tableau's permissions.

The AI labs are coming down from the top, with generation nobody can match and governance still arriving. Claude Dashboards shows the query behind every chart and when each chart last refreshed, a level of query transparency BI tools don't usually put in front of business users. Anthropic also positions it as working alongside BI tools for quick, exploratory questions, with a hand-off to Sigma, Hex, Omni and others when a question needs deeper analysis (Looker and Tableau are listed as coming soon). In this framework's terms, that's an L1 product that sends you to the governed layer when the stakes go up. It's an honest description of where each side is strong today.

Four principles

1. Definitions before displays. A metric gets defined once, in the semantic layer, and every app consumes that definition. Once an app embeds its own calculation you have two sources of truth, and eventually a meeting about why they disagree. Looker built its product around this idea with LookML more than a decade ago, and it's still the clearest example of a metric defined once and reused everywhere.

2. Run as the user, not as the app. Queries and actions execute with the requester's identity and permissions, down to the data and back up to whatever the app triggers. That can happen in two ways. The warehouse can see the user directly, through per-user credentials. Or a governed semantic layer can compile that user's row and column policy into every query it generates, on every surface, even though the warehouse sees one connection. Looker's access filters and Tableau's data policies work the second way, and it counts. What doesn't count is an app that connects to the warehouse with its own credentials and enforces permissions only in its interface. That app has stepped around the governed layer, and every user of it effectively holds the app's access. Pull up your warehouse query log and check where the app's queries come from. If it's a service account that isn't your governed semantic layer, your row-level security is a rumor.

3. Writes are transactions, and derived data keeps its lineage. A write-back needs the acting identity, the data it was based on, its approval state, and a way to reverse it. Whatever the app writes becomes someone else's source data tomorrow, so it should carry its lineage with it. When a number is wrong in a quarter-end review, "the app did it" isn't an answer. You need to know which app ran, as whom, on what data, and who approved it, and you can only answer that if the platform recorded the derivation at write time. I make the formal case for derivation-aware access control in agent systems in "Lineage-Aware Memory Governance" (IEEE Access, 2026). The short version is that access decisions have to know where data came from.

4. Generated apps are drafts, and drafts need owners. A prompt-generated app is unreviewed code whose author happens to be a model. Treat it like unreviewed code from a new hire: fine for exploration, gated before production, with the gate proportional to the autonomy level. Light for L0, mandatory for L3. Anthropic ships Dashboards switched off by default for Enterprise organizations, so an admin has to turn it on, and that's the right default for this category. Every app that does reach production also needs a named owner, a review date, and a way to be retired. BI teams spent a decade producing dashboard sprawl, thousands of unowned near-duplicates. Generated apps can produce the same sprawl in a quarter, and these ones hold credentials and can write.

Where governance has to live

Every platform here is some arrangement of the same parts: sources, a data platform with its own security model, a governance layer (definitions, identity, policy, lineage), an app runtime, and the surfaces where people meet the app, from portals and embeds to chat tools and AI assistants. The architectural question that matters is whether the governance layer is a plane every runtime and surface passes through, or a feature each app implements for itself.

Feature-level governance breaks down once the app is distributed. MCP is about to put the same app inside your portal, your Slack, your CRM and two AI assistants. If policy lives in the app, you maintain it in five places and the weakest copy defines your actual security. If it lives in the plane, enforced when the query executes against definitions written once, a new surface inherits it. Sigma argues for this with its warehouse-native design, which runs live queries with governance at the source. Tableau argues for it with Tableau Knowledge and its auto-built knowledge graph. They put it at different layers, but the instinct is the same, and I think it's correct.

Failure modes

These are the ones I see most often. Name them in reviews.

The service-account masquerade. The app queries the warehouse with its own service account, outside the governed layer, and per-user security exists only in its UI. Check the warehouse log.

Forked definitions. The app computes its own version of a governed metric. To test, change the certified definition. If the app's number doesn't move, it's a fork.

Stale data that looks live. A "live" badge on a dashboard fed by an hourly pipeline, refreshing at per-click warehouse prices. Or an upstream schema change makes the app's queries fail quietly, and the first person to notice is an executive quoting an old number. Ask for the refresh contract per widget, and ask which contract tests catch a schema change.

Unreviewed promotion. An app built for exploration picks up a submit button along the way, and nobody reclassifies or re-reviews it. There is probably an L3 capability in your estate right now that no one remembers approving.

Scorecard

For vendor evaluations and release gates. Mark each question Described, Partially described, or Unknown — anything the platform's public material does not describe is Unknown, never absent. An app that fails question 2 or 4 shouldn't run above L1.

Do the metrics resolve to the governed semantic layer, or does the app define its own?
Does every query and action run under the end user's identity, either directly or through a governed layer that enforces that user's policy?
Is row/column policy enforced at execution time on every surface where the app appears?
Can every output be traced to its inputs, definitions and derivations?
Can a reviewer see the generated SQL and logic before release?
Is the app in a registry, with an owner and a review date?
Is freshness a stated contract, and do contract tests catch schema or definition changes before users do?

Using this

The canonical copy of this framework lives at github.com/sangaraju1988/data-app-framework, versioned, with issues open. Cite and fork that copy and file corrections there; this post is the announcement. The first product mapping, for Claude Dashboards, is already in the repo. Vendors, publish your mapping against the levels and the scorecard. Builders, classify what you're shipping before your security team does it for you.

Venkata M Sangaraju is a Sr. Database Engineer working on governed conversational analytics and agent memory governance, and the author of "Lineage-Aware Memory Governance" (IEEE Access, 2026).

Your agent's skills have no lockfile — a derivation graph for skill provenance

vsangaraju — Sat, 24 Oct 2026 20:01:53 +0000

Venkata M Sangaraju, Sr. Database Engineer

Repo: github.com/sangaraju1988/skill-derivation-graph — spec v0.1, reference implementation, Zenodo DOI 10.5281/zenodo.23623206. Discussion: agentskills/agentskills#620.

A skill written for one business unit's reporting — how ambiguous cases should be handled — was copied by another team and built on. The original was later updated because some of the metrics it referenced had been deprecated. The copy never got that change. The other team's agent kept failing tool calls on the deprecated metrics, retried, tried to work around them on its own, and eventually errored out. It took a deep investigation to trace the failures back to a stale copy.

One line saying "derived from X at ref Y" would have turned that into a quick lookup.

That is the whole problem. Agent skills spread by copying, and a copy carries no record of where it came from. There is no lockfile for skills.

The scale of the copy problem

"Skill Constellations" (Seddik, UBC Okanagan, 8 October 2026, arXiv:2610.11169) reconstructed a dated copy network over 2,193,119 skill adoptions on GitHub. Half of all SKILL.md files are verbatim copies of another file. And when a source skill is fixed, only 11.1% of changes to a clone group reach every copy. Most copies keep running the old version.

So the ecosystem has the copy behaviour of a software supply chain and none of its provenance. A patched skill leaves most of its descendants silently running the vulnerable version, and nobody loading a skill can cheaply ask: is this current, and what is it derived from?

Anthropic donated Agent Skills to the Agentic AI Foundation (AAIF, Linux Foundation) on 22 October 2026, and the starting roadmap covers versioned releases, a formal proposal process, and security guidance. Versioning answers "which release is this?" It does not answer "which skills descend from this one, and are they still carrying the old version?" That second question is what this post is about.

The proposal: two optional keys

The full spec is in the repo — Skill Derivation Graph, v0.1 — but the part I'm proposing to the AAIF Agent Skills repo is deliberately small: two optional string keys under the existing metadata map in SKILL.md.

metadata:
  derived-from: "https://github.com/acme/skills/tree/3f2c9e1/doc-summarizer"
  derived-from-hash: "sha256:9b1d…"

derived-from is where this skill was copied or adapted from, pinned to a ref (commit SHA, tag, whatever the author uses). No version format is imposed.
derived-from-hash is the SHA-256 of the parent's normalised SKILL.md at that ref (line endings normalised to LF, trailing whitespace stripped, these two keys excluded).

Verbatim vs. modified is not declared anywhere. Tooling derives it: hash the copy's own normalised SKILL.md (with the two keys removed) and compare it to derived-from-hash. Same hash, verbatim copy. Different, modified fork.

Agents that don't know the keys ignore them. Nothing is required of anyone. And because the lineage lives in SKILL.md itself, it survives the copy — which is the point, since a root manifest doesn't travel with a skill copied as a directory.

What tooling can do with it

When a source publishes a fix, a tool walks forward through the lineage from the patched version:

Verbatim copies of the vulnerable version inherit the bug by construction, so they get flagged stale — blocked by default, or warned loudly, per the consumer's policy.
Modified forks get flagged for review. The fix may or may not apply, so a human or a diff decides. No silent upgrades.
Broken or missing lineage means "staleness unknown" — never "current." A consumer that can't reach a derivation index has to say so rather than reporting a skill as current.

A deliberately pinned old version is a warning, not a block: a newer version exists, and you load the old one only because you chose to.

The underlying model is a small graph: skills and skill versions as nodes, with three edges — DERIVED_FROM (copied or adapted from), SUPERSEDES (replaces a previous version), PATCHES (a fix release for a vulnerable version). The walk is a breadth-first traversal from the patched version over child edges, carrying a flag for whether any modified edge was crossed on the path. The full semantics, the provenance record schema, and the versioning rule are in the spec.

What it is not

Three limits, stated plainly, because a provenance proposal that hides them isn't honest:

It is not trust. The hash is for identity, not trust. A lineage claim is self-declared and unverified — a malicious author can claim a reputable parent. Consumers should weight provenance by author trust and treat unattributed authors accordingly.
It is not a registry. The graph is registry-agnostic. It works over skills wherever they live, and nothing here requires AAIF adoption to be useful locally: a team can run the graph over its own skill corpus today.
It is not versioning. It sits alongside versioning. AAIF versioning gives each release a stable version; the provenance record references those versions and adds the parent pointers and content hashes that make derivation walkable.

The repo: spec, code, and a benchmark with no results yet

The repo holds three things:

SPEC.md — the v0.1 specification: the derivation-graph model, the provenance record schema, the versioning rule, the fix-propagation mechanism, and a benchmark definition.
A reference implementation — standard-library-only Python. Content hashing of a SKILL.md file, a small derivation graph, and a fix-propagation walk, with a worked 3-node example (source → verbatim copy → modified fork) and unit tests.
A benchmark definition with no results. The spec defines five metrics — fix reach %, time-to-fix, stale-descendant count, fork-detection accuracy, unresolved share — and a synthetic-corpus protocol for measuring them, including the key control: report the same metrics with the derivation graph removed (hash-only matching, no parent pointers) to show what the graph adds. No measurements exist yet, and none are claimed. If the graph adds nothing, the spec has failed — that's the test.

The spec is also archived on Zenodo (DOI 10.5281/zenodo.23623206) so it can be cited.

The discussion

The proposal is live as agentskills/agentskills Discussion #620, which also relates it to the existing threads: versioning (#420), upstream-source declaration (#436), the skill package manifest (#210), licensing and attribution (#379), and provenance attestation (#358) / supply-chain trust (#418). It ends with three questions for maintainers and implementers:

Is a shared key name worth converging on, so independent scanners and registries interoperate?
Is the hash normalisation — LF line endings, trailing whitespace stripped, these two keys excluded — enough, or should the hash cover the whole skill directory?
If this doesn't meet the bar for the spec, would a short "recommended conventions" note in the docs be acceptable?

Try to break it

If you maintain skills, vendor them, or run an index over them: try the convention on a copied skill, run the propagation walk over your own corpus, and tell me where it's wrong. The sharpest feedback is a case where the two keys don't survive a real copy workflow, or where the verbatim/fork distinction by hash gives the wrong answer. Issues are open on the repo.

github.com/sangaraju1988/skill-derivation-graph

Venkata M Sangaraju is a Sr. Database Engineer working on governed conversational analytics and agent memory governance, and the author of "Lineage-Aware Memory Governance" (IEEE Access, 2026).

The demo works. Production doesn't: what breaks when you put an LLM on your company's data

vsangaraju — Thu, 22 Oct 2026 19:54:17 +0000

Every conversational analytics demo follows the same script. Someone connects a model to a clean warehouse schema, types "show me revenue by region last quarter," and gets a correct chart in four seconds. The room nods. Then somebody says the sentence that has launched a thousand doomed pilots: "Now let's roll it out to the whole company."

I have spent the last few years doing exactly that rollout — building governed conversational analytics on Looker and its semantic layer, most recently exposing it to AI agents through the Model Context Protocol (MCP). The rollout I work on now runs at enterprise scale, and the metric that matters isn't the user count. It's accuracy — the kind you can defend in a board meeting — achieved not by using a bigger model, but by refusing to let the model touch raw data at all.

Here is what actually breaks between the demo and production, and why the fix is architectural, not prompt-based.

Break #1: Your company doesn't have one definition of anything

Ask a demo dataset for "revenue" and there's one column. Ask a real company and you get a knife fight. Finance's revenue is recognized revenue, net of refunds, on an accrual basis. Sales' revenue is bookings. Support's "active customer" hasn't opened a ticket in 90 days; Growth's "active customer" logged in this week. These aren't errors. They're legitimate, battle-tested definitions that happen to share a name.

An LLM pointed at raw tables has no way to know which definition you meant — so it guesses, fluently and confidently, and it guesses differently on Tuesday than it did on Monday. This is the failure mode that kills trust in production, and it never appears in demos because demo schemas have one table per concept.

The fix is unglamorous: a semantic layer where each metric is defined exactly once, in code, with the business logic owned by the people accountable for it. In Looker, that's LookML — measures, dimensions, and joins curated by data teams, version-controlled, reviewable. When the model asks for "revenue," it isn't writing SQL against a guess. It's invoking a definition that already survived your company's internal arguments. The semantic layer is the contract between the LLM and your data: the model decides what the user is asking, the layer decides what the data means.

This is also why text-to-SQL demos age so badly. Generated SQL re-derives your business logic from scratch on every single question, with a probabilistic engine. A semantic layer derives it once, deterministically, and reuses it forever.

Break #2: The model is the wrong place to enforce access control

The second thing that breaks is security, and it breaks quietly. In a traditional BI setup, access control lives close to the data: row-level filters, column restrictions, user attributes. Put an LLM in front of the warehouse with a broad service account — the default in most quick integrations — and you have just handed every user the union of all permissions the agent can reach, wrapped in a friendly chat interface.

Natural language makes this worse, not better. Nobody types "SELECT salary FROM…" into a chat box by accident; they ask "who on the team got the biggest raise?" A model eager to be helpful will route around your intent unless the data layer itself says no.

The principle that works in production: the agent inherits the requester's permissions, never its own, and enforcement happens at the data layer, not in the prompt. With a semantic layer in the middle, every query the agent issues carries the user's attributes, and the same row- and column-level rules that govern your dashboards govern the conversation. "Please ignore your restrictions" is not an attack a data layer has to care about. I've written elsewhere about the harder version of this problem — what happens when agents start caching and sharing results with each other — but the baseline rule is simple: if a control only exists in the system prompt, it doesn't exist.

Break #3: "Mostly right" is a failing grade

In a demo, a slightly wrong number is a chuckle. In production, one executive catches one wrong number in one board meeting, and your rollout is dead for a year — not because the system is bad, but because trust in data tools is binary in a way trust in search results isn't.

This is why the architecture above isn't optional decoration. Deterministic definitions (the semantic layer), deterministic permissions (the data layer), and a model constrained to intent-parsing and presentation is how you get from "impressive demo" to answer accuracy you can defend in production. It also gives you something a raw LLM integration never will: an audit trail. Every answer can be traced to a governed query against named definitions, which matters the first time someone asks "where did this number come from?" in a regulated business.

Connecting this to agents through MCP is what makes it timely. When we exposed the semantic layer through Looker's MCP server — including to desktop agent clients across the company — the agents didn't get new powers. They got a well-lit doorway into powers that were already governed. That's the pattern I'd argue for everywhere: don't build a special, weaker governance regime for AI. Route AI through the governance you already trust, and let the model be the interface rather than the authority.

What I'd tell any team starting today

Start with twenty questions your business actually asks, define those metrics properly in a semantic layer, and only then connect the model. Resist the warehouse-wide text-to-SQL launch; it's a demo that scales its failure modes along with its coverage. Enforce access where the data lives. Measure accuracy like an SLO, with a golden set of questions and known-good answers, and publish the number internally — nothing builds adoption like a team willing to be measured.

The demo is the easy part; it's supposed to work. Production is where the same KPI has three definitions, the helpful model outruns its permissions, and one wrong number costs you the room. All three failures are fixable — but they're fixed in your data architecture, not in your prompt.

Venkata M Sangaraju is a Sr Database Engineer working on governed conversational analytics and AI agents over enterprise data. He spoke on this work at Google Cloud Next '26 and co-authored "Lineage-Aware Memory Governance" (IEEE Access, 2026).

I built a small library so my LLM agent stops double-charging people

vsangaraju — Fri, 24 Jul 2026 21:03:30 +0000

Agents that call tools over a network eventually retry a call they shouldn't have. If the tool isn't idempotent, that retry becomes a duplicate charge, a duplicate email, a duplicate order — quietly, with nothing in the logs to flag it. It's a decades-old distributed-systems problem wearing a new agent costume, and as far as I could tell, nobody had shipped a small, pip-installable fix for the agent-shaped version of it. So I built one. It's called latch. Worth saying up front: this is not a novel idea, and I'd rather say so myself than have someone point it out in the comments.

The bug, live

Here's the whole thing in one file, no library involved:

def charge_card(order_id, amount):
    time.sleep(0.4)  # the payment API is a little slow today
    ledger[order_id] += 1
    return {"order_id": order_id, "status": "charged"}

def agent_charge_with_naive_retry(order_id, amount, max_retries=2):
    for attempt in range(max_retries):
        thread = threading.Thread(target=lambda: charge_card(order_id, amount))
        thread.start()
        thread.join(timeout=0.2)  # the agent's own client-side timeout
        if thread.is_alive():
            continue  # "no response, let's retry"
        return  # got a response

The payment call takes 0.4s. The agent gives up waiting after 0.2s and retries. The first attempt is still running in the background — it wasn't cancelled, it can't be safely cancelled, Python threads don't work that way. So it finishes on its own time and charges the card. Then the retry charges it again. The agent's own view of the world is "everything's fine, got a response eventually." The ledger says otherwise.

This is examples/naive_agent_example.py in the repo, if you want to run it and watch it happen rather than take my word for it.

None of this is exotic. It's the same class of problem as "what happens when a payment gateway's webhook fires twice," which every backend engineer who's touched Stripe has dealt with. The fix has a name — idempotency keys — and it's been standard practice in backend systems for a couple of decades. It just hadn't shown up as an off-the-shelf thing you pip install for the agent-tool-calling version of the same problem. That's the gap.

What latch actually is

A small Python library, no required dependencies, that wraps a tool function so calling it twice with the same idempotency_key only executes it once:

from latch import idempotent

@idempotent()
def create_order(order_id: str, amount: float) -> dict:
    return payments_api.charge(order_id, amount)

create_order(order_id="A1", amount=42.0, idempotency_key="run-7-step-3")
create_order(order_id="A1", amount=42.0, idempotency_key="run-7-step-3")  # cache hit, no second charge

idempotency_key is required, not auto-generated from the arguments. I went back and forth on this — auto-hashing the args is tempting and would make the decorator feel more magical. I didn't do it, because whether two calls with slightly different arguments should count as "the same operation" is a judgment call only the caller has the context to make. Guessing wrong silently is worse than making the caller type one extra kwarg.

Idempotency turned out to be the smallest piece of a slightly bigger problem, so three more decorators and one small orchestration class rode along on the same package over the following weeks:

what	decorator	stops
idempotency	`@idempotent`	duplicate side effects on retry
circuit breaker	`@circuit_breaker`	hammering a dependency that's already down
timeout	`@with_timeout`	a hung call blocking the whole agent loop
budget guardrail	`@budget_guardrail`	a runaway loop burning unbounded cost
saga	`Saga`	a multi-step plan (charge → book flight → book hotel) left half-done when step 3 fails

They're independent — use one, use all five, doesn't matter — but they compose on the same function:

@budget_guardrail(max_calls=100, window_seconds=3600)
@circuit_breaker(failure_threshold=5, recovery_timeout=30.0)
@with_timeout(seconds=10)
@idempotent()
def charge_card(order_id: str, amount: float) -> dict:
    ...

There's also an opt-in tracing hook (tracer=, subscribe to what each primitive is doing — cache hits, circuit trips, saga rollbacks) and a small chaos-injection module for testing your own retry logic against simulated failure/latency instead of hoping it works. Neither is load-bearing to the pitch, so I'll leave the details to the README.

Does it actually help, or is that just a nice story

I didn't want to ship this on vibes, so there's a benchmark that runs the exact double-charge scenario above, twice, under identical injected latency — once with nothing, once with @with_timeout wrapping @idempotent:

$ python benchmarks/chaos_benchmark.py --seed 1 --operations 30

metric                             naive   protected
----------------------------------------------------
orders attempted                      30          30
orders reported successful            21          30
orders reported failed                 9           0
total real charges issued             65          30
orders double-charged                 21           0
idempotency cache hits               n/a          18

30 simulated orders, same seed, same latency curve on both sides. The naive column: 21 of 30 got double-charged, and the agent's own success/failure reporting was wrong for a third of the batch (9 reported as failed outright, despite the card sometimes still getting charged). The protected column: zero double charges, and the "failure" case just becomes a cache hit on the retry instead of a second execution. A second run at a different seed (seed 7, 20 orders) landed at 12 double-charged naive vs. 0 protected — same shape, not a cherry-picked seed.

The part I actually want to talk about

Before I wrote a "ready for public use" README section, I made myself run a real audit against my own already-published package — not just re-reading the code, but fresh-install smoke tests, every snippet in the docs executed for real, and — this is the part that mattered — actual multi-threaded and asyncio.gather concurrency tests. My existing 111 tests were all sequential: call, retry, assert. Nothing had ever hit the decorator with two threads at once.

Turns out that matters. I found four real, silent bugs in code that was already on PyPI:

@idempotent didn't dedupe functions that return None — store.get() is not None can't tell "nothing cached" from "cached None," so a fire-and-forget tool (a delete, a notification) never got deduped at all.
Two decorated functions sharing a store could collide on the same key and silently return each other's cached result.
Under real concurrency — not sequential retries, actual threads racing — the "check cache, then execute" sequence wasn't atomic. N threads calling the same key at the same instant could all see a cache miss and all execute.
The circuit breaker's half-open state let unlimited concurrent calls through as "trial" calls, when the whole point of half-open is exactly one trial call.

Every one of these is the kind of bug that doesn't crash, doesn't throw, doesn't show up in a code review — it just quietly does less than it promises to. I'd rather find that myself before anyone else's production traffic does. Fixed all four, added 17 regression tests specifically targeting concurrency (not just logic), and wrote up exactly what was wrong in the changelog instead of quietly folding it into "misc fixes." If you're evaluating a library like this for anything real, "here's what I found wrong with my own code and how" is more useful to you than a changelog that only ever says "improvements."

What it's not

It's Alpha software and I'd rather undersell it than have you find the ceiling the hard way:

Saga has no persistence. Compensation runs in-process; if the process dies mid-saga, nothing resumes automatically. If you need crash-durable multi-step workflows, that's a job for Temporal or Step Functions, not this.
The circuit breaker and budget guardrail hold state per-process. Run five replicas of your service, you get five independent circuits that don't talk to each other, unless you build that coordination yourself.
A sync @with_timeout can't actually kill the underlying call — Python has no safe way to force-kill a thread. It unblocks the caller and raises, but the original call might still be running in the background. This is exactly the scenario @idempotent exists for, which is why they're meant to be stacked, not used as substitutes for each other.

None of that is a secret dealbreaker — it's just the actual scope, written down instead of implied away.

Prior art, one more time

To be clear about credit: SagaLLM (arXiv 2503.11951) already applies the Saga pattern to multi-agent LLM planning, Robust Agent Compensation (ACM CAIS) covers agents learning to compensate for their own failures, and ReliabilityBench is a whole benchmark for exactly this space. Circuit breakers and timeouts are decades-old patterns with existing Python libraries (pybreaker, tenacity) that do them well in a general context. I'm not claiming to have invented any of this. What I think was actually missing was the boring part — a small, tested, zero-dependency, pip install-able version scoped specifically to the shape of an LLM tool call, that a working developer can drop into an agent loop this afternoon without reading a paper first. If that's wrong and something like this already exists and I missed it, I'd genuinely like to know — tell me in the comments.

Try it

pip install latch-idempotent

Repo: github.com/sangaraju1988/latch
PyPI: pypi.org/project/latch-idempotent
The double-charge demo above: examples/naive_agent_example.py / examples/resilient_agent_example.py

If you're building anything that calls a tool with a real side effect and retries on failure — which, if you're building agents, you already are, whether you've thought about it that way or not — I'd be curious whether this is useful to you, or where it falls over. Issues and PRs are welcome, and honestly the "where does this break" feedback is worth more to me right now than stars.