DEV Community: Viktor Kessler

Two Owners, One Dataset: Why Governance Has to Shift Left

Viktor Kessler — Mon, 19 Oct 2026 12:05:08 +0000

Max runs the order system. Every order is a row in his Postgres database, and he knows exactly what each column means, who may read it and why. Alice runs analytics. She works on a copy of the same orders in the lakehouse, exported overnight, cleaned, joined and secured again with her own rules.

On Monday the board asks how many orders were cancelled last quarter. Max says one number. Alice says another. Both are right about their own data, and neither owns the answer.

This is not a story about bad engineers. It is the normal shape of a data platform: data is born in one place, copied to another, and every copy gets a new owner and a new set of rules. We have spent a decade making the lakehouse faster and more open. We have spent much less time asking who owns the data once it leaves home.

Every hand-over is a governance break

Follow the orders from Max's table to Alice's dashboard and count the hand-overs: export, land, clean, join, secure, publish. Each hop is a place where ownership changes hands and access rules are written again, by someone else, in another tool. Four things go wrong, every time.

Two owners, one dataset. Max owns the source, Alice owns the copy. Nobody owns the truth end to end, so every disagreement becomes a meeting.
A governance break at every hop. Each copy gets new permissions. A rule that masks a customer's IBAN in Postgres does not travel with the export. Policies drift, and lineage is lost on the way.
Unstructured data stays ungoverned. Contracts, scans and PDFs sit in buckets with no catalog, no owner and no policy. They are exactly what we now feed to AI.
Agents multiply the problem. Every new agent gets its own extract and its own keys. There is no central control and no audit of who did what.

None of this is a tooling failure. It is what happens when governance is something we add at the end of the pipeline instead of something the data carries from the start.

The problem is not the bytes. Moving data is often the right call. The problem is that ownership and rules do not move with it.

Why "fix it later" stopped working

For years the copy chain was annoying but tolerable. A data team could reconcile the numbers, rebuild the permissions and explain the difference in a footnote. Three things have changed that.

AI reads everything. A model or a retrieval pipeline does not stop at the tables someone curated. It reads the PDFs, the images and the raw exports, the data that never had an owner. If you cannot say what went into an answer, you cannot defend the answer.

Agents act, and they act fast. An agent working for a user reads from Postgres, writes to the lake and calls the next system in seconds. Every extract it takes is another copy with another set of keys. Governance that runs once a quarter cannot keep up with software that runs every second.

Regulators want proof, not intent. In financial services, BCBS 239 asks banks to show where risk data comes from and that it is accurate. DORA asks them to prove control over their ICT systems. "We copied it, cleaned it and re-secured it" is not proof. It is a chain of trust with a weak link at every hop.

Shift left: govern where data is born

Software engineering learned this lesson years ago. Testing and security moved "left", closer to where code is written, because finding a problem in production costs far more than preventing it at the source. The data world has started the same move for quality, with data contracts at the producer.

Access governance has not moved yet. We still secure data where it lands: in the warehouse, in the BI tool, in the vector store. Shifting left means the opposite. The owner of the data at its source, Max, defines who may see what, once, and that definition follows the data wherever it is read.

In practice that means three things:

One catalog that reaches the source. The Postgres tables, the Iceberg tables in the lake and the files in object storage are all visible in one place, with one owner each.
One policy language for all of it. A tag like pii on a column means the same rule whether the column is read in Postgres, in the lake or by an agent.
No re-owned copies. Data may still move, from Postgres into Iceberg for example, but it moves as the owner's data product: same owner, same policy, same lineage. Consumers read that product instead of building their own secured copy.

Alice does not lose anything. She still builds her analysis. She just builds it on Max's data product rather than on a private copy of it.

Continuous governance: decide at every access

Shifting left fixes where rules are written. Continuous governance fixes when they are applied.

Most organizations govern in batches. Permissions are granted in tickets, copied into each engine and reviewed once a quarter in a spreadsheet. Between reviews, nobody can say with confidence who can read what right now. When someone leaves a team, the revoke reaches one system and not the other four.

Continuous governance replaces that with three simple properties:

Define once. Rules live in one place as policy as code, not as grants scattered across engines.
Enforce at every access. Each read is checked against the current policy at the moment it happens, by whoever is asking: an analyst in Snowflake, a Spark job, an agent acting for a user. Access comes as short-lived credentials, not standing passwords.
Prove it all the time. Every allow and every deny lands in one audit log. The answer to "who could see this table last Tuesday" is a query, not a project.

This is also how governance scales to agents. An agent acting for Bob should get exactly Bob's permissions, evaluated the same way as Bob's own query, and leave the same trail. If the rule changes, the next request already follows the new rule.

The difference is not more security on the right. It is fewer places where security has to be rebuilt.

A path, not a big bang

Nobody rebuilds their data estate to fix ownership. The good news is that you don't have to. Shift left works best as a sequence, where each step pays off on its own:

Start with the lakehouse. Put one catalog and one policy set in front of your Iceberg tables, so every engine reads and writes through the same rules.
Bring in the other catalogs. Federate the tables your Snowflake and Databricks teams own. They keep ownership; everyone else finds every table, and its rules, in one place.
Bring in the files. Give every image, PDF and document an owner, a policy and a version, before it feeds a model.
Reach the source. Govern Postgres tables in the same catalog, and replace standing database passwords with short-lived credentials per user.
Treat agents as first-class. Same policies, same audit log, acting as the user they work for.

This is the path we are building Lakekeeper for, an open-source Apache Iceberg REST catalog with Cedar policy as code. Where it stands today:

Step	Status in Lakekeeper
One catalog and policy set for Iceberg, short-lived storage credentials	Available today
Images, PDFs and documents as governed Datasets	Preview, usable today
Postgres tables and per-user Postgres credentials via pg_lakekeeper	Next
Federation of Unity Catalog and Snowflake Horizon	On the roadmap
Per-user decisions inside Snowflake and Databricks	Depends on engines passing end-user identity

We would rather be precise about that table than promise the whole picture at once. Each row is a step a team can take on its own.

One owner, one number

Go back to Monday's board meeting. In a shifted-left world, Max owns the orders from the moment a row is written to the moment it appears in the board report. Alice builds her analysis on his data product. The orders still land in the lake as Iceberg tables, but Max publishes them, owns them and sets their rules, and nobody re-secures them on the way. The rule that hides customer IBANs is written once and applies in Postgres, in the lake and to the agent that drafts the summary. When the board asks how many orders were cancelled, there is one number, and an audit log that shows who could see what along the way.

That is the real promise of shift left and continuous governance. Not more controls, but fewer places where control can break.

If you are working through the same problem, I would like to hear where your copy chain hurts most. Write to me or find us on the Lakekeeper Discord.

Extending Zero Trust to Your Agents' Memory

Viktor Kessler — Tue, 22 Sep 2026 11:01:16 +0000

Assume the agent is compromised. Then decide where its memory lives.

I ended a piece two weeks ago with a claim: a catalog is where governed agent memory starts, and a control plane is what it has to become. Claims are cheap. So we built it, as a runnable example — two agents keeping private memory in one catalog, proposing skills they are unable to publish, and a human promoting them. I work on the catalog in question, so read the comparisons below with that in mind.

The frame is zero trust, pointed at something that has mostly escaped it: nothing in the agent's code enforces any of this. The agent is assumed hostile, and every refusal comes from the catalog.

It works. Rather more interesting is what had to change on the way: four points where the system, rather than an argument, decided how this had to be built.

The boundary is the dataset, not the record

The obvious design is one memory store with a column for whose memory it is. Nearly everything does this: mem0 scopes with a user_id, a vector database with a metadata filter, Databricks with a key. It is the shape anyone would draw first.

It cannot be enforced. A catalog that vends credentials issues them for a location: a prefix in object storage, scoped to a dataset. Everything under that prefix is reachable by whoever holds the keys. There is no row-level anything, because at the moment of enforcement there are no rows — there are bytes at a path, and a credential that either covers the path or does not.

And there is a second leak that row scoping does not even reach. Memory is not only entries; it is the embeddings built over them. One shared recall dataset lets anyone who can read it vector-search every memory it indexes, whatever the paths inside look like. Similarity does not respect a whose column.

So a scope is a dataset — entries and embeddings together. If two agents must not read each other's memory, they get two datasets, in two namespaces, with two grants. Not two row filters.

Dataset here means the catalog's unit of registration, not an Iceberg table: a dataset of entries, a Lance index beside it, each a separate object with its own grant. Only the decision log at the end of this piece is Iceberg.

That sounds like a limitation and is closer to a clarification. Row-level isolation in a memory service is a promise the application makes; dataset-level isolation is a promise the storage layer keeps. One of those survives a bug in the agent framework.

It has a cost, and I would rather state it than not: per-user isolation means a dataset per user. Where a team or a tenant is the real boundary, one dataset for the group is correct and cheaper. The rule that falls out is short: the dataset is the unit of access, so it has to equal the unit of trust.

You cannot grant write without read

The design had agents filing skill proposals into a shared queue they could write but not read. An inbox: you may post, you may not rummage.

The model will not express it. Reading data resolves through a relation that includes writing it, so granting modify confers select — and can_drop, can_rename and can_set_protection with it. A shared queue would have been readable by every agent that could write to it, and droppable by any one of them.

So each proposer gets its own queue — its own dataset, named for it. Proposals stay private, and attribution stops being a field the caller fills in and becomes a fact about which dataset the write landed in. An agent cannot file under another agent's name because it has no credentials for that dataset.

The right to label is its own grant

The catalog has governance tags — a controlled vocabulary you attach to objects. pii, sensitivity=restricted, a lifecycle tag of your own.

The useful part is not the word, it is who may apply it. Attaching a tag is delegated separately from the right to change the data, so a compliance reviewer can label something they cannot read — and someone who can rewrite a dataset cannot quietly strip the label that marks it sensitive, because removing a tag is gated on the tag rather than on the object. Classification becomes a power in its own right, held by whoever should hold it.

What a tag does not do, under the relationship model this build runs, is decide access: no authorizer reads the value, and the grant stays the gate. Under Cedar — which the catalog also supports — policy decides from attributes on the resource, and that is the direction this is going. The label and the decision converge.

A governance agent must not judge

Human review does not scale. Forty proposals arrive overnight, one matters, and the reviewer reads all forty or none. The obvious fix is an agent that triages them, and it is a trap.

A governance agent is, by construction, a machine that reads attacker-controlled text. A proposed skill is exactly that — it was written by an agent that may itself have been compromised, which is the case the review exists to catch. That is assume-breach applied to an agent rather than a network: the proposer is treated as already hostile, and every refusal in this piece has to hold in that case. Put a model in charge of the verdict and you have built something that reads hostile input and holds the publishing credential. A skill that cannot get past a regex can absolutely get past a model by addressing the reviewer instead of the task.

So the verdicts are deterministic. Rules, not judgement: every proposal comes out ok, warning or critical, and the queue is ordered worst first. Triage decides what the human reads first, never what happens to it. The model writes a one-line summary alongside and touches nothing else. Rules cannot be argued with, they replay identically tomorrow, and they can be shown to an auditor.

Then the same question one level up: who may change the rules? Because whoever can weaken a rule turns triage into theatre — silently, with no diff for anyone to review.

So the rules are governed data, not code. They live in the catalog as a policy document. The governance agent reads them and cannot write them. Which comes down to one sentence:

The governance agent can read the rules it enforces. It cannot change them, and it cannot approve anything.

Both refusals come from the catalog. Neither is a check in its own code that a clever prompt could talk past.

What a denial actually looks like

The shape of the thing, once all of that settles:

  agent_memory/
    ├── agent_a/     entries + recall     agent-a: read+write · agent-b: invisible
    ├── agent_b/     entries + recall     agent-b: read+write · agent-a: invisible
    └── shared/      entries + recall     both read, neither writes
  skills/
    ├── proposed/
    │     ├── <agent-a>   its own queue, and no sight of any other
    │     └── <agent-b>
    ├── approved          agents READ · only the reviewer WRITES   ← the wall
    ├── rejected          refusals with reasons, readable by the proposer
    └── decisions         every approval and refusal, as a table
  governance/
    └── policy            the triage rules · read by the agent, written by nobody else

The two halves are granted at different levels, and the asymmetry is not an oversight. A namespace grant flows downward — select on skills would hand over every proposal queue beneath it — so skills are granted dataset by dataset. Memory is granted on the namespace, because there the boundary runs around a whole scope rather than between siblings. Inheritance is convenient everywhere except at the line you are trying to hold.

The rejected tier is where the loop closes: a refusal is readable by the agent that filed it, so a proposal that fails comes back with the reason it failed.

And the output of the run that proves it, which I find more convincing than any diagram:

  PASS  agent-b denied agent-a's memory (404)
  PASS  denied scope drops out of the fan-out
  PASS  shared tier is read-only for agents (AccessDenied)
  PASS  agent-a cannot approve its own skill (AccessDenied)
  PASS  agent-b cannot see agent-a's queue (NotFoundError)
  PASS  governance agent cannot approve (NotFoundError)
  PASS  governance agent cannot weaken its rules (AccessDenied)

All of it runs locally. ./up.sh brings up the catalog, an identity provider, object storage, Jupyter and two local models — no API key, no cloud account — and four notebooks walk it end to end: setup, an agent learning, the denials above, and the governance agent. The third and fourth are where the refusals happen.

Two details in there are worth pulling out.

It is 404, not 403. The catalog does not admit that a dataset you may not see exists. A denial that says "forbidden" has already told you something.

And "denied scope drops out of the fan-out" is the property I like most. When an agent searches its memory, it asks every scope it knows about. The ones it may not read return nothing and fall out of the results. The agent never enumerates its own permissions, never carries a list of what it is allowed to touch, never has code that could get that list wrong. It asks for everything and the catalog answers by refusing.

That is the difference between governance in the application and governance underneath it. One is a list the agent consults. The other is a door that does not open.

When not to use any of this

Most production "skills" today are markdown files in a repository, reviewed by pull request, shipped on deploy. That is already governance: versioned, attributed, reviewed, auditable, free.

If your skills are human-authored and ship on deploy, use git. Nothing here beats it, and an answer that cannot say so is not worth trusting on the rest.

The same honesty is owed to the memory layer you already run. mem0 and Zep do a great deal this does not — extraction, consolidation, resolving one fact against a contradicting one, letting old entries decay. None of that is on offer here, and none of it is what this replaces. The question it answers is the one underneath: where the bytes live, and who holds keys to them. Nor is it true that they lack access control. Zep binds attribute-based policies to the API key an agent authenticates with and filters results by data class — a real authorization layer, better than most of what is deployed.

The difference is where the refusal happens. Zep evaluates the policy and then reads the data: the service is in the path, holding storage credentials for everything it protects. A vended credential puts the boundary underneath. The agent holds keys for one prefix and nothing else, so a request for bytes it may not read cannot be formed, let alone evaluated — there is no service in the path to have a bug in.

The part I would actually argue for, though, is duller than any of that. These products bring their own permission model, and whatever backend they write to — your Postgres included — those permissions are theirs rather than your platform's. So "who may read this" gets answered twice, in two vocabularies, with two trails to reconcile at audit. Here memory is a dataset beside every other dataset: the same grants your tables already use — readable by whatever engine reads your tables — one log, and a decision log an auditor can query from Trino or DuckDB without knowing the memory layer exists.

Two complete governance stacks side by side are not a separation of powers — they are the same powers twice, and separation needs one constitution with different hands beneath it.

There is also a half of this with no equivalent in that market at all. Those tools store facts. None of them govern a procedure the agent is going to execute.

The sharper version is not "use git" but "have the agent open a pull request", and for many teams that is the right answer — you inherit review, history and CI for nothing. It loses in three places. A pull request is global where approval has to be per tenant. A merge ships on the next deploy where approval has to take effect on the next run. And an agent holding repository credentials can merge its own work: there the control is process and CI, here it is the absence of the ability. That last one is the whole distinction.

What neither reaches is the case this was built for: an agent writing a procedure at three in the morning, at volume, with nothing to review until you build the machinery to make it reviewable.

The nearer-term version needs no autonomous agents at all. On a shared agent platform, who may publish a skill that another team's agents will execute? That is the same question as who may push to production, and it arrives the moment a second team joins.

One table earns Iceberg

The last of the four, and the only one where the answer was yes.

I argued for an Iceberg index over agent memory, on the grounds that memory should be queryable data. Then I dropped it, for the reason above: memory is one dataset per scope, so there is nothing central to query, and agents read it through an SDK rather than SQL.

Decisions are different, and it took someone asking "is that written anywhere?" to see it. Approvals and refusals are central rather than per-scope, so there is one table, and the readers are auditors writing SQL. The log is append-only, multi-writer and time-ordered — which is Iceberg's actual job, and precisely what a plain object store cannot arbitrate.

So every approval and refusal appends a row: when, which skill and version, who proposed it, who signed, what the rules flagged, the reviewer's reason, and which version of the rules was in force. That last column separates "we did not check for that then" from "we checked and missed it" — a distinction that decides how a post-mortem goes.

The catalog's own audit log already records that a write happened and who made it. What it cannot know is why — that is application meaning, and it needs somewhere to live.

What this does not fix

It governs shared, durable memory. It cannot stop an agent keeping something in its context window or writing to its own disk — the claim is about what persists and is shared, not about what a model can hold in mind.

The rules catch phrasing, not intent. A carefully worded malicious procedure passes every regex in the set. Triage buys attention, not safety.

And the reviewer is still the ceiling. The honest path past them is a policy principal applying an approval under stated conditions — never a model's judgement about text it was handed.

None of that undermines the part that works. A denied agent gets no keys, so it cannot read the bytes. It runs the same code as the permitted one, and the difference is a grant. Zero trust comes down to that: the boundary does not live inside the thing you are not trusting.

The catalog was where it started. Memory and skills are data now, governed like the rest of it — and governance that only works inside one product is not governance you can move.

The Ungoverned State of Agent Memory

Viktor Kessler — Thu, 10 Sep 2026 09:17:19 +0000

Why local agent state is a security nightmare, and how a control plane fixes it.

Your coding agents have started writing things down.

Not the code — notes about the work. Who you are. What you corrected last week. Why the deploy failed in March. Every serious coding agent now keeps a memory folder, and it fills up quietly while people work.

Nobody governs any of it.

I wrote a few months ago, borrowing Jamin Ball's framing, that the clearinghouse for AI agents has to hold four things: memory, context, execution, and governance. Memory is the one already happening — on every laptop in your company, in a format nobody chose.

Two folders that never met

Open two coding agents on the same machine, same repository, same person, and you get two memory systems that have never heard of each other.

Claude Code keeps a folder per project and writes a note the moment it decides something matters. Codex pools its notes for the whole machine and merges them later, in the background. Both are sensible. Neither team had reason to coordinate with the other.

That is the point. Two agents, one employee, one laptop — and what each learns about your codebase sits in a separate folder that will never be reconciled, searched together, or deleted together. Multiply that across a company and you have a filing system nobody designed.

The contractor problem

There is a sharper version of this, on a lot of laptops right now: the freelancer who works for two clients. One machine, two companies that are not supposed to know anything about each other. Possibly competitors.

Claude Code's per-project folders happen to keep them apart. Codex's memory does not work that way — one pool for the person, fed into the next session no matter which client it is for.

Nothing here is misbehaving. The tool was never told that the person at the keyboard has two employers, because there is nowhere in the format to say so.

So one client's architecture and incident history gets folded into a summary, and handed to an agent working for the other. No breach. No alert. No log entry. And no NDA that anticipated it.

Nobody owns the format

You would expect the Model Context Protocol to have settled this. It has not. MCP covers tools, resources, and prompts. It says nothing about memory.

Into that gap fell an example implementation, shipped as a demo and copied widely because it was there. It lost data — two writes at once, one silently gone — for the better part of two years. Volunteers fixed that copy a few weeks ago. Everything that inherited the pattern did not get the patch, and most never will.

A newer proposal, memorywire, at least gives memory a shared vocabulary — but a vocabulary tells you how to say "forget this," not which stores exist to say it to, or who is allowed to say it.

Every vendor governs. Nobody governs across.

It would be wrong to call enterprise memory tools ungoverned. They are siloed, which is a different problem.

AWS, Zep, Mem0 and Databricks all ship real controls — customer-held encryption keys, audits, legal holds, catalog permissions. Each can answer honestly, inside its own walls: is this encrypted, can we delete it, who is allowed to read it.

None of them can answer it about the others. No shared identity, no shared retention rule, no shared delete button across four stores and a laptop folder that all know something about the same customer.

Governance grew up per vendor, correctly. It stops at the vendor's edge, the way a passport is real, enforceable, and good only in the country that issued it.

We have seen this before. A decade ago every warehouse governed its own data perfectly and none of it moved. Iceberg broke that open: the table became something any engine could read, with a catalog above it holding the rules. Agent memory is roughly where the warehouses were — governed inside each vendor, portable nowhere.

And you cannot fix that by asking four vendors to agree. Interoperability holds only where it is enforced — somewhere everyone already passes through: where the data sits, and where permission to read it is issued.

The question nobody can answer

Try this today. Show me everything any agent, on any tool, knows about this person — and delete all of it.

There is no query that answers it. Not because any vendor is careless, but because there is no catalog of memory the way there is a catalog of tables. Nobody can even list what exists.

It is a security problem too

Picture the attack. Someone plants an instruction inside a document your agent reads — a README, a ticket. The agent decides it is worth remembering and writes it down. Weeks later it fires, in a session with nothing to do with where it came from.

This is memory poisoning, and OWASP now ranks it in its top ten for agentic applications. Prompt injection ends when the session ends. This gets written down.

It is also close to invisible: poisoned memory looks like the model being unreliable, so it gets debugged instead of investigated. Memory is not unaudited. There is no audit to fail.

Where this has to go

A table format is not a memory format, and I am not going to pretend otherwise. But a catalog does not need to hold the memory — only the record of it and the rules over it. That part already works: files and datasets on object storage can be registered as governed objects, with access granted per request and per identity, and the data itself never moving.

A memory store is already a location plus a policy. That is what it is on every laptop today, minus the registration.

But registration is where this starts, not where it ends. An inventory tells you what exists; it does not stand between the agent and the data at the moment the agent asks — the only place a rule is really enforced. That needs a control plane: one place where every request is cleared or refused, and the decision written down either way.

And notes are not the only thing accumulating. The same background pass writes the agent's skills too. A note is something the agent believes; a skill is something it will run. Govern the notes, ignore the skills, and you have governed the safer half.

The catalog is where it starts. The control plane is what it has to become.