A risk manager once walked over to my desk holding a printout. One number on it was circled. Not wrong, exactly. Just different from the number on the same report the day before, and nobody had changed anything.
I spent most of that afternoon working backwards. The report read a view, the view read a table, the table was loaded by a job, the job pulled from a vendor file, and the vendor had quietly resent the file at 11am with two rows corrected. Fine. Annoying, but fine. The thing I remember is how much of the afternoon was spent proving what hadn't happened. Nobody edited the loader. Nobody touched the table. Once I could show that, the corrected file was the only suspect left.
In practice, that's what I use lineage for. It's an alibi.
The suspect list just got longer
In that story, every step in the chain was built by a person and left alone until a person changed it. The list of things that could have moved was short, and the humans who could have moved them all sat within shouting distance.
Now picture the same report, except the loader was written by an agent. It was told "pull the daily positions file from this vendor into this table," wrote a Python script to do it, ran it, noticed a column had shifted last Thursday, and rewrote its own script to cope.
Every one of those is a legitimate action. But walk back to the circled number and try to build the alibi. You need the run that landed the row and the version of the script it ran under, the one patched Thursday or the one before. Did the patch change what the column means? And which key did the patching, a person or a scheduled job with nobody watching?
"It came from the positions table" answers none of that.
Four questions, in order
There are four questions, and they have to be answerable in this order, because each one narrows the next.
Start with the run. A row needs to carry the identity of the run that wrote it. On the row, as a column, because the log somewhere else rotates after a week. Then when the risk manager circles a number, you can select the row and read off the run id and when it landed.
Then the script. For agent-written feeds this is the one that keeps me up. The script that ran is a commit, and the row should carry the commit. I wrote back in July that agent code belongs in git and every run should be pinned to a SHA. With a SHA on the row, "the script that ran Thursday" is a diff you can open.
Then the source, which is not the same thing as the tap. When a script edit switches the vendor, rows already landed should keep the source they were stamped with. Otherwise the graph tells you where the data comes from now, which is a different question from where the circled number came from.
Last, whose key ran it. That could be a person logged in, an agent holding an issued key, or an assistant acting on behalf of a person. Record the denials too. An agent that was refused permission to change something is evidence, because it tells you what almost happened. In an audit, who moved it is the half they ask about.
Claims versus evidence
One thing I got wrong early, and I think most lineage tools still get wrong: drawing the graph from configuration and calling it done.
Configuration is a claim. It says pipeline A is supposed to land into table B. Whether it did, when, how many rows, and whether it failed halfway through on Tuesday, none of that is in the config. So the graph should carry what actually traversed each hop, and an edge with nothing behind it should be drawn faint, because it's still a hypothesis.
The same discipline applies inside the row. You can ask a model to read a transformation script and tell you which output column came from which input field. That's inference, and it has to say so. Anything it can't point to evidence for should come back as "unresolved." A lineage tool that fills gaps with plausible answers is worse than one with visible gaps. A gap at least sends you looking.
The agent needs the alibi too
I've been writing this as though the reader of the lineage is a person. More and more often it's an agent.
An agent about to change a tap should ask what's downstream of it first. Before it answers a question from a dataset, it should check whether anything has landed there lately and which run did it. None of that works if lineage is only a screen. It has to be something an agent can call as a tool, returning the records a person would read.
Where I landed
This is the ground I've been covering in Datris over the last few weeks: an opt-in provenance stamp that puts run id, ingest time, definition version, script commit and source on every row a pipeline lands; run-level lineage recorded for every run; evidence on every edge of the graph; column lineage in two labeled tiers, exact and inferred; and an audit log that names the actor on every write. An agent can call all of it as tools.
But you don't need my product to take the test. Go find a number on a dashboard that an agent-built feed produced. Try to build the alibi. Which run, which script, which source, who. Count how many of those you can answer from the row itself, and how many need a person who remembers.
The risk manager with the printout is coming back. This time, the loader has no one to shout across the room to.
Todd Fearn is the founder of Datris.ai, an open-source, agent-native data platform built on the Model Context Protocol, and he runs IData Corporation, a data engineering consultancy for financial services firms. He has spent about thirty years building production data infrastructure inside institutions like Goldman Sachs, Bridgewater Associates, Deutsche Bank, and Freddie Mac.
Top comments (0)