There's a question I keep coming back to with AI and data engineering:
Do we move the data to the agent, or move the agent to where the data already lives?
Because we're starting to put agents everywhere.
In the warehouse.
On the source databases.
Inside BI.
Writing transforms.
Watching queries.
Building DAGs.
Debugging pipelines.
And if you aren't careful, you end up with five agents, five sets of credentials, five different copies of context and a developer laptop somehow holding the whole thing together.
That doesn't feel like where this should end up.
Take a boring data stack
Nothing exotic.
You've got a handful of MSSQL databases.
Fivetran moves the data into Snowflake.
You've got some DAGs taking it through bronze, silver and gold.
Then BI sits on top.
Pretty normal.
Swap MSSQL for MongoDB, Snowflake for Databricks and Fivetran for Hevo if you want.
The names change.
The problem really doesn't.
Now add agents.
Where do they go?
The annoying answer is "everywhere"
Ask a data engineer where the agent belongs and you're probably going to get the classic answer:
"It depends."
And, unfortunately, they're right.
If you're doing BI, maybe the agent should sit near the warehouse and help users build queries and reports.
If you're using it for ETL design, maybe it needs access to your transformation layer, DAGs or SSIS packages.
Maybe it's managing transforms directly inside something like Hevo.
If it's acting more like a DBA, it probably needs to see query plans, database metrics, indexes and workload history.
Maybe another agent is responsible for data quality.
Another one owns schema changes.
Another handles lineage.
There are about a zillion places where an agent can be useful.
The interesting problem starts when those agents need each other.
Your org chart doesn't care about your pipeline
The database team might own the source.
Data Engineering owns ingestion and transforms.
Analytics owns the models.
BI owns what the business actually sees.
Dev might own the application producing the data in the first place.
Those teams aren't always under the same person.
Sometimes they barely talk.
The data doesn't care.
A bad query in the application can hammer the OLTP database, break an ingestion SLA, delay a transform and leave a dashboard showing garbage.
That's one problem.
Organizationally, it might belong to four teams.
And now we're about to recreate that exact same mess with agents.
A DBA agent discovers the issue.
A DE agent needs to change something downstream.
A dev agent needs to fix the source query.
A BI agent needs to know the data is currently questionable.
Those agents need shared context around the same goal, even if they have completely different permissions and responsibilities.
Otherwise we're right back to using people as the transport layer.
So who is actually running these things?
You can absolutely give one data engineer a monster setup.
Claude Code.
Five MCP servers.
Snowflake credentials.
Access to MSSQL.
AWS.
Databricks.
GitHub.
Whatever else you need.
Run the whole thing from their laptop.
And honestly?
It'll probably work.
It may even work really well.
Right up until you show the architecture diagram to Security.
There's also a more practical problem.
What happens when something occurs at 2 AM?
How do I trigger an agent running on someone's laptop because a CDC event showed up?
What happens when a FIFO event needs processing?
What if a database crosses a threshold and I want an agent immediately investigating it?
You can have the laptop wake up every so often and check something.
That's not the same thing.
If agents are supposed to become part of the data infrastructure, they need to behave like infrastructure.
Stop making the agent come looking for work
This is where I think the placement gets more obvious.
Put the agents near the tooling that's already moving the data.
Your ETL infrastructure already has a huge advantage.
It can reach the source.
It can reach the destination.
It already understands when jobs start and stop.
It sees failures.
It sees schemas.
It sees transforms.
It can react to events.
And, most importantly, it's already running 24/7.
You don't have to give some random new service network access to every database in the company.
You don't need a developer's laptop waiting around for something interesting to happen.
The infrastructure already exists.
Put the agent there.
That doesn't mean stuffing your warehouse into an LLM
There's another version of this conversation that immediately turns into:
"How do we get all our enterprise data into the agent?"
I don't think that's usually the right question.
Most of the time, don't move the data unless you actually need to.
Let the database do database things.
If the agent needs to understand a billion-row table, it probably shouldn't pull a billion rows into context.
Have it inspect the schema.
Look at statistics.
Run a query.
Sample the data.
Ask the warehouse to aggregate it.
Use the database as the compute layer it already is.
Then give the agent the result it actually needs to make the next decision.
That's very different from creating another giant copy of your data just so an LLM can stare at it.
Sometimes moving data does make sense
There are still cases where moving some of it is useful.
Maybe you want an isolated subset of production data for investigation.
Maybe the agent needs a smaller analytical working set.
Maybe you want to correlate data from several OLTP systems without hammering any of them.
Maybe you want cheap local execution for exploration before touching the warehouse.
Fine.
Move the slice you need.
But now that movement should be part of the pipeline too.
It should have lineage.
Permissions.
Retention.
A reason for existing.
"Claude needed it" probably shouldn't be your entire data governance strategy.
The agent belongs in the control plane more than the data plane
This is the distinction I think matters.
The agent doesn't necessarily need to hold the data.
It needs to understand where the data is, how it's moving and what it's allowed to do about it.
A pipeline fails.
An agent can inspect the source.
Check whether the schema changed.
Look at the ingestion job.
Check the target.
Inspect the transform.
Figure out what downstream models are affected.
Maybe fix the DAG.
Maybe create the migration.
Maybe tell another agent that the application needs a change instead.
That's much closer to a control plane than some giant AI database.
And now agent-to-agent communication actually makes sense.
The DBA agent doesn't need the BI agent's credentials.
The BI agent doesn't need production write access.
They need to be able to share enough context to work toward the same result.
Same problem.
Different domains.
Different permissions.
This is also where humans should show up less
There's no reason a pipeline failure should require three Slack threads before anyone starts fixing it.
The agents can do quite a bit before that point.
Discover the issue.
Figure out where it started.
Determine what's affected.
Gather the evidence.
Hand work to the appropriate domain.
Make low-risk changes.
Run validation.
Then bring a person in when there's actually a decision worth making.
Not because an agent needed someone to copy an error message from Snowflake into Jira.
We've already got APIs for that.
The boring infrastructure matters
Once you host agents in the data layer, all the unexciting stuff becomes really important.
Network access.
SSH tunnels.
Private connectivity.
Credential scoping.
Audit logs.
Shared state.
Tool permissions.
Agent-to-agent messaging.
Which agent can change what.
Which agent can only inspect it.
Which actions require approval.
This is the stuff that makes the difference between "I made an AI demo against Snowflake" and actually letting agents participate in your data platform.
The LLM is almost the easy part.
This is basically where I've landed
I don't think there's one database where "the agent" belongs.
There are going to be agents across the stack.
But I do think there should be a place where those agents run, share context, receive events and interact with the infrastructure without turning every engineer's computer into a tiny integration server.
For data, the ETL layer is a pretty damn good place to do it.
It already sits between the systems.
It already has the connectivity.
It already owns workflows.
It already reacts to changes.
We're just giving it something smarter than another DAG operator.
Shameless plug: this is the direction I'm building SchemaBounce around.
Transforms and DAG management are in the same environment as the agents. There's managed DuckDB when you actually do want to offload a working subset of OLTP or warehouse data, a SQL workbench, private VPN and SSH connectivity, A2A messaging between agents, and permission boundaries so the DBA agent doesn't automatically get to become the everything agent.
Basically, the boring pieces you start needing as soon as you try to take "AI-managed data infrastructure" past a laptop demo.
I've seen a lot of tools bolt an agent onto one part of the data stack.
Query the warehouse with AI.
Generate SQL with AI.
Build a DAG with AI.
Useful stuff.
But that's not really the problem I'm interested in anymore.
I want the source database, ingestion layer, transforms, warehouse and BI side to be able to work on the same problem without a human spending their day moving context between them.
The data already has a pipeline.
Maybe the agents should use it too.
Top comments (0)