Every time this comes up, someone credits the same setup: Athena on Iceberg is where "the code is the spec" — where you open Git and read the whole system, catalog to transforms to schema, without logging into anything. In my experience they're not wrong.
What I want to argue is that they're right for mostly the wrong reason.
The reason people reach for is the engine — serverless, open format, nothing to provision. But the thing that actually keeps code as the spec, when it does, is something you could have applied to almost any engine. And the thing that breaks it, when it breaks, has nothing to do with Iceberg at all.
So here's the split I've landed on, for now: whether "code is the spec" holds is about 90% discipline and 10% engine. Athena/Iceberg earns that 10% honestly — but 10% is all it earns, and I keep watching people mistake it for the whole thing. (Just where my own tinkering has led — not advice.)
I should say up front that I'm still in the middle of this. What the best declarative setup for an agent actually looks like — how you turn a system into a spec it can read and act on without guessing — is something I'm actively testing, not something I've settled. Read what follows as a working idea at a particular moment, written down partly so I can find out where it's wrong.
To see where the 10% actually lives, it helps to notice that a stateful system always keeps two copies of itself.
The spec has two copies
One copy is declared: the code you wrote, the schema you committed, the transforms in dbt, the catalog in Terraform. The other is realized: the state the engine accumulates while running — statistics, physical layout, caches, maintenance history, tuning knobs someone set at 2am.
"Code is the spec" is really a claim about the distance between those two. When the declared copy explains almost everything about the realized one, you can reason about the system by reading Git. When it doesn't, you can't.
Some of that state is declarable — you can pin a distribution or sort strategy in DDL, dbt, or Terraform. But plenty of it isn't, or drifts away from what you declared: which strategy the engine is actually using once automatic optimization kicks in, when the last vacuum finished, whether statistics are fresh, which materialized view is stale, whether that partition is really being pruned. To answer those you log in and ask the running system. Now your spec is half in the repo and half in someone's head.
So the real question isn't "is my code readable." It's: how much of the truth escaped the code into runtime state I can't see? I've started calling that the hidden-state surface area. Shrink it and the code stays the spec. Let it grow and the code becomes a hopeful sketch.
I argued a while back that rigor compresses — the more precisely you can state a fact, the fewer tokens it takes to hand over. This is the same idea pointed at a platform instead of a function: the smaller the hidden-state surface, the shorter and truer the account of the system you can hand to anyone, human or agent.
Why this is mostly discipline
Here's the part I want to be honest about first, because it's the 90%: the engine doesn't make your system legible. Declarative rigor does.
A catalog expressed as data contracts, transforms as a single dbt DAG, infra as Terraform — that's what makes design intent followable in code. And it's almost entirely engine-agnostic. You can point all of it at Redshift and get something a newcomer can read top to bottom. You can also build an Iceberg-on-Athena setup that's an undocumented swamp of hand-run ALTERs and orphaned partitions nobody dares touch.
I've done both. The readable one wasn't readable because of the storage format. It was readable because the rules were declared, in one place, and enforced. "Iceberg, therefore legible" is a claim I'd push back on if someone else made it — so I should hold myself to the same standard.
If you do nothing else, do this part. It's where the leverage is.
But the engine sets the difficulty
Now the 10% — small, but not nothing. Given equal discipline, engines differ in how much state they insist on keeping inside themselves versus how much they externalize where you can see it. That doesn't decide whether you can make code the spec; it decides how much discipline it costs to keep it that way.
Athena on Iceberg externalizes an unusual amount. The stuff that's normally locked inside a proprietary engine — snapshot history, the manifest of what files exist, the current schema, the partition spec — lives as open metadata that isn't confined to one vendor's internal system tables. You can read it through standard metadata tables ($snapshots, $manifests, $history), through the catalog, or straight from object storage. It's still state, and it still changes at runtime — but the format is open and portable, so the same facts stay reachable from more than one tool.
Redshift keeps more of that truth behind its own interfaces. Materialized-view freshness, VACUUM/ANALYZE history, statistics, workload-management tuning, and — with automatic table optimization or AUTO distribution — the effective physical layout, which can shift with data volume regardless of what you declared. It's all observable; you just query the system views for it.
| Athena / Iceberg | Redshift | |
|---|---|---|
| Visible in Git / code | Terraform, dbt, contracts, SQL/YAML | Terraform, dbt, DDL, contracts |
| Runtime state lives... | open metadata (snapshots, manifests, schema, partition spec) | engine-internal: MV freshness, vacuum/analyze history, stats, effective dist/sort under AUTO, tuning |
| To inspect it you... | read open metadata tables, the catalog, or object storage | query the engine's system views |
| Compute infra to manage? | no dedicated warehouse compute | provisioned: yes · serverless: mostly abstracted |
| Hidden-state surface | typically smaller, more externalized | typically larger, more engine-bound |
The honest framing: it's not that one is readable and one isn't. Both expose their state — the difference is how. One hands you facts in an open format that multiple compatible tools can inspect; the other hands them back through its own interfaces. Same level of legibility, less engine-specific discipline required to preserve it — not necessarily less operational work overall. That's the whole 10%. (This is for the batch-oriented, per-tenant workload I keep describing; a giant, high-churn Iceberg estate grows its own large state surface — manifests, small files, snapshot sprawl, catalog commits, cross-engine compatibility — and the gap narrows.)
Why I actually care: the guesser downstream
Here's where it stops being an aesthetics argument for me.
The real line isn't S3 versus a database. It's this: a fact exposed through an inspectable interface the agent can actually reach is verified; a fact outside its observable context must be estimated. A Redshift VACUUM history the agent is allowed to query is verified; an Iceberg snapshot it has no catalog access or tooling for is a guess. The engine doesn't decide it — reachable evidence does. What Iceberg tends to do is make that evidence easier to expose safely and portably: open metadata is simple to grant read access to without also handing over the keys to a live warehouse.
And the consumer of those facts, more and more, is an agent. These days I'm rarely the one reading the schema first — Claude Code or Codex is, running in the repo, editing a migration, rewriting a transform. What it can see is the declared copy plus whatever interfaces I've wired it into; everything else it infers. The larger the hidden-state surface — or the harder its evidence is to reach — the more of my system the agent is quietly guessing about, and the more defensive lines I tack onto prompts: "check the effective sort strategy first," "don't assume the MV is fresh," "verify before you migrate." When the fact is reachable, it can state it flatly and move on; when it isn't, the best it can do is hedge — and hedging costs tokens and trust.
And this isn't OLAP-only. The agent touches both tiers — on OLTP it edits schema, writes migrations, reasons about a tenant_id guard; on OLAP it changes a dbt model, reads a partition spec, checks whether a snapshot rolled forward. The same question decides it either way: can it reach the evidence for what it's about to do? Legibility-for-agents is one property spanning the whole platform, not a warehouse feature.
It's the same move I make everywhere else
Once I saw "shrink the hidden-state surface" as the goal, I noticed I was already doing it at the architecture level, not just the table level.
The platform I work on keeps three kinds of state deliberately apart — durable relational truth in Postgres, hot disposable state in DynamoDB, private local work in per-user SQLite — and the thing I'm most careful about isn't any one store. It's the crossings between them.
OLTP (write / update) OLAP (read / aggregate)
┌───────────────────────────┐ ┌──────────────────────┐
│ control-db (Postgres) │ │ Athena / Iceberg │
│ durable, relational │ │ S3 Tables (open │
│ operational_state (Dynamo) │ ==3==> │ metadata mart) │
│ hot, TTL'd │ bridges │ DuckDB (local) │
│ metadata.db (SQLite) │ │ │
│ local, private │ │ │
└───────────────────────────┘ └──────────────────────┘
the only sanctioned OLTP→OLAP paths are three named bridges,
and Postgres reaches the lake through exactly one VPC Lambda.
The point of narrowing the crossings to a countable few — and letting Postgres reach the analytics lake through a single path — is the same point as preferring open metadata. Every uncounted pathway is hidden state: a way the system can be true in production in a shape the code never described. Three named bridges I can hold in my head. Twenty accidental ones I cannot, and neither can an agent.
Same principle at two scales: make the crossings and the state legible, so the code keeps describing the system. That the pattern reappears from a single table up to the whole control-plane-to-lake boundary is, honestly, most of why I trust it — though that's a hunch I hold loosely, not a thesis I'd defend to the death.
And this is where I've started spending real budget on purpose. Making OLTP and OLAP legible to an agent — declared schemas, reachable metadata, counted crossings, a system view the agent is actually allowed to query — used to read as tidiness, a nice-to-have for when you had time. With Claude Code and Codex doing more of the actual editing, it's stopped feeling like tidiness and started feeling like the interface I hand the most productive worker on the team. An hour spent making a fact reachable is an hour I don't spend, over and over, writing "be careful, verify this" into prompts — and the payoff compounds every time the agent runs. So I treat agent-legibility as infrastructure worth building, the way you'd invest in tests or types, not polish applied at the end. Whether it fully pencils out depends on how much of your work the agents are really doing; for the amount they're doing in mine, it already has.
Objections I'd raise myself
- "So you're saying Iceberg is more readable than Redshift." No — I'm saying it externalizes more state, which lowers the discipline cost of legibility. A disciplined Redshift shop beats a sloppy Iceberg one every time. The 90% dominates; the engine only sets the exchange rate.
- "Open metadata isn't zero hidden state." Correct. Compaction, small-file cleanup, maintenance jobs — Iceberg has operational state too. It's smaller and observable-from-outside, not absent. If I implied "no hidden state," that's overreach.
-
"Athena/Iceberg has its own tax." It does — and it's not about S3 consistency (object storage is strongly consistent now). It's concurrent-write conflicts, catalog commit semantics, small files, compaction, snapshot expiration, and orphan-file cleanup. Athena needs
OPTIMIZE/VACUUMagainst Iceberg tables too. Legibility isn't free; I'm trading one set of runtime concerns for another, and I want that trade stated plainly, not smuggled in. - "This is just a warehouse-vs-lakehouse take in disguise." Partly. But the axis I care about isn't performance or cost — it's the size of the gap between declared and realized. Two separate arguments; I don't want to lean on the popular one to prop up mine.
Where this leaves me
The design goal I keep circling back to is small: shrink the distance between what the code says and what's true at runtime — because that distance is precisely where humans and agents are forced to guess.
Most of the shrinking is discipline: declare the rules, put them in one place, enforce them — the 90%, and no engine does it for you. The last 10% is picking tools that don't quietly hoard state inside themselves, but hand you their truth as metadata you can read, crossings you can count, facts you can verify. Do the 90% and almost any engine can be the spec; do the 10% too and the code stays the spec with less of you holding it together by hand. That's the honest version of what people reach for when they credit Athena/Iceberg — not that the engine makes code the spec, but that it asks less of you to keep it that way.
I want to be honest about how provisional this still is. I don't think anyone, me least of all, has settled what declarative management and code-as-spec should look like once the primary reader is an agent rather than a person. I'm running my own small experiments — which facts to externalize, which interfaces to expose, where the verified/estimated line should sit — and expect to be revising this for a while. The 90/10 is a heuristic I find useful today, not a law I'd stand behind tomorrow.
That, for now, is the shape I'm designing toward — a little more on purpose, a little less out of habit, and still figuring out the edges.
Top comments (5)
I really like the distinction between declared state and realized state. One thing I’d add is that agents don’t only guess hidden runtime state—they also guess hidden business assumptions. A schema can tell an agent that status = ACTIVE exists, but not necessarily what “ACTIVE” means in terms of business rules. That’s why, in my experience, “code is the spec” gradually becomes “code + contracts + policies + business events are the spec.” The less implicit knowledge lives only in people’s heads, the less the agent has to estimate.
Thank you for the thoughtful comment, I completely agree.
Especially in the database and API layers, I’ve increasingly been asking Claude Code and Codex to make contracts and policies explicit and encode them directly into the implementation wherever possible.
For business events, I try to model the relevant states explicitly and include them in each event. That said, I still feel there’s more work to do in how those states—and the business meaning behind them—are modeled. It’s something I’m continuing to think through and refine.
That resonates with me. One thing I’ve been finding is that making business events explicit captures what happened, but agents often still need to infer why it happened. That’s where policies and business intent become just as important as state. An event like OrderApproved is useful, but the rule explaining why approval was granted is often what keeps an agent from making the wrong assumption later. It feels like we’re gradually moving from “code as the spec” toward “decision-making as the spec.”
Exactly. Capturing why a decision was made—and making that reasoning accessible to agents—feels much harder than making state explicit. Even with code, policies, and ADRs in place, one key question is how to represent decision context so agents can interpret it reliably.
I think that’s the next interesting challenge. Rather than trying to encode all of the decision context, I’ve been thinking about making the decision evidence explicit. Context is still open to interpretation, but evidence is something an agent can verify. Instead of only recording OrderApproved, exposing the policy version, matched rules, risk score, feature flags, or other inputs that led to the decision gives the agent a much firmer foundation for reasoning. The less it has to infer, the more deterministic its behavior becomes.