DEV Community: Evan Rosa

The Medallion Architecture, Explained: Raw, Clean, Ready

Evan Rosa — Wed, 19 Aug 2026 17:32:10 +0000

The medallion architecture organizes a data platform into three layers, Raw, Clean, and Ready, that take source data and refine it step by step into trusted, business-ready tables. It is the most common blueprint for the modern data lakehouse, and it is the foundation OptimaFlo is built on. This guide explains each layer in plain English, then shows how to stand one up without hiring a data team.

What is the medallion architecture?

The medallion architecture is a data design pattern popularized by Databricks. It splits data processing into three progressively cleaner layers. You may know these layers by their traditional names, bronze, silver, and gold; we use clearer terms: Raw, Clean, and Ready. Each stage adds structure, quality, and business meaning:

Raw holds data exactly as it arrived.
Clean holds cleaned, validated, joined data.
Ready holds business-ready metrics for dashboards and reports.

The point is simple. You never transform data in one big, fragile step. You move it through three small, reliable ones, and every stage stays queryable and auditable.

The three layers

Raw: untouched source data

Raw is your system of record. Data lands here exactly as the source sent it, with zero transforms and full history. Nothing is thrown away. If a downstream rule turns out to be wrong, you can always reprocess from Raw.

Because Raw keeps everything, the table format matters. OptimaFlo stores every layer in an open lakehouse table format, which gives Raw ACID transactions, schema evolution, and time travel. You can query the table as it looked last Tuesday, not just today.

Clean: cleaned and conformed

Clean is where source data becomes reliable. This layer validates types, removes duplicates, fixes nulls, and joins sources into clean, conformed tables. It is the layer most analysts actually build on.

Clean is also where SQL lives. In a traditional stack this is the dbt layer, written and maintained by data engineers. The transforms are version-controlled and reviewed, because a bad Clean query quietly poisons everything downstream.

Ready: business-ready

Ready is the layer your business sees. It holds aggregated metrics, star schemas, and the tables that power dashboards and reports. Ready tables are usually updated incrementally, so a daily metric refreshes in seconds instead of rebuilding from scratch.

Here is how the three layers compare:

Layer	Holds	Transforms	Typical consumer
Raw	Source data, full history	None	Reprocessing, audits
Clean	Cleaned, validated, joined	Type, dedupe, join	Analysts, data scientists
Ready	Metrics, star schema	Aggregate, model	Dashboards, executives

Why data teams adopt it

The pattern is popular because it solves real problems at once:

Reliability. Failures are isolated to one hop, not the whole pipeline.
Reproducibility. Keeping Raw plus lakehouse time travel means any result can be rebuilt exactly.
Incremental refinement. Each layer does one job, so changes are small and safe.
Auditability. You can trace any Ready number back through Clean to the original source record.

The catch: it usually takes a whole stack and a team

The architecture is elegant. Building it is not. A typical stack stitches together six to ten tools: ingestion (Fivetran), a warehouse (Snowflake), transforms (dbt), orchestration (Airflow), a BI tool (Tableau), and data observability (Monte Carlo). Each one is a contract, a bill, and a thing that breaks at 2 a.m.

It also takes people. Someone has to write the Clean and Ready SQL, wire up the orchestration, and own the schemas. For a data-rich team without a dedicated data engineering function, that is the wall most projects hit.

How OptimaFlo builds the medallion architecture for you

OptimaFlo is one platform that runs the full Raw, Clean, Ready flow inside your own cloud. It is built to remove the stack and help overwhelmed teams at the same time.

LLM-native. Describe the table you want in plain language. OptimaFlo generates the Clean and Ready SQL, you review and approve it, and it ships. The AI writes the data engineering, you keep the judgment.
Open lakehouse foundation. Every layer is an open table, so ACID guarantees, schema evolution, and time travel apply end to end, not just in one tool. No proprietary lock-in.
Bring your own cloud (BYOC). The platform runs in your environment. Your source data is stored in your account, never ours.
The right compute, automatically. OptimaFlo matches the compute to the size of your data: a light local engine for small jobs, your own warehouse as they grow. You never size or manage clusters.
One platform, not ten. Ingestion, transforms, orchestration, dashboards, and data quality checks live in one place.

The result: you can go from your first data source connection to a live, Ready-backed dashboard in a single afternoon. See the getting started guide for the five-minute version, or the architecture overview for the full picture.

What a Clean transform looks like

A Clean model is just SQL: take the Raw table, cast the types, drop the bad rows, and write a clean table. OptimaFlo generates SQL like this, and you approve it before it runs:

SELECT
    order_id,
    customer_id,
    CAST(order_total AS DECIMAL(12, 2)) AS order_total,
    DATE_TRUNC('day', ordered_at)       AS order_day
FROM raw.orders
WHERE order_total IS NOT NULL;

That clean.orders table then feeds a Ready metric, like daily revenue by customer segment, which lands in your dashboard.

Key takeaways

The medallion architecture refines data through three layers: Raw source data, Clean validated data, and Ready business-ready tables.
Its real cost is the multi-tool stack and the data team needed to run it.
OptimaFlo collapses that into one LLM-native platform on an open lakehouse, running in your own cloud.

Ready to build your first Raw-to-Ready pipeline? Start building with OptimaFlo or read the getting started guide.

Connecting an LLM to Your Data Is the 21% Solution.

Evan Rosa — Wed, 19 Aug 2026 17:18:10 +0000

Connecting an LLM to Your Data Is the 21% Solution.

I hear the same question in almost every early conversation: why do I need a platform when I can point Claude at Postgres and start asking questions?

It is a fair question. The connectors are real. Claude wires into Snowflake and Postgres, ChatGPT reaches warehouses over MCP bridges, Gemini sits inside BigQuery. The demo is genuinely impressive.

We also now have public numbers on what happens after the demo.

The numbers

Anthropic measured it on their own analytics workload: "Without skills, Claude's ability to answer analytics questions accurately didn't exceed 21% on our evals." Same model you would connect to your database. Their data, their questions, 21%.

The benchmarks agree. Spider 2.0 is built from real enterprise warehouse workflows: thousand-column schemas, hundred-line queries, multiple SQL dialects. GPT-4o scores 10.1% on it. o1-preview reaches 17.1%. The same models clear 85% on the older academic benchmark, which is why the demo feels like magic and the fourth week does not.

BIRD explains the collapse. Every BIRD question ships with a hand-written hint explaining what the fields mean. Remove the hints and GPT-4 falls from 54.89% to 34.88%. Your users do not write hints. They ask "how many active customers do we have" and assume the model knows what active means. It does not. It picks one of your four dozen candidate columns and sums with confidence.

That is the dangerous part. A wrong number does not look wrong. Anthropic names it the silent failure and admits their hardened stack reduces it rather than eliminating it.

What the 74 points cost

Anthropic closed the gap from 21% to above 95%. Not with a bigger model, and not with more data access: they gave the agent grep access to thousands of their own correct past queries and accuracy moved less than one point.

What worked was structure. Governed canonical datasets. A semantic layer where a human owns every metric definition. Lineage. Skills that encode how a careful analyst works. An eval harness watching all of it. Four layers, built and maintained by one of the best data teams in the world. I wrote about who gets to build that scaffolding when the post first landed.

The other vendors built the same conclusion into their products. Snowflake restricts its own MCP server to semantic views because raw schemas lack the meaning an analyst needs. Google wraps Gemini in a Knowledge Catalog and a Looker semantic layer and calls the bundle the product. The connector is the interface; the governed layer underneath is what they sell.

What a connector never builds

Accuracy aside, there is a category error in the pitch. A connector reads tables. Your data problem is that nobody is building the tables.

The connector will not land your sources in a warehouse. It will not model Raw into Clean into Ready. It runs when you open a chat and stops when you close it: ChatGPT sessions drop their state on disconnect and run nothing on a schedule. It watches nothing overnight. It re-learns your business every session, because chat memory is not a semantic layer.

And the plumbing itself is on you. Anthropic archived its reference Postgres MCP server, the one without query timeouts, row caps, or column blocklists, behind a plain warning: no security guarantees. It still gets about 312k installs a month, mostly from people wiring up exactly the setup in the pitch.

Where I land

OptimaFlo runs on these same models. You bring your own Claude, GPT, or Gemini key. I have no argument with the engine.

What we sell is the 74 points: the governed datasets, the semantic layer, the pipelines, the quality checks, and the eval discipline that Anthropic staffed a team to build, shipped as a product one data owner can run, on open Iceberg tables in your own cloud. A human approves the SQL before it touches your tables. The chat interface is in there too. It is the last step, and it is the easy one.

If your warehouse is already modeled and governed, a $20 chat seat on top of it is a great deal, and I say so in our side-by-side comparison. If nobody is building that warehouse, the seat has nothing to stand on.

The 21% is what the model brings. The rest is the job. Come see how we do it, or kick the tires in a 7-day pilot.

Apache Iceberg in Your Own Cloud, Without the Overhead

Evan Rosa — Wed, 19 Aug 2026 16:36:57 +0000

You do not need a data team to run Apache Iceberg. That used to be the catch, and it is no longer true. A single person can create, query, and maintain Iceberg tables today, because managed platforms took the hard parts off your plate.

So the interesting question is not whether you can run Iceberg. You can. The question is where your data lives while you do it, and what you trade away for the easy path.

Two ways to run Iceberg in 2026

There are two real options, and they pull in opposite directions.

Managed services. Platforms like Snowflake run Iceberg for you. They own the metadata catalog, the storage optimization, and the governance. You focus on pipelines and analytics, and one person can do it. The easiest on-ramp by far.

Self-managed open source. You host Iceberg on your own cloud object storage, like AWS S3 or Azure Blob, and query it with engines like Spark or Trino. This is the path that introduces operational overhead. Someone configures IAM, stands up the catalog, and keeps it running. That someone is usually a data engineer.

Both paths give you real Iceberg. The difference is what you give up.

The tradeoff nobody names: convenience versus control

Managed Iceberg is convenient, and the convenience has a price. Your tables and the optimization around them live inside the vendor's platform. You query open data, but the operational center of gravity sits with them.

Self-managed Iceberg is the opposite. The data stays in your own account. It stays portable, so any engine can read it and you are not tied to one vendor's roadmap or pricing. You pay for that freedom in setup and upkeep, which is why most small teams never choose it. The control is worth it. The overhead is the wall.

That leaves a gap. What if you want the control of self-managed Iceberg, in your own cloud, without paying the operational tax that normally comes with it?

Where OptimaFlo fits

OptimaFlo is built for exactly that gap. It runs the open, in-your-cloud version of Iceberg, and takes the setup work off you.

Your data lands in your own cloud account under a bring-your-own-cloud model, stored as open Iceberg tables across the medallion architecture: Raw source data, Clean modeled data, and Ready business metrics. The tables stay yours and stay portable. Here is what OptimaFlo takes off your plate.

Catalog and access setup, handled at connect time

The first hurdle in self-managed Iceberg is plumbing: standing up the catalog and wiring the cloud permissions so engines can read and write safely. OptimaFlo provisions that when you connect a source, so you do not hand-configure IAM roles or run a catalog server yourself.

The SQL, written by AI

Iceberg gives you durable tables. It does not write the transforms that fill them. You describe the table you want in plain language, OptimaFlo generates the Clean and Ready SQL, and you review and approve it before it runs. You keep the judgment and skip the hand-coding.

The right engine, picked for you

Iceberg separates storage from compute, so the same tables can run on different engines. OptimaFlo picks for you: small jobs on a light, cheap engine, larger ones on your warehouse. You never size a cluster.

Who this is for

You have data in your own cloud and a reason to keep it there: portability, governance, or just not wanting a vendor to hold it. You like the open, self-managed model, but you do not have an engineer to stand up catalogs and wire cloud permissions. That is the person OptimaFlo is built for. You get the control of self-managed Iceberg with an operational load closer to a managed service.

Key takeaways

You do not need a data team to use Iceberg. Managed services already make it a one-person job.
The real choice is convenience versus control: managed keeps your data in a vendor's platform, self-managed keeps it open and yours but adds operational overhead.
OptimaFlo removes that overhead for the in-your-cloud path: catalog and IAM at connect time, AI-generated SQL, and automatic engine selection.

Want open Iceberg in your own cloud without the setup work? Start building with OptimaFlo or read the getting started guide.