TL;DR
A metadata graph links datasets, columns, jobs, runs, dashboards and owners as nodes and edges, so an engineer can ask what depends on a table before changing it. Most teams keep that knowledge in separate tools: the orchestrator, the warehouse, the transformation layer and BI. As AI coding tools send more pipeline changes through review, the cost of not knowing what sits downstream grows with every merge. The guide walks through what to model, where the raw metadata comes from, how to build the graph in five steps, and how to keep it accurate.
Every data pipeline sits inside a set of relationships that nobody wrote down in one place. A table feeds a model, the model feeds a dashboard, a scheduled job copies the same table into a CRM, and a notebook built two years ago reads it every Monday. Someone knows each link, in some tool, but no one knows the whole chain. The longer an estate has been running, the wider that gap gets, and the more a small change to one table depends on luck to stay safe.
Teams feel this most when the pace of change picks up. In dbt Labs' 2025 State of Analytics Engineering report, 56% of respondents identified data quality as a problem and 80% said they use AI in some way in their workflows. More pipeline changes are coming, many drafted with AI help, and each one needs someone to know what it touches. A metadata graph answers that question directly. By the end, you will know what the graph contains, where its raw material comes from, how to build one in five steps, and how to keep it accurate as the stack changes.
What a Metadata Graph Is in Plain Terms
A graph is a set of things, called nodes, and the relationships between them, called edges. In a metadata graph the nodes are the parts of a data estate: a table, a column, a pipeline job, one run of that job, a dashboard, the team that owns something. The edges record how they connect. This job reads that table, this job wrote that table, this column is derived from those two columns, this dashboard queries that view. A catalog page lists the same facts one asset at a time. A graph stores the connections as data, so you can follow them in either direction.
The questions it answers are traversal questions. What is downstream of this table? Which source columns feed this metric? What ran against this table last night, and how long did it take? A list of assets handles these poorly, because each answer depends on hopping across several relationships, and often across several tools, without losing the thread.
A concrete estate makes this easier to follow, so the rest of this guide uses one. A retail analytics team runs Snowflake as the warehouse, dbt for transformation, Airflow for orchestration, a BI tool for dashboards and a reverse-ETL job that pushes revenue figures into the CRM. An engineer, working with an AI coding assistant, changes how net_revenue is calculated in the fct_orders model. The change passes its tests. The graph shows what the tests do not: fct_orders feeds a revenue dashboard, the CRM sync and a customer lifetime value model, and the lifetime value model feeds a retention dashboard. Four downstream assets, three of them outside the dbt project.
A practical model needs only a few node and edge types to start:
Datasets and columns: tables, views and files, plus the columns inside them.
Jobs and runs: a job is the definition, such as a dbt model build or an Airflow task. A run is one execution, with a start time, end time, status and runtime numbers.
Consumers: dashboards, reverse-ETL syncs, notebooks and applications that read the data.
Owners: the team or person to ask when something changes.
Edges: reads, writes, derived from (column to column), run of, consumed by and owned by.
Where the Raw Metadata Comes From in a Real Stack
No single system holds all of it, so the graph is assembled from several feeds. The most portable feed is OpenLineage, an open specification for lineage events. It defines three core entities. A job is a process that consumes or produces datasets, a run is one occurrence of a job in time, and a dataset is a discrete collection of data, such as a table. Each run has a runId, a UUID generated by the client and kept constant across that run's updates, with UUIDv7 recommended. A run event is sent when something significant happens, usually when a job starts and finishes, so the first state is usually START and the last usually COMPLETE.
Facets attach detail to those entities. A schema facet records a dataset's columns, and a SQL facet can carry the text of the query a job ran. The column lineage facet lists, for each output column, the input fields it came from, and gives each one a transformation type. DIRECT marks an output that is derived from the input, with subtypes IDENTITY, TRANSFORMATION and AGGREGATION. INDIRECT marks an input that influences the output without being part of its value, with subtypes such as JOIN, GROUP_BY, FILTER and SORT. That vocabulary is worth adopting for the whole graph, because it separates a column that is copied from one that only filters rows.
The warehouse is the second feed, and it records what happened instead of what a tool declared. On Snowflake, the ACCESS_HISTORY view in the ACCOUNT_USAGE schema records the objects each query accessed and the objects it modified. For write operations, the objects_modified column carries directSources and baseSources fields, and Snowflake notes that the baseSources columns facilitate column lineage. The view requires Enterprise Edition or higher, can lag by up to 180 minutes, and reaches back 365 days. The OBJECT_DEPENDENCIES view adds structural links, such as a view that references a table or a dynamic table that references a source, for Snowflake objects only.
Snowsight also draws a lineage graph for Enterprise Edition accounts. It traces data movement from statements such as CTAS, INSERT and MERGE, and keeps column and object lineage for one year. Its documented limits are the kind of gap a graph builder has to plan around. Lineage was introduced in November 2024, so data movement before that date is not available. It cannot track data movement that results from separate, disjointed queries. Objects in shared databases and in the SNOWFLAKE database are excluded.
Neither feed is complete on its own. Events tell you what a tool declared, warehouse history tells you what the warehouse saw, and some tools emit no lineage events at all. That is why graph builders usually add a third feed, the SQL text itself, parsed to recover column-level relationships.
Building the Graph in Five Practical Steps
The sequence below works whether the graph lives in a graph database, a relational schema or a set of tables in the warehouse. The data model counts for more than the storage engine.
Fix naming first. Pick one identifier scheme for datasets and columns and apply it to every source. OpenLineage identifies a dataset by a namespace and a name, such as a warehouse account and a database.schema.table path. Without a shared scheme,
PROD.MARTS.FCT_ORDERSfrom the warehouse andprod.marts.fct_ordersfrom an orchestrator become two nodes, and the graph quietly splits in half. Normalizing case and quoting is the first job.Ingest events and warehouse snapshots. Accept OpenLineage run events over HTTP and store each one with its timestamp. Run a separate collector for warehouse metadata on a schedule that fits the source. A view with up to 180 minutes of latency, like ACCESS_HISTORY, will always trail real time, so record when each row was loaded as well as when the query ran.
Derive column edges by parsing SQL. For each job, take the query text, parse it with a parser that understands the warehouse dialect, and trace each output column back to its input columns. Record a transformation type using the same DIRECT and INDIRECT vocabulary as OpenLineage, so edges from different feeds can be compared. Where the parser has to guess, such as a SELECT * with no known schema or a match made on column name alone, store the edge with an inferred flag.
Keep history. Store each edge with the time range in which it was valid instead of overwriting it. A table of edges with valid_from and valid_to columns is enough. Then "what depended on this column before last Tuesday's deploy" is a query instead of an investigation.
Query it. A downstream lookup is a recursive query over the edge table. The version below assumes a table edges(src, dst) and returns everything downstream of fct_orders with its distance in hops. The depth limit stops the recursion if the data contains a cycle.
WITH RECURSIVE downstream(node, depth) AS (
SELECT dst, 1 FROM edges WHERE src = 'marts.fct_orders'
UNION
SELECT e.dst, d.depth + 1
FROM edges e JOIN downstream d ON e.src = d.node
WHERE d.depth < 10
)
SELECT node, MIN(depth) AS hops
FROM downstream
GROUP BY node
ORDER BY hops, node;
Run against the retail estate, the query returns the revenue dashboard, the lifetime value model and the CRM sync at one hop, and the retention dashboard at two. Adding the validity range from step four turns the same query into a point-in-time lookup.
Keeping the Graph Fresh and Trustworthy Over Time
Each feed has its own latency, so the graph should record when every edge was last confirmed and by which source. An engineer who sees that an edge came from warehouse history three hours ago reads it differently from one that came from a manifest file last month. A graph that serves stale edges without saying so is more dangerous than one with visible gaps.
The same applies to how an edge was obtained. Label each one as declared by an emitter, observed in warehouse history, parsed from SQL or inferred. Parsed lineage is a model of the query text and can miss dynamic SQL or behavior that only appears at runtime. Inferred edges, such as a match on column name when no other evidence exists, are useful leads and should never be shown with the same confidence as an observed write.
Coverage is the third discipline. Keep a list of the systems that feed the graph and a list of systems that are known to exist but have no feed yet. A stored procedure that builds tables, a notebook that exports files, a sync job nobody registered: if the graph cannot see them, an answer of "nothing downstream" can be wrong. Surfacing those systems as unmapped lets a reader tell the difference between a table nobody uses and a table used by something the graph cannot see.
What Data Engineers Do With the Graph Day to Day
The first use is the impact check before a merge. In the retail example, the engineer changing net_revenue queries the graph for that column. Column-level edges narrow the answer from every consumer of fct_orders to the consumers that read the changed column, and the result names the revenue dashboard, the CRM sync and, through the lifetime value model, the retention dashboard. The review conversation starts from a list instead of from a guess.
The second use depends on whether the graph also stores run metrics. If each run records duration, rows returned, bytes returned and bytes scanned, the graph can compare two versions of the same job. Suppose the new calculation also rewrites a join, and the new version scans four times the bytes of the old one. A dependency list would call the change safe. The metric comparison shows the cost before it reaches the warehouse bill.
The third use is investigation after something breaks. When a dashboard shows a wrong number on Tuesday, the engineer walks upstream from the dashboard, lists which jobs wrote each upstream table in the window, and compares schema and column edges before and after the incident. With history stored, that comparison is a query. The fourth is migration planning: traversing from every source system to its consumers shows what has to move together, because an estate cannot be migrated until it has been mapped.
How Datameer Builds and Uses the Metadata Graph
Datameer builds this graph for data engineering teams, with catalog, lineage and versioned metadata history in one place. Metadata arrives from systems that emit OpenLineage, from Snowflake and from other non-dbt sources, so the graph spans platforms instead of stopping at the warehouse. Column-level lineage is derived by parsing each job's SQL. Where an edge is inferred, for example by matching column names, Datameer marks it as inferred instead of presenting it as certain. Systems that show up in the metadata but are not yet mapped are flagged, which gives the coverage list described earlier.
The graph is used at two points in a change. On a GitHub pull request, Datameer posts an impact report that lists the downstream assets the change touches and compares runtime metrics between the two versions of the job: duration, result rows, result bytes, bytes scanned and records read. In the retail example, the report on the net_revenue change would name the revenue dashboard, the CRM sync and the retention dashboard, and would show how the new version's bytes scanned compare with the current one.
After something breaks, the same graph supports investigation. An engineer can ask a question in the pull request thread, in Slack, in a web chat or from the CLI, and the agent works from the catalog, the lineage, point-in-time reads of earlier metadata versions and operational logs from tools such as Datadog, all in one context. Dependency lists tell you which models and dashboards are affected. Adding run metrics to the same graph shows what the change does to cost, such as a version that scans four times the bytes.
Common Mistakes When Teams Build Their Own Graph
Modeling too much at the start. Teams add every entity type before one traversal works. Begin with datasets, columns, jobs and runs, and add dashboards and owners as their feeds arrive.
Trusting design-time lineage alone. Declared dependencies in a project file show intent. Warehouse history shows what executed. The gap between them is where surprises live.
Overwriting instead of versioning. Without history, the graph can describe today's estate but cannot answer what changed on the day an incident began.
Reading a missing edge as no dependency. Absence from the graph has two possible causes: nothing reads the table, or something reads it that the graph cannot see. Tracking coverage separates the two.
Where Should You Start With Your Own Graph
A metadata graph turns scattered facts about tables, columns, jobs and runs into relationships you can traverse. The practical path is short: normalize identifiers, ingest OpenLineage events and warehouse history, parse SQL for column edges and label the inferred ones, store history, and publish the list of systems the graph cannot see yet. Choose one pipeline with a known consumer outside the transformation layer, prove the traversal on it, and widen from there.
To see how this works on a live estate, book a Datameer demo. For background on why static catalogs fall behind and what changes when metadata is kept current, read the guide to active metadata management.
FAQ
What is the difference between a metadata graph and a data catalog?
A catalog is an inventory of assets with descriptions, owners and tags. A metadata graph stores the relationships between those assets as queryable edges, with history. Many catalogs include some lineage, and a useful test is whether you can traverse relationships in both directions and compare them over time.
Do I need a graph database to build one?
No. The recursive query in this guide runs on a plain table of edges in any database that supports recursive common table expressions. A dedicated graph database becomes worth considering when traversals get deep or frequent enough that query time becomes a problem.
How is column-level lineage produced in practice?
There are three routes. A tool can emit it, for example through the OpenLineage column lineage facet. The warehouse can record it, as Snowflake does with source columns in ACCESS_HISTORY for write operations. Or a parser can derive it from the query text. Parsing works wherever the SQL is readable, but it should flag any edge it had to infer.
Is OpenLineage required to build a metadata graph?
No. It is an open, portable event format that adds run-level context such as status and parent runs. A graph can also be built from warehouse history and SQL parsing alone, and the two approaches work well together.
Which systems can the Datameer graph include?
Datameer builds its graph from systems that emit OpenLineage, from Snowflake and from other non-dbt sources. Systems that appear in the metadata but are not yet mapped are flagged instead of being silently ignored.
How does Datameer show uncertain column lineage?
Column lineage is derived by parsing each job's SQL. Edges that rely on inference, such as a match on column name, are marked as inferred, so a reviewer can tell which links are parsed with confidence and which are working assumptions.
Top comments (0)