---
title: "Your Schema Changed. Congratulations, You've Just Inherited 47 Broken Pipelines."
published: false
tags: [dataengineering, python, opensource, dbt]
---
It's 9:47 AM on a Tuesday. A product manager renames a column in a source table — `user_id` becomes `customer_id`, totally reasonable, five minutes of work. By 2 PM, three dashboards are blank, an ML feature pipeline is silently feeding stale data to a model in production, and your Slack is a graveyard of confused @ mentions.
You spend the next four hours playing archaeologist in your own infrastructure, grepping through dbt YAML files, reading Airflow DAG source code, and cross-referencing a Confluence page that was last updated in 2022. You find the broken consumers. Eventually. Most of them.
This is not a tooling problem that requires better documentation or more disciplined engineers. It's a *visibility* problem. And visibility problems have a specific kind of solution.
---
## The Lineage Gap Nobody Talks About
Data lineage isn't a new idea. Every modern data warehouse has *some* version of it. But there's a gap between "lineage that exists in one tool" and "lineage that reflects how your actual pipelines work."
Your dbt models know about each other. Your Airflow DAGs know about their own tasks. Your Spark jobs know about their inputs and outputs — if someone remembered to log them. But none of these tools talk to each other, which means when a schema change ripples *across* tool boundaries, you're flying blind.
That's the gap [DataLineage](https://github.com/datalineage) was built to close.
---
## What DataLineage Actually Does
DataLineage runs as a lightweight service that connects to your existing stack — dbt, Airflow, Spark, and custom ETL — and builds a unified dependency graph across all of them. No manual configuration of relationships. No YAML files describing what you already know. It discovers the graph by reading what your tools already produce: dbt manifests, Airflow task metadata, Spark execution plans, and a thin SDK for anything custom.
The moment a schema changes, you query the graph and get back every downstream consumer, ranked by dependency depth, with the tool context preserved. You know *what* breaks, *where* it lives, and *how far* the blast radius extends.
Three things matter here:
**Auto-discovery, not auto-configuration.** You don't tell DataLineage about your pipelines. It reads your existing artifacts and infers the graph. Your dbt `manifest.json` already contains model dependencies. Your Airflow metadata database already contains task relationships. DataLineage connects the dots between them.
**Cross-tool lineage as a first-class feature.** A dbt model that feeds an Airflow DAG that triggers a Spark job that writes to a table consumed by three more dbt models — that full chain is visible as a single graph, not three disconnected fragments.
**Real-time impact analysis.** Before you rename that column, run an impact query. Get back the full list of affected consumers in under a second. Make the change knowing exactly what you're touching.
---
## Quick Start
bash
pip install datalineage
python
from datalineage import LineageClient
client = LineageClient(api_key="your_key")
impact = client.impact_analysis(table="raw.events", column="user_id")
print(impact.affected_nodes)
That's the core loop. Point it at a table and column, get back a structured list of everything downstream. The `affected_nodes` list includes node type (dbt model, Airflow task, Spark job), tool context, and dependency depth.
---
## A Real-World Scenario: The Column Rename That Didn't Hurt
Let's say you're migrating your user identifier scheme. The source table `raw.users` currently has `legacy_user_id` and you're standardizing on `user_id`. You want to deprecate `legacy_user_id` eventually, but first you need to know who's still depending on it.
python
from datalineage import LineageClient
from datalineage.models import ImpactReport
client = LineageClient(api_key="your_key")
Get full impact report before touching anything
report: ImpactReport = client.impact_analysis(
table="raw.users",
column="legacy_user_id",
depth=None # traverse the full graph, no depth limit
)
Group affected nodes by tool
by_tool = report.group_by_tool()
print(f"dbt models affected: {len(by_tool.get('dbt', []))}")
print(f"Airflow DAGs affected: {len(by_tool.get('airflow', []))}")
print(f"Spark jobs affected: {len(by_tool.get('spark', []))}")
Print the full dependency chain
for node in report.affected_nodes:
print(f"[depth={node.depth}] {node.tool}/{node.name} → {node.owner_team}")
Sample output:
plaintext
dbt models affected: 12
Airflow DAGs affected: 3
Spark jobs affected: 1
[depth=1] dbt/stg_users → analytics-eng
[depth=1] dbt/stg_user_events → analytics-eng
[depth=2] dbt/fct_user_sessions → analytics-eng
[depth=2] airflow/user_cohort_export → data-platform
[depth=3] dbt/rpt_weekly_active_users → analytics-eng
[depth=3] spark/user_feature_pipeline → ml-platform
...
Now you have a migration checklist. You know which teams to notify, in what order, and you can track remediation by querying the graph after each fix. When `affected_nodes` is empty, you're done.
This is the difference between a schema migration that takes a week of careful coordination and one that takes an afternoon.
---
## The API-First Design Decision
DataLineage exposes everything through a REST API with a Python client that wraps it. This wasn't an accident — it means the lineage graph is queryable from CI/CD pipelines, dbt hooks, Airflow sensors, or anywhere else you want to embed impact awareness.
A common pattern we've seen: add a DataLineage impact check as a pre-merge CI step for dbt PRs. If a PR modifies a model that has more than N downstream consumers, automatically request review from the teams that own those consumers. No more "I didn't know that model was used by the ML team."
python
In your CI pipeline
import sys
from datalineage import LineageClient
client = LineageClient(api_key=os.environ["DATALINEAGE_KEY"])
changed_models = get_changed_dbt_models() # your CI helper
for model in changed_models:
impact = client.impact_analysis(table=model)
if len(impact.affected_nodes) > 10:
print(f"⚠️ {model} affects {len(impact.affected_nodes)} downstream nodes")
print("Requesting cross-team review...")
sys.exit(1) # block merge, require manual approval
---
## Where We Are
DataLineage is in public beta. The Python client is stable. The connectors for dbt, Airflow, and Spark are production-ready. We're actively working on Flink support and a graph visualization UI (because sometimes you need to *see* the graph, not just query it).
The API is free during beta with generous rate limits. We're not going to surprise you with a pricing page before you've had a chance to actually use the thing.
---
## Try It
If you've ever spent an afternoon debugging a broken pipeline that broke because of a change you didn't know existed, DataLineage was built for you.
- ⭐ **[Star us on GitHub](https://github.com/datalineage/datalineage)** — it helps more data engineers find the project
- 🔑 **[Get a free API key](https://datalineage.io/signup)** — beta access, no credit card
- 📖 **[Read the docs](https://docs.datalineage.io)** — including connector setup guides for dbt, Airflow, and Spark
We're in the DataLineage Discord if you have questions, run into issues, or want to tell us what connectors to build next. See you there.
Top comments (0)