---
title: "Your Pipeline Is Lying to You (And You Won't Know Until It's Too Late)"
published: false
tags: [dataengineering, python, opensource, dbt]
---
It's 2:47 AM. Your phone is buzzing. The ML model that drives 30% of your company's revenue recommendations is returning garbage. You SSH in, grep through logs, and eventually trace it back to a "harmless" column rename that happened three weeks ago in a Postgres table upstream. Three weeks. The damage has been compounding silently ever since.
You didn't break anything. You just didn't *know* what you'd break.
This is the dirty secret of modern data stacks: we've gotten incredibly good at building pipelines and remarkably bad at understanding them. We have dbt for transformation, Airflow for orchestration, Spark for scale, and a graveyard of custom ETL scripts held together with cron jobs and optimism. Each tool knows its own world. None of them talk to each other.
So when something changes upstream, you're not doing impact analysis. You're doing archaeology.
---
## Introducing DataLineage
[DataLineage](https://github.com/datalineage/datalineage) automatically discovers and tracks data dependencies across your entire pipeline stack — dbt, Airflow, Spark, and custom ETL — without requiring you to annotate a single thing manually. When a schema changes, you see *every downstream consumer affected*, across tool boundaries, in real time.
Not a diagram you drew once and forgot to update. Not a wiki page from 2022. The actual, living dependency graph of your data.
---
## Quick Start
bash
pip install datalineage
python
from datalineage import LineageClient
client = LineageClient(api_key="your_key")
impact = client.get_impact("schema.orders", change="rename_column", column="user_id")
print(impact.affected_assets) # Every downstream model, job, and feature that just broke
That's it. No YAML config. No agents to deploy. No "enterprise onboarding call."
DataLineage connects to your existing tools through their APIs and metadata stores — dbt's manifest, Airflow's DAG definitions, Spark's query history — and stitches the dependency graph together automatically.
---
## The Real-World Problem: Schema Changes Don't Respect Tool Boundaries
Here's a scenario that's more common than anyone admits.
You have a dbt model `fct_orders` that reads from a raw table `raw.orders`. That model feeds an Airflow DAG that trains a churn prediction model. That DAG writes features to a feature store. Those features power a Spark batch job that scores your entire user base nightly.
A data analyst renames `customer_id` to `user_id` in `raw.orders` because the naming is inconsistent with everything else. Totally reasonable. They check: does dbt compile? Yes. Does the dbt test pass? Yes (they updated the model). Ship it.
What they couldn't see: the Airflow DAG that bypasses dbt and reads `raw.orders` directly. The Spark job that joins on `customer_id` from a different source. The feature pipeline that's been hardcoding that column name for eight months.
Let's see how DataLineage handles this:
python
from datalineage import LineageClient
from datalineage.models import SchemaChange, ChangeType
client = LineageClient(api_key="your_key")
Register the proposed change before you make it
change = SchemaChange(
asset="raw.orders",
change_type=ChangeType.RENAME_COLUMN,
before="customer_id",
after="user_id"
)
report = client.analyze_impact(change)
print(f"Direct consumers: {len(report.direct_consumers)}")
print(f"Total affected assets: {len(report.all_affected)}")
print(f"Critical path broken: {report.has_critical_path_impact}")
for asset in report.all_affected:
print(f" [{asset.tool}] {asset.name} — {asset.impact_type}")
Output:
plaintext
Direct consumers: 3
Total affected assets: 11
Critical path broken: True
[dbt] fct_orders — BREAKING (column reference)
[airflow] dag:churn_model_training — BREAKING (direct SQL query)
[airflow] dag:weekly_cohort_report — INDIRECT (via fct_orders)
[spark] job:nightly_user_scoring — BREAKING (join key)
[dbt] dim_customers — INDIRECT (via fct_orders)
... 6 more
The analyst sees this *before* merging. Not three weeks later at 2:47 AM.
---
## How Auto-Discovery Actually Works
DataLineage doesn't ask you to declare dependencies. It finds them.
For **dbt**, it parses your compiled `manifest.json` and extracts the full ref() and source() graph — including column-level lineage where dbt exposes it.
For **Airflow**, it hooks into the metadata database and parses DAG definitions to extract SQL queries, dataset references, and task dependencies.
For **Spark**, it processes query execution logs and the Spark SQL query plan to reconstruct what read from what.
For **custom ETL**, you get a lightweight decorator:
python
from datalineage import track
@track(reads=["raw.orders", "raw.products"], writes=["analytics.order_summary"])
def run_custom_etl():
# Your existing code, untouched
...
Everything flows into a unified graph. Cross-tool edges are first-class citizens, not afterthoughts.
---
## API-First, Because Your Stack Is Unique
Every data stack is a snowflake (the bad kind). DataLineage is built API-first so you can integrate it into your existing workflows rather than replacing them.
The Python client is the primary interface, but every operation is a REST call underneath. That means:
- **CI/CD integration**: Run impact analysis on every PR that touches schema definitions
- **Slack alerts**: Webhook when a breaking change is detected in a critical pipeline
- **dbt Cloud integration**: Trigger lineage refresh after every production job run
- **Custom dashboards**: Pull the graph data into whatever observability tool you already use
We're not trying to be your data catalog. We're trying to be the dependency engine that makes everything else smarter.
---
## What's Next
DataLineage is in public beta. What's working today:
- [x] dbt lineage (model + column level)
- [x] Airflow DAG parsing
- [x] Spark query log analysis
- [x] Real-time impact analysis API
- [x] Python client
On the roadmap:
- [ ] Fivetran + Airbyte source tracking
- [ ] GitHub PR integration (comment impact reports automatically)
- [ ] Column-level lineage across all tools
- [ ] Slack + PagerDuty alerting
---
## Try It
If you've ever spent more than an hour tracing a broken pipeline back to its source, DataLineage is for you.
**⭐ [Star us on GitHub](https://github.com/datalineage/datalineage)** — it genuinely helps us understand who's interested and what to build next.
**🚀 [Try the API](https://datalineage.io/signup)** — free tier includes up to 500 assets and unlimited impact analyses.
**💬 [Join our Discord](https://discord.gg/datalineage)** — we're actively building with early users. Your pipeline's weird edge case is exactly what we want to hear about.
The 2:47 AM call is optional. The broken pipeline is not. Let's fix the second one.
Top comments (0)