Introducing DataLineage -- Automated Data Pipeline Lineage Tracking

---
title: "Your Schema Changed. Congratulations — You Just Broke 47 Things."
published: false
tags: [dataengineering, python, opensource, dbt]
---

It's 2:47 PM on a Thursday. A product manager pings you: "Hey, can we just rename `user_id` to `customer_id` in the users table? Should be quick, right?"

You stare at the message. You've been in this industry long enough to know that "quick rename" is the data engineering equivalent of "we just need to move one wall" in construction. Somewhere downstream, a Spark job is reading that column. Three dbt models depend on it. An Airflow DAG feeds a dashboard that the CEO checks every morning. You don't know exactly which ones. Nobody does. The knowledge lives in a combination of Slack threads, a Confluence page last updated in 2022, and the brain of a senior engineer who's on PTO in Portugal.

So you do what every data engineer does: you grep through repos, open eight browser tabs, and spend four hours building a mental map of something that should have been documented automatically.

That's the problem DataLineage solves.

---

## What DataLineage Actually Does

DataLineage crawls your pipeline stack — dbt, Airflow, Spark, and custom ETL scripts — and builds a unified dependency graph across all of them. Not a static diagram you maintain by hand. A live graph that updates as your pipelines change.

The core value proposition is dead simple: **when something changes, you know immediately what breaks.**

- Auto-discovers dependencies by parsing dbt manifests, Airflow DAG definitions, and Spark execution plans
- Stitches cross-tool lineage together (yes, it understands that your Airflow DAG triggers a dbt model that feeds a Spark job)
- Runs real-time impact analysis when a schema change is detected
- Exposes everything through an API with a first-class Python client

No YAML files to maintain. No manual graph updates. It reads what you've already written.

---

## Quick Start

bash
pip install datalineage

Point it at your stack and let it discover:

python
from datalineage import LineageClient

client = LineageClient(api_key="your_key")
graph = client.discover(sources=["dbt://./my_project", "airflow://localhost:8080"])
print(graph.summary())

That's it. You now have a queryable dependency graph of your entire pipeline. The `graph.summary()` output tells you how many nodes were discovered, how many cross-tool edges were found, and flags any circular dependencies it caught along the way.

---

## A Real-World Scenario: The Thursday Rename

Let's go back to that `user_id` → `customer_id` rename. Here's how you'd handle it with DataLineage:

python
from datalineage import LineageClient, SchemaChange

client = LineageClient(api_key="your_key")

Define the proposed change

change = SchemaChange(
table="warehouse.users",
column_rename={"user_id": "customer_id"}
)

Run impact analysis before touching anything

impact = client.analyze_impact(change)

print(f"Directly affected nodes: {len(impact.direct)}")
print(f"Transitively affected nodes: {len(impact.transitive)}")
print()

for node in impact.direct:
print(f" [{node.tool}] {node.name} — {node.owner_email}")

Output:

Directly affected nodes: 6

Transitively affected nodes: 41

[dbt] stg_users — [e-mail]

[dbt] fct_orders — [e-mail]

[spark] user_feature_pipeline — [e-mail]

[airflow] daily_user_export — [e-mail]

[dbt] dim_customers — [e-mail]

[custom_etl] segment_sync — [e-mail]

Now you have something concrete to bring back to that PM. Not "I think it touches some things" — a precise list of 47 nodes (6 direct, 41 transitive), organized by tool, with owner contacts already attached.

You can also generate a migration checklist:

python
checklist = impact.generate_migration_plan()
checklist.export_markdown("rename_migration.md")

This produces a step-by-step document with the recommended order to update each dependent, based on topological sort of the dependency graph. Update the leaf nodes first, work your way up to the source. No more accidentally breaking a downstream model because you updated the upstream first.

---

## The Cross-Tool Part Is the Hard Part (We Did It For You)

Most lineage tools work great *within* a single tool. dbt's built-in lineage is genuinely excellent — inside dbt. But the moment your dbt model output gets picked up by a Spark feature pipeline, that lineage chain breaks. You're back to tribal knowledge.

DataLineage resolves cross-tool edges by matching on table and column identifiers across tool boundaries. When your Airflow DAG runs a `SparkSubmitOperator` that reads from `warehouse.fct_orders`, and `fct_orders` is a dbt model, DataLineage connects those dots. The graph spans the entire chain from raw source to final consumer, regardless of which tool owns each step.

This matters most for ML teams. Feature pipelines are notorious for having opaque upstream dependencies. An ML engineer shouldn't have to reverse-engineer three layers of dbt models to figure out where a feature value comes from — or why it suddenly started returning nulls after a schema migration.

---

## API-First Design

Everything in DataLineage is accessible via REST API, which means it fits into existing workflows without requiring anyone to change how they work.

The Python client is a thin, ergonomic wrapper. But if you want to integrate lineage queries into your own tooling, CI pipelines, or Slack bots, the raw API is fully documented and straightforward.

A common pattern: add a lineage impact check to your PR pipeline. Before any schema migration merges, a GitHub Action calls `analyze_impact` and posts the affected node count as a PR comment. Your team sees the blast radius before the change lands in production.

python

In your CI script

impact = client.analyze_impact(change)
if len(impact.transitive) > 10:
post_pr_comment(f"⚠️ This change affects {len(impact.transitive)} downstream nodes. Review required.")

Small addition. Significant reduction in Thursday-afternoon incidents.

---

## What's Next

We're actively building:
- **Slack integration** — get notified when a schema change affects pipelines you own
- **Column-level lineage** — trace a single column through transformations across tools
- **Snowflake + BigQuery query log parsing** — auto-discover dependencies from warehouse query history

---

## Try It

The API is live and the Python client is on PyPI. If you're managing a pipeline stack across more than one tool and you've ever spent an afternoon manually tracing a dependency, this is worth ten minutes of your time.

**[⭐ Star us on GitHub](https://github.com/datalineage/datalineage)** — it helps more than you'd think.

**[Try the API →](https://datalineage.io/signup)** — free tier covers most small-to-medium stacks.

If you hit something broken or missing, open an issue. We read them all. The Thursday rename scenario above came directly from a user report, and it shipped as a feature two weeks later.

---

*Questions? Drop them in the comments or find us on the dbt Slack (#tools-and-integrations). We're there.*