The Heisenberg Problem: Why Observing Your Data Pipeline Breaks Your Documentation
Or: How to stop lying to your auditors (and yourself)
There's a principle in quantum mechanics that says the act of observing a particle changes its behavior. Your data lineage documentation has the opposite problem: the moment you stop observing it, it collapses into a superposition of "probably still accurate" and "completely wrong."
You know the drill. Six months ago, someone built a meticulous lineage diagram in Lucidchart. Three sprints later, the ETL got refactored. Two months after that, a new Snowflake schema appeared. Last Tuesday, someone quietly renamed a column. Today, your compliance audit starts at 9 AM.
The spreadsheet is open. The cursor is blinking. The coffee is cold.
Let's talk about why this keeps happening — and then let's actually fix it.
The Fundamental Lie We Tell Ourselves
Manual lineage documentation fails for the same reason manual testing fails at scale: it requires humans to do something boring, consistently, forever. We are spectacularly bad at this.
But there's a deeper architectural problem hiding underneath the human problem. Most teams treat lineage as a documentation artifact rather than a system property. You wouldn't document your database schema in a Google Sheet and call it a day — you'd introspect it programmatically. Lineage deserves the same treatment.
Consider what your data actually knows about itself right now:
-- Your Snowflake query history knows exactly what touched what
SELECT
query_id,
query_text,
database_name,
schema_name,
execution_status,
start_time
FROM snowflake.account_usage.query_history
WHERE query_type IN ('INSERT', 'CREATE_TABLE_AS_SELECT', 'MERGE')
AND start_time >= DATEADD(day, -7, CURRENT_TIMESTAMP())
ORDER BY start_time DESC;
That's not documentation. That's evidence. The difference matters enormously when an auditor asks you to prove that PII from your CRM never touches your analytics warehouse without masking.
What Real Lineage Actually Looks Like
Before we get into the fix, let's be precise about what we're solving. Data lineage has three layers that most teams conflate:
1. Technical Lineage — Column A in Table X is derived from Column B in Table Y via transformation Z. Pure mechanics.
2. Operational Lineage — When did this transformation run? Did it succeed? What version of the transform logic was used? This is where incidents live.
3. Business Lineage — This "Revenue" metric in the dashboard traces back to this definition in the data contract, which was approved by Finance on this date. This is where auditors live.
A spreadsheet might capture a snapshot of layer one. It captures layers two and three almost never. This is why your compliance team and your data engineering team are essentially speaking different languages while standing in the same room.
The 5-Minute Fix (For Real This Time)
Here's where DataLineage enters the picture — and I want to show you exactly what happens under the hood, because the magic is less magic and more clever instrumentation.
Step 1: Connect your sources (2 minutes)
from datalineage import LineageClient
client = LineageClient(api_key="your_key_here")
# Connect to your warehouse — DataLineage uses read-only
# query history introspection, not query interception
client.connect_source(
name="production_snowflake",
type="snowflake",
config={
"account": "your-account.snowflakecomputing.com",
"warehouse": "COMPUTE_WH",
"role": "LINEAGE_READER", # Principle of least privilege
"database": "PROD_DB"
}
)
Notice what's happening here: DataLineage isn't a proxy sitting between your application and your database. It's reading query history and metadata, which means zero latency impact on your production systems. This is a non-negotiable design requirement for any lineage tool you should trust.
Step 2: Tag your sensitive assets (1 minute)
# Define your compliance domains
client.tag_assets([
{
"table": "PROD_DB.RAW.CUSTOMER_PII",
"tags": ["gdpr", "ccpa", "pii"],
"owner": "data-platform@yourcompany.com",
"classification": "restricted"
},
{
"table": "PROD_DB.ANALYTICS.REVENUE_METRICS",
"tags": ["financial", "sox-relevant"],
"owner": "finance-data@yourcompany.com",
"classification": "confidential"
}
])
Step 3: Let it run (2 minutes of your time, continuous thereafter)
# Start the lineage crawler — this runs as a background job
# scanning query history on your configured interval
lineage_job = client.start_crawler(
sources=["production_snowflake"],
scan_interval_minutes=15,
backfill_days=90 # Reconstruct historical lineage from query history
)
print(f"Crawler started: {lineage_job.id}")
print(f"Initial backfill ETA: {lineage_job.estimated_completion}")
# > Crawler started: clj_7f3a9b2c
# > Initial backfill ETA: ~4 minutes
Within minutes, you have a queryable, automatically-maintained graph of your entire data flow.
The Part That Actually Matters to Your Auditor
Here's where this stops being a developer toy and starts being a compliance asset:
# Generate a compliance report for a specific table
report = client.compliance_report(
table="PROD_DB.ANALYTICS.REVENUE_METRICS",
format="pdf",
include=[
"full_upstream_lineage",
"transformation_history",
"access_log_summary",
"quality_metrics_timeline",
"data_contract_status"
]
)
# Or query the lineage graph programmatically
upstream = client.get_upstream_lineage(
table="PROD_DB.ANALYTICS.REVENUE_METRICS",
depth=5, # How many hops back to trace
include_transformations=True
)
for node in upstream.nodes:
if "pii" in node.tags:
print(f"⚠️ PII exposure path: {node.full_path}")
print(f" Masking applied: {node.masking_confirmed}")
print(f" Last verified: {node.last_scan}")
When your auditor asks "show me every place customer email addresses are used," that's a three-second query, not a three-day investigation. When they ask for a timestamped record of data flow changes over the last 90 days, you generate a PDF. You don't open a spreadsheet.
The Uncomfortable Truth About Your Current Setup
Here's a question worth sitting with: If your lineage documentation is wrong, when would you find out?
With a spreadsheet, the answer is "when something breaks or someone asks." With automated lineage, the answer is "immediately, with a Slack notification and a diff of what changed."
That's not just a compliance improvement. That's a fundamentally different relationship with your own infrastructure — one where the systems tell you what they're doing rather than requiring you to remember to write it down.
The gap between your data engineering team and your governance team isn't a people problem. It's a tooling problem. People built spreadsheets because that's what was available. Now something better is available.
Get Started
The DataLineage open-source core — including the crawler engine, lineage graph schema, and local visualization UI — is available on GitHub. The cloud connectors for Snowflake, BigQuery, Redshift, dbt, and Airflow are all in the repo, along with a Docker Compose setup that gets you running locally in under five minutes.
→ github.com/datalineage/datalineage-core
Your next audit doesn't have to start with a cold cup of coffee and a stale spreadsheet. It can start with a query.
Found a bug? Want to add a connector for your stack? PRs are open and the maintainers are responsive. The issue tracker is the right place to start.
Top comments (0)