DEV Community: Ahmed Moussa

How to Automate SOC2 and GDPR Compliance Scans with ComplianceWeave

Ahmed Moussa — Sat, 15 Aug 2026 05:45:02 +0000

---
title: "I Automated My SOC2 Audit Prep in an Afternoon (And You Can Too)"
published: false
tags: [security, devops, python, compliance]
---

Picture this: it's 11 PM, three weeks before your SOC2 audit, and you're manually screenshotting AWS console pages into a Google Doc labeled `evidence_FINAL_v3_USE_THIS_ONE.docx`.

We've all been there. Or we've *heard* the war stories.

This tutorial is the exit ramp from that highway to misery. We're going to wire up **ComplianceWeave** to your infrastructure and have it do the evidence-hunting while you do literally anything else.

By the end, you'll have a Python script that scans your environment, surfaces gaps, triggers remediation, and hands you a polished audit report — automatically.

Let's build it.

---

## What We're Working With

ComplianceWeave exposes three endpoints that, chained together, form a complete compliance automation loop:

| Endpoint | What it does |
|---|---|
| `POST /compliance/scan` | Kicks off an infrastructure scan against your chosen frameworks |
| `GET /compliance/reports` | Fetches generated audit-ready reports |
| `POST /compliance/remediate` | Auto-remediates fixable violations |

We'll use all three. In order. Like sensible engineers.

---

## Prerequisites

bash
pip install requests python-dotenv rich


We're using `rich` for readable terminal output because staring at raw JSON at 11 PM is a form of self-harm.

Create a `.env` file:

bash
COMPLIANCEWEAVE_API_KEY=your_api_key_here
COMPLIANCEWEAVE_BASE_URL=https://api.complianceweave.io/v1


---

## Step 1: Set Up Your Client

First, let's build a small, reusable client so we're not copy-pasting headers into every request.

python

compliance_client.py

import os
import time
import requests
from dotenv import load_dotenv
from rich.console import Console
from rich.panel import Panel

load_dotenv()
console = Console()

class ComplianceWeaveClient:
def init(self):
self.api_key = os.getenv("COMPLIANCEWEAVE_API_KEY")
self.base_url = os.getenv("COMPLIANCEWEAVE_BASE_URL")

    if not self.api_key:
        raise EnvironmentError(
            "COMPLIANCEWEAVE_API_KEY not set. "
            "Check your .env file."
        )

    self.session = requests.Session()
    self.session.headers.update({
        "Authorization": f"Bearer {self.api_key}",
        "Content-Type": "application/json",
        "Accept": "application/json",
    })

def _request(self, method, endpoint, **kwargs):
    url = f"{self.base_url}{endpoint}"
    try:
        response = self.session.request(method, url, **kwargs)
        response.raise_for_status()
        return response.json()
    except requests.exceptions.HTTPError as e:
        console.print(f"[bold red]HTTP Error:[/bold red] {e.response.status_code} — {e.response.text}")
        raise
    except requests.exceptions.ConnectionError:
        console.print("[bold red]Connection failed.[/bold red] Is the API URL correct?")
        raise
    except requests.exceptions.Timeout:
        console.print("[bold red]Request timed out.[/bold red] Try again or check your network.")
        raise


**Why a session object?** It reuses the TCP connection across requests, which matters when you're hitting the API repeatedly during a scan-poll loop. Small wins add up.

---

## Step 2: Trigger a Compliance Scan

Now let's initiate a scan. ComplianceWeave lets you specify which frameworks to check — we'll target SOC2 and GDPR simultaneously.

python

scan.py

from compliance_client import ComplianceWeaveClient, console
from rich.progress import Progress, SpinnerColumn, TextColumn
import time

def run_scan(client, frameworks=None, environment="production"):
if frameworks is None:
frameworks = ["SOC2", "GDPR"]

console.print(Panel(
    f"Starting scan for: [bold cyan]{', '.join(frameworks)}[/bold cyan]\n"
    f"Environment: [bold]{environment}[/bold]",
    title="🔍 ComplianceWeave Scan"
))

payload = {
    "frameworks": frameworks,
    "environment": environment,
    "options": {
        "deep_scan": True,         # Check resource-level configs, not just account-level
        "include_remediation": True # Flag auto-fixable issues
    }
}

response = client._request("POST", "/compliance/scan", json=payload)
scan_id = response.get("scan_id")

if not scan_id:
    raise ValueError("No scan_id returned. Something went wrong on the API side.")

console.print(f"[green]✓ Scan initiated.[/green] ID: [bold]{scan_id}[/bold]")

# Poll until the scan completes
with Progress(
    SpinnerColumn(),
    TextColumn("[progress.description]{task.description}"),
    transient=True,
) as progress:
    task = progress.add_task("Scanning infrastructure...", total=None)

    while True:
        status_response = client._request("GET", f"/compliance/scan/{scan_id}")
        status = status_response.get("status")

        if status == "completed":
            progress.stop()
            break
        elif status == "failed":
            raise RuntimeError(f"Scan failed: {status_response.get('error', 'Unknown error')}")

        time.sleep(10)  # Don't hammer the API

console.print(f"[green]✓ Scan complete.[/green]")
return status_response


**Expected output:**

plaintext
╭──────────────── 🔍 ComplianceWeave Scan ────────────────╮
│ Starting scan for: SOC2, GDPR │
│ Environment: production │
╰─────────────────────────────────────────────────────────╯
✓ Scan initiated. ID: scan_a3f92b1c
⠸ Scanning infrastructure...
✓ Scan complete.


The polling loop is intentional. Scans against real infrastructure take time — fighting that with impatient code just gets you incomplete results.

---

## Step 3: Auto-Remediate What You Can

Before pulling the report, let's fix the low-hanging fruit. ComplianceWeave flags issues it can remediate automatically (think: misconfigured S3 bucket ACLs, overly permissive security groups, missing encryption settings).

python

remediate.py

from compliance_client import ComplianceWeaveClient, console
from rich.table import Table

def remediate_violations(client, scan_results):
violations = scan_results.get("violations", [])
auto_fixable = [v for v in violations if v.get("auto_remediable") is True]

if not auto_fixable:
    console.print("[yellow]No auto-remediable violations found.[/yellow] Manual review required for all issues.")
    return []

table = Table(title="Auto-Remediating Violations")
table.add_column("Resource", style="cyan")
table.add_column("Issue", style="yellow")
table.add_column("Framework", style="magenta")

for v in auto_fixable:
    table.add_row(v["resource_id"], v["description"], v["framework"])

console.print(table)

violation_ids = [v["violation_id"] for v in auto_fixable]

payload = {
    "violation_ids": violation_ids,
    "dry_run": False  # Set True to preview changes without applying
}

result = client._request("POST", "/compliance/remediate", json=payload)

fixed = result.get("remediated", [])
failed = result.get("failed", [])

console.print(f"[green]✓ Fixed {len(fixed)} violation(s).[/green]")
if failed:
    console.print(f"[red]✗ {len(failed)} remediation(s) failed.[/red] Check the report for details.")

return fixed


> **Best practice:** Always run with `dry_run: True` first in a staging environment. Automated remediation is powerful. So is accidentally locking yourself out of a resource.

---

## Step 4: Pull Your Audit-Ready Report

The scan is done, violations are patched. Time to collect your evidence.

python

report.py

from compliance_client import ComplianceWeaveClient, console
import json
from datetime import datetime

def fetch_report(client, scan_id, output_format="pdf"):
console.print(f"\n[bold]Fetching report for scan:[/bold] {scan_id}")

params = {
    "scan_id": scan_id,
    "format": output_format,   # "pdf", "json", or "csv"
    "include_evidence": True,  # Attach raw evidence artifacts
    "frameworks": ["SOC2", "GDPR"]
}

report = client._request("GET", "/compliance/reports", params=params)

# Save the report locally
timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
filename = f"compliance_report_{timestamp}.json"

with open(filename, "w") as f:
    json.dump(report, f, indent=2)

# Print a human-readable summary
summary = report.get("summary", {})
console.print(f"\n[bold green]Report Summary[/bold green]")
console.print(f"  Overall Score:     {summary.get('compliance_score', 'N/A')}%")
console.print(f"  Controls Passed:   {summary.get('controls_passed', 0)}")
console.print(f"  Controls Failed:   {summary.get('controls_failed', 0)}")
console.print(f"  Report saved to:   [cyan]{filename}[/cyan]")

return report, filename


**Expected output:**

plaintext
Fetching report for scan: scan_a3f92b1c

Report Summary
Overall Score: 91%
Controls Passed: 147
Controls Failed: 13
Report saved to: compliance_report_20240315_143022.json


---

## Step 5: Wire It All Together

python

main.py

from compliance_client import ComplianceWeaveClient, console
from scan import run_scan
from remediate import remediate_violations
from report import fetch_report

def main():
client = ComplianceWeaveClient()

# 1. Scan
scan_results = run_scan(
    client,
    frameworks=["SOC2", "GDPR", "HIPAA"],
    environment="production"
)

# 2. Remediate what we can
remediate_violations(client, scan_results)

# 3. Re-scan to capture remediation (optional but recommended for auditors)
final_results = run_scan(client, frameworks=["SOC2", "GDPR", "HIPAA"])

# 4. Generate the report
report, filename = fetch_report(
    client,
    scan_id=final_results["scan_id"]
)

console.print(f"\n[bold green]✓ Done.[/bold green] Hand [cyan]{filename}[/cyan] to your auditor.")

if name == "main":
main()


---

## What You Actually Built

In ~150 lines of Python, you now have:

- **Automated evidence collection** across SOC2, GDPR, HIPAA, and ISO 27001
- **Intelligent remediation** that fixes what it safely can
- **Audit-ready reports** generated without a single screenshot

The real win isn't the code — it's the meeting you won't have to schedule to explain why the evidence doc is named `FINAL_v3_USE_THIS_ONE`.

Schedule this as a weekly cron job. Let ComplianceWeave run continuously. Show up to your audit with a folder of timestamped reports instead of a folder of regrets.

Your future self, three weeks before the next audit, will thank you.

---

*Questions? Drop them in the comments. I check. Eventually.*

Introducing ComplianceWeave -- Automated Compliance Monitoring for DevSecOps Teams

Ahmed Moussa — Sat, 15 Aug 2026 05:45:00 +0000

---
title: "I Automated the Part of My Job That Was Making Me Quit"
published: false
tags: [devsecops, compliance, python, security]
---

There's a specific kind of dread that hits on a Tuesday afternoon when your Slack lights up with: *"Audit prep starts next week. Can you pull together the evidence package?"*

You know what that means. Three weeks of:
- Digging through CloudTrail logs like an archaeologist
- Exporting CSVs from five different dashboards
- Writing the same "here's what we do for access control" paragraph for the fourth time this year
- Discovering, on day twelve, that one Lambda function has been logging to an unencrypted bucket since March

I've been that engineer. I've also been the one who *caused* that Lambda situation. Neither role is fun.

This is why we built **ComplianceWeave** — continuous infrastructure compliance monitoring that generates audit-ready reports automatically, across SOC2, GDPR, HIPAA, and ISO 27001, without the quarterly fire drill.

---

## The Actual Problem (Not the Sales Version)

Point-in-time audits are a lie we tell ourselves.

You scramble, you clean up, you document, you pass. Then six weeks later someone deploys a new service and forgets to tag it, or a contractor account doesn't get deprovisioned, or MFA enforcement drifts because someone was fighting a production incident and temporarily loosened a policy.

Compliance isn't a state you achieve. It's a state you *maintain*. And maintaining it manually, across multiple frameworks that all care about slightly different things, is a full-time job that nobody actually has time for.

ComplianceWeave treats compliance like you treat uptime: something you monitor continuously, alert on when it breaks, and have runbooks for.

---

## Quick Start

bash
pip install complianceweave


Set your API key and point it at your infrastructure:

python
import complianceweave as cw

client = cw.Client(api_key="cw_live_...")
scan = client.scan(frameworks=["soc2", "gdpr"], target="aws://us-east-1")
print(scan.summary())


That's it. Within minutes you'll have a gap analysis across both frameworks, ranked by severity, with control IDs mapped to the actual resources causing the finding.

No YAML manifests. No agent to deploy. No "contact sales to enable this feature."

---

## Real-World Use Case: Catching Drift Before Your Auditor Does

Here's a pattern I see constantly: a team passes their SOC2 Type II audit in Q1, then spends the rest of the year building features. By Q4, when the next audit window opens, they've accumulated months of compliance drift they don't know about.

Let's say you want to run ComplianceWeave as part of your CI/CD pipeline — not just scanning prod, but catching violations before they ever merge.

python
import complianceweave as cw
import sys

client = cw.Client(api_key="cw_live_...")

Scan a staging environment after infrastructure changes

scan = client.scan(
frameworks=["soc2", "hipaa"],
target="aws://us-west-2",
environment="staging",
severity_threshold="medium"
)

Get auto-generated remediation plans for any findings

if scan.has_violations():
for violation in scan.violations:
print(f"[{violation.severity.upper()}] {violation.control_id}: {violation.title}")
print(f" Resource: {violation.resource_arn}")
print(f" Fix: {violation.remediation.description}")
print(f" Terraform: {violation.remediation.iac_snippet}\n")

# Fail the pipeline if critical violations exist
if scan.critical_count > 0:
    print(f"❌ {scan.critical_count} critical violation(s) found. Blocking deploy.")
    sys.exit(1)

print(f"✅ Scan complete. {scan.passing_controls}/{scan.total_controls} controls passing.")


**Sample output:**

plaintext
[HIGH] CC6.1: Encryption at rest not enabled
Resource: arn:aws:rds:us-west-2:123456789:db/user-data-staging
Fix: Enable storage encryption on RDS instance. Note: requires snapshot restore for existing instances.
Terraform: storage_encrypted = true

[MEDIUM] GDPR-Art32: Data processing logs retained beyond policy window
Resource: arn:aws:logs:us-west-2:123456789:log-group:/app/user-events
Fix: Set retention policy to 90 days per your DPA.
Terraform: retention_in_days = 90

✅ Scan complete. 47/49 controls passing.


The remediation plans aren't generic advice — they're generated from the actual resource configuration, with IaC snippets you can drop directly into your Terraform. When your auditor asks "how do you ensure encryption at rest?", your answer isn't a doc you wrote last February. It's a dashboard showing continuous enforcement and a git history of remediations.

---

## Multi-Framework, Without the Redundancy

One thing that's genuinely annoying about compliance is that SOC2, HIPAA, and ISO 27001 all care about encryption, access control, and audit logging — they just call them different things and ask for slightly different evidence.

ComplianceWeave maps controls across frameworks automatically. If your S3 bucket encryption satisfies SOC2 CC6.1, we tell you it also satisfies HIPAA §164.312(a)(2)(iv) and ISO 27001 A.10.1.1. You fix it once, it counts everywhere.

python

See which frameworks a single remediation satisfies

violation = scan.violations[0]
print(violation.cross_framework_mappings)

{

"soc2": "CC6.1",

"hipaa": "§164.312(a)(2)(iv)",

"iso27001": "A.10.1.1"

}


For startups navigating their first SOC2 while also trying to land an enterprise customer who wants HIPAA attestation, this is the difference between two parallel workstreams and one.

---

## Generating Audit Reports

When the auditor actually shows up:

python
report = client.generate_report(
framework="soc2",
period_start="2024-01-01",
period_end="2024-12-31",
format="pdf" # or "json", "csv"
)

report.save("soc2_type2_evidence_2024.pdf")


The report includes control status over time (not just current state), evidence timestamps, resource inventory, and a change log of every violation and remediation. It's the audit package, assembled automatically from twelve months of continuous monitoring.

---

## What We're Not

We're not a SIEM. We're not trying to replace your security team's threat detection. We're not a checkbox tool that gives you a passing score while your infrastructure is actually on fire.

ComplianceWeave is infrastructure compliance infrastructure. It's the layer that answers "are we compliant?" continuously, so that question stops being a project and starts being a metric.

---

## Try It

The API is live. The Python client is open source.

- **⭐ Star the client library**: [github.com/complianceweave/complianceweave-python](https://github.com/complianceweave/complianceweave-python)
- **🚀 Start a free scan**: [complianceweave.io/start](https://complianceweave.io/start) — no credit card, scans your first framework free
- **📖 Read the docs**: [docs.complianceweave.io](https://docs.complianceweave.io)

If you've got questions about integrating into your pipeline, or want to talk through a specific framework requirement, drop a comment below. I read all of them.

---

*Built by engineers who have been on both sides of the audit table. We know what auditors actually look for, and we know what "three weeks of evidence collection" does to a team's morale.*

ComplianceWeave vs Vanta, Drata, Secureframe: Which Compliance Automation Tool Should You Use?

Ahmed Moussa — Sat, 08 Aug 2026 06:45:00 +0000

---
title: "Compliance Automation in 2024: ComplianceWeave vs. The Field (An Engineer's Honest Take)"
published: false
tags: [security, devops, compliance, tooling]
---

# Compliance Automation in 2024: ComplianceWeave vs. The Field (An Engineer's Honest Take)

There's a particular kind of dread that descends on engineering teams roughly six weeks before an audit. You know the feeling — suddenly everyone is a compliance expert, Slack fills with panicked messages about evidence screenshots, and someone is manually exporting CSV files from five different dashboards at 11pm.

Compliance automation tools exist to prevent exactly this. But the market has gotten crowded, and the differences between products are genuinely meaningful depending on *who you are* as a team. Let's cut through the noise.

---

## The Landscape at a Glance

Before diving deep, here's where the major players land across the dimensions that actually matter to engineering teams:

| Feature | ComplianceWeave | Vanta | Drata | Tugboat Logic |
|---|---|---|---|---|
| **Pricing model** | Usage-based + self-hosted tier | Per-employee SaaS | Per-employee SaaS | Per-framework SaaS |
| **Self-hosted option** | ✅ Yes | ❌ No | ❌ No | ❌ No |
| **API-first design** | ✅ Yes | Partial (read-only API) | Partial (read-only API) | ❌ No |
| **Python client** | ✅ Official | ❌ Community only | ❌ No | ❌ No |
| **Multi-framework single scan** | ✅ Yes | ✅ Yes | ✅ Yes | ⚠️ Add-on cost |
| **Frameworks covered** | SOC2, GDPR, HIPAA, ISO 27001 | SOC2, ISO 27001, HIPAA + others | SOC2, ISO 27001, HIPAA + others | SOC2, ISO 27001, GDPR |
| **Audit-ready report generation** | ✅ Automated | ✅ Automated | ✅ Automated | ⚠️ Semi-manual |
| **GUI dashboard** | ✅ Yes | ✅ Polished | ✅ Polished | ✅ Yes |
| **Continuous monitoring** | ✅ Yes | ✅ Yes | ✅ Yes | ⚠️ Scheduled scans |
| **Community/ecosystem** | Growing (GitHub-active) | Large, mature | Large, mature | Smaller |
| **Best for** | Dev-led teams, regulated infra | Sales-led compliance | Mid-market SaaS | SMBs, consultants |

---

## Deep Dive: What Each Tool Actually Does Well

### ComplianceWeave — Built for Teams Who Automate Everything

ComplianceWeave's core thesis is that compliance should live in your CI/CD pipeline, not in a browser tab someone checks quarterly. The API-first architecture isn't a checkbox feature — it means you can trigger compliance scans as part of a deployment gate, pull findings into your internal dashboards, or build custom alerting logic without waiting for a vendor to ship a native integration.

The Python client is genuinely useful. You can write something like:

python
from complianceweave import Scanner

scanner = Scanner(frameworks=["SOC2", "HIPAA"])
report = scanner.run(infrastructure="aws-prod")
report.export("audit-2024-q3.pdf")


...and have that living in a Makefile or GitHub Action. For teams that treat infrastructure as code, this fits naturally.

The self-hosted option is the other major differentiator. If you're in healthcare, financial services, or any regulated industry where sending infrastructure metadata to a third-party SaaS creates its own compliance problem — this matters enormously. Most competitors simply don't offer it.

**Where ComplianceWeave is weaker:** The ecosystem is younger. Integrations with niche SaaS tools (HR platforms, certain ticketing systems) are thinner than Vanta or Drata. The GUI, while functional, lacks the polished onboarding experience that makes Vanta so easy to hand to a non-technical compliance officer. Community resources — tutorials, pre-built policy templates, forum activity — are growing but not yet at the depth of the more established players.

---

### Vanta — The Safe Choice for Most Startups

Vanta built its reputation by making SOC2 feel achievable for early-stage companies. The onboarding experience is genuinely excellent — connect your AWS/GCP/Azure account, link your HR system, and within a day you have a live compliance dashboard. The GUI is polished enough that your Head of Sales can pull it up during a customer security review without embarrassment.

The partner network with auditing firms is a real advantage. Vanta has pre-negotiated relationships with auditors who know the platform, which can meaningfully compress your audit timeline.

**Where Vanta is weaker:** The per-employee pricing scales painfully for larger teams. The API is read-only, meaning you can *observe* your compliance posture programmatically but you can't *drive* it. If you want compliance embedded in your deployment pipeline rather than bolted on afterward, you'll be working around the tool rather than with it.

---

### Drata — Best-in-Class for Mid-Market

Drata targets the same space as Vanta but with more depth on the compliance workflow side — evidence collection, personnel training tracking, vendor risk management. For a 200-person company with a dedicated security team, Drata's additional process tooling earns its price.

**Where Drata is weaker:** Similar API limitations to Vanta. No self-hosting. Pricing is opaque until you talk to sales, which is a friction point for engineering-led evaluations.

---

### Tugboat Logic — The Consultant-Friendly Option

Tugboat Logic takes a more consultative approach, with strong support for teams that want human guidance through their first framework certification. If you're a small company doing SOC2 for the first time with no dedicated security staff, the guided approach has real value.

**Where Tugboat Logic is weaker:** Less automation depth, no continuous monitoring in the same sense as the others, and the multi-framework story requires add-on purchases.

---

## When to Use Each

**Choose ComplianceWeave if:**
- Your team treats infrastructure as code and wants compliance in the same mental model
- You're in a regulated industry where self-hosting isn't optional
- You need to cover multiple frameworks simultaneously without paying per-framework fees
- You want to build compliance checks into CI/CD pipelines, not just run them manually

**Choose Vanta if:**
- You're a startup doing your first SOC2 and want to move fast with minimal engineering overhead
- You have a non-technical compliance owner who needs a great GUI
- You value auditor relationships and want a streamlined path to certification

**Choose Drata if:**
- You're 100+ people and need compliance workflow management (training, vendor risk, personnel tracking) beyond just infrastructure scanning
- You have a dedicated security team that will get value from deeper process tooling

**Choose Tugboat Logic if:**
- You want guided, consultative support through your first certification
- Budget is a primary constraint and you're comfortable with more manual steps

---

## The Honest Bottom Line

None of these tools is universally best. Vanta and Drata win on polish, ecosystem maturity, and ease of onboarding for non-engineers. ComplianceWeave wins on programmability, self-hosting, and the ability to treat compliance as a first-class engineering concern rather than an annual fire drill.

The question worth asking your team: *Do you want a compliance dashboard, or do you want compliance infrastructure?* If it's the latter, ComplianceWeave's approach is architecturally different in ways that matter — not just a feature list difference, but a different philosophy about where compliance lives in your stack.

For teams that have already automated their security scanning, dependency auditing, and infrastructure provisioning, the idea of *not* being able to script your compliance checks will feel like a step backward. That's the team ComplianceWeave was built for.

---

*Pricing information and feature sets change frequently — verify current details directly with each vendor before making a purchasing decision. This comparison reflects publicly available information as of late 2024.*

How to Track Data Pipeline Dependencies Automatically with DataLineage

Ahmed Moussa — Sat, 08 Aug 2026 05:45:01 +0000

---
title: "Stop Playing Data Detective: Let DataLineage Trace Your Pipeline Dependencies"
published: false
tags: [dataengineering, python, tutorial, dbt]
---

# Stop Playing Data Detective: Let DataLineage Trace Your Pipeline Dependencies

Picture this: it's 3pm on a Friday. Someone renamed a column in a source table. By Monday morning, three dashboards are broken, one Airflow DAG is throwing cryptic errors, and your Spark job silently swallowed bad nulls for 48 hours before anyone noticed.

You've been there. We've all been there.

The problem isn't that pipelines break — it's that when they break, you're essentially an archaeologist. You dig through dbt docs, grep through DAG definitions, and DM the person who "probably knows" what consumes that table. It's manual, it's slow, and it scales terribly.

This tutorial walks you through DataLineage — a tool that automatically maps dependencies across your entire stack (dbt, Airflow, Spark, custom ETL) so that when something changes, you know *exactly* what's downstream before it bites you.

Let's build something real.

---

## What We're Building

By the end of this post, you'll have a Python script that:

1. Traces the full dependency graph for any dataset
2. Runs an impact analysis *before* a schema change ships
3. Handles errors gracefully so your CI pipeline doesn't silently lie to you

---

## Prerequisites

bash
pip install requests python-dotenv


Grab your API key from the DataLineage dashboard and store it safely:

bash

.env

DATALINEAGE_API_KEY=your_key_here
DATALINEAGE_BASE_URL=https://api.datalineage.io/v1


---

## Step 1: Trace a Dataset's Dependency Graph

The first thing you'll want to do is ask DataLineage: *"who touches this dataset, and what does it touch?"*

We hit `POST /lineage/trace` with the dataset identifier — this can be a table name, a dbt model, or a Spark output path.

python
import os
import requests
from dotenv import load_dotenv

load_dotenv()

BASE_URL = os.getenv("DATALINEAGE_BASE_URL")
HEADERS = {
"Authorization": f"Bearer {os.getenv('DATALINEAGE_API_KEY')}",
"Content-Type": "application/json",
}

def trace_lineage(dataset_id: str, depth: int = 3) -> dict:
"""
Initiate a lineage trace for a given dataset.

Args:
    dataset_id: The unique identifier for your dataset 
                (e.g., 'warehouse.analytics.user_events')
    depth: How many hops upstream/downstream to traverse

Returns:
    The trace job response containing a lineage ID
"""
payload = {
    "dataset_id": dataset_id,
    "depth": depth,
    "include_upstream": True,
    "include_downstream": True,
}

response = requests.post(
    f"{BASE_URL}/lineage/trace",
    json=payload,
    headers=HEADERS,
    timeout=30,
)

# Don't silently swallow HTTP errors — surface them immediately
response.raise_for_status()

return response.json()

--- Run it ---

if name == "main":
result = trace_lineage("warehouse.analytics.user_events")
print(f"Trace initiated. Lineage ID: {result['lineage_id']}")
print(f"Status: {result['status']}")


**Expected output:**

plaintext
Trace initiated. Lineage ID: lin_8f3a92bc
Status: processing


> **Best practice:** Store `lineage_id` — you'll need it to fetch results and run impact analysis. Consider logging it to your observability stack alongside any schema migration you run.

---

## Step 2: Fetch the Full Lineage Graph

Tracing is async (these graphs can be large), so we poll `GET /lineage/{id}` until the job completes. Here's a polling wrapper that won't hammer the API:

python
import time

def fetch_lineage(lineage_id: str, poll_interval: int = 2, max_wait: int = 60) -> dict:
"""
Poll for lineage results until complete or timeout.

Args:
    lineage_id: The ID returned from trace_lineage()
    poll_interval: Seconds between polls
    max_wait: Maximum seconds to wait before giving up

Returns:
    The complete lineage graph

Raises:
    TimeoutError: If the trace doesn't complete within max_wait
    RuntimeError: If the trace job failed on the server side
"""
elapsed = 0

while elapsed < max_wait:
    response = requests.get(
        f"{BASE_URL}/lineage/{lineage_id}",
        headers=HEADERS,
        timeout=30,
    )
    response.raise_for_status()
    data = response.json()

    if data["status"] == "complete":
        return data

    if data["status"] == "failed":
        raise RuntimeError(
            f"Lineage trace failed: {data.get('error', 'Unknown error')}"
        )

    print(f"  Still processing... ({elapsed}s elapsed)")
    time.sleep(poll_interval)
    elapsed += poll_interval

raise TimeoutError(f"Lineage trace {lineage_id} did not complete within {max_wait}s")

def summarize_lineage(lineage_data: dict) -> None:
"""Print a human-readable summary of the lineage graph."""
nodes = lineage_data.get("nodes", [])
edges = lineage_data.get("edges", [])

print(f"\n{'='*50}")
print(f"Lineage Graph: {lineage_data['dataset_id']}")
print(f"{'='*50}")
print(f"Total nodes: {len(nodes)}")
print(f"Total dependencies: {len(edges)}")

print("\nDownstream consumers:")
for node in nodes:
    if node["direction"] == "downstream":
        print(f"  [{node['tool']}] {node['name']}")

--- Run it ---

if name == "main":
trace = trace_lineage("warehouse.analytics.user_events")
lineage = fetch_lineage(trace["lineage_id"])
summarize_lineage(lineage)


**Expected output:**

plaintext
Still processing... (2s elapsed)

==================================================

Lineage Graph: warehouse.analytics.user_events

Total nodes: 14
Total dependencies: 19

Downstream consumers:
[dbt] model.analytics.weekly_retention
[dbt] model.analytics.revenue_attribution

[airflow] dag.nightly_user_export
[spark] job.ml_feature_pipeline
[custom_etl] sync.salesforce_enrichment


Fourteen nodes. Nineteen edges. From a single table. This is exactly why manual tracing doesn't scale.

---

## Step 3: Run Impact Analysis Before a Schema Change

This is where DataLineage earns its keep. Before you rename that column, you ask: *"what breaks?"*

`POST /lineage/impact` takes your proposed change and returns a severity-ranked list of affected consumers.

python
def analyze_impact(dataset_id: str, proposed_changes: list[dict]) -> dict:
"""
Assess the downstream impact of proposed schema changes.

Args:
    dataset_id: The dataset you're planning to modify
    proposed_changes: List of change descriptors
        Each dict should have: 'type' (rename/drop/type_change), 
        'field', and optionally 'new_name' or 'new_type'

Returns:
    Impact report with affected consumers ranked by severity
"""
payload = {
    "dataset_id": dataset_id,
    "changes": proposed_changes,
}

response = requests.post(
    f"{BASE_URL}/lineage/impact",
    json=payload,
    headers=HEADERS,
    timeout=30,
)
response.raise_for_status()
return response.json()

def print_impact_report(report: dict) -> None:
"""Display a color-coded impact report (terminal-friendly)."""
severity_icons = {
"critical": "🔴",
"high": "🟠",
"medium": "🟡",
"low": "🟢",
}

affected = report.get("affected_consumers", [])

print(f"\n{'='*50}")
print(f"Impact Analysis Report")
print(f"{'='*50}")
print(f"Total affected consumers: {len(affected)}")

if not affected:
    print("✅ No downstream consumers affected. Safe to proceed.")
    return

for consumer in sorted(affected, key=lambda x: x["severity_rank"]):
    icon = severity_icons.get(consumer["severity"], "⚪")
    print(f"\n{icon} {consumer['name']}")
    print(f"   Tool: {consumer['tool']}")
    print(f"   Affected fields: {', '.join(consumer['affected_fields'])}")
    print(f"   Owner: {consumer.get('owner', 'Unknown')}")
    print(f"   Reason: {consumer['reason']}")

--- Run it ---

if name == "main":
changes = [
{
"type": "rename",
"field": "user_id",
"new_name": "customer_id",
},
{
"type": "drop",
"field": "legacy_segment",
}
]

report = analyze_impact("warehouse.analytics.user_events", changes)
print_impact_report(report)


**Expected output:**

plaintext

Impact Analysis Report

Total affected consumers: 6

🔴 model.analytics.revenue_attribution
Tool: dbt
Affected fields: user_id, legacy_segment
Owner: data-team@company.com
Reason: Direct field reference in JOIN condition

🟠 dag.nightly_user_export
Tool: airflow
Affected fields: user_id
Owner: platform-team@company.com
Reason: Field used in SELECT and WHERE clause

🟡 job.ml_feature_pipeline
Tool: spark
Affected fields: legacy_segment
Owner: ml-team@company.com
Reason: Feature derived from legacy_segment


Now you have a contact list and a blast radius — before you've touched a single line of SQL.

---

## Putting It All Together

Here's a minimal CLI wrapper you can drop into your CI/CD pipeline as a pre-migration gate:

python
import sys

def pre_migration_check(dataset_id: str, changes: list[dict]) -> int:
"""
Returns exit code 0 if safe, 1 if critical impacts found.
Designed for use in CI pipelines.
"""
print(f"Running pre-migration check for: {dataset_id}")

report = analyze_impact(dataset_id, changes)
print_impact_report(report)

critical_count = sum(
    1 for c in report.get("affected_consumers", [])
    if c["severity"] == "critical"
)

if critical_count > 0:
    print(f"\n❌ {critical_count} critical impact(s) found. Resolve before proceeding.")
    return 1

print("\n✅ No critical impacts. Proceeding with migration.")
return 0

if name == "main":
exit_code = pre_migration_check(
dataset_id="warehouse.analytics.user_events",
changes=[{"type": "rename", "field": "user_id", "new_name": "customer_id"}]
)
sys.exit(exit_code)


Add this to your GitHub Actions workflow before any migration step runs, and you've turned a Friday-afternoon fire drill into a boring, automated checklist item.

---

## Key Takeaways

- **Trace first, change second.** Running `POST /lineage/trace` before any schema work costs you 30 seconds and can save you hours.
- **Impact analysis is a contract.** Share the report with consumer team owners before you ship — it's a conversation starter, not just a debugging tool.
- **Automate the gate.** The pre-migration check pattern above is trivially embeddable in any CI system. Make it a required check.
- **Depth matters.** The `depth` parameter on trace is worth tuning — a depth of 2 might miss a dbt model that feeds a Spark job that feeds a dashboard.

Data dependencies are invisible until they're not. DataLineage makes them visible on purpose, before something goes wrong.

Now go rename that column — you've earned it.

Introducing DataLineage -- Automated Data Pipeline Lineage Tracking

Ahmed Moussa — Sat, 08 Aug 2026 05:45:00 +0000

---
title: "Your Schema Changed. Congratulations, You've Just Inherited 47 Broken Pipelines."
published: false
tags: [dataengineering, python, opensource, dbt]
---

It's 9:47 AM on a Tuesday. A product manager renames a column in a source table — `user_id` becomes `customer_id`, totally reasonable, five minutes of work. By 2 PM, three dashboards are blank, an ML feature pipeline is silently feeding stale data to a model in production, and your Slack is a graveyard of confused @ mentions.

You spend the next four hours playing archaeologist in your own infrastructure, grepping through dbt YAML files, reading Airflow DAG source code, and cross-referencing a Confluence page that was last updated in 2022. You find the broken consumers. Eventually. Most of them.

This is not a tooling problem that requires better documentation or more disciplined engineers. It's a *visibility* problem. And visibility problems have a specific kind of solution.

---

## The Lineage Gap Nobody Talks About

Data lineage isn't a new idea. Every modern data warehouse has *some* version of it. But there's a gap between "lineage that exists in one tool" and "lineage that reflects how your actual pipelines work."

Your dbt models know about each other. Your Airflow DAGs know about their own tasks. Your Spark jobs know about their inputs and outputs — if someone remembered to log them. But none of these tools talk to each other, which means when a schema change ripples *across* tool boundaries, you're flying blind.

That's the gap [DataLineage](https://github.com/datalineage) was built to close.

---

## What DataLineage Actually Does

DataLineage runs as a lightweight service that connects to your existing stack — dbt, Airflow, Spark, and custom ETL — and builds a unified dependency graph across all of them. No manual configuration of relationships. No YAML files describing what you already know. It discovers the graph by reading what your tools already produce: dbt manifests, Airflow task metadata, Spark execution plans, and a thin SDK for anything custom.

The moment a schema changes, you query the graph and get back every downstream consumer, ranked by dependency depth, with the tool context preserved. You know *what* breaks, *where* it lives, and *how far* the blast radius extends.

Three things matter here:

**Auto-discovery, not auto-configuration.** You don't tell DataLineage about your pipelines. It reads your existing artifacts and infers the graph. Your dbt `manifest.json` already contains model dependencies. Your Airflow metadata database already contains task relationships. DataLineage connects the dots between them.

**Cross-tool lineage as a first-class feature.** A dbt model that feeds an Airflow DAG that triggers a Spark job that writes to a table consumed by three more dbt models — that full chain is visible as a single graph, not three disconnected fragments.

**Real-time impact analysis.** Before you rename that column, run an impact query. Get back the full list of affected consumers in under a second. Make the change knowing exactly what you're touching.

---

## Quick Start

bash
pip install datalineage

python
from datalineage import LineageClient

client = LineageClient(api_key="your_key")
impact = client.impact_analysis(table="raw.events", column="user_id")
print(impact.affected_nodes)


That's the core loop. Point it at a table and column, get back a structured list of everything downstream. The `affected_nodes` list includes node type (dbt model, Airflow task, Spark job), tool context, and dependency depth.

---

## A Real-World Scenario: The Column Rename That Didn't Hurt

Let's say you're migrating your user identifier scheme. The source table `raw.users` currently has `legacy_user_id` and you're standardizing on `user_id`. You want to deprecate `legacy_user_id` eventually, but first you need to know who's still depending on it.

python
from datalineage import LineageClient
from datalineage.models import ImpactReport

client = LineageClient(api_key="your_key")

Get full impact report before touching anything

report: ImpactReport = client.impact_analysis(
table="raw.users",
column="legacy_user_id",
depth=None # traverse the full graph, no depth limit
)

Group affected nodes by tool

by_tool = report.group_by_tool()

print(f"dbt models affected: {len(by_tool.get('dbt', []))}")
print(f"Airflow DAGs affected: {len(by_tool.get('airflow', []))}")
print(f"Spark jobs affected: {len(by_tool.get('spark', []))}")

Print the full dependency chain

for node in report.affected_nodes:
print(f"[depth={node.depth}] {node.tool}/{node.name} → {node.owner_team}")


Sample output:

plaintext
dbt models affected: 12
Airflow DAGs affected: 3
Spark jobs affected: 1

[depth=1] dbt/stg_users → analytics-eng
[depth=1] dbt/stg_user_events → analytics-eng
[depth=2] dbt/fct_user_sessions → analytics-eng
[depth=2] airflow/user_cohort_export → data-platform
[depth=3] dbt/rpt_weekly_active_users → analytics-eng
[depth=3] spark/user_feature_pipeline → ml-platform
...


Now you have a migration checklist. You know which teams to notify, in what order, and you can track remediation by querying the graph after each fix. When `affected_nodes` is empty, you're done.

This is the difference between a schema migration that takes a week of careful coordination and one that takes an afternoon.

---

## The API-First Design Decision

DataLineage exposes everything through a REST API with a Python client that wraps it. This wasn't an accident — it means the lineage graph is queryable from CI/CD pipelines, dbt hooks, Airflow sensors, or anywhere else you want to embed impact awareness.

A common pattern we've seen: add a DataLineage impact check as a pre-merge CI step for dbt PRs. If a PR modifies a model that has more than N downstream consumers, automatically request review from the teams that own those consumers. No more "I didn't know that model was used by the ML team."

python

In your CI pipeline

import sys
from datalineage import LineageClient

client = LineageClient(api_key=os.environ["DATALINEAGE_KEY"])
changed_models = get_changed_dbt_models() # your CI helper

for model in changed_models:
impact = client.impact_analysis(table=model)
if len(impact.affected_nodes) > 10:
print(f"⚠️ {model} affects {len(impact.affected_nodes)} downstream nodes")
print("Requesting cross-team review...")
sys.exit(1) # block merge, require manual approval


---

## Where We Are

DataLineage is in public beta. The Python client is stable. The connectors for dbt, Airflow, and Spark are production-ready. We're actively working on Flink support and a graph visualization UI (because sometimes you need to *see* the graph, not just query it).

The API is free during beta with generous rate limits. We're not going to surprise you with a pricing page before you've had a chance to actually use the thing.

---

## Try It

If you've ever spent an afternoon debugging a broken pipeline that broke because of a change you didn't know existed, DataLineage was built for you.

- ⭐ **[Star us on GitHub](https://github.com/datalineage/datalineage)** — it helps more data engineers find the project
- 🔑 **[Get a free API key](https://datalineage.io/signup)** — beta access, no credit card
- 📖 **[Read the docs](https://docs.datalineage.io)** — including connector setup guides for dbt, Airflow, and Spark

We're in the DataLineage Discord if you have questions, run into issues, or want to tell us what connectors to build next. See you there.

ComplianceWeave vs Vanta, Drata, Secureframe: Which Compliance Automation Tool Should You Use?

Ahmed Moussa — Sat, 01 Aug 2026 06:45:00 +0000

---
title: "Compliance Automation Tools in 2024: A Developer's Honest Field Guide"
published: false
tags: [security, devops, compliance, tooling]
---

# Compliance Automation Tools in 2024: A Developer's Honest Field Guide

Let me tell you something nobody puts in their compliance tool landing page: **audits are boring in a very specific way**. Not the thrilling chaos of a production outage — the slow, grinding boredom of hunting down a screenshot of an access log from eight months ago at 11pm before a SOC2 review.

Compliance automation tools exist to make that particular flavor of suffering optional. But they're not all solving the same problem for the same person. This guide is written for developers and platform engineers who are being handed compliance ownership (congratulations, probably) and need to make a real decision.

I've tried to be genuinely fair here. Where a competitor is better, I'll say so.

---

## The Contenders

We're looking at **ComplianceWeave** alongside three established players: **Vanta**, **Drata**, and **Tugboat Logic** (now part of OneTrust). These represent the mainstream of the compliance automation market.

---

## Feature Comparison

| Feature | ComplianceWeave | Vanta | Drata | Tugboat Logic |
|---|---|---|---|---|
| SOC2 | ✅ | ✅ | ✅ | ✅ |
| GDPR | ✅ | Partial | ✅ | ✅ |
| HIPAA | ✅ | ✅ | ✅ | ✅ |
| ISO 27001 | ✅ | ✅ | ✅ | ✅ |
| Multi-framework single scan | ✅ | ❌ | ❌ | ❌ |
| API-first architecture | ✅ | ❌ | ❌ | ❌ |
| Self-hosted option | ✅ | ❌ | ❌ | ❌ |
| Python client | ✅ | Limited | ❌ | ❌ |
| GUI dashboard | ✅ | ✅ | ✅ | ✅ |
| Auditor portal | In roadmap | ✅ | ✅ | ✅ |
| Automated evidence collection | ✅ | ✅ | ✅ | ✅ |
| Pricing transparency | ✅ | ❌ | ❌ | ❌ |

---

## Deep Dive: Each Tool

### Vanta

Vanta is the market leader for a reason. Their integrations catalog is enormous — if you use a SaaS tool, Vanta probably connects to it. Their auditor-facing portal is polished and well-regarded by the actual humans who will sign off on your SOC2. If you're a startup that needs to close enterprise deals and wants the path of least resistance, Vanta is a defensible choice.

**Where it struggles:** Vanta is a GUI-first product. Automating evidence collection *into* your CI/CD pipeline or programmatically querying your compliance posture isn't really the use case they're optimizing for. Pricing is also opaque — expect a sales call before you see a number. For teams managing compliance across multiple frameworks simultaneously, you'll be running separate workflows.

### Drata

Drata competes closely with Vanta and has arguably better UX in some areas. Their continuous monitoring story is strong, and they've invested heavily in policy templates. Drata also has a more developer-friendly feel than Vanta, though it's still fundamentally a SaaS dashboard product.

**Where it struggles:** Like Vanta, there's no self-hosting, no real API-first story, and no Python client worth mentioning. If your compliance data sovereignty is a concern (regulated industries, government contracts), running Drata means trusting a third party with your infrastructure metadata.

### Tugboat Logic / OneTrust

Tugboat Logic was acquired by OneTrust and has evolved into an enterprise-grade GRC platform. If you're a larger organization that needs to map compliance into a broader risk management program, the OneTrust ecosystem is genuinely powerful.

**Where it struggles:** The acquisition brought enterprise complexity. Onboarding is slower, pricing is enterprise-tier, and the developer experience is an afterthought. For a 20-person engineering team that just needs SOC2 Type II, this is likely overkill.

### ComplianceWeave

ComplianceWeave takes a meaningfully different architectural bet: **compliance as code, not compliance as a dashboard**. The API-first design means you can trigger scans from a GitHub Action, pull results into your internal tooling, or build custom alerting on top of compliance drift. The Python client is a genuine first-class citizen, not a thin wrapper.

The multi-framework single scan is the feature I find most compelling. Running one scan that simultaneously evaluates against SOC2, GDPR, and ISO 27001 isn't just a time saver — it changes how you think about compliance. Overlapping controls become visible. You stop doing redundant work.

The self-hosted option matters more than it might seem. For companies in healthcare, finance, or defense contracting, sending infrastructure metadata to a third-party SaaS is a non-starter. ComplianceWeave gives you the option to keep that data inside your perimeter.

**Where it struggles:** The auditor-facing portal is still on the roadmap, which is a real gap. Vanta and Drata have spent years building relationships with auditing firms, and their portals make the auditor's job easier. ComplianceWeave's auto-generated reports are solid, but if your auditor is used to logging into a Vanta portal, there may be friction. Community and ecosystem are also smaller — fewer pre-built integrations, fewer Stack Overflow answers, fewer consultants who know the tool.

---

## Pricing

Vanta, Drata, and Tugboat Logic all require sales conversations before you see pricing. Expect $10,000–$30,000/year for a typical startup depending on scope. ComplianceWeave publishes its pricing directly, which is itself a signal about who they're building for.

---

## When to Use Each

**Use Vanta if:** You're a Series A/B startup, you need SOC2 Type II to close enterprise deals in the next 6 months, you want the most auditor-recognized name on your report, and you're comfortable with a SaaS-only, GUI-first workflow.

**Use Drata if:** You want something similar to Vanta with slightly better UX, or you've had a specific recommendation from your auditing firm.

**Use Tugboat Logic / OneTrust if:** You're an enterprise with existing GRC programs, you need to map compliance into broader risk frameworks, and you have budget and implementation time to match.

**Use ComplianceWeave if:** You're an engineering-led team that wants compliance to live in your infrastructure-as-code workflow, not a separate SaaS dashboard. If you're managing multiple frameworks, need self-hosting for data sovereignty, or want to build compliance monitoring into your deployment pipeline programmatically — this is the tool designed for that use case. The smaller community is a real tradeoff, but the architectural philosophy is the right one for teams who treat compliance as an engineering problem.

---

## The Honest Summary

Most compliance tools are built for the *buyer* of compliance (the CISO, the VP of Sales who needs the SOC2 logo). ComplianceWeave is built for the *builder* of compliance — the engineer who has to actually implement and maintain it.

Neither approach is wrong. They're solving for different people in the same process.

Know which one you are, and pick accordingly.

---

*Have you used any of these tools in production? Drop your experience in the comments — especially if your auditor had opinions.*

How to Track Data Pipeline Dependencies Automatically with DataLineage

Ahmed Moussa — Sat, 01 Aug 2026 05:45:01 +0000

---
title: "Stop Playing Detective: Automate Data Lineage Tracing Across Your Entire Pipeline Stack"
published: false
tags: [dataengineering, python, tutorial, dataquality]
---

# Stop Playing Detective: Automate Data Lineage Tracing Across Your Entire Pipeline Stack

You know the scene. A Slack message arrives at 9 AM: *"The revenue dashboard looks wrong."* You open your laptop and begin the archaeological dig — tracing data backward through Airflow DAGs, dbt models, Spark jobs, and three custom ETL scripts written by someone who left the company in 2022. Two hours later, you've found the culprit: a schema change in an upstream table that nobody knew had seventeen downstream consumers.

This tutorial will show you how to use **DataLineage** to make that investigation take thirty seconds instead of two hours.

---

## What We're Building

By the end of this tutorial, you'll have a Python script that:

1. Registers your pipeline assets with DataLineage
2. Traces dependencies between them automatically
3. Runs an impact analysis before any schema change ships

We'll simulate a realistic stack: an Airflow-ingested raw table, a dbt model that transforms it, and a Spark job that feeds a downstream dashboard.

---

## Prerequisites

bash
pip install requests python-dotenv


Set your API key in a `.env` file:

conf
DATALINEAGE_API_KEY=your_key_here
DATALINEAGE_BASE_URL=https://api.datalineage.io/v1


---

## Step 1: Build a Reusable Client

Before touching the API, let's write a thin client wrapper. This keeps authentication and error handling in one place — a pattern your future self will thank you for.

python
import os
import requests
from dotenv import load_dotenv

load_dotenv()

class DataLineageClient:
def init(self):
self.base_url = os.getenv("DATALINEAGE_BASE_URL")
self.headers = {
"Authorization": f"Bearer {os.getenv('DATALINEAGE_API_KEY')}",
"Content-Type": "application/json",
}

def _handle_response(self, response: requests.Response) -> dict:
    try:
        response.raise_for_status()
        return response.json()
    except requests.exceptions.HTTPError as e:
        error_body = response.json() if response.content else {}
        raise RuntimeError(
            f"API error {response.status_code}: "
            f"{error_body.get('message', str(e))}"
        ) from e
    except requests.exceptions.ConnectionError:
        raise RuntimeError("Could not reach DataLineage API. Check your network and base URL.")

def trace(self, payload: dict) -> dict:
    response = requests.post(
        f"{self.base_url}/lineage/trace",
        json=payload,
        headers=self.headers,
        timeout=10,
    )
    return self._handle_response(response)

def get_lineage(self, lineage_id: str) -> dict:
    response = requests.get(
        f"{self.base_url}/lineage/{lineage_id}",
        headers=self.headers,
        timeout=10,
    )
    return self._handle_response(response)

def impact_analysis(self, payload: dict) -> dict:
    response = requests.post(
        f"{self.base_url}/lineage/impact",
        json=payload,
        headers=self.headers,
        timeout=10,
    )
    return self._handle_response(response)

client = DataLineageClient()
print("Client initialized successfully.")


**Expected output:**

plaintext
Client initialized successfully.


---

## Step 2: Register Your Pipeline Assets

Now let's tell DataLineage about our pipeline. We'll model three assets that represent a real-world ingestion → transformation → consumption chain.

python
pipeline_assets = [
{
"asset_id": "raw.orders",
"asset_type": "table",
"tool": "airflow",
"schema": {
"columns": ["order_id", "customer_id", "amount", "created_at"],
},
"description": "Raw orders ingested from the transactional DB via Airflow.",
},
{
"asset_id": "dbt.orders_daily",
"asset_type": "model",
"tool": "dbt",
"schema": {
"columns": ["order_date", "customer_id", "total_amount", "order_count"],
},
"description": "Daily order aggregation built on raw.orders.",
"upstream": ["raw.orders"],
},
{
"asset_id": "spark.revenue_summary",
"asset_type": "dataset",
"tool": "spark",
"schema": {
"columns": ["week", "region", "revenue"],
},
"description": "Weekly revenue rollup consumed by the executive dashboard.",
"upstream": ["dbt.orders_daily"],
},
]

trace_result = client.trace({
"assets": pipeline_assets,
"auto_discover": True,
})

lineage_id = trace_result["lineage_id"]
print(f"Lineage graph registered. ID: {lineage_id}")
print(f"Assets discovered: {trace_result['asset_count']}")
print(f"Edges mapped: {trace_result['edge_count']}")


**Expected output:**

plaintext
Lineage graph registered. ID: lin_8f3a92bc
Assets discovered: 3
Edges mapped: 2


The `auto_discover: true` flag tells DataLineage to also scan your connected integrations for any assets it can infer automatically — useful when you have undocumented dependencies lurking in legacy scripts.

---

## Step 3: Inspect the Full Lineage Graph

With the graph registered, let's pull it back and walk the dependency tree programmatically.

python
import json

lineage = client.get_lineage(lineage_id)

print("\n=== Lineage Graph ===\n")
for node in lineage["nodes"]:
upstream_list = ", ".join(node.get("upstream", [])) or "None (source)"
print(f" [{node['tool'].upper()}] {node['asset_id']}")
print(f" └─ Upstream: {upstream_list}")
print(f" └─ Columns: {', '.join(node['schema']['columns'])}\n")


**Expected output:**

plaintext
=== Lineage Graph ===

[AIRFLOW] raw.orders
└─ Upstream: None (source)
└─ Columns: order_id, customer_id, amount, created_at

[DBT] dbt.orders_daily
└─ Upstream: raw.orders
└─ Columns: order_date, customer_id, total_amount, order_count

[SPARK] spark.revenue_summary
└─ Upstream: dbt.orders_daily
└─ Columns: week, region, revenue


This is the map you'd normally reconstruct manually from scattered YAML files and Slack threads.

---

## Step 4: Run Impact Analysis Before a Schema Change

Here's where DataLineage earns its keep. Before you rename `amount` to `order_amount` in `raw.orders`, let's see exactly what breaks.

python
proposed_change = {
"asset_id": "raw.orders",
"change_type": "column_rename",
"details": {
"old_name": "amount",
"new_name": "order_amount",
},
"lineage_id": lineage_id,
}

impact = client.impact_analysis(proposed_change)

print("\n=== Impact Analysis Report ===\n")
print(f"Change: rename '{proposed_change['details']['old_name']}' "
f"→ '{proposed_change['details']['new_name']}' "
f"in {proposed_change['asset_id']}\n")

if not impact["affected_assets"]:
print("✅ No downstream consumers affected. Safe to proceed.")
else:
print(f"⚠️ {len(impact['affected_assets'])} downstream asset(s) affected:\n")
for asset in impact["affected_assets"]:
severity = asset.get("severity", "unknown").upper()
print(f" [{severity}] {asset['asset_id']} ({asset['tool']})")
print(f" Reason: {asset['reason']}")
print(f" Suggested fix: {asset.get('suggested_fix', 'Manual review required')}\n")


**Expected output:**

plaintext
=== Impact Analysis Report ===

Change: rename 'amount' → 'order_amount' in raw.orders

⚠️ 2 downstream asset(s) affected:

[HIGH] dbt.orders_daily (dbt)
Reason: References column 'amount' in aggregation logic.
Suggested fix: Update SUM(amount) to SUM(order_amount) in orders_daily.sql

[HIGH] spark.revenue_summary (spark)
Reason: Inherits 'amount' column through dbt.orders_daily transformation.
Suggested fix: Re-run dbt.orders_daily after fix; verify Spark job schema mapping.


Two assets. Flagged. With suggested fixes. In under a second.

---

## Step 5: Wire It Into Your CI/CD Pipeline

The real power comes from making this check automatic. Here's a minimal GitHub Actions step that blocks a PR if an impact analysis returns high-severity hits:

python

impact_gate.py — run this in CI before schema migrations ship

import sys

def check_impact_gate(lineage_id: str, asset_id: str, change: dict) -> None:
impact = client.impact_analysis({
"asset_id": asset_id,
"lineage_id": lineage_id,
**change,
})

high_severity = [
    a for a in impact.get("affected_assets", [])
    if a.get("severity") == "high"
]

if high_severity:
    print(f"❌ CI gate failed: {len(high_severity)} high-severity impact(s) detected.")
    for asset in high_severity:
        print(f"   - {asset['asset_id']}: {asset['reason']}")
    sys.exit(1)  # Fail the pipeline
else:
    print("✅ Impact gate passed. No high-severity consumers affected.")
    sys.exit(0)

check_impact_gate(
lineage_id=lineage_id,
asset_id="raw.orders",
change={"change_type": "column_rename", "details": {"old_name": "amount", "new_name": "order_amount"}},
)


Drop this into a CI step with `python impact_gate.py` and schema changes can never silently break downstream consumers again.

---

## What You've Built

In about fifty lines of Python, you've replaced a two-hour manual investigation with an automated, CI-integrated lineage system that:

- **Maps** your full dbt + Airflow + Spark dependency graph
- **Surfaces** every downstream consumer before a breaking change ships
- **Blocks** high-risk schema changes at the PR level

The archaeological digs are over. Your 9 AM Slack messages just got a lot less stressful.

---

*Have a multi-tool pipeline with custom ETL? The `POST /lineage/trace` endpoint accepts a `custom_etl` asset type — drop your questions in the comments.*

Introducing DataLineage -- Automated Data Pipeline Lineage Tracking

Ahmed Moussa — Sat, 01 Aug 2026 05:45:00 +0000

---
title: "Your Pipeline Is Lying to You (And You Won't Know Until It's Too Late)"
published: false
tags: [dataengineering, python, opensource, dbt]
---

It's 2:47 AM. Your phone is buzzing. The ML model that drives 30% of your company's revenue recommendations is returning garbage. You SSH in, grep through logs, and eventually trace it back to a "harmless" column rename that happened three weeks ago in a Postgres table upstream. Three weeks. The damage has been compounding silently ever since.

You didn't break anything. You just didn't *know* what you'd break.

This is the dirty secret of modern data stacks: we've gotten incredibly good at building pipelines and remarkably bad at understanding them. We have dbt for transformation, Airflow for orchestration, Spark for scale, and a graveyard of custom ETL scripts held together with cron jobs and optimism. Each tool knows its own world. None of them talk to each other.

So when something changes upstream, you're not doing impact analysis. You're doing archaeology.

---

## Introducing DataLineage

[DataLineage](https://github.com/datalineage/datalineage) automatically discovers and tracks data dependencies across your entire pipeline stack — dbt, Airflow, Spark, and custom ETL — without requiring you to annotate a single thing manually. When a schema changes, you see *every downstream consumer affected*, across tool boundaries, in real time.

Not a diagram you drew once and forgot to update. Not a wiki page from 2022. The actual, living dependency graph of your data.

---

## Quick Start

bash
pip install datalineage

python
from datalineage import LineageClient

client = LineageClient(api_key="your_key")
impact = client.get_impact("schema.orders", change="rename_column", column="user_id")
print(impact.affected_assets) # Every downstream model, job, and feature that just broke


That's it. No YAML config. No agents to deploy. No "enterprise onboarding call."

DataLineage connects to your existing tools through their APIs and metadata stores — dbt's manifest, Airflow's DAG definitions, Spark's query history — and stitches the dependency graph together automatically.

---

## The Real-World Problem: Schema Changes Don't Respect Tool Boundaries

Here's a scenario that's more common than anyone admits.

You have a dbt model `fct_orders` that reads from a raw table `raw.orders`. That model feeds an Airflow DAG that trains a churn prediction model. That DAG writes features to a feature store. Those features power a Spark batch job that scores your entire user base nightly.

A data analyst renames `customer_id` to `user_id` in `raw.orders` because the naming is inconsistent with everything else. Totally reasonable. They check: does dbt compile? Yes. Does the dbt test pass? Yes (they updated the model). Ship it.

What they couldn't see: the Airflow DAG that bypasses dbt and reads `raw.orders` directly. The Spark job that joins on `customer_id` from a different source. The feature pipeline that's been hardcoding that column name for eight months.

Let's see how DataLineage handles this:

python
from datalineage import LineageClient
from datalineage.models import SchemaChange, ChangeType

client = LineageClient(api_key="your_key")

Register the proposed change before you make it

change = SchemaChange(
asset="raw.orders",
change_type=ChangeType.RENAME_COLUMN,
before="customer_id",
after="user_id"
)

report = client.analyze_impact(change)

print(f"Direct consumers: {len(report.direct_consumers)}")
print(f"Total affected assets: {len(report.all_affected)}")
print(f"Critical path broken: {report.has_critical_path_impact}")

for asset in report.all_affected:
print(f" [{asset.tool}] {asset.name} — {asset.impact_type}")


Output:

plaintext
Direct consumers: 3
Total affected assets: 11
Critical path broken: True

[dbt] fct_orders — BREAKING (column reference)
[airflow] dag:churn_model_training — BREAKING (direct SQL query)
[airflow] dag:weekly_cohort_report — INDIRECT (via fct_orders)
[spark] job:nightly_user_scoring — BREAKING (join key)
[dbt] dim_customers — INDIRECT (via fct_orders)
... 6 more


The analyst sees this *before* merging. Not three weeks later at 2:47 AM.

---

## How Auto-Discovery Actually Works

DataLineage doesn't ask you to declare dependencies. It finds them.

For **dbt**, it parses your compiled `manifest.json` and extracts the full ref() and source() graph — including column-level lineage where dbt exposes it.

For **Airflow**, it hooks into the metadata database and parses DAG definitions to extract SQL queries, dataset references, and task dependencies.

For **Spark**, it processes query execution logs and the Spark SQL query plan to reconstruct what read from what.

For **custom ETL**, you get a lightweight decorator:

python
from datalineage import track

@track(reads=["raw.orders", "raw.products"], writes=["analytics.order_summary"])
def run_custom_etl():
# Your existing code, untouched
...


Everything flows into a unified graph. Cross-tool edges are first-class citizens, not afterthoughts.

---

## API-First, Because Your Stack Is Unique

Every data stack is a snowflake (the bad kind). DataLineage is built API-first so you can integrate it into your existing workflows rather than replacing them.

The Python client is the primary interface, but every operation is a REST call underneath. That means:

- **CI/CD integration**: Run impact analysis on every PR that touches schema definitions
- **Slack alerts**: Webhook when a breaking change is detected in a critical pipeline
- **dbt Cloud integration**: Trigger lineage refresh after every production job run
- **Custom dashboards**: Pull the graph data into whatever observability tool you already use

We're not trying to be your data catalog. We're trying to be the dependency engine that makes everything else smarter.

---

## What's Next

DataLineage is in public beta. What's working today:

- [x] dbt lineage (model + column level)
- [x] Airflow DAG parsing
- [x] Spark query log analysis
- [x] Real-time impact analysis API
- [x] Python client

On the roadmap:

- [ ] Fivetran + Airbyte source tracking
- [ ] GitHub PR integration (comment impact reports automatically)
- [ ] Column-level lineage across all tools
- [ ] Slack + PagerDuty alerting

---

## Try It

If you've ever spent more than an hour tracing a broken pipeline back to its source, DataLineage is for you.

**⭐ [Star us on GitHub](https://github.com/datalineage/datalineage)** — it genuinely helps us understand who's interested and what to build next.

**🚀 [Try the API](https://datalineage.io/signup)** — free tier includes up to 500 assets and unlimited impact analyses.

**💬 [Join our Discord](https://discord.gg/datalineage)** — we're actively building with early users. Your pipeline's weird edge case is exactly what we want to hear about.

The 2:47 AM call is optional. The broken pipeline is not. Let's fix the second one.

ComplianceWeave vs Vanta, Drata, Secureframe: Which Compliance Automation Tool Should You Use?

Ahmed Moussa — Sat, 25 Jul 2026 06:45:00 +0000

---
title: "Compliance Automation Tools in 2024: A Developer's Honest Breakdown"
published: false
tags: [security, devops, compliance, tooling]
---

# Compliance Automation Tools in 2024: A Developer's Honest Breakdown

Let me tell you something nobody in a vendor-written comparison will admit upfront: **most compliance automation tools were built for auditors, not engineers.** They assume you want to click through dashboards, export PDFs manually, and schedule calls with "compliance success managers."

That assumption shapes everything — the UX, the pricing model, the integration story.

This post tries to cut through that. I've looked at ComplianceWeave alongside the established players (Vanta, Drata, and Tugboat Logic) with one lens: *what does this actually feel like to operate as a developer or DevOps engineer responsible for compliance?*

Fair warning: I'll tell you when the alternatives are genuinely better. Spoiler — sometimes they are.

---

## The Landscape at a Glance

| | **ComplianceWeave** | **Vanta** | **Drata** | **Tugboat Logic** |
|---|---|---|---|---|
| **Primary User** | Developer / DevOps | GRC / Founder | GRC / Founder | GRC Manager |
| **API Access** | First-class (full REST + Python client) | Limited (read-mostly) | Limited | Minimal |
| **Self-Hosted** | ✅ Yes | ❌ No | ❌ No | ❌ No |
| **Frameworks** | SOC2, GDPR, HIPAA, ISO 27001 (one scan) | SOC2, HIPAA, ISO 27001 (separate) | SOC2, HIPAA, ISO 27001 (separate) | SOC2, ISO 27001 |
| **Multi-Framework Scan** | ✅ Single scan, mapped across | ❌ Per-framework runs | ❌ Per-framework runs | ❌ Per-framework runs |
| **Audit-Ready Reports** | Auto-generated | Semi-automated | Semi-automated | Manual assembly |
| **Pricing Model** | Usage-based / self-hosted | Per-seat + integrations | Per-seat + integrations | Per-user |
| **Community / OSS** | Growing (GitHub presence) | Large (partner ecosystem) | Large (partner ecosystem) | Small |
| **Ease of Setup** | Moderate (requires config) | Very easy | Very easy | Moderate |
| **Best For** | Infra-heavy teams, CI/CD integration | Early-stage startups | Mid-market SaaS | Enterprise GRC |

---

## Feature Deep Dive

### ComplianceWeave

The architectural bet here is interesting: **API-first** means compliance checks can live inside your existing pipelines. You can trigger a SOC2 evidence collection run from a GitHub Action. You can query your current control status from a Slack bot. You can diff your compliance posture between last Tuesday and today.

The multi-framework scan is genuinely useful — if you're pursuing SOC2 *and* GDPR simultaneously (common for companies with EU customers), you're not running two separate tooling stacks and reconciling overlapping controls by hand. ComplianceWeave maps shared controls once.

The self-hosted option matters for two audiences: companies with strict data residency requirements, and security-conscious teams who simply don't want their infrastructure evidence sitting in a third-party SaaS. That's a real concern, not paranoia.

**Where it falls short:** The setup experience requires more configuration than the SaaS alternatives. If you want something running in 30 minutes with zero engineering involvement, this isn't it. The community is also younger — fewer pre-built integrations and less Stack Overflow-style tribal knowledge floating around.

---

### Vanta

Vanta's strength is **time to value for non-technical founders.** Connect your AWS account, your GitHub, your HR system — and you have a live compliance dashboard in under an hour. The partner network (auditors who know Vanta's export format) is genuinely valuable when you're approaching your first SOC2 audit.

The tradeoff: you're in a walled garden. The API is read-mostly and not designed for automation. You can't easily pull Vanta data into your own dashboards or trigger checks programmatically. For a developer who wants compliance as code, it feels like a detour.

**Vanta is the right call if:** You're a non-technical founder or early-stage startup that needs SOC2 quickly and doesn't have engineering cycles to spare on tooling configuration.

---

### Drata

Drata sits in similar territory to Vanta but with slightly more polish on the continuous monitoring story and a stronger mid-market positioning. The evidence collection automation is solid, and their control mapping is well-maintained.

Like Vanta, the API surface is limited. Drata is also GUI-first in a way that feels intentional — the product is designed around the assumption that a compliance manager, not an engineer, is the daily user.

**Drata is the right call if:** You're a mid-market SaaS company with a dedicated GRC or security person who will own the tool day-to-day.

---

### Tugboat Logic

Tugboat Logic (now part of OneTrust) leans hardest into the enterprise GRC buyer. The policy management features are more mature, and the workflow tooling for getting controls reviewed and approved across large organizations is better than the others listed here.

The developer experience is the weakest of the four. This is a tool built for compliance teams, full stop.

**Tugboat Logic is the right call if:** You're in a larger organization where compliance is owned by a dedicated GRC team and the engineering team's role is answering requests, not running the tooling.

---

## Pricing: What We Know

Pricing in this space is famously opaque. Here's the honest summary:

- **Vanta and Drata** both use per-seat + integration pricing that scales quickly. Expect $10K–$30K/year for a typical Series A startup, more as you add frameworks.
- **Tugboat Logic** is enterprise-priced and typically requires a sales conversation.
- **ComplianceWeave** uses usage-based pricing with a self-hosted tier that can dramatically reduce cost for teams with the infrastructure to run it.

If cost is a primary driver and you have engineering resources, ComplianceWeave's self-hosted path is worth evaluating seriously.

---

## Community and Ecosystem

This is where the established players have a real edge. Vanta and Drata have large auditor partner networks, active user communities, and years of accumulated documentation and workarounds. When you hit an edge case at 11pm before your audit, someone has probably already solved it.

ComplianceWeave's community is smaller but growing, and the API-first design means integrations are more composable — you're not waiting for a vendor to build the Datadog connector.

---

## When to Use Each

**Use ComplianceWeave when:**
- You want compliance integrated into CI/CD pipelines, not managed separately
- You're pursuing multiple frameworks simultaneously
- Data residency or security requirements make SaaS evidence storage a concern
- You have engineering resources and want to treat compliance as infrastructure

**Use Vanta when:**
- You need SOC2 quickly with minimal engineering involvement
- You're pre-Series A and speed matters more than flexibility
- You'll be working closely with a Vanta-familiar auditor

**Use Drata when:**
- You have a dedicated GRC or security hire who will own the tool
- You want strong continuous monitoring with a polished GUI
- You're mid-market SaaS with standard cloud infrastructure

**Use Tugboat Logic when:**
- Compliance is owned by a dedicated enterprise GRC team
- Policy management and approval workflows matter as much as technical controls
- You're already in the OneTrust ecosystem

---

## The Honest Conclusion

There's no universally best tool here. The right answer depends almost entirely on *who in your organization owns compliance* and *how tightly you want it integrated with your engineering workflow.*

The interesting thing about ComplianceWeave is that it's making a different bet than the incumbents — that compliance should be code, not a SaaS dashboard your engineers file tickets into. If that bet resonates with how your team works, it's worth a serious look. If it doesn't, Vanta and Drata have earned their market position for good reasons.

Pick the tool that fits your team's actual workflow, not the one with the best demo.

DataLineage vs OpenLineage, Marquez, DataHub: Which Data Lineage Tool Should You Use?

Ahmed Moussa — Sat, 25 Jul 2026 05:45:04 +0000

---
title: "Data Lineage Tools in 2024: An Honest Field Guide (With Scars to Prove It)"
published: false
tags: [dataengineering, dataquality, tooling, opensource]
---

# Data Lineage Tools in 2024: An Honest Field Guide (With Scars to Prove It)

There's a specific kind of pain that data engineers know intimately: a schema changes upstream, something breaks downstream, and you spend three hours playing archaeological detective through Slack threads and stale Confluence docs to understand *why*.

Data lineage tools exist to prevent exactly that. But the space is crowded, the marketing is loud, and "automatic" means something different on every vendor's website. This post is my attempt at a genuinely fair comparison — the kind I wish existed when I was evaluating these tools.

I'll cover **DataLineage**, **OpenMetadata**, **Marquez**, and **Atlan**. Four tools with meaningfully different philosophies. Let's get into it.

---

## The Core Problem Each Tool Is Trying to Solve

Before the table, a quick framing. These tools aren't identical products competing for the same user — they have distinct centers of gravity:

- **DataLineage** is laser-focused on *pipeline impact analysis* — specifically, "what breaks if I change this?"
- **OpenMetadata** is a full data catalog that includes lineage as one feature among many
- **Marquez** is an open standard and metadata server built around the OpenLineage spec
- **Atlan** is an enterprise data catalog with strong governance features and lineage visualization

Understanding this helps you pick the right tool rather than the most-hyped one.

---

## Comparison Table

| Dimension | DataLineage | OpenMetadata | Marquez | Atlan |
|---|---|---|---|---|
| **Setup complexity** | Low (zero-config discovery) | Medium (connectors + config) | Medium (instrumentation required) | High (enterprise onboarding) |
| **Auto-discovery** | ✅ Yes | ⚠️ Partial (connector-dependent) | ❌ Requires OpenLineage instrumentation | ⚠️ Partial (crawler-based) |
| **dbt support** | ✅ Native | ✅ Native | ✅ Via OpenLineage | ✅ Native |
| **Airflow support** | ✅ Native | ✅ Native | ✅ Via provider | ✅ Native |
| **Spark support** | ✅ Native | ⚠️ Limited | ✅ Via listener | ⚠️ Limited |
| **Custom ETL support** | ✅ Yes | ❌ Manual only | ⚠️ Requires SDK | ❌ Manual only |
| **Impact analysis API** | ✅ Dedicated endpoint | ❌ | ❌ | ⚠️ UI only |
| **Data catalog features** | ❌ Not the focus | ✅ Full catalog | ❌ Lineage only | ✅ Full catalog |
| **Governance / RBAC** | ⚠️ Basic | ✅ Strong | ❌ | ✅ Enterprise-grade |
| **License** | BSL 1.1 (free non-prod) | Apache 2.0 | Apache 2.0 | Proprietary SaaS |
| **Self-hosted option** | ✅ | ✅ | ✅ | ❌ |
| **Community size** | Small / growing | Large | Medium | Vendor-driven |
| **Pricing (prod)** | Paid (contact) | Free (self-host) | Free (self-host) | Expensive (enterprise) |

---

## Honest Takes, Tool by Tool

### DataLineage

**What it does well:** The zero-config auto-discovery claim holds up better than most. Rather than requiring you to annotate your pipelines or instrument your code, DataLineage parses your existing dbt manifests, Airflow DAG structure, and Spark query plans to build a dependency graph automatically. For teams with messy, organically grown pipelines — which is most teams — this is genuinely valuable.

The impact analysis endpoint is the feature that makes engineers' eyes light up. You can query it programmatically: "if column `user_id` in `raw.events` changes type, what downstream models and dashboards are affected?" This is CI/CD-friendly in a way that most competitors aren't.

**Where it falls short:** DataLineage is not a data catalog. If you need column-level business glossary, data quality scoring, or access governance, you'll need another tool. The community is also small — don't expect a rich ecosystem of plugins or a busy Slack community yet. The BSL 1.1 license is worth reading carefully; it's free for non-production use, but production deployments require a commercial license, which puts it in a different budget conversation than Apache-licensed alternatives.

**Best for:** Data engineering teams who want fast time-to-value on lineage specifically, and who are running heterogeneous stacks.

---

### OpenMetadata

**What it does well:** OpenMetadata is probably the most complete open-source data catalog available right now. Lineage is one well-implemented feature inside a broader platform that includes data quality, profiling, glossary, and RBAC. If you need a single pane of glass for your data assets, it's a serious contender. The community is active and the connector library is extensive.

**Where it falls short:** Setup is not trivial. You're deploying a multi-service application, and getting lineage working well requires configuring connectors for each source. Auto-discovery is connector-dependent — if your ETL is custom Python scripts, you're writing manual lineage. There's no dedicated impact analysis API; lineage is primarily a visual exploration tool.

**Best for:** Teams who need a full data catalog and are willing to invest in setup. Especially strong if you have a dedicated data platform team.

---

### Marquez

**What it does well:** Marquez is the reference implementation of the OpenLineage specification — an open standard for lineage metadata. If you care about vendor neutrality and interoperability, Marquez is philosophically appealing. It's lightweight, Apache-licensed, and the OpenLineage spec has solid adoption across the ecosystem.

**Where it falls short:** Marquez requires instrumentation. Your pipelines need to emit OpenLineage events, which means adding providers or listeners to Airflow, Spark, etc. For legacy or custom pipelines, this is real engineering work. The UI is functional but minimal. Think of it as infrastructure, not a product.

**Best for:** Teams building a custom lineage solution who want a standards-based backend. Also good if you're already heavily invested in the OpenLineage ecosystem.

---

### Atlan

**What it does well:** Atlan is the enterprise choice — polished UI, strong governance features, Slack/Jira integrations, and a sales team that will hold your hand through onboarding. Lineage visualization is excellent. If budget isn't a constraint and you need executive-friendly dashboards, Atlan delivers.

**Where it falls short:** It's expensive, it's SaaS-only (no self-hosting), and the pricing is opaque. Auto-discovery is crawler-based and less comprehensive than DataLineage's approach for complex pipeline stacks. Impact analysis is UI-driven, not API-driven, which limits automation use cases.

**Best for:** Enterprise data teams with governance requirements, compliance needs, and budget to match.

---

## When to Use Each: The Decision Tree

**Choose DataLineage if:**
- You need lineage working *fast* without a large setup investment
- Your stack is heterogeneous (dbt + Airflow + Spark + custom scripts)
- You want to integrate impact analysis into your CI/CD pipeline programmatically
- You're a startup or scale-up that doesn't need a full catalog yet

**Choose OpenMetadata if:**
- You need a full data catalog, not just lineage
- You have a dedicated data platform team to manage it
- Open-source and self-hosted is a hard requirement
- Community and plugin ecosystem matter to you

**Choose Marquez if:**
- You're building a custom metadata platform and want a standards-based foundation
- Your team already uses OpenLineage-compatible tools
- You want maximum flexibility and don't need a polished UI

**Choose Atlan if:**
- You're in an enterprise environment with compliance and governance requirements
- Budget is not the primary constraint
- You need a tool your non-technical stakeholders can actually use

---

## Final Thought

The "best" data lineage tool is the one your team will actually maintain. A zero-config tool with 80% coverage that's running in production beats a perfectly configured tool that's been in "evaluation" for six months.

Start with your most painful problem — broken pipelines, schema drift, compliance audits — and work backward to the tool that solves it most directly. All four tools here solve real problems. None of them solves all of them.

---

*Have experience with any of these tools? Corrections, additions, or war stories welcome in the comments.*

How to Track Data Pipeline Dependencies Automatically with DataLineage

Ahmed Moussa — Sat, 25 Jul 2026 05:45:03 +0000

---
title: "Your Pipeline is Lying to You (And Here's How to Catch It)"
published: false
tags: [dataengineering, python, tutorial, dataquality]
---

# Your Pipeline is Lying to You (And Here's How to Catch It)

Last Tuesday at 2 AM, a Slack message woke up a data engineer somewhere. A dashboard was broken. The on-call scrambled through dbt models, Airflow DAGs, and three Spark jobs trying to answer one question: *who touched what?*

Four hours later, they found it. A column rename. Innocent. Undocumented.

This tutorial is about never having that 2 AM call again.

We'll use **DataLineage** — a tool that automatically maps dependencies across your entire pipeline stack — to build a system that *knows* what breaks before you deploy the thing that breaks it.

---

## What We're Actually Building

By the end of this post, you'll have a Python workflow that:

1. Traces a schema change through your entire pipeline
2. Identifies every downstream consumer affected
3. Runs an impact assessment *before* you merge that PR

No more archaeological digs through YAML files at midnight.

---

## Prerequisites

bash
pip install requests python-dotenv


You'll need a DataLineage API key. Set it in a `.env` file:

properties
DATALINEAGE_API_KEY=your_key_here
DATALINEAGE_BASE_URL=https://api.datalineage.io/v1


Let's write a small client wrapper we'll reuse throughout:

python
import os
import requests
from dotenv import load_dotenv

load_dotenv()

def _handle_response(self, response: requests.Response) -> dict:
    """Centralized error handling — because silent failures are the worst kind."""
    try:
        response.raise_for_status()
    except requests.exceptions.HTTPError as e:
        error_body = response.json().get("error", "No error detail provided")
        raise RuntimeError(
            f"DataLineage API error [{response.status_code}]: {error_body}"
        ) from e
    return response.json()


> **Best practice:** Always surface the API's error message, not just the status code. "422" tells you nothing. "Column 'user_id' not found in registered schema" tells you everything.

---

## Step 1: Trace a Node Through the Pipeline

Every table, model, or dataset in DataLineage is a **node**. When you want to understand a node's full family tree — what it depends on, what depends on it — you call `POST /lineage/trace`.

Let's trace our `orders` dbt model:

python
def trace_lineage(client: DataLineageClient, node_id: str, depth: int = 3) -> dict:
"""
Trace upstream and downstream dependencies for a given node.

Args:
    node_id: The unique identifier for your table/model/dataset
    depth:   How many hops to traverse (default 3 is usually enough)
"""
payload = {
    "node_id": node_id,
    "depth": depth,
    "include_metadata": True  # Grab schema info, owner, last modified
}

response = client._handle_response(
    requests.post(
        f"{client.base_url}/lineage/trace",
        json=payload,
        headers=client.headers
    )
)
return response

Run it

client = DataLineageClient()
lineage = trace_lineage(client, node_id="dbt.production.orders")

print(f"Node: {lineage['node']['name']}")
print(f"Upstream dependencies: {len(lineage['upstream'])}")
print(f"Downstream consumers: {len(lineage['downstream'])}")


**Expected output:**

plaintext
Node: dbt.production.orders
Upstream dependencies: 4
Downstream consumers: 11


Eleven downstream consumers. That's eleven things that could break if someone renames a column. Let's see exactly what they are.

---

## Step 2: Retrieve and Parse the Full Lineage Graph

The trace job runs asynchronously for large graphs. Use the returned `lineage_id` to fetch results:

python
import time

def get_lineage_result(client: DataLineageClient, lineage_id: str, timeout: int = 30) -> dict:
"""
Poll for lineage results with a timeout.
Large graphs can take a few seconds to compute.
"""
deadline = time.time() + timeout

while time.time() < deadline:
    response = client._handle_response(
        requests.get(
            f"{client.base_url}/lineage/{lineage_id}",
            headers=client.headers
        )
    )

    status = response.get("status")

    if status == "complete":
        return response["graph"]
    elif status == "failed":
        raise RuntimeError(f"Lineage computation failed: {response.get('reason')}")

    # Still processing — wait and retry
    time.sleep(2)

raise TimeoutError(f"Lineage graph not ready after {timeout}s. Try increasing timeout.")

def print_dependency_tree(graph: dict) -> None:
"""Human-readable summary of what we found."""
print("\n📊 LINEAGE SUMMARY")
print("=" * 50)

print("\n⬆️  UPSTREAM (this node depends on):")
for node in graph.get("upstream", []):
    tool = node["metadata"].get("tool", "unknown")
    print(f"  └─ [{tool.upper()}] {node['name']}")

print("\n⬇️  DOWNSTREAM (depends on this node):")
for node in graph.get("downstream", []):
    tool = node["metadata"].get("tool", "unknown")
    owner = node["metadata"].get("owner", "unassigned")
    print(f"  └─ [{tool.upper()}] {node['name']}  (owner: {owner})")

Fetch and display

lineage_id = lineage["lineage_id"]
graph = get_lineage_result(client, lineage_id)
print_dependency_tree(graph)


**Expected output:**

plaintext

📊 LINEAGE SUMMARY

⬆️ UPSTREAM (this node depends on):
└─ [DBT] dbt.production.raw_orders
└─ [DBT] dbt.production.customers
└─ [SPARK] spark.etl.order_enrichment
└─ [AIRFLOW] airflow.dag.payment_sync

⬇️ DOWNSTREAM (depends on this node):
└─ [DBT] dbt.production.revenue_monthly (owner: analytics@company.com)
└─ [SPARK] spark.ml.churn_features (owner: ml-team@company.com)
└─ [DBT] dbt.production.executive_kpis (owner: data@company.com)
... and 8 more


Now you can see the blast radius. The ML team's churn model feeds off `orders`. So does the executive KPI dashboard. One column rename, two very unhappy teams.

---

## Step 3: Run an Impact Assessment Before You Merge

This is the step that turns DataLineage from a debugging tool into a **prevention** tool.

Before any schema change, call `POST /lineage/impact` with the proposed change:

python
def assess_schema_change_impact(
client: DataLineageClient,
node_id: str,
proposed_changes: list[dict]
) -> dict:
"""
Simulate a schema change and identify every affected downstream consumer.

proposed_changes format:
    [{"type": "rename_column", "from": "old_name", "to": "new_name"}]
    [{"type": "drop_column",   "name": "column_name"}]
    [{"type": "change_type",   "column": "col", "from": "INT", "to": "STRING"}]
"""
payload = {
    "node_id": node_id,
    "changes": proposed_changes,
    "severity_threshold": "low"  # Catch even minor breakages
}

response = client._handle_response(
    requests.post(
        f"{client.base_url}/lineage/impact",
        json=payload,
        headers=client.headers
    )
)
return response

def format_impact_report(impact: dict) -> None:
"""Print a CI-friendly impact report."""
affected = impact.get("affected_nodes", [])
severity_counts = {"high": 0, "medium": 0, "low": 0}

print("\n🔍 IMPACT ASSESSMENT REPORT")
print("=" * 50)
print(f"Proposed change: {impact['change_summary']}")
print(f"Total affected nodes: {len(affected)}\n")

for node in affected:
    severity = node["impact_severity"]
    severity_counts[severity] += 1
    icon = {"high": "🔴", "medium": "🟡", "low": "🟢"}.get(severity, "⚪")

    print(f"{icon} [{severity.upper()}] {node['name']}")
    print(f"   Reason: {node['break_reason']}")
    print(f"   Owner:  {node['metadata'].get('owner', 'unknown')}\n")

print("SEVERITY SUMMARY:")
for level, count in severity_counts.items():
    print(f"  {level.capitalize()}: {count}")

# Exit non-zero if high-severity issues exist — useful in CI pipelines
if severity_counts["high"] > 0:
    print("\n❌ High-severity impacts detected. Review before merging.")
    return False

print("\n✅ No high-severity impacts. Proceed with caution.")
return True

Simulate renaming 'customer_id' to 'user_id' in orders

impact = assess_schema_change_impact(
client,
node_id="dbt.production.orders",
proposed_changes=[
{"type": "rename_column", "from": "customer_id", "to": "user_id"}
]
)

safe_to_merge = format_impact_report(impact)


**Expected output:**

plaintext

🔍 IMPACT ASSESSMENT REPORT

Proposed change: Rename column 'customer_id' → 'user_id' in dbt.production.orders
Total affected nodes: 7

🔴 [HIGH] spark.ml.churn_features
Reason: Column 'customer_id' referenced in JOIN condition (line 34)
Owner: ml-team@company.com

🔴 [HIGH] dbt.production.executive_kpis
Reason: Column 'customer_id' used in GROUP BY clause
Owner: data@company.com

🟡 [MEDIUM] airflow.dag.weekly_report
Reason: Column referenced in downstream SELECT *
Owner: analytics@company.com

SEVERITY SUMMARY:
High: 2
Medium: 1
Low: 4

❌ High-severity impacts detected. Review before merging.


---

## Putting It All Together: A Pre-Merge Check Script

Here's a complete script you can drop into your CI pipeline:

python

!/usr/bin/env python3

"""
pre_merge_lineage_check.py

Usage: python pre_merge_lineage_check.py --node dbt.production.orders \
--change rename_column --from customer_id --to user_id
"""

import argparse
import sys

def main():
parser = argparse.ArgumentParser(description="DataLineage pre-merge impact check")
parser.add_argument("--node", required=True)
parser.add_argument("--change", required=True, choices=["rename_column", "drop_column", "change_type"])
parser.add_argument("--from", dest="from_name", required=True)
parser.add_argument("--to", dest="to_name")
args = parser.parse_args()

client = DataLineageClient()

change = {"type": args.change, "from": args.from_name}
if args.to_name:
    change["to"] = args.to_name

print(f"🔎 Assessing impact of {args.change} on {args.node}...")

impact = assess_schema_change_impact(client, args.node, [change])
safe = format_impact_report(impact)

sys.exit(0 if safe else 1)  # Non-zero exit blocks the merge in CI

if name == "main":
main()


Add this to your GitHub Actions workflow and schema changes *cannot* merge without an impact review.

---

## The Bigger Picture

Manual dependency tracing is archaeology. You dig through layers of YAML, Spark configs, and Airflow DAGs hoping you've found everything — knowing you probably haven't.

DataLineage makes the invisible visible. Your pipeline has a graph structure whether you document it or not. This just makes it queryable.

The 2 AM Slack message? It becomes a PR comment: *"This rename affects 7 nodes. Here are the owners. Here's why it breaks."*

Ship with confidence. Your on-call engineer will thank you.

---

*Have a horror story about undocumented pipeline dependencies? Drop it in the comments — misery loves company.*

ComplianceWeave vs Vanta, Drata, Secureframe: Which Compliance Automation Tool Should You Use?

Ahmed Moussa — Sat, 18 Jul 2026 06:45:00 +0000

---
title: "Compliance Automation in 2024: ComplianceWeave vs. The Field (An Engineer's Honest Take)"
published: false
tags: [security, devops, compliance, tooling]
---

# Compliance Automation in 2024: ComplianceWeave vs. The Field (An Engineer's Honest Take)

I want to start with a confession: I used to spend three weeks every year in what I privately called "evidence purgatory" — hunting down screenshots, exporting logs, and pestering teammates for policy documents. Compliance automation tools promised to end that. Some delivered. Some traded one kind of pain for another.

This post is my attempt to give you an honest map of the landscape, including where ComplianceWeave genuinely wins, where it doesn't, and how to decide which tool fits your situation.

---

## Why This Space Is Harder Than It Looks

Compliance automation sounds simple: scan your infrastructure, check it against a framework, produce a report. The complexity hides in the edges. Which frameworks? Whose interpretation of a control? Can your security team *read* the output, or does it require a consultant to decode it? Can your *developers* actually integrate it into a workflow they already use?

Different tools have made different bets on these questions. Let's look at how they stack up.

---

## The Contenders

For this comparison, I'm evaluating **ComplianceWeave** alongside three representative categories of alternatives you'll encounter in the wild: **enterprise GRC platforms** (think legacy, GUI-heavy, consultant-friendly), **modern SaaS compliance tools** (slick dashboards, venture-backed, GUI-first), and **open-source DIY frameworks** (maximum control, maximum assembly required).

---

## Feature Comparison

| Feature | ComplianceWeave | Enterprise GRC | Modern SaaS | Open-Source DIY |
|---|---|---|---|---|
| SOC 2 support | ✅ | ✅ | ✅ | ⚠️ Partial |
| GDPR support | ✅ | ✅ | ⚠️ Partial | ⚠️ Partial |
| HIPAA support | ✅ | ✅ | ⚠️ Varies | ⚠️ Partial |
| ISO 27001 support | ✅ | ✅ | ⚠️ Add-on | ❌ Rare |
| Multi-framework single scan | ✅ | ❌ | ❌ | ❌ |
| API-first design | ✅ | ❌ | ❌ | ✅ |
| Self-hosted option | ✅ | ✅ | ❌ | ✅ |
| Python client | ✅ | ❌ | ❌ | Varies |
| Auto-generated audit reports | ✅ | ✅ | ✅ | ❌ |
| GUI dashboard | ⚠️ Basic | ✅ Excellent | ✅ Excellent | ❌ |
| Vendor ecosystem integrations | ⚠️ Growing | ✅ Extensive | ✅ Strong | ❌ |
| Pricing transparency | ✅ | ❌ | ⚠️ | ✅ |

---

## Breaking It Down

### ComplianceWeave

**Where it genuinely shines:**

The multi-framework single scan is the feature that made me stop and pay attention. Running separate scans for SOC 2 *and* ISO 27001 *and* HIPAA — and then reconciling overlapping controls across three different reports — is a time sink that most tools ignore entirely. ComplianceWeave treats overlapping controls as a first-class problem worth solving.

The API-first architecture is the other standout. If you've ever wanted to trigger a compliance check as part of a CI/CD pipeline, or pull findings into your own internal dashboard, or automate remediation workflows — you can actually do that here. The Python client is clean and reasonably documented, which matters more than it sounds. Most security tooling has Python bindings that feel like an afterthought.

The self-hosted option is significant for regulated industries. Healthcare organizations and financial services companies often cannot send infrastructure telemetry to a third-party SaaS. Having a self-hosted path isn't a checkbox — it's a blocker removed.

**Where alternatives are stronger:**

I won't pretend the GUI is a highlight. If your CISO or compliance officer lives in dashboards and needs to walk auditors through findings visually, ComplianceWeave's interface will feel spartan compared to the polished modern SaaS competitors. Those tools have invested heavily in making compliance *legible* to non-engineers, and it shows.

The vendor integration ecosystem is still maturing. Enterprise GRC platforms have spent years building connectors to every HR system, ticketing tool, and cloud provider under the sun. ComplianceWeave's integration library is growing but not yet at parity.

Community and third-party resources are thinner. If you get stuck, you're more likely to find a Stack Overflow answer or a detailed tutorial for the more established players.

---

### Enterprise GRC Platforms

These tools were built for large compliance teams, not engineering teams. The audit trail features are often genuinely excellent — years of refinement based on actual auditor feedback. If your organization has a dedicated GRC team and a budget to match, the depth here is real.

The downsides are real too: implementation timelines measured in months, pricing that requires a procurement conversation, and developer experience that ranges from "cumbersome" to "actively hostile."

---

### Modern SaaS Compliance Tools

The best of these tools are genuinely impressive at what they do. Beautiful dashboards, strong integrations with AWS/GCP/Azure, and enough polish that you can hand the interface to a non-technical stakeholder without embarrassment.

The trade-offs: they're GUI-only (automation and scripting are second-class citizens), they're cloud-hosted with no self-hosted option, and multi-framework support often means purchasing separate add-ons. They're also optimized for SOC 2 specifically — HIPAA and ISO 27001 coverage can feel bolted on.

---

### Open-Source DIY Frameworks

Maximum flexibility, zero licensing cost, and full control over your data. Also: you are the product team now. You'll spend real engineering hours wiring things together, writing your own report templates, and keeping up with framework updates. For a team with strong security engineering capacity and specific requirements that no commercial tool meets, this is a legitimate path. For most teams, the total cost of ownership is higher than it appears.

---

## Pricing Reality Check

Enterprise GRC: expect five-figure annual contracts, often with mandatory professional services.

Modern SaaS: typically $500–$2,000/month depending on scope, with per-framework or per-integration pricing adding up quickly.

ComplianceWeave: pricing is publicly available and usage-based, which makes budgeting predictable. Self-hosted reduces ongoing costs further.

Open-source: $0 in licensing, but budget for engineering time honestly.

---

## When to Use Each

**Choose ComplianceWeave if:**
- Your team is engineering-led and wants compliance integrated into existing workflows
- You need to demonstrate compliance across multiple frameworks simultaneously (common in B2B SaaS selling to enterprise and healthcare customers)
- Data residency or regulatory requirements make SaaS-hosted tools a non-starter
- You want to automate evidence collection via API rather than manage it manually

**Choose an enterprise GRC platform if:**
- You have a large, dedicated compliance team with complex workflow and approval requirements
- Your organization has existing procurement relationships and needs deep vendor integrations
- Auditor familiarity with the platform is a meaningful advantage for you

**Choose a modern SaaS compliance tool if:**
- SOC 2 is your primary (or only) framework
- Your compliance owner is non-technical and needs an excellent GUI experience
- You're a smaller team that wants fast time-to-value and don't have self-hosting requirements

**Choose open-source DIY if:**
- You have strong security engineering capacity in-house
- Your requirements are genuinely unusual and no commercial tool fits
- You're comfortable owning the maintenance burden long-term

---

## The Honest Bottom Line

No tool in this space is a perfect fit for everyone. The compliance automation market has historically optimized for auditor satisfaction and enterprise sales cycles — which is why the developer experience across most tools is an afterthought.

ComplianceWeave's bet is that infrastructure teams shouldn't have to context-switch into a separate compliance universe. Whether that bet is right for your organization depends on who owns compliance at your company, what your regulatory surface looks like, and whether your team will actually use a tool that requires reading API documentation.

If the answer to that last question is "yes, enthusiastically" — ComplianceWeave is worth a serious look.

---

*Have experience with any of these tools? Corrections and counterpoints welcome in the comments.*