DEV Community: Bobby Ray

What Should a Real Data Engineer Assessment Actually Test?

Bobby Ray — Thu, 27 Aug 2026 23:26:31 +0000

What Should a# What Should a Real Data Engineer Assessment Actually Test?

Data Engineering interviews are surprisingly inconsistent.

One company may focus heavily on SQL. Another may ask Python coding questions. Another may go deep on Spark, data modeling, cloud architecture, or distributed systems. And sometimes candidates still get generic algorithm questions that have little connection to the work they would actually do.

That raises a better question:

If you had 60–90 minutes to evaluate a Data Engineer, what would actually be worth testing?

I think the answer should be much closer to real engineering work.

1. SQL reasoning, not SQL trivia

A useful assessment can test:

joins and cardinality
window functions
aggregations
duplicate handling
null behavior
query correctness
performance trade-offs
debugging incorrect results

Instead of asking:

What is the syntax for ROW_NUMBER()?

give the candidate a realistic dataset and ask:

We need the most recent valid record for each customer. The source contains duplicates and late-arriving updates. How would you solve it?

That tells you much more.

2. Data modeling

Useful questions should involve:

grain
facts and dimensions
slowly changing dimensions
normalization vs. denormalization
analytical access patterns
historical correctness

Instead of:

Define SCD Type 2.

try:

The business needs to report revenue using the customer segment that was valid when the transaction happened. How would you model that?

Now the candidate has to reason.

3. Pipeline reliability

Production pipelines fail.

A practical assessment should test whether the candidate understands:

retries
idempotency
late-arriving data
duplicate events
backfills
partial failures
schema changes
observability
data quality

A candidate who can build a pipeline but cannot reason about failure modes is not finished yet.

4. Python in a Data Engineering context

Python matters, but the problem should match the role.

Useful areas include:

parsing
transformation
validation
APIs
file processing
testing
error handling
memory/performance reasoning

For many Data Engineering roles, this is more useful than an unrelated algorithm puzzle.

The question should be:

Can this person write maintainable data-processing code?

not:

Did this person memorize the exact interview pattern we selected?

5. Engineering judgment

Give the candidate an imperfect system.

Ask:

What would you change first?
What assumptions worry you?
Where would you add monitoring?
What would fail at 10× the data volume?
Which parts would you keep simple?
What would you not optimize yet?

There is rarely one perfect answer.

That is the point.

Strong engineers explain trade-offs.

6. Communication

A Data Engineer needs to explain:

why a model was chosen
why a pipeline is failing
what a data-quality issue means
which trade-offs matter
what should be fixed first

I would rather see a clear, defensible decision than a technically clever solution nobody can maintain.

7. Respect the candidate's time

Candidates should know before starting:

estimated time
format
what skills are being evaluated
whether work can be saved
whether they can resume later
what happens after submission

Assessment difficulty can be meaningful.

Assessment access should not be difficult.

A better principle

A good Data Engineering assessment should answer:

Can this person reason through realistic Data Engineering problems?

It should not primarily answer:

Can this person optimize for our interview format?

What I am experimenting with

I've been working with Anzibloom, a technical hiring platform experimenting with role-specific assessment before employer introductions.

The idea is:

Practical assessment
        ↓
Verified technical evidence
        ↓
Relevant opportunities

The goal is not to make assessment easier.

It is to make the assessment more representative of the work—and make the resulting evidence more useful than a disposable screening test.

If you work in Data Engineering, I would especially like your criticism:

What would you include in a 60–90 minute Data Engineer assessment, and what would make you stop halfway through one?

If you want to see the approach in practice, you can check out the Data Engineering assessment on Anzibloom.

Contract Versioning in Production Pipelines: Registry, CLI, and Run History

Bobby Ray — Thu, 27 Aug 2026 08:51:55 +0000

Dataset contracts are easy to write once. They are hard to operate when you cannot answer: which version failed, who pinned it, and whether the registry agrees with the YAML on disk.

This article closes the loop on Data Quality Contracts in Production Pipelines â€” registry â†’ CLI resolution â†’ Airflow scheduling â†’ versioned run history, with ingestion quarantine in production-data-pipeline.

Portfolio: br413.github.io Â· Quality layer: data-quality-observability Â· Ingestion: production-data-pipeline v0.2.1

Why versioning matters at the dataset boundary

Row-level quarantine catches poison pills during API ingestion. Dataset contracts catch schema drift, stale facts, and broken foreign keys before promote.

Without versioning you get:

A contract file changes; scheduled runs still report failures against an unknown baseline
On-call cannot tell whether freshness regressed on v1.0 or v1.1
Reviewers cannot see breaking changes without diffing YAML by hand

A lightweight registry fixes discoverability and pins â€” without standing up Confluent Schema Registry for a portfolio-scale stack.

Layer 1: File-based registry

contracts/registry.yml is the canonical catalog:

contracts:
  orders:
    current: "1.0"
    path: orders.yml
  customers:
    current: "1.0"
    path: customers.yml

Semver policy (PATCH / MINOR / MAJOR) lives in ADR 0002. Breaking changes require:

Major bump in the contract YAML
current update in the same PR
Entry in contracts/CHANGELOG.md

Layer 2: CLI resolves by name

Operators no longer pass file paths:

python -m src.dqo.cli run \
  --contract orders \
  --data data/samples/orders.csv \
  --references data/samples

The CLI loads registry.yml, resolves orders.yml, and runs schema / null / freshness / RI checks.

Layer 3: Airflow uses registry names

The dqo_contract_checks DAG runs registry-backed tasks:

run_orders_checks    â†’ --contract orders
run_customers_checks â†’ --contract customers

Optional DQO_WEBHOOK_URL routes contract failures the same way ingestion alerts work in the pipeline repo.

Layer 4: Versioned run history

Each run persists which contract version was evaluated:

2026-07-14T10:00:00+00:00  run-id  v1.0  passed

On-call can correlate freshness regressions to a semver pin instead of guessing from git history.

Layer 5: Cross-repo pins from ingestion

Ingestion quarantine and dataset contracts are separate boundaries. production-data-pipeline declares pins in config:

pins:
  orders: orders@1.0
  customers: customers@1.0

dqo:
  project_root: ../data-quality-observability

This makes the full-stack demo config-driven â€” not buried in a README footnote.

CI guards (the part teams skip)

Registry consistency runs in CI:

Every registry path exists on disk
current matches the contract YAML version
CHANGELOG documents the active version
PRs that bump version must update registry and CHANGELOG

python scripts/validate_registry.py

Skipping this step is how "silent schema drift" returns â€” with extra YAML files.

How this fits the portfolio stack

production-data-pipeline (ingestion + quarantine + contract pins)
    â†” data-quality-observability (registry + CLI + history)
    â†” Airflow DAGs (scheduled checks)
    â†” Dev.to series (public narrative)

Each layer answers a different reviewer question:

Quarantine â€” Can you isolate bad rows without aborting the batch?
Contracts â€” Can you define dataset quality explicitly?
Versioning â€” Can you operate those contracts over time?

What I would add next

Export landed tables to CSV/Parquet and wire dqo into the pipeline Airflow DAG after dbt
registry_revision (git SHA) in run history for full audit trails
One upstream merge on Airflow or dbt docs to balance portfolio depth with OSS signal

Related writing

If you operate contract versioning differently â€” Glue Registry, Data Contract CLI, or dbt exposures only â€” I am interested in how you draw the boundary between row-level and dataset-level gates.

What I Learned Contributing to Prefect, dbt, and Airflow (An Honest OSS Retrospective)

Bobby Ray — Fri, 21 Aug 2026 20:45:26 +0000

Portfolio repos prove you can build. Upstream merges prove you can collaborate with teams that maintain the tools production platforms run on. Over roughly ninety days I ran both tracks in parallel â€” portfolio releases, Dev.to writing, and OSS contributions to Prefect, dbt docs, Airflow, and Meltano â€” without backdating history or republishing private employer work.

Portfolio: br413.github.io Â· 90-day plan: github.com/br413/br413

Why upstream, not just portfolio

A strong GitHub profile needs more than greenfield demos:

Hiring signal â€” judgment inside someone else's codebase, not only your own repo boundaries
Operational credibility â€” fixes that reflect how platforms fail at 2 AM, not tutorial happy paths
Collaboration proof â€” you can respond to review feedback and respect maintainer direction

My portfolio stack â€” production-data-pipeline, data-quality-observability, lakehouse-platform-starter â€” gave real context for what to fix upstream. The rule I followed: comment on the issue before opening the PR.

What merged (and why those landed)

PR	Project	Change	Why it merged
Prefect #22500	Prefect	Kubernetes readiness vs liveness probes	Small, verifiable ops detail; maintainer-aligned
dbt docs #9606	dbt docs	Prefixed custom schema troubleshooting	Deployment pitfall many teams hit silently
Airflow #71158	Airflow	Metrics vs traces `otel_*` config clarity	Docs clarity; merged after second reviewer

Pattern: documentation and operational clarity beat drive-by feature PRs for early upstream contributions. Both changes were easy to review, tied to real production confusion, and did not require deep codebase archaeology.

What's still open (and what that teaches)

As of late August 2026, four PRs remain in flight:

PR	Status	Lesson
dbt docs #9781	Awaiting review	Issue-linked docs fixes still wait on maintainer bandwidth
Meltano #10253	Awaiting review	Tie PRs to maintainer-requested issues (#6289)
Prefect #22533	Changes requested â†’ addressed	Automated review catches doc accuracy gaps; respond precisely
Airflow #70171	Open	Provider PRs need patience; keep CI green, don't churn

Airflow #70185 was closed when the maintainer wanted a proper OpenLineage facet instead of my initial approach. That was the right outcome â€” don't force the wrong abstraction to keep a PR open.

What I would do differently

Fewer open PRs at once â€” after ~4 in flight, review bandwidth becomes the bottleneck, not ideas
Rebase early â€” Airflow moves fast; waiting weeks breaks CI on unrelated upstream changes
Portfolio first, then upstream narrative â€” shipping quarantine/DLQ in production-data-pipeline v0.2.1 made the data quality contracts article credible
Close gracefully â€” a withdrawn or closed PR with a clear maintainer reason is better than a stale open one

The weekly rhythm that worked

Day	Activity
Mon	One upstream comment + one small portfolio commit (docs/tests)
Wed	OSS PR work, rebase, or CI fix
Fri	README/ADR cross-link, plan update, or writing

Minimum bar: three public commit days per week. Consistency beats hero days for both the contribution graph and maintainer trust.

How portfolio and OSS reinforce each other

production-data-pipeline (ingestion + quarantine)
    â†” data-quality-observability (contracts)
    â†” Dev.to articles (public narrative)
    â†” upstream fixes (Prefect / Airflow / dbt / Meltano ops + docs)

Each layer answers a different reviewer question:

Portfolio â€” Can you design and ship a production-style stack?
Writing â€” Can you explain trade-offs clearly?
Upstream â€” Can you improve tools other teams already depend on?

Honest scorecard (90-day closeout)

Outcome	Target	Status
Upstream merges	5+	3 â€” in progress
Dev.to articles	3	4 âœ“
Portfolio release	v0.2.1 + quarantine	âœ“
Green contribution weeks	10+ consecutive	On track with Mon/Wed/Fri rhythm

The merge count is below target. That is worth stating plainly. The work is still credible because the open PRs are real, issue-linked, and actively maintained â€” not abandoned drive-bys.

Rules I kept (and recommend)

Never backdate commits â€” the activity graph reflects real work only
Comment before PR on upstream issues
Prefer data-platform repos (dbt, Airflow, Prefect, Meltano) over unrelated forks
One meaningful merge beats five cosmetic self-PRs
Profile, portfolio site, and resume must agree

If you're starting a similar push

Pick one upstream project you already use in production. Find a docs gap or ops footgun you have actually hit. Comment on the issue. Open a small PR. Ship one portfolio release that gives you standing to write about the same problem space.

Then repeat on a weekly cadence for ninety days.

Related writing

If this helped, leave a comment â€” I am interested in how other data engineers approach upstream contributions without turning it into performance theater.

Data Quality Contracts in Production Pipelines (Without a Separate Platform Team)

Bobby Ray — Fri, 21 Aug 2026 16:59:17 +0000

Production pipelines fail in two ways: loudly at 3 a.m., or quietly in the CEO's dashboard. Data quality contracts give you a third path â€” catch drift at the boundary, persist evidence, and route bad rows before they poison silver models.

Portfolio: br413.github.io Â· Quality layer: data-quality-observability Â· Ingestion quarantine: production-data-pipeline v0.2.1

The problem: silent failure modes

Most teams discover data quality problems downstream:

Schema drift lands in bronze; dbt tests fail hours later (or worse, pass on stale assumptions)
One poison-pill API record aborts an entire ingestion page â€” valid events in the same batch never land
Operators grep logs instead of querying failure artifacts

The fix is not "more dbt tests." It is layered gates with durable evidence at each boundary.

What a contract is (and isn't)

A data contract is a machine-readable agreement about a dataset: column types, null rules, freshness windows, and referential integrity. It is a gate at the dataset boundary, not a replacement for warehouse-level tests.

Example from my data-quality-observability project:

name: orders
version: "1.0"
description: Order facts contract for retail analytics

freshness:
  column: updated_at
  max_age_hours: 48

columns:
  order_id:
    type: string
    nullable: false
    unique: true
  customer_id:
    type: string
    nullable: false
  order_total:
    type: number
    nullable: false
  status:
    type: string
    nullable: false
  updated_at:
    type: datetime
    nullable: false

referential_integrity:
  - column: customer_id
    references:
      table: customers
      column: customer_id

Run the contract against sample data:

python -m src.dqo.cli run \
  --contract contracts/orders.yml \
  --data data/samples/orders.csv \
  --references data/samples

The CLI validates schema, nulls, uniqueness, freshness, and foreign keys â€” then persists run history and routes alerts.

What contracts are not: they do not replace dbt source freshness or uniqueness tests. They complement them by running before promote with auditable run history.

Two layers in a real stack

Production quality work spans two boundaries in my portfolio:

Layer	When	Project
Row-level quarantine	During ingestion	production-data-pipeline â€” invalid events â†’ `bronze.quarantine_events`
Dataset contracts	After landing / before promote	data-quality-observability â€” run history + alerts

flowchart TD
    API[External API] --> Ingest[Ingestion connector]
    Ingest -->|valid record| Bronze[bronze.raw_events]
    Ingest -->|validation failure| Quarantine[bronze.quarantine_events]
    Bronze --> Silver[dbt stg_events]
    Quarantine -.->|excluded by NOT EXISTS| Silver
    Silver --> Gold[fct_daily_event_metrics]
    Bronze --> Contract[YAML contract checks]
    Contract --> History[Run history store]
    Contract --> Alerts[Alert router]

Row-level quarantine at ingestion

Before v0.2.1, my pipeline used fail-fast batch validation: one bad record aborted the entire page. ADR 0004 introduces per-record routing:

Pass â†’ land in bronze.raw_events
Fail (recoverable) â†’ write to bronze.quarantine_events with rule name, message, and raw payload
Checkpoint â†’ quarantined event_id values are marked processed so poison pills do not block retries

Enable quarantine during ingestion:

python -m src.pipeline.ingestion \
  --source sample \
  --storage postgres \
  --pipeline-name sample-ingestion \
  --enable-quarantine \
  --alert-on-quarantine

The quarantine table schema:

CREATE TABLE IF NOT EXISTS bronze.quarantine_events (
    event_id TEXT NOT NULL,
    occurred_at TIMESTAMPTZ,
    payload JSONB NOT NULL,
    failed_rule TEXT NOT NULL,
    failure_message TEXT NOT NULL,
    pipeline_name TEXT NOT NULL,
    run_id TEXT NOT NULL,
    quarantined_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    PRIMARY KEY (pipeline_name, event_id, run_id)
);

Silver models exclude quarantined IDs explicitly:

select
    event_id,
    occurred_at,
    (payload ->> 'value')::numeric as event_value,
    ingested_at
from {{ source('bronze', 'raw_events') }} as r
where r.payload ? 'value'
  and not exists (
    select 1
    from {{ source('bronze', 'quarantine_events') }} as q
    where q.event_id = r.event_id
  )

Operator triage becomes a SQL query instead of a log grep:

SELECT event_id, failed_rule, failure_message, quarantined_at
FROM bronze.quarantine_events
WHERE pipeline_name = 'sample-ingestion'
ORDER BY quarantined_at DESC
LIMIT 20;

The ingestion summary now includes records_quarantined alongside fetched, inserted, and skipped counts.

Dataset contracts after landing

The quality observability layer runs after data lands but before you trust it for analytics:

Data contract (YAML)
    â†“
Quality check suite (schema Â· null Â· unique Â· freshness Â· RI)
    â†“
Run summary â†’ history store (SQLite / PostgreSQL)
    â†“
Alert router (console Â· JSONL file Â· webhook)

Schedule it with the included Airflow DAG (dqo_contract_checks) or run ad hoc from CI.

Alert routing that on-call will answer

Alerts must be actionable. Both projects route to console, file, and webhook channels:

Event	Source	When
`zero_record_ingestion`	production-data-pipeline	Ingestion succeeded but landed zero new records
`ingestion_quarantine`	production-data-pipeline	One or more records routed to quarantine
Contract failure	data-quality-observability	Schema, freshness, or RI check failed

Webhook payloads include structured summaries â€” pipeline name, quarantine count, failed rule â€” so on-call can triage without opening the repo.

Tie alerts to operations runbooks: query the quarantine table, inspect the failed payload, fix the source or rule, then replay or manually promote the row.

CI as contract enforcement

Green CI means the contract suite ran, not that production data is clean:

production-data-pipeline â€” pytest covers quarantine routing, webhook event types, and dbt integration (including quarantine exclusion in stg_events)
data-quality-observability â€” pytest covers check logic; integration tests mock webhook delivery success and failure

GitHub Actions runs both suites on every push. Contract YAML changes trigger the same gates as code changes â€” drift in the contract file is caught before merge.

Practical adoption path

You do not need a dedicated data quality platform team to start:

Pick one high-value dataset â€” orders, events, or your most-used fact table
Add freshness + required-field checks before adding referential integrity complexity
Enable quarantine/DLQ when API sources are messy or batch fail-fast is blocking valid records
Wire alerts to your existing webhook or Slack path â€” reuse the same router for ingestion and contract failures
Document recovery in an operations runbook so 2 AM triage does not depend on tribal knowledge

How this connects to article #1

My first Dev.to article covered incremental ingestion, checkpoints, and medallion layering with dbt. That pipeline intentionally ended with "at scale I would add a dead-letter queue and a separate quality layer."

This article is the follow-through: quarantine at ingestion (v0.2.1) and contracts at the quality boundary (data-quality-observability), wired together as a production-style stack.

Try it yourself

Quality contracts:

git clone https://github.com/br413/data-quality-observability.git
cd data-quality-observability
python -m venv .venv
# Windows: .\.venv\Scripts\Activate.ps1
pip install -r requirements.txt
pytest
python -m src.dqo.cli run --contract contracts/orders.yml --data data/samples/orders.csv --references data/samples

Ingestion quarantine:

git clone https://github.com/br413/production-data-pipeline.git
cd production-data-pipeline
pip install -r requirements.txt
docker compose up -d
python -m src.pipeline.ingestion \
  --source sample --storage postgres --pipeline-name sample-ingestion \
  --enable-quarantine --alert-on-quarantine
python -m src.pipeline.run_dbt --target dev

Both repos include ADRs, operations runbooks, and CI with PostgreSQL service containers.

Summary

Data quality is a stack of boundaries, not a single tool. Row-level quarantine keeps poison pills from blocking valid ingestion. Dataset contracts catch drift before promote. Persisted failure artifacts and webhook alerts turn silent dashboard bugs into triageable incidents.

Related projects

production-data-pipeline v0.2.1 â€” quarantine/DLQ at ingestion
data-quality-observability â€” YAML contracts and run history
cloud-lakehouse-blueprint â€” platform governance and lineage
Portfolio site Â· GitHub

If this helped, star the repos or leave a comment â€” I am interested in how other teams layer contracts and quarantine in production.

Building a Production Data Pipeline with Incremental Loading and dbt

Bobby Ray — Sun, 19 Jul 2026 19:15:54 +0000

Operational analytics breaks when pipelines silently drop records, re-process duplicates, or push schema drift into dashboards. This article walks through a production-style data pipeline pattern: incremental API ingestion, explicit checkpoints, medallion-style layering, and orchestration with Apache Airflow and dbt.

Portfolio: br413.github.io Â· Source code: production-data-pipeline

The problem

Most demo pipelines assume APIs are always available, schemas never change, and re-runs are harmless. Production systems need:

Incremental loading â€” fetch only new data since the last successful run
Idempotency â€” duplicate deliveries must not inflate metrics
Separable layers â€” raw landing, validated staging, and curated aggregates tested independently
Operational visibility â€” know when a run ingested zero records or failed mid-page

If your pipeline cannot recover at 2 AM without tribal knowledge, it is not production-ready.

Architecture overview

External API
    â†“
Ingestion connector (incremental, idempotent)
    â†“
Raw / bronze layer (PostgreSQL landing)
    â†“
Validated / silver layer (dbt stg_events)
    â†“
Curated / gold layer (fct_daily_event_metrics)
    â†“
Downstream consumers

Each boundary has a clear responsibility. Ingestion handles pagination and checkpoint advancement. Bronze stores append-only raw events. dbt owns transformation logic and data tests. Airflow schedules and retries the workflow.

Key design decisions

Decision	Why
Checkpoint + processed-ID tracking	Timestamp watermarks fail when APIs return out-of-order events
PostgreSQL bronze landing	DB constraints enforce idempotency; matches warehouse patterns
Quality gates before bronze	Block bad records at ingestion, not after they pollute raw history
dbt for silver/gold	Declarative transforms with built-in tests and clearer ownership

Incremental ingestion with checkpoints

The ingestion connector tracks its read position in a checkpoint store. Two options:

File-based checkpoints â€” lightweight for local development
PostgreSQL metadata tables â€” durable for shared environments

# Local run with file checkpoint
python -m src.pipeline.ingestion \
  --source sample \
  --checkpoint .checkpoints/sample.json

# Production-style run with PostgreSQL
python -m src.pipeline.ingestion \
  --source sample \
  --storage postgres \
  --pipeline-name sample-ingestion

Key behaviors:

Checkpoints advance only after successful page processing â€” a timeout mid-page does not skip data
Stable event IDs suppress duplicates â€” re-delivered events are ignored via a processed-ID set
Quality gates run before landing â€” schema validation, required fields, and freshness checks block bad records from bronze

Bronze, silver, and gold with dbt

After landing in bronze.raw_events, dbt builds two layers:

Layer	Model	Purpose
Silver	`stg_events`	Validated staging with typed columns
Gold	`fct_daily_event_metrics`	Daily aggregates for analytics

python -m src.pipeline.run_dbt --target dev

dbt tests enforce uniqueness, not-null constraints, and source freshness. When a test fails, the pipeline stops â€” downstream consumers never see broken data.

This is the medallion architecture in miniature: raw â†’ validated â†’ curated, each layer independently testable.

Orchestration with Airflow

The production_sample_ingestion DAG runs two tasks:

ingest_sample_events â€” Python ingestion into bronze
run_dbt_models â€” dbt run + test for silver and gold

Default retry policy: 2 retries with a 5-minute delay. For local testing:

airflow dags test production_sample_ingestion 2026-07-14

Failure modes and mitigations

Failure	Mitigation
Duplicate delivery	Processed-ID set suppresses re-ingestion
Partial page read	Checkpoint advances only after full page success
API timeout	Airflow retry + documented manual rerun
Schema drift	Python quality gate blocks landing; dbt tests catch transform issues
dbt test failure	DAG stops before downstream use
Zero-record success	Webhook alerts for empty ingestion runs

Document these in an operations runbook. Pipelines that cannot be recovered at 2 AM are not production-ready.

Observability: metrics and alerts

Recent additions to the project include:

Structured ingestion summary metrics â€” records fetched, inserted, skipped per run
Webhook alerts for zero-record runs and Airflow task failures
Local validation script (scripts/check.ps1) for pre-commit confidence

Observability is not optional. A pipeline that runs successfully but ingests zero records is often worse than one that fails loudly.

What I would do differently at scale

This portfolio project targets a single API source and PostgreSQL landing. In a larger platform I would add:

Dead-letter queue for records that fail quality gates
Schema registry or contract versioning for API drift
Lineage tracking from source through gold (see my cloud-lakehouse-blueprint)
Separate quality observability layer (see data-quality-observability)

Try it yourself

git clone https://github.com/br413/production-data-pipeline.git
cd production-data-pipeline
python -m venv .venv
source .venv/bin/activate  # Windows: .\.venv\Scripts\Activate.ps1
pip install -r requirements.txt
pytest
docker compose up -d
python -m src.pipeline.ingestion --source sample --storage postgres --pipeline-name sample-ingestion
python -m src.pipeline.run_dbt --target dev

The repo includes architecture docs, ADRs, CI with PostgreSQL service containers, and a v0.1.0 release.

Summary

Production data pipelines are defined by how they handle failure, not how they handle the happy path. Incremental checkpoints, idempotent loads, medallion layering, and testable transformation boundaries give you a foundation that scales from portfolio projects to real cloud data platforms.

Related projects

data-quality-observability â€” contract-driven quality checks
cloud-lakehouse-blueprint â€” medallion lakehouse with Terraform
Portfolio site Â· GitHub â€” more data engineering work

If this helped, star the repo on GitHub or leave a comment â€” I am interested in feedback from other data engineers and platform architects building similar systems.