DEV Community: Bruin

What Is a Data Orchestrator? And When a Small Team Does Not Need One

Kateryna Kozachenko — Sun, 25 Oct 2026 19:38:46 +0000

This article was first published on the Bruin blog.

A data orchestrator is the system that runs a pipeline's tasks in the right order, on a schedule, retries the ones that fail, and records what ran and when. Apache Airflow is the archetype: you describe tasks and their dependencies as a DAG, and the orchestrator's scheduler, workers, and metadata database do the rest. Dagster and Prefect are the modern alternatives. The question a small team should ask first is not which orchestrator but whether they need one, because a pipeline framework that resolves its own dependency graph, such as Bruin, removes the layer entirely for pipelines made of loads, transformations, and checks.

What an orchestrator actually does

Four jobs, and every orchestrator does all four:

Job	What it means	Airflow's implementation
Dependency resolution	Run B after A, C after both	The DAG, written in Python
Scheduling	Start the DAG at 02:00 daily, or when a file lands	The scheduler process
Execution	Run the tasks, in parallel where possible, with retries	Workers, plus an executor to distribute them
Bookkeeping	Which run did what, when, with what result	The metadata database and the web UI

The value is real. Without those four, a pipeline is a cron entry and a prayer. The cost is also real: each of the four is a component someone operates.

Why it became a layer of its own

Orchestrators exist because the modern data stack was assembled from single-purpose tools. Fivetran loads, dbt transforms, Great Expectations checks, a BI tool reads. None of them knows about the others, so a fifth tool has to know about all of them and call them in order. That is what most Airflow DAGs in data teams are: glue that runs an ingestion sync, then a dbt run, then a check, then a dashboard refresh.

That glue has a price. A self-hosted Airflow is a scheduler, workers, a metadata database, a web server, a deployment image, secrets handling, and someone who understands all of it. Managed Airflow removes the servers but keeps the concepts and the bill: Amazon MWAA's own pricing example for a small environment with typical retention comes to about $449 a month before the warehouse and the tools it orchestrates. Astronomer is the other managed route. Either way the team is paying to run a tool whose job is to call other tools.

When you do not need one

If the DAG mostly runs data work, loads, SQL and Python transformations, quality checks, and a schedule, then the orchestrator is doing dependency resolution for tasks that could declare their dependencies themselves. A pipeline framework that owns all of those steps does not need a separate system to order them.

That is the design of Bruin. Each asset is a file that declares what it depends on, and the SQL is parsed for the rest:

/* @bruin
name: mart.orders
type: sf.sql
depends: [raw.orders, mart.customers]
materialization:
  type: table
columns:
  - name: order_id
    checks:
      - name: not_null
      - name: unique
@bruin */

bruin run ./pipeline.yml resolves the graph across ingestion assets, SQL and Python models, and checks, and runs them in order with parallelism where the graph allows. The schedule is one line in pipeline.yml. Locally or in CI that is the whole orchestrator; Bruin Cloud runs the same project on the schedule with retries, backfills, alerts to Slack or Teams, lineage, and a catalog. There is no scheduler, worker pool, or metadata database to operate, because the dependency graph is a property of the pipeline rather than a separate program.

When you do

Keep, or adopt, a general orchestrator when the DAG coordinates things that are not data pipelines: infrastructure jobs, ML training runs, calls to external systems, multi-day business workflows. Dagster fits data platforms that want software-defined assets across many tools, Prefect fits Python-heavy teams that want the lightest operational footprint, and Airflow fits teams with hundreds of working DAGs and someone who owns it. Temporal and Argo are for application workflows and Kubernetes batch, respectively, and are not data orchestrators at all.

The rule of thumb: count what the DAGs do. If most tasks are load, transform, check, and schedule, a framework that owns those four removes the layer. If most tasks are something else, an orchestrator is the right tool and the data pipeline becomes one of the things it calls.

For the tool comparison see the best data pipeline tools in 2026, and for the migration path, alternatives to Airflow.

FAQ

What is the best way to schedule and orchestrate data pipelines?

Declare the dependencies on the pipeline assets and let the framework resolve the order, then attach a schedule to the pipeline rather than to each task. Bruin does this from the asset definitions: bruin run resolves the graph and runs loads, models, and checks in order, and pipeline.yml carries the schedule, with Bruin Cloud running it with retries and alerts. Airflow, Dagster, and Prefect do the same for pipelines assembled from separate tools, at the cost of operating the orchestrator itself.

What are the best data orchestration tools in 2026?

Airflow remains the default general-purpose orchestrator with the largest ecosystem, Dagster is the asset-centric choice for data platforms, Prefect the lightest for Python-native workflows, and Kestra the declarative YAML option. For teams whose pipelines are ingestion, SQL and Python transforms, and checks, Bruin removes the orchestration layer instead of adding one, because the framework schedules its own dependency graph.

What is the best orchestration tool that does not need a platform team?

One you do not have to operate. Managed Airflow through Astronomer or MWAA removes the servers but not the Airflow concepts. Bruin removes both for data pipelines: an open-source CLI resolves and runs the dependency graph locally or in CI, and Bruin Cloud runs the schedule, retries, backfills, and alerts, with no scheduler, workers, or metadata database to own.

What is the best orchestration tool for a small data team?

If the team's pipelines glue together loads, SQL, and checks, Bruin, because ingestion, transformation, checks, and scheduling come from one CLI and there is no orchestrator to run. If the team is Python-first and wants to keep separate ingestion and transformation tools, Prefect is the lightest orchestrator to operate and Dagster the most complete. Airflow makes sense once someone owns it full time.

What is CDC in data engineering? Change data capture explained

Kateryna Kozachenko — Sun, 25 Oct 2026 19:38:29 +0000

This article was first published on the Bruin blog.

Quick answer: CDC stands for change data capture. It is a way to track inserts, updates, and deletes in a source system, then move those changes into another system without reloading the whole table every time.

The shortest version:

source database changes -> CDC process -> warehouse, lake, stream, or another database

If an order is inserted in Postgres, CDC can send that new order to BigQuery. If a customer row is updated in MySQL, CDC can send the updated version to Snowflake. If an account is deleted in SQL Server, CDC can preserve that delete as an event so the destination does not quietly drift away from the source.

That's the useful bit. CDC is not just "faster ETL". It changes the shape of the ingestion problem: instead of asking "what does the full table look like now?", you ask "what changed since the last point I safely processed?"

This article covers the broader capture method. For the continuous, low-latency delivery model, see CDC streaming.

For data teams, this matters because most company data does not sit still. Product events arrive, payments settle, subscriptions change status, tickets get merged, rows get corrected, and someone eventually deletes the thing everyone assumed would never be deleted. The classic nightly full refresh works for small tables. It gets painful once tables are large, freshness matters, or downstream systems need a proper history of changes.

What CDC means

Change data capture is the process of detecting row-level changes in a source system and making those changes available to another system.

A CDC event usually answers a few practical questions:

Which table changed?
Which row changed?
Was it an insert, update, or delete?
What did the row look like before?
What does the row look like after?
When did the change happen?
Where in the source log did this change come from?

A simplified CDC event might look like this:

{
  "table": "public.orders",
  "operation": "update",
  "primary_key": { "order_id": 1042 },
  "before": {
    "status": "pending",
    "updated_at": "2026-07-13T08:14:10Z"
  },
  "after": {
    "status": "paid",
    "updated_at": "2026-07-13T08:16:03Z"
  },
  "source_position": "log-sequence-number-or-binlog-offset"
}

That source_position part is easy to ignore when you are learning CDC, but it is one of the most important pieces. A CDC pipeline needs to know where it stopped so it can resume without skipping changes or processing the same change incorrectly. In PostgreSQL, this is tied to logical decoding and replication slots. In MySQL, it is commonly tied to the binary log. In SQL Server, CDC reads from the transaction log into change tables.

The names vary by database, but the pattern is similar: the database already records changes so it can commit transactions, recover from failure, and replicate data. CDC uses that record as the source of truth for downstream movement.

How CDC works

There are a few ways to capture changes, but log-based CDC is the one most engineers mean when they say CDC.

1. The database writes a change

An application inserts, updates, or deletes a row. The database commits the transaction and records it in its internal log.

Examples:

PostgreSQL exposes changes through logical decoding, which can stream SQL-level modifications to external consumers.
MySQL records changes in the binary log, often called the binlog.
SQL Server CDC reads from the transaction log and stores changes in relational change tables.
Debezium connectors commonly capture MySQL changes from the binlog and PostgreSQL changes from a logical replication stream, as described in the Debezium architecture docs.

2. A CDC reader consumes the log

The CDC reader connects to the source database and reads committed changes in order. It does not need to run select * from orders every five minutes. It reads the database's change stream.

This usually reduces source load compared with full-table extraction, especially when a table is large but only a small percentage of rows change during each sync.

3. The pipeline stores an offset

The pipeline records the last safe position it processed. That offset might be a log sequence number, binlog file and position, timestamp plus version, or another source-specific cursor.

This is what lets the pipeline restart after a deploy, network failure, worker crash, or source maintenance window.

4. The destination applies the changes

The destination might receive the changes as:

an append-only raw change table
a current-state replica table
a stream topic
a slowly changing dimension table
a queue for downstream applications

Analytics teams often keep both:

raw CDC events for auditability and replay
current tables for normal BI, AI analysis, dashboards, and modelling

That split is boring, but it saves you later. Raw events are useful when something goes wrong and you need to replay or explain history. Current tables are useful because most business users want to query "orders" rather than reconstruct a table from every historical mutation.

CDC vs ETL vs ELT

CDC is not a replacement for ETL or ELT. It is a method of extraction.

Concept	What it means	Where CDC fits
ETL	Extract, transform, load	CDC can be the extract step
ELT	Extract, load, transform	CDC can load raw changes first, then transform in the warehouse
Data ingestion	Moving data from source to destination	CDC is one ingestion strategy
Replication	Keeping another system close to the source	CDC may power replication, but it is not limited to source-shaped copies
Streaming	Moving events continuously	CDC can produce a stream, but CDC can also be consumed in batches

The difference sounds pedantic until you design the pipeline.

If you say "we need ETL", you still have to decide whether extraction is full refresh, timestamp incremental, API pagination, event export, or CDC. If you say "we need CDC", you still have to decide what happens after the changes land: raw storage, merges, transformations, checks, lineage, documentation, and access control.

This is why CDC is usually one part of a broader data platform, not the whole platform.

CDC vs incremental loading

CDC and incremental loading overlap, but they are not the same thing.

Incremental loading means "only load new or changed data since the last run." CDC is one way to do that.

The common incremental strategies look like this:

Strategy	How it finds changes	Good for	Main risk
Timestamp cursor	Reads rows where `updated_at > last_run_time`	Simple tables with reliable timestamps	Missed deletes, clock issues, late updates
Increasing ID	Reads rows where `id > last_seen_id`	Append-only tables	Cannot detect updates or deletes
Full diff	Compares source and destination snapshots	Smaller tables where correctness matters	Expensive at scale
Log-based CDC	Reads transaction logs or change streams	Large mutable tables, low-latency syncs	More setup, permissions, and operational detail
Trigger-based CDC	Writes changes through database triggers	Databases without usable logs	Adds write-path overhead and schema complexity

If a table is append-only, an increasing ID cursor might be enough. If rows are updated and deleted all day, CDC becomes much more attractive.

The mistake is treating every table the same. A small reference table can be full-refreshed. A huge orders table probably should not be. An event table may only need append mode. A customer table with merges, deletes, and compliance requirements may need CDC or SCD Type 2.

Types of CDC

Log-based CDC

This is the usual preferred option for production databases. The CDC tool reads the transaction log, binlog, write-ahead log, redo log, or logical replication stream.

Why teams like it:

lower source query load than repeated full scans
captures inserts, updates, and deletes
preserves ordering better than timestamp scans
can support lower-latency syncs
usually does not require changing application code

The tradeoff: you need source-specific setup. Permissions, retention, replication slots, binlog settings, replica identity, networking, and failover behaviour matter.

Trigger-based CDC

Trigger-based CDC adds database triggers that write changes to audit or shadow tables. A pipeline then reads those tables.

This can work, especially when the database does not expose a useful log. But it adds logic to the write path. That means more performance risk, more migration work, and more things to review when schema changes happen.

Query-based CDC

Query-based CDC polls the source with a cursor, usually updated_at, created_at, or an incrementing ID.

This is the easiest model to understand and often the first one teams implement. It is also the easiest one to get subtly wrong.

The usual problems:

hard deletes disappear unless you soft-delete rows
updates can be missed if timestamps are unreliable
backfilled rows can fall outside the cursor window
bulk updates can overload the next sync
clock and timezone issues create awkward edge cases

Query-based incremental loading is still useful. It just should not be called "proper CDC" unless it really captures the mutations you care about.

When CDC is worth it

CDC is worth considering when at least one of these is true:

tables are too large for regular full refreshes
downstream data needs to be fresh within minutes
updates and deletes matter
you need an audit trail of row-level changes
you want to replicate operational database changes into a warehouse or lakehouse
source systems are sensitive and you want to reduce repeated heavy reads
downstream consumers need to replay events from a known point

Good CDC use cases:

syncing Postgres, MySQL, or SQL Server into Snowflake, BigQuery, Redshift, Databricks, or DuckDB
keeping customer, subscription, payment, order, ticket, or inventory tables fresh
feeding event-driven systems from database mutations
maintaining current-state and historical tables side by side
reducing batch windows for large mutable datasets

CDC is especially useful when the business keeps asking why yesterday's dashboard is not fresh enough, but the data team knows that full-refreshing the source every 15 minutes would be irresponsible.

Popular CDC tools

If you search for "CDC tools", you will usually see the same names come up: Debezium, Fivetran, Airbyte, Estuary Flow, AWS Database Migration Service, Google Datastream, Oracle GoldenGate, Qlik Replicate, and now Bruin with ingestr for code-first CDC inside a governed data pipeline.

They are not interchangeable. Some are CDC engines. Some are managed ELT platforms. Some are cloud migration services that happen to support ongoing replication. Some are better when Kafka is already part of your stack. Some are better when your team wants a YAML asset in Git and a scheduler around it.

Here is the practical split.

Tool	Best fit	What to watch
Debezium	Open-source log-based CDC into Kafka Connect, Debezium Server, or streaming infrastructure	You operate the runtime, connectors, offsets, schema changes, and downstream sinks
Fivetran	Managed database replication into warehouses with less infrastructure work	Cost, vendor control, sync frequency, and connector-specific behaviour
Airbyte	Open-source or managed connector platform with CDC support for common databases	Airbyte's CDC model is scheduled sync, not an infinite stream, so log retention and sync cadence matter
Estuary Flow	Streaming-first CDC and materialization for low-latency data movement	Another streaming platform to operate, govern, and connect to the rest of your stack
AWS DMS	Database migration and ongoing replication inside AWS	AWS explicitly notes that DMS CDC is not real-time replication and latency depends on workload, network, replication instance, and target capacity
Google Datastream	Serverless CDC on Google Cloud, especially into BigQuery and Cloud Storage	Strong fit for GCP-centred stacks, less portable if your CDC architecture needs to stay cloud-neutral
Oracle GoldenGate	Enterprise replication, Oracle-heavy estates, and heterogeneous database environments	Powerful, but usually more enterprise-ops heavy than a simple warehouse ingestion job
Qlik Replicate	Enterprise CDC and replication across many source and target systems	Similar tradeoff: broad enterprise capability, but with platform cost and operational ownership
Bruin + ingestr	Code-first ingestion assets, CDC, transformations, checks, lineage, and governance in one workflow	Best when CDC should be part of a reviewed data pipeline, not a disconnected sync job

Honestly, the right choice is less about the CDC acronym and more about the operating model.

If Kafka is already the centre of the architecture, Debezium is a natural fit. If the team wants a managed warehouse connector and does not want to run capture infrastructure, Fivetran or Airbyte may be the simpler path. If the project is a cloud migration, AWS DMS or Datastream can be the right boring answer. If the estate is Oracle-heavy or enterprise replication is the main job, GoldenGate or Qlik Replicate deserve a look.

And if the question is "how do we put CDC inside the same repo where we define transformations, tests, owners, metadata, and lineage?", that is where Bruin fits.

How Bruin supports CDC with ingestr

Bruin's CDC support lives in the ingestr asset type. In other words, CDC is not configured as a one-off side process that nobody can find later. It is an asset in the pipeline, alongside SQL assets, Python assets, checks, ownership metadata, and downstream lineage.

A simple PostgreSQL CDC asset looks like this:

name: raw.orders
type: ingestr
connection: bigquery

parameters:
  source_connection: pg_prod
  source_table: public.orders
  destination: bigquery
  cdc: "true"
  cdc_mode: stream

The Bruin docs describe CDC support for PostgreSQL, MySQL/MariaDB, Vitess, and PlanetScale sources through ingestr assets:

PostgreSQL CDC in Bruin uses PostgreSQL logical replication. The source needs wal_level=logical, a publication, and a replication slot.
MySQL CDC in Bruin uses binary-log replication. MySQL needs settings like log_bin=ON, binlog_format=ROW, and binlog_row_image=FULL.
Vitess CDC in Bruin uses Vitess VStream through vtgate's gRPC API.
PlanetScale CDC in Bruin uses PlanetScale's hosted psdbconnect API over TLS.

Under the hood, ingestr's CDC docs expose the lower-level source schemes and runtime controls. For example, PostgreSQL CDC can use postgres+cdc://, MySQL CDC can use mysql+cdc://, and PlanetScale CDC can use ps_mysql+cdc://. ingestr also has streaming flush controls like --flush-interval and --flush-records, and stream monitoring metrics for replication lag and durable row counts.

Bruin adds the pipeline layer around that:

CDC assets are defined in Git.
Destination application uses merge semantics for CDC, so updates and deletes can be applied by primary key.
Primary keys are determined from the source for CDC assets where the source exposes them.
Pipeline runs can include CDC ingestion, transformations, checks, and downstream assets in the same graph.
Bruin Cloud can add scheduling, catalog, lineage, access controls, SSO, audit logs, and observability around those assets.

That's the main difference. A CDC tool can move changes. A governed CDC pipeline also tells you who owns the data, which downstream models depend on it, what checks ran, what changed in Git, and whether the tables are trustworthy enough for dashboards or an AI data analyst.

When CDC is not worth it

CDC has a cost. Not always a vendor cost, but definitely an operational cost.

You may not need CDC when:

the table is tiny
the table is append-only
daily freshness is enough
deletes do not matter
a source API already gives reliable incremental exports
the source database team will not grant replication or log access
your downstream models are not ready to handle late-arriving changes

CDC can make a messy pipeline faster, but it will not make it governed.

If there is no ownership, no tests, no lineage, no clear schema-change policy, and no destination contract, CDC just moves the chaos more quickly. This is where teams get disappointed: they add a powerful capture layer, then discover their warehouse models still cannot explain what "active customer" means.

What can go wrong with CDC

The failure modes are not exotic. They are normal data engineering problems with sharper edges.

Schema changes

Source tables change. Columns are added, renamed, removed, widened, narrowed, or repurposed. CDC pipelines need a schema-change policy:

Should new columns appear automatically?
Should type changes be blocked?
Should the raw event stream preserve unknown fields?
Who reviews downstream model impact?

Without this, one application migration can break ingestion, transformations, dashboards, and AI answers at the same time.

Deletes

Deletes are the reason many timestamp-based incremental jobs lie.

If a row is physically deleted from the source, a normal updated_at query will not see it. CDC can capture delete events, but the destination still needs to decide what to do with them:

delete the row from the current table
mark it as deleted
keep it in raw history only
apply a retention policy

That choice is a business and governance decision, not just an engineering one.

Backfills and snapshots

Most CDC pipelines start with an initial snapshot, then switch to ongoing changes.

That handoff is critical. If the snapshot and log stream are not coordinated, you can miss changes that happen during the snapshot or apply duplicates after it.

Good CDC tools handle this carefully. Homegrown scripts often do not, mostly because the happy path looks deceptively simple.

Ordering and idempotency

CDC events need to be applied in a way that produces a correct destination state even if a worker retries. That usually means designing around primary keys, source positions, transaction boundaries, and idempotent merges.

If "run it again" creates duplicates, the pipeline is not finished.

Source retention

Database logs are not kept forever. If a CDC process stops for too long and the source removes the old log segment, the pipeline may no longer be able to resume from its saved offset. Then you are looking at a re-snapshot or some careful repair work.

This is why monitoring matters. CDC needs lag alerts, not just job-success alerts.

How CDC fits into a governed data pipeline

A practical CDC pipeline usually has these layers:

source database
  -> CDC capture
  -> raw change events
  -> current-state tables
  -> transformations
  -> checks
  -> catalog, lineage, dashboards, AI analyst

The capture layer is only the beginning.

For analytics and AI workflows, the downstream context is just as important:

catalog so people know what tables mean
lineage so changes can be traced from source to dashboard
quality checks so broken loads do not become trusted answers
asset tiers so important tables are handled differently from scratch work
audit trails so production changes have a review history
access controls so raw operational data is not exposed casually

This is the part Bruin cares a lot about. The open-source side gives you local, code-first pipeline work with Bruin CLI and ingestion with ingestr. Bruin Cloud adds orchestration, catalog, lineage, checks, access controls, SSO, audit logs, and observability around those workflows.

CDC is strongest when it is not a disconnected sync job. It should be visible inside the same workflow where transformations, checks, and downstream data products are reviewed.

A simple CDC decision checklist

Before choosing CDC, ask:

Does the source expose a reliable change log or CDC feature?
Can we get the permissions needed to read it?
Do we need deletes, or only inserts and updates?
Do we need raw history, current-state tables, or both?
How fresh does the destination actually need to be?
What happens when the schema changes?
What happens when the pipeline is down for two hours?
How do we monitor lag?
Can the destination apply events idempotently?
Who owns the downstream model if the source semantics change?

If those questions feel annoying, that's the point. CDC is not hard because the acronym is hard. It is hard because it touches database internals, pipeline reliability, schema contracts, and business meaning at the same time.

FAQ

What is CDC in data engineering?

CDC means change data capture. It is the process of identifying inserts, updates, and deletes in a source system and moving those changes to another system, usually a warehouse, lakehouse, stream, or operational database.

What is the difference between CDC and ETL?

ETL is the broader process of extracting, transforming, and loading data. CDC is a capture strategy inside that process. Instead of extracting a whole table every time, CDC captures only the rows that changed.

Is CDC the same as replication?

No. Replication usually tries to keep another database copy close to the source shape. CDC emits a stream or table of change events that downstream systems can transform, audit, replay, or merge into different models.

When should a team use CDC?

Use CDC when full refreshes are too slow, source tables are large, downstream data needs to stay fresh, or deletes and updates matter. It is especially useful for database-to-warehouse ingestion and event-driven data pipelines.

What is log-based CDC?

Log-based CDC reads the database transaction log, such as the PostgreSQL logical replication stream, MySQL binary log, or SQL Server transaction log, instead of repeatedly querying source tables for changes.

What are the most popular CDC tools?

Popular CDC tools include Debezium, Fivetran, Airbyte, Estuary Flow, AWS Database Migration Service, Google Datastream, Oracle GoldenGate, Qlik Replicate, and Bruin with ingestr. The right choice depends on whether you want open-source streaming CDC, managed ELT, cloud migration replication, enterprise database replication, or CDC as part of a governed pipeline.

Does Bruin support CDC?

Yes. Bruin supports CDC through ingestr assets. Bruin docs describe CDC support for PostgreSQL, MySQL/MariaDB, Vitess, and PlanetScale sources using cdc: "true". Bruin keeps CDC in the same asset graph as transformations, checks, ownership metadata, catalog, lineage, and Cloud observability.

Does CDC require Kafka?

No. Kafka is common in CDC architectures, especially with Debezium, but it is not mandatory. CDC changes can flow into warehouses, object storage, queues, streams, or another database depending on the tool and destination.

Can CDC capture deletes?

Yes, proper CDC can capture deletes if the source and capture method support them. The destination still needs a clear policy for applying those deletes, preserving them in history, or marking rows as deleted.

Is CDC always real-time?

No. CDC can be near real-time, but it can also be consumed in micro-batches or scheduled jobs. The important part is that the pipeline reads changes from a reliable change source instead of repeatedly extracting the whole table.

The useful way to think about CDC

CDC is a better question to ask of operational data:

What changed, in what order, and how do we safely apply it downstream?

Once you think about it that way, the rest of the design gets clearer. You need a source of changes, a stored position, an idempotent destination write, a schema policy, monitoring, and enough governance that people can trust the tables built from those changes.

The capture part is technical. The trust part is the real work.

What Is a Data Quality Check? Types, Where to Run Them, and Examples

Kateryna Kozachenko — Sun, 25 Oct 2026 19:38:12 +0000

This article was first published on the Bruin blog.

A data quality check is a rule that runs against a table, or a column in it, and fails when the data breaks the rule. "No nulls in order_id", "every status is one of four values", "the newest row is less than six hours old". The check either passes or it fails, and what happens on failure is the design decision that separates a useful check from a dashboard nobody reads: a blocking check stops the pipeline, so bad data never reaches the tables downstream. In Bruin, checks are declared on the column inside the asset that produces it and block by default; dbt tests, Great Expectations, and Soda run the same kinds of rules as a separate step.

The types of data quality check

Most checks fall into six families. The names below are Bruin's built-in check names; every framework has an equivalent.

Family	What it asserts	Built-in check	Catches
Completeness	The column has a value	`not_null`	A source that started sending empty fields
Uniqueness	No duplicates	`unique`	A load that ran twice, a join that fanned out
Validity	Values are from an allowed set or shape	`accepted_values`, `pattern`	Upstream schema drift, a new status nobody mapped
Range	Numbers and dates sit inside bounds	`positive`, `non_negative`, `min`, `max`	Unit errors, currency mix-ups, future dates
Referential integrity	Every foreign key points at a real row	`relationships`	Orphaned facts after a dimension reload
Freshness and volume	The table was updated recently and has the expected rows	custom SQL check	The pipeline that silently stopped

The first two families on a primary key, plus a freshness check on the load timestamp, catch most of the incidents a data team ever gets paged for.

Where the check runs matters more than which checks you write

There are two places a check can live. It can sit beside the pipeline, as a separate suite of expectations or a YAML scan file, run by a step the orchestrator triggers between load and transform. Or it can sit inside the pipeline, declared on the asset that produces the table and executed as part of building it.

Beside the pipeline is how Great Expectations and Soda Core work, and it suits a platform team that wants one central library of rules over many producers. The cost is drift: rename a column in the transformation and the expectation that references it is now wrong in a different repository, and nobody finds out until it fails for the wrong reason.

Inside the pipeline is how Bruin and dbt tests work. The check is in the same file as the SQL that produces the column, so rename the column and the check moves with it, delete the asset and the check goes with it. There is nothing to keep in sync.

What a check looks like in practice

In Bruin the checks are part of the asset header, in the SQL or Python file that builds the table:

/* @bruin
name: mart.orders
type: sf.sql
depends: [raw.orders]
materialization:
  type: table
columns:
  - name: order_id
    type: integer
    primary_key: true
    checks:
      - name: not_null
      - name: unique
  - name: status
    type: string
    checks:
      - name: accepted_values
        value: [placed, paid, shipped, refunded]
  - name: order_total
    type: float
    checks:
      - name: non_negative
custom_checks:
  - name: loaded in the last 6 hours
    query: SELECT max(updated_at) > current_timestamp - interval '6 hours' FROM mart.orders
    value: 1
@bruin */

SELECT order_id, status, order_total, updated_at FROM raw.orders

bruin run builds the table and runs the checks in the same step. Every check is blocking by default, so a failure stops the downstream assets; a check you only want to observe gets blocking: false. The custom_checks block takes any SQL that returns a value to compare, which is how freshness, row counts, and business rules such as "no order total above a limit without a manager flag" are expressed.

The dbt equivalent is a tests: list under the column in the model's YAML, run by dbt test or dbt build. Great Expectations expresses the same rules as expectations in a suite and runs them from a checkpoint. Soda writes them as checks for orders: with lines like missing_count(order_id) = 0.

When a check fails

A failed check should do three things: stop the assets that depend on the table, tell a person in the channel they already watch, and leave enough context to diagnose the cause. Bruin does the first two from the pipeline definition, with notifications routed to Slack or Microsoft Teams, and the third comes from lineage, which shows which upstream asset fed the bad rows. The alternative, a check that logs a warning and lets the pipeline continue, is how a wrong number ends up in a board deck.

Checks are the start of data quality, not the whole of it

Checks are a gate: they catch what you thought to write a rule for. They do not notice that a table's row count is 40% below its usual Tuesday, because no rule said what Tuesday usually looks like. That is the job of an observability tool such as Elementary, Monte Carlo, or Anomalo, and most teams end up with both a gate and a monitor once they have more tables than rules. For the tool-by-tool view see the best data quality tools in 2026, and for the full strategy including data contracts, data quality and testing strategies for modern pipelines.

FAQ

What is the best way to add data quality checks to a pipeline?

Declare the check on the asset that produces the table, so it runs every time the table is built and stops the pipeline when it fails. Start with not_null and unique on every primary key, accepted_values on every status column, and a freshness check on every scheduled load. In Bruin the checks sit in the columns block of the SQL or Python asset and run with bruin run; dbt tests, Great Expectations suites, and Soda scans are the equivalents that run as a separate step.

How do I validate data as part of a pipeline?

Make validation a stage of the pipeline rather than a job beside it. A blocking check fails the asset and stops its downstream assets, so bad rows never reach a dashboard. Bruin does this by default for every check declared on an asset; in dbt you get the same by running dbt build, which stops on a failing test; with Great Expectations or Soda you wire a checkpoint or scan into the orchestrator between load and transform.

What are the most important data quality checks?

Four cover most incidents: not_null and unique on primary keys, which keep joins safe; accepted_values on categorical columns, which catch upstream schema drift; a range check such as non_negative or max on amounts, which catches unit and currency errors; and a freshness check on the load timestamp, which catches the pipeline that silently stopped. Add referential integrity between fact and dimension tables once those four are in place.

What is the difference between a data quality check and data observability?

A check is a gate: a rule you wrote, run at build time, that blocks bad data. Observability is a monitor: it watches tables after the fact and flags anomalies you did not write a rule for. Checks are cheap and prevent incidents; observability costs a licence and reports them. Most teams need a gate first and add a monitor when the number of tables outgrows the rules they can write.