Bruin: SQL + Python Pipelines in One Framework With Built-In Quality Checks

bruin is the open-source data pipeline framework that collapses four tools into one: it runs your SQL transforms, executes your Python jobs, ingests data from external sources, and enforces data-quality checks — all from a single Go binary and a folder of files you keep in version control. You do not wire dbt to Airbyte to Great Expectations to an orchestrator and hope the seams hold. You write assets — SQL files, Python files, or small YAML files — each carrying its own metadata in a comment block, and you type bruin run.

That is a different shape from the modern stack most teams assembled over the last five years, where transformation, ingestion, testing, and scheduling are four separate products glued together with CI scripts. This guide walks through the four ideas an interviewer will actually probe — the asset model with YAML-in-comment metadata, materialization strategies, built-in and custom quality checks, and ingestion plus the dependency-driven DAG — and pairs each with a Solution-Tail interview answer: code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works.

When you want hands-on reps immediately after reading, drill the pipeline-design practice library →, rehearse the load-shape decisions on the ETL practice set →, and harden your test suite on the data-quality practice set →.

---

On this page

• Why Bruin puts the whole pipeline in one framework
• Assets: SQL, Python & YAML-in-comment metadata
• Materialization: how a SELECT becomes a table
• Built-in & custom quality checks
• Ingestion with ingestr & multi-destination DAGs
• Cheat sheet — Bruin recipes
• Frequently asked questions
• Practice on PipeCode

---

1. Why Bruin puts the whole pipeline in one framework

Bruin is one tool for SQL, Python, ingestion, and quality — that single fact decides where it fits

The one-sentence invariant: Bruin treats every step of a pipeline — extract, transform, and test — as an asset in one project, so you stop stitching four products together and run the whole graph with bruin run. Everything that makes Bruin attractive to a data engineering team follows from that. There is no separate ingestion service, no separate test framework, no separate scheduler required to make a pipeline correct; a Bruin project is a folder of asset files plus a pipeline.yml that the CLI reads, orders, executes, and validates.

What lives inside one Bruin project.

• Ingestion. An ingestr asset pulls data from an external source (an API, a database, a bucket) into a destination, with no connector code to write.
• Transformation. A SQL asset (bq.sql, sf.sql, duckdb.sql, pg.sql, and more) or a Python asset runs your business logic and materializes a table or view.
• Quality. Column-level and custom checks are declared in the same file as the asset they guard, and run right after it produces data.
• Orchestration. Bruin reads each asset's depends list, builds a DAG, and runs assets in topological order — no external orchestrator needed for a single pipeline.

Where Bruin sits against the alternatives.

• vs dbt. dbt is transform-only: it compiles and runs SQL (and, more recently, Python models) but leaves ingestion and orchestration to other tools. Bruin covers the same SQL-modelling ground plus native Python assets, ingestr-based ingestion, and a built-in runner. If your answer to "how does raw data arrive?" is "a different tool," Bruin folds that step in.
• vs dbt + Airbyte + Great Expectations + Airflow. The classic modern stack is four products and the glue between them. Bruin puts ingestion, transformation, quality, and single-pipeline scheduling behind one CLI and one config language, so there are fewer moving parts to break.
• vs a hand-rolled Python stack. A pile of scripts has no schema for metadata, no dependency graph, no first-class checks, and no consistent run semantics. Bruin gives you all four without a platform to operate.

What interviewers listen for.

• Do you say "Bruin is SQL + Python + ingestion + quality in one framework" rather than "another dbt"? — senior signal.
• Do you place dbt as "transform-only" and Bruin as "end-to-end assets" unprompted? — required framing.
• Do you mention that quality checks live next to the asset and can block downstream — not in a separate suite? — the whole point.
• Do you note that bruin run needs no external orchestrator for a single pipeline, while still deploying to Airflow when you want it? — senior signal.

Worked example — one asset that materializes a table

Detailed explanation. The canonical Bruin "hello world" is a single SQL file whose top comment block is YAML metadata and whose body is a SELECT. Bruin reads the metadata to learn the asset's name, platform, and how to materialize it, then wraps your query in the right DDL. You never write CREATE TABLE — you declare materialization: type: table and Bruin does it.

Question. Write one Bruin SQL asset that turns a two-row SELECT into a DuckDB table dashboard.hello, with no CREATE TABLE written by you.

Input.

what you write	what Bruin uses it for
@bruin comment block	asset metadata (name, type, materialization)
SELECT body	the query whose result becomes the table

Code.

/* @bruin
name: dashboard.hello
type: duckdb.sql
materialization:
  type: table
@bruin */

select 1 as id, 'ada' as name
union all
select 2 as id, 'linus' as name

Step-by-step explanation. The block between /* @bruin and @bruin */ is parsed as YAML: name: dashboard.hello becomes the target schema.table, type: duckdb.sql tells Bruin to run the body as DuckDB SQL, and materialization: type: table tells it to persist the result. When you run bruin run assets/hello.sql, Bruin generates a create+replace statement (the default table strategy), executes your SELECT, and stores the rows in dashboard.hello. The SQL body stays a plain SELECT — all the persistence logic lives in the metadata.

Output.

Bruin created	value
table	dashboard.hello
columns	id, name (inferred from the SELECT)
rows	2
strategy applied	create+replace (table default)

Rule of thumb. If a step can be expressed as "metadata plus a body," it is a Bruin asset — a SQL query, a Python script, or an ingestion spec differ only in the body, never in the project shape.

---

2. Assets: SQL, Python & YAML-in-comment metadata

The asset — code plus its YAML metadata in one file — is Bruin's entire mental model

Bruin has exactly one core abstraction you compose, and an interviewer who asks "walk me through Bruin's model" wants it crisp: everything is an asset, and an asset is code with a block of YAML metadata attached to it. Learn the three file shapes and where the metadata goes, and the whole framework snaps into focus.

The three asset file shapes.

• SQL asset (.sql). The metadata lives between /* @bruin and @bruin */ markers at the top of the file; the SQL query body follows underneath. Definition and query stay in one file — Bruin will not let you split them across a sibling .asset.yml.
• Python asset (.py). The metadata lives between """@bruin and @bruin""" docstring markers; the Python script follows. Same file, same idea.
• YAML asset (<name>.asset.yml). A standalone YAML file with only metadata and no code body — used for asset types that have no inline code, such as ingestr, sensor, and seed. The .asset.yml suffix is required; a plain .yml is treated as config and ignored.

The metadata keys that matter.

• name. The asset's identity, following the schema.table convention (dots separate segments). It is optional — if omitted, Bruin infers it from the file path relative to assets/, so assets/analytics/orders.sql becomes analytics.orders.
• type. How the asset executes: bq.sql, sf.sql, pg.sql, duckdb.sql, python, ingestr, r, and more. The type binds the asset to a platform.
• depends. The list of upstream assets this one waits for. This list is the edge set of the DAG — Bruin runs an asset only after every asset in its depends has succeeded.
• materialization, columns, custom_checks. How the result is persisted, the typed columns with their quality checks, and any SQL-expressed custom checks — all covered in the next sections.

Why the asset is the powerful unit.

• Because metadata sits next to the code, everything about a table — its owner, its dependencies, its tests — is readable in one file and versioned in one commit.
• Because SQL and Python are both just assets, a single pipeline can mix a duckdb.sql staging model and a python enrichment job in the same DAG, with a dependency edge between them.
• Because name follows schema.table, the asset graph mirrors your warehouse layout, and Bruin's VS Code extension can render lineage straight from the depends edges.

Worked example — a Python asset that depends on a SQL asset

Detailed explanation. Real pipelines mix languages. Here a SQL asset builds a staging.players table, and a Python asset depends on it to compute something SQL is awkward at. The depends edge is what makes Bruin run the SQL asset first and the Python asset second, in one command.

Question. Define a duckdb.sql asset staging.players and a python asset analytics.player_features that depends on it, so bruin run executes them in the right order.

Input. Two files under assets/.

Code.

/* @bruin
name: staging.players
type: duckdb.sql
materialization:
  type: table
@bruin */

select 'ada' as name, 42 as rating
union all
select 'linus' as name, 51 as rating

"""@bruin
name: analytics.player_features
type: python
depends:
  - staging.players
@bruin"""

print("computing features from staging.players")
result = None  # a real asset would read the upstream table and write a result

Step-by-step explanation. The SQL file's @bruin block names it staging.players and materializes it as a table. The Python file's docstring block names it analytics.player_features, sets type: python, and lists staging.players under depends. When you run bruin run, Bruin parses both files, sees the edge staging.players → analytics.player_features, topologically sorts them, and executes the SQL asset before the Python asset. Neither file references the other's code — the depends metadata alone encodes the order.

Output.

asset	type	runs when
staging.players	duckdb.sql	first (no upstream)
analytics.player_features	python	after staging.players succeeds

Rule of thumb. One file = one asset. Put the language in type, put the order in depends, and let Bruin, not a scheduler you configure separately, decide execution order.

Bruin interview question on the asset model

Question. An interviewer gives you a folder of SQL and Python files and asks: how does Bruin know an asset's table name, which platform runs it, and what must complete before it — without any central manifest listing all of this? Show a SQL asset that answers all three, and explain where each answer comes from.

Solution Using inline YAML metadata and path-based name inference

Code.

/* @bruin
name: analytics.daily_revenue
type: bq.sql
owner: [e-mail]
depends:
  - staging.orders
  - staging.refunds
materialization:
  type: table
  strategy: create+replace
@bruin */

select
  order_date,
  sum(amount) as gross,
  sum(refund) as refunds,
  sum(amount) - sum(refund) as net
from staging.orders o
left join staging.refunds r using (order_id, order_date)
group by order_date

Step-by-step trace.

question	answer	where it comes from
table name?	analytics.daily_revenue	the name key (or the file path if omitted)
which platform?	BigQuery	type: bq.sql
what runs first?	staging.orders, staging.refunds	the depends list

1. Bruin scans every file under assets/ and parses each file's inline @bruin block — there is no central manifest; the metadata is distributed into the files themselves.
2. name: analytics.daily_revenue sets the target schema.table; had it been omitted, Bruin would have inferred it from the path (assets/analytics/daily_revenue.sql).
3. type: bq.sql binds the asset to the BigQuery connection so the body runs as BigQuery SQL.
4. Each entry in depends becomes an inbound edge, so Bruin runs staging.orders and staging.refunds before this asset and fails fast if either upstream fails.

Output:

resolved property	value
target	analytics.daily_revenue (BigQuery)
upstream edges	staging.orders, staging.refunds
execution position	after both upstreams succeed

Why this works — concept by concept:

• Inline metadata — because the YAML lives in the same file as the code, an asset is self-describing; there is no separate registry to keep in sync with the files.
• Name inference — the schema.table convention (explicit or path-derived) means the asset graph mirrors the warehouse layout without redundant configuration.
• Type binds platform — type alone routes the body to BigQuery, Snowflake, DuckDB, or Python, so one project can span engines.
• depends is the DAG — the dependency list is the single source of execution order; Bruin never guesses, and lineage is exact.
• Cost — parsing is O(assets) at startup; the DAG build is O(V + E) topological sort, negligible next to query runtime.

Pipelines
Topic — pipelines
Pipeline-design and DAG-ordering problems

Practice →

ETL Topic — etl Extract-transform-load asset problems

Practice →

---

3. Materialization: how a SELECT becomes a table

Bruin wraps your SELECT in the load strategy you choose — from full refresh to merge and SCD2

The feature that turns Bruin from "a SQL runner" into "a warehouse-loading framework" is materialization: you write a plain SELECT, and Bruin applies the DDL and DML needed to persist that result the way you asked. Say it in one breath: type decides table-or-view, strategy decides how rows hit the table. Pick wrong and you get a data-quality bug; pick right and you never hand-write a MERGE again.

The two materialization types.

• type: table. Persist the query result as a table. This is the workhorse and the one that carries a strategy.
• type: view. Persist the query as a view — no data is copied, the query re-runs on read.

The strategies that matter (all on type: table).

• create+replace (default). Overwrite the whole table with the new result every run. A full refresh — simple and correct for small tables, expensive for large ones.
• delete+insert. Incremental: needs an incremental_key; Bruin loads the query into a temp table, deletes the target rows whose key values appear in the new batch, then inserts. Good for reprocessing a partition.
• truncate+insert. Full replacement that keeps the existing table's schema, permissions, and indices — TRUNCATE then INSERT, no DROP.
• append. Only add the new rows, never overwrite — the correct choice for immutable event logs.
• merge. Upsert: mark columns with primary_key: true and Bruin updates matched rows and inserts new ones. Use update_on_merge to mark which columns get overwritten on a match, or merge_sql for a custom expression like GREATEST(target.col, source.col).
• time_interval. Incrementally load a time window; needs incremental_key and time_granularity (date or timestamp), and you pass --start-date / --end-date.
• ddl. Create an empty table from the columns definition only — no query body — when you want the structure created exactly once.
• scd2_by_column / scd2_by_time. Maintain full Slowly-Changing-Dimension history with automatic _valid_from, _valid_until, and _is_current columns, tracking changes by column diff or by a time-based incremental key.

Worked example — an incremental merge on a primary key

Detailed explanation. The most common non-trivial materialization is a merge: you have a dimension keyed by an id, rows change, and you want to update the changed ones and insert the new ones without duplicating. Bruin generates the MERGE for you from the columns metadata — you mark the key and the updatable columns and write a plain SELECT.

Question. Materialize a dim.customers table that upserts on customer_id, overwriting email and tier when a customer already exists.

Input.

customer_id	email	tier
5	ada@x	gold
6	linus@x	silver

Code.

/* @bruin
name: dim.customers
type: bq.sql
materialization:
  type: table
  strategy: merge

columns:
  - name: customer_id
    type: integer
    primary_key: true
  - name: email
    type: string
    update_on_merge: true
  - name: tier
    type: string
    update_on_merge: true
@bruin */

select customer_id, email, tier
from staging.customers

Step-by-step explanation. strategy: merge tells Bruin to generate a MERGE statement instead of overwriting the table. Bruin reads the columns block: customer_id is primary_key: true, so it becomes the match key; email and tier carry update_on_merge: true, so they are the columns overwritten with source values when a row matches. On each run, Bruin runs your SELECT as the source, matches on customer_id, updates the matched rows' email/tier, and inserts the unmatched rows. Rows deleted at the source stay in the target — merge never deletes, unlike delete+insert.

Output.

customer_id	result on next run
5 (exists)	email/tier updated in place
6 (exists)	email/tier updated in place
9 (new)	inserted

Rule of thumb. Use merge for mutable dimensions keyed by an id; use append for immutable events; reach for delete+insert only when the source truly removes rows and the target must mirror that.

Bruin interview question on tracking history

Question. Product prices change over time and the business needs to answer "what was this product's price on any past date," not just its current price. Which Bruin materialization strategy gives you that with no hand-written history SQL, and what columns does it add?

Solution Using the scd2_by_column strategy

Code.

/* @bruin
name: dim.product_catalog
type: bq.sql
materialization:
  type: table
  strategy: scd2_by_column

columns:
  - name: id
    type: integer
    primary_key: true
  - name: name
    type: string
  - name: price
    type: float
@bruin */

select id, name, price
from staging.products

Step-by-step trace.

run	id 1 price	action taken	resulting rows for id 1
1 (full refresh)	29.99	insert current version	1 current row
2	39.99	expire old, insert new	1 historical + 1 current
3	39.99 (unchanged)	no change detected	unchanged

1. strategy: scd2_by_column makes Bruin compare each incoming row against the current stored version, keyed by the primary_key column id.
2. On the first run (--full-refresh), each product is inserted as a current version with _is_current = true, _valid_from = now, and _valid_until = 9999-12-31.
3. When price changes on run 2, Bruin marks the old row _is_current = false and sets its _valid_until to the change time, then inserts a new current row with the updated price.
4. A row that disappears from the source is expired (_is_current = false); an unchanged row is left alone. You never wrote a line of history-tracking SQL.

Output:

id	price	_is_current	_valid_from	_valid_until
1	29.99	false	2024-01-01	2024-01-02
1	39.99	true	2024-01-02	9999-12-31

Why this works — concept by concept:

• scd2_by_column — a declarative strategy that detects changes in any non-key column and versions the row, so history is a config choice, not bespoke SQL.
• Reserved validity columns — _valid_from, _valid_until, and _is_current are added and maintained by Bruin, giving you point-in-time queries with a simple where _valid_from <= d and _valid_until > d.
• primary_key drives detection — the key identifies the entity across runs; changes to non-key columns trigger a new version, unchanged rows are skipped.
• Idempotent history — re-running with the same source inserts nothing new, because no change is detected — retries are safe.
• Cost — the change comparison is O(rows in the batch) against the current-version slice of the target, far cheaper than rebuilding a history table by hand.

ETL
Topic — etl
Load-mode and incremental-materialization problems

Practice →

Transform Topic — data-transformation Upsert, merge and SCD2 history problems

Practice →

---

4. Built-in & custom quality checks

Quality checks live in the asset file and run right after it — a blocking failure stops the DAG

The feature that sells Bruin to a data engineer who has been burned by silent data corruption is that tests are part of the asset, not a separate suite you hope someone wired up. You declare column-level checks and SQL-based custom checks in the same @bruin block as the query, and Bruin runs them immediately after the asset produces data. A failing check with blocking: true fails the asset and prevents its downstream from running — the bad data never propagates.

Built-in column checks.

• Existence and uniqueness. not_null (no nulls in the column) and unique (no value appears twice) — the two you attach to almost every key.
• Sign and range. positive, negative, non_negative, and min / max with a value threshold (numbers or dates) constrain the domain.
• Membership and shape. accepted_values with a value list restricts to an enum; pattern with a regex enforces a format.
• Referential integrity. relationships verifies every non-null value exists in a parent column named by the column's foreign_key metadata — a foreign-key test with no join written by you.

Custom checks — for logic the built-ins can't express.

• A custom_checks entry is a SQL query plus an expected value. Bruin runs the query and passes the check only if the returned integer equals value.
• Encode business invariants. "Row count is greater than zero," "client X has exactly 15 credits for June," "no order has a negative net after refunds" — anything you can phrase as a query returning a number.
• blocking and description. Each check can set blocking (whether a failure stops downstream) and a human description for the report.

How checks run and gate the pipeline.

• After the asset, not before. Checks execute once the asset has produced its table, then validate the produced data.
• blocking defaults to true. A blocking failure marks the asset failed and stops its downstream; set blocking: false to record the failure without blocking.
• retries. A check can retry on failure independently, resolving through the chain check → asset → pipeline.
• Run checks alone. bruin run --only checks assets/my_asset.sql re-validates without recomputing the table.

Worked example — column checks on a keyed table

Detailed explanation. The everyday quality pattern is a handful of column checks that encode what "correct" means for a table: the key is unique and never null, a count is positive, a status is one of a fixed set. You attach them in the columns block and Bruin runs them after the asset.

Question. Add checks to a dataset.player_stats table so that name is unique and not null, player_count is not null and positive, and status is one of active or inactive.

Input.

columns:
  - name: name        # unique, not null
  - name: player_count  # not null, positive
  - name: status      # accepted_values: active/inactive

Code.

/* @bruin
name: dataset.player_stats
type: duckdb.sql
materialization:
  type: table
depends:
  - dataset.players

columns:
  - name: name
    type: string
    checks:
      - name: not_null
      - name: unique
  - name: player_count
    type: integer
    checks:
      - name: not_null
      - name: positive
  - name: status
    type: string
    checks:
      - name: accepted_values
        value: [active, inactive]
@bruin */

select name, count(*) as player_count, 'active' as status
from dataset.players
group by name

Step-by-step explanation. Bruin first runs the query and materializes dataset.player_stats. Then it runs the checks column by column: not_null and unique on name, not_null and positive on player_count, and accepted_values on status. Each check is a generated SQL query counting violations; zero violations passes. Because blocking defaults to true, any failing check fails the asset and stops anything that depends on it — so a downstream dashboard model never reads a table with a duplicate key.

Output.

column	check	passes when
name	not_null, unique	no nulls, no duplicates
player_count	not_null, positive	no nulls, all > 0
status	accepted_values	every value in {active, inactive}

Rule of thumb. Attach not_null + unique to every key as a reflex; add accepted_values and positive wherever the domain is known — the checks cost one query each and catch the drift that silently corrupts downstream tables.

Bruin interview question on business-rule validation

Question. A finance table tier2.client_credits must satisfy a rule the built-in checks can't express: for June 2024, client X should have exactly 15 rows where credits_spent = 1. If it doesn't, the run must fail before any downstream asset reads the table. How do you encode that in Bruin?

Solution Using a blocking custom check

Code.

/* @bruin
name: tier2.client_credits
type: bq.sql
materialization:
  type: table

custom_checks:
  - name: Client X has 15 credits for June 2024
    description: Guards against the ACME-1234 miscount regression.
    value: 15
    blocking: true
    query: |
      SELECT count(*)
      FROM `tier2.client_credits`
      WHERE client = 'client_x'
        AND date_trunc(start_date, month) = '2024-06-01'
        AND credits_spent = 1
@bruin */

select client, start_date, credits_spent
from staging.credits

Step-by-step trace.

stage	what runs	outcome
1	asset query materializes tier2.client_credits	table built
2	custom check query runs, returns a count	e.g. 15
3	Bruin compares count to value: 15	equal → pass; not equal → fail
4	on fail with blocking: true	asset marked failed, downstream stopped

1. Bruin materializes the table first, then executes the custom_checks query against the just-written data.
2. The query returns a single integer — the number of matching credit rows for client X in June 2024.
3. Bruin passes the check only if that integer equals the declared value of 15; any other number fails.
4. Because blocking: true, a failure marks tier2.client_credits as failed and prevents every asset that depends on it from running, so the miscount never reaches a report.

Output:

returned count	check result	downstream
15	pass	runs
14 or 16	fail (blocking)	stopped

Why this works — concept by concept:

• Custom check — an arbitrary SQL query plus an expected value lets you encode any business invariant the built-in checks can't express.
• Expected-value equality — the pass condition is "query returns exactly value," which turns a fuzzy business rule into a deterministic gate.
• blocking gate — blocking: true converts a failed check into a hard stop, so corrupt data cannot flow downstream — a failed run is recoverable, a silently wrong report is not.
• Co-located with the asset — the check lives in the same file as the table it guards, so the invariant is versioned and reviewed alongside the query that must satisfy it.
• Cost — each check is one aggregate query, O(rows scanned by the query); use partitioned predicates to keep the scan cheap on large tables.

Quality
Topic — data-quality
Built-in and custom data-quality-check problems

Practice →

Validation Topic — data-validation Referential-integrity and constraint-validation problems

Practice →

---

5. Ingestion with ingestr & multi-destination DAGs

An ingestr asset lands raw data, SQL and Python transform it, and bruin run executes the whole DAG

The last piece that makes Bruin end-to-end is ingestion: instead of a separate connector platform, you declare an ingestr asset — a small YAML file — and Bruin pulls data from a source into a destination as the first node of the same DAG that transforms and tests it. ingestr is Bruin's open-source ingestion CLI, and one line of config replaces a hand-written extract job.

The ingestr asset.

• type: ingestr in a .asset.yml. No code body — just parameters naming the source and destination.
• parameters. source_connection and source_table say where data comes from; destination says where it lands. The connections themselves live in .bruin.yml.
• Any source to any destination. ingestr covers databases, SaaS APIs, and file sources, so "there is no connector for this" is rarely the blocker it is with a hand-rolled script.

The runner and its flags.

• bruin run. Execute the whole pipeline — every asset, in dependency order.
• bruin run assets/x.sql. Run a single asset (and, with --downstream, everything that depends on it).
• bruin run --tag daily. Run only assets carrying a tag — the same graph, sliced by schedule or domain.
• bruin run --full-refresh. Drop and rebuild instead of incrementally updating, for a clean reload.
• bruin run --start-date … --end-date …. Bound the window for time_interval and ingestr assets.
• bruin validate. Statically check every asset — parse the metadata, verify types and dependencies — without running anything, so CI catches a broken pipeline before it touches data.

The DAG and multi-destination pipelines.

• depends builds the graph. Bruin reads every asset's depends, assembles a DAG, and runs it in topological order — ingest first, then staging, then marts, then checks.
• One pipeline can span platforms. An ingestr asset can land data in Postgres, a duckdb.sql asset can transform it, and a bq.sql asset can publish it — Bruin routes each asset to its own connection by type.
• No external orchestrator for a single pipeline. bruin run is the scheduler for one graph; when you need cluster-scale scheduling, Bruin can deploy the same assets to Airflow.

Worked example — an ingestr asset feeding a SQL transform

Detailed explanation. The everyday ingestion pattern is a two-asset chain: an ingestr asset lands a raw table, and a SQL asset depends on it to build a clean model. Running the pipeline executes them in order without you sequencing anything by hand.

Question. Ingest a profiles table from a chess source into DuckDB, then build a dataset.player_stats aggregate that depends on it.

Input. Two files under assets/.

Code.

name: dataset.players
type: ingestr
parameters:
  destination: duckdb
  source_connection: chess-default
  source_table: profiles

/* @bruin
name: dataset.player_stats
type: duckdb.sql
materialization:
  type: table
depends:
  - dataset.players
@bruin */

select name, count(*) as player_count
from dataset.players
group by name

Step-by-step explanation. The .asset.yml file declares an ingestr asset: it reads the chess-default connection's profiles table and lands it in DuckDB as dataset.players. The SQL asset lists dataset.players under depends, so Bruin knows the ingestion must finish before the aggregate runs. bruin run parses both, orders them ingest → transform, executes ingestr to load the raw table, then runs the SELECT to build dataset.player_stats.

Output.

asset	type	produces
dataset.players	ingestr	raw profiles landed in DuckDB
dataset.player_stats	duckdb.sql	aggregated counts (runs second)

Rule of thumb. Model ingestion as an asset, not a pre-step: put the source in an ingestr .asset.yml and let every downstream model depends on it, so the whole extract-transform chain runs and validates in one bruin run.

Bruin interview question on orchestrating a mixed pipeline

Question. You have four assets — an ingestr load, a SQL staging model, a Python enrichment job, and a final SQL mart with quality checks. An interviewer asks: how does Bruin run these in the correct order across two platforms in one command, and how would you verify the graph is valid before it touches production data?

Solution Using depends-driven topological execution and bruin validate

Code.

/* @bruin
name: mart.daily_active
type: bq.sql
materialization:
  type: table
  strategy: merge
depends:
  - raw.events          # ingestr asset
  - staging.sessions    # duckdb.sql asset
  - features.user_score # python asset

columns:
  - name: user_id
    type: integer
    primary_key: true
    checks:
      - name: not_null
      - name: unique
@bruin */

select user_id, active_date, score
from staging.sessions s
join features.user_score f using (user_id)

Step-by-step trace.

asset	type	depends on	run position
raw.events	ingestr	—	1
staging.sessions	duckdb.sql	raw.events	2
features.user_score	python	staging.sessions	3
mart.daily_active	bq.sql	all three	4, then checks

1. Bruin parses all four assets and builds the DAG from their depends edges: ingest → staging → features → mart.
2. bruin validate first runs statically — it parses every @bruin block, checks that referenced dependencies exist, and verifies types and column definitions, all without executing a single query.
3. On bruin run, Bruin executes in topological order, routing each asset to its platform by type (ingestr, DuckDB, Python, BigQuery) — one command, two-plus platforms.
4. After mart.daily_active materializes via merge, its not_null and unique checks on user_id run; a blocking failure would stop anything downstream of the mart.

Output:

step	command	effect
pre-flight	bruin validate	graph + metadata verified, nothing run
execute	bruin run	four assets in order, then checks

Why this works — concept by concept:

• depends-driven DAG — the union of every asset's depends list is the graph; Bruin topologically sorts it, so order is derived, never hand-maintained.
• Cross-platform routing — type sends each asset to its own connection, so a single DAG spans ingestr, DuckDB, Python, and BigQuery in one run.
• validate before run — static validation catches missing dependencies and metadata errors in CI, before any query touches production data.
• Checks as the final gate — quality checks on the mart run after materialization and block downstream on failure, closing the loop from ingest to validated output.
• Cost — the DAG sort is O(V + E); total runtime is dominated by the assets themselves, and bruin run adds no orchestrator overhead for a single pipeline.

ETL
Topic — etl
Ingestion and extract-load pipeline problems

Practice →

Pipelines Topic — pipelines DAG-orchestration and multi-platform pipeline problems

Practice →

---

Cheat sheet — Bruin recipes

Minimal SQL asset.

/* @bruin
name: dashboard.hello
type: duckdb.sql
materialization:
  type: table
@bruin */

select 1 as id

Python asset with a dependency.

"""@bruin
name: analytics.features
type: python
depends:
  - staging.players
@bruin"""

print("build features here")

ingestr asset (any source to any destination).

name: raw.profiles
type: ingestr
parameters:
  destination: bigquery
  source_connection: chess-default
  source_table: profiles

Merge / upsert materialization.

/* @bruin
name: dim.customers
type: bq.sql
materialization:
  type: table
  strategy: merge
columns:
  - name: id
    type: integer
    primary_key: true
  - name: email
    type: string
    update_on_merge: true
@bruin */

select id, email from staging.customers

Column + custom quality checks.

/* @bruin
name: dataset.orders
type: duckdb.sql
materialization:
  type: table
columns:
  - name: order_id
    type: integer
    checks:
      - name: not_null
      - name: unique
custom_checks:
  - name: table is not empty
    query: SELECT count(*) FROM dataset.orders
    value: 1
    blocking: true
@bruin */

select order_id from staging.orders

Common bruin commands.

bruin init default my-pipeline      # scaffold a project
bruin validate                      # static-check every asset, run nothing
bruin run                           # run the whole DAG in order
bruin run assets/orders.sql --downstream
bruin run --tag daily --full-refresh
bruin run --only checks assets/orders.sql

Materialization picker.

Situation	Strategy
Small, re-pullable table	create+replace
Append-only events	append
Mutable rows keyed by an id	merge + primary_key
Reprocess a partition	delete+insert + incremental_key
Full change history required	scd2_by_column / scd2_by_time

---

Frequently asked questions

What is Bruin?

Bruin is an open-source data pipeline framework — a single Go CLI plus a VS Code extension — that runs SQL transforms, Python jobs, ingestion, and data-quality checks from one project. You define assets (SQL, Python, or YAML files) whose metadata lives in an inline @bruin comment block, declare dependencies with depends, and run the whole DAG with bruin run. It is a tool you install and version in your repo, not a hosted platform.

How is Bruin different from dbt?

dbt is transform-only: it compiles and runs SQL (and Python) models but leaves ingestion, testing infrastructure, and scheduling to other tools. Bruin covers the same SQL-modelling ground and adds native Python assets, ingestr-based ingestion, built-in column and custom quality checks, and a runner that executes the DAG without a separate orchestrator. In short, Bruin aims to be SQL + Python + ingestion + quality in one framework, where dbt is the transform layer of a larger stack.

How do Bruin quality checks work?

You declare checks in the same @bruin block as the asset. Column checks like not_null, unique, positive, accepted_values, pattern, min/max, and relationships attach to a column; custom_checks are SQL queries with an expected integer value. Checks run right after the asset materializes, and a failing check with blocking: true (the default) fails the asset and stops its downstream, so bad data never propagates. You can re-run just the checks with bruin run --only checks.

What materialization strategies does Bruin support?

For type: table, Bruin supports create+replace (full overwrite, the default), append (immutable insert), delete+insert and time_interval (incremental with an incremental_key), truncate+insert (replace data, keep the table), merge (upsert on a primary_key), ddl (create an empty typed table), and scd2_by_column / scd2_by_time for full history with _valid_from / _valid_until / _is_current columns. You write a plain SELECT and Bruin generates the DDL and DML for the strategy you chose. type: view persists the query as a view instead.

What is ingestr and how does Bruin use it?

ingestr is Bruin's open-source ingestion CLI that copies data from any supported source (databases, SaaS APIs, files) into any supported destination. In Bruin you use it by declaring an ingestr asset — a .asset.yml file with parameters naming the source_connection, source_table, and destination, with the connection secrets kept in .bruin.yml. That asset becomes the first node of your DAG, so ingestion, transformation, and quality checks all run in the same bruin run.

Do I need an orchestrator to run Bruin?

Not for a single pipeline. bruin run reads the depends edges, builds the DAG, and executes assets in topological order itself, so it is the scheduler for one graph — you can run it locally, in CI, or in a container. When you need cluster-scale scheduling across many pipelines, Bruin can deploy the same assets to Airflow, and settings like rerun_cooldown translate to Airflow retry semantics. Use bruin validate in CI to catch a broken graph before it runs.

Practice on PipeCode

Pipecode.ai is Leetcode for Data Engineering — every Bruin idea above, from the depends-driven DAG to the merge materialization and the blocking custom check, maps to a hands-on practice room where you build the pipeline against real graded inputs. PipeCode pairs each reading with 450+ DE-focused problems and a real-time scoring engine, so your answer to "how would you gate this pipeline on data quality?" holds up under a senior interviewer's depth probes.

Practice pipeline problems now →
Data-quality drills →