DEV Community: allenhori

zhao-dbt-plan: a microbatch cascading time-window planner for dbt

allenhori — Thu, 13 Aug 2026 13:07:24 +0000

A few days ago I posted about zhao-cli, a breaking-change
gate for dbt (link at the bottom if you missed it). This is the second tool in the same family, built to fix a problem I kept hitting myself. I think this one is honestly the
bigger deal.

The problem

dbt's microbatch incremental strategy applies one flat --event-time-start/--event-time-end window across an entire --select. That's fine until a rolling-window model reads a wider span than its immediate upstream was just recomputed for, at which point dbt has no way to know it needs a wider batch too. A backfill to a daily model silently corrupts every downstream rolling aggregate that reads across the backfilled day, and dbt only ever re-triggers the model you actually selected.

zhao-dbt-plan: the microbatch cascading window planner

zhao-dbt-plan reads your compiled manifest, walks the DAG within whatever you --select, and computes the correct, minimal per-model expanded window -- as a plan you review, never a command it runs for you. It never executes dbt build or dbt run on your behalf; that boundary is permanent, not a v1 scope cut. You decide how to actually run the plan -- raw dbt, Dagster, Airflow, a Databricks Asset Bundle, whatever you already use.

$ zhao-dbt-plan --select tag:microbatch_demo --event-time-start 2026-07-01 --event-time-end 2026-07-01 --pretty

[layer 0] mb_daily [2026-07-01 .. 2026-07-01]
  [layer 1] mb_rolling_7d [2026-06-28 .. 2026-07-05]
    [layer 2] mb_rolling_14d [2026-06-26 .. 2026-07-06]
      [layer 3] mb_summary [2026-06-25 .. 2026-07-07]
      [layer 3] mb_wide [2026-04-07 .. 2026-07-11]
warning: mb_wide: expanded window (96 days) exceeds max_window_expansion_days (90)

One day's backfill at the top cascades into a correctly-sized, per-model window all the way down the DAG, and it flags the one model whose expansion is wide enough to be worth a second look, instead of silently recomputing (or silently under-computing) it. --html renders the same plan as an interactive report:

Live demo · Repo

How this compares

I'd rather be precise here than let anyone assume I haven't looked: this specific problem, correctly widening a downstream rolling model's window when only part of its upstream dependency was recomputed -- isn't solved by dbt's own native tooling. dbt's lookback config is single-model and self-contained; it doesn't propagate anything to downstream models at all.

SQLMesh does handle the underlying problem, natively. lookback there triggers automatic cascading restatement through the whole downstream chain, because SQLMesh owns persistent per-interval state across the DAG. That's a genuinely different, arguably more elegant mechanism
than what zhao-dbt-plan does. It's a reason to consider SQLMesh on its own merits, not a knock against it. But it only applies if you're on SQLMesh's engine; it's not something you get by adding a tool on top of dbt. I haven't found anything, official or community -- that fills this gap for dbt itself, so I can't claim it's the only thing that's ever tried; I can say it's solving a real, confirmed pain point in dbt's own microbatch design that I hit myself and couldn't find an existing answer for.

Making the plan actually apply: `zhao_utils`

There's one gap worth being upfront about: zhao-dbt-plan computes the correct window, but on its own it can't make dbt's compiled SQL actually use it, dbt's own microbatch ref() filtering can't be overridden by a project macro (confirmed against dbt-core's real behavior, not
assumed -- it's resolved outside normal macro dispatch, to build the dependency graph statically before Jinja even renders). So a plain ref() to a widened upstream still silently gets dbt's narrow, single-batch default, no error, no warning, just a rolling-window model quietly computing on too little data.

![Setup: a one-time wrapper macro in your own project, safe to blanket-replace ref with wref across your whole project since it's a no-op without meta.zhao. Effect: plain ref() reads a 1-day window and silently under-computes; wref() reads the correct 8-day window, matching the plan zhao-dbt-plan already computed]

I built a small, separately-licensed (Apache-2.0), separately-repo'd package for this:
zhao_utils -- wref() ("windowed ref"), a
drop-in ref() replacement, plus two boundary helpers, reading the exact same meta.zhao block
the planner already does. Completely optional. If you've already hand-rolled this pattern yourself (the documented dbt way to do a rolling-window read), you don't need it; zhao-dbt-plan itself works identically either way. It's for whoever's starting that pattern fresh.

Two install paths, both through GitHub, neither requiring a manually-tracked local copy:

As a real dbt package (packages.yml + dbt deps, versioned, upgradeable), macros called namespaced: {{ zhao_utils.wref('mb_daily') }}.
As a single file you copy straight into your own project's macros/ folder, no package management at all, and calls are bare by default: {{ wref('mb_daily') }}.

Both are genuinely tested, not just the primary path with a workaround bolted on: real dbt build runs against DuckDB for both install modes, plus compile-verified against dbt Fusion and a real Databricks workspace. Full details, including the exact optional/validated-argument behavior, are in the package's own README.

Built and tested honestly

Built and tested solo, as honestly as I could. I haven't been able to cover every setup alone, so if something doesn't fit yours, an issue or a PR is genuinely welcome.

Install

curl -fsSL https://raw.githubusercontent.com/allenhori/zhao-dbt-plan/master/scripts/install.sh | sh
# or: cargo install zhao-dbt-plan

zhao-dbt-plan runs as a zhao Addon too (zhao dbt-plan ...), if you'd rather invoke it through zhao-cli directly, see the Addon contract
if you want to build your own.

Licensing -- stated plainly, not buried

zhao-cli (the first tool in this family) is Apache-2.0 -- fully permissive. zhao-dbt-plan is
AGPLv3 instead -- copyleft. If you modify it or run it as part of a network service, that has
to stay open too. Not because I'm precious about it -- I just don't want someone quietly wrapping
it into a paid product without contributing back. Both are free to use today, either way; the
license difference is about what happens if you modify and redistribute, not about whether you
can use the tool.

Repos: zhao-dbt-plan · zhao_dbt_utils · zhao-cli

If you try this and it breaks on your setup, please open an issue, that's exactly the kind of real-world coverage I can't get building this alone.

zhao-cli: a free, deterministic breaking-change gate for dbt

allenhori — Sat, 08 Aug 2026 09:42:29 +0000

I kept hitting the same problem on almost every dbt project I worked on: a PR changes a column somewhere mid-DAG, and the only way to know what it actually breaks downstream is to either read the SQL by hand across every model that might reference it, or rebuild the whole project/downstream in CI and wait.

Neither scales past a DAG of any real size. So over the last few weeks, after hours, I built zhao-cli to fix that for myself , and since it turned out to actually work, I'm putting it out there.

This is a solo, weekend project. I've tested it against real dbt projects and, for the trickier calls (like whether adding a field to a STRUCT column should be treated as breaking), I went and verified it against a live Databricks workspace rather than guessing. But I haven't been able to cover every adapter/warehouse combination alone -- if you hit something that doesn't match your setup, an issue or a PR is genuinely welcome. I'd rather ship something honestly labeled
"early" than oversell it.

The problem

dbt's own state:modified comparison is syntactic -- any compiled-SQL text change counts as "modified," and everything downstream is assumed affected. Teams end up either rebuilding their whole downstream cone on every PR (slow CI), or leaning on a human reviewer to catch a removed column, a narrowed type, or a loosened join by reading SQL -- something nobody reliably does across a DAG of any real size.

zhao-cli: the breaking-change gate

zhao parses the compiled SQL itself and computes real column-level lineage between two states of your project, classifies each change against a fixed Rule catalog, and reports the exact models each change actually reaches -- never the whole DAG, never a guess.

$ zhao check --against main

Changed:
  model model.jaffle_shop.stg_customers:
    - column removed: last_name

Downstream impact:
  model model.jaffle_shop.dim_customers:
    [BREAKING] last_name removed from model model.jaffle_shop.stg_customers breaks reference via last_name (column-removed-with-active-references)

Summary: 1 model(s) changed, 1 column(s) changed, 1 breaking, 0 warning

Impacted models: dim_customers

That's the whole review, in one CI step: exactly what changed, exactly what it breaks, and exactly which models to re-validate, not a guess at the whole downstream cone.

The analysis itself is entirely local: no LLM, no account, and it never reads or sends your actual data, nothing installed in your warehouse beyond what dbt run already needs. The one place a network call happens is resolving a git-native Baseline (dbt compile/dbt deps, the same as running dbt yourself) -- pass --state with an already-compiled manifest to skip that entirely, for a genuinely zero-network-call run.

It also exports an interactive, self-contained lineage graph, click a model or column to trace exactly what depends on it and what it depends on, search, filter, all in one offline HTML file:

Live demo · Repo

How this compares

I'd rather be precise here than let anyone assume I haven't looked:

SQLMesh already does this well sqlmesh plan uses column-level lineage to classify changes as breaking or non-breaking, natively, for free. If you're already on SQLMesh, you have this.
zhao-cli exists for the much larger population of teams already on dbt who don't want to migrate platforms just to get it. It brings the same category of protection to a dbt project as it already stands.

Install

curl -fsSL https://raw.githubusercontent.com/allenhori/zhao-cli/master/scripts/install.sh | sh
# or: cargo install zhao-cli

Licensing

zhao-cli is Apache-2.0 -- fully permissive, use it however you want, no strings.

Why "zhao(曌)"

Named for the character Empress Wu Zetian invented for herself: 明 (sun and moon) over 空 (sky), "illuminating everything below." Felt like an honest fit for a tool whose entire job is showing you exactly what a change touches, instead of leaving you to trace the DAG by hand.

Repo: zhao-cli

There's a second tool in the same family, zhao-dbt-plan, tackling a sharper dbt gap around microbatch backfills, more on that in a follow-up post in a few days.

If you try this and it breaks on your setup, please open an issue, that's exactly the kind of real-world coverage I can't get building this alone.

DEV Community: allenhori

zhao-dbt-plan: a microbatch cascading time-window planner for dbt

The problem

zhao-dbt-plan: the microbatch cascading window planner

How this compares

Making the plan actually apply: zhao_utils

Built and tested honestly

Install

Licensing -- stated plainly, not buried

zhao-cli: a free, deterministic breaking-change gate for dbt

The problem

zhao-cli: the breaking-change gate

How this compares

Install

Licensing

Why "zhao(曌)"

Making the plan actually apply: `zhao_utils`