Backfills have always been a critical - but sometimes tedious - part of managing robust data pipelines. Whether you're dealing with schema drift, outdated destination tables, or bad source data, initiating a full reset of your pipeline used to require multiple steps.
Not anymore.
With Estuary’s new Dataflow Reset feature, you can perform a clean-sweep backfill in just one step - reloading your sources, refreshing schemas, re-triggering derivations, and updating destination tables - all at once.
What Is a Dataflow Reset?
A Dataflow Reset is Estuary’s one-click solution to refresh your entire dataflow. It works from top to bottom:
- Re-extracts data from the source
- Re-runs all derivations
- Recalculates schemas using updated data
- Rebuilds destination tables
This isn't just a re-run - it's a recalibration. If your schemas previously became too broad (due to inconsistent or junk data), the reset starts fresh and reflects the true shape of your source.
When Should You Use It?
The new Dataflow Reset option is ideal for scenarios like:
- Structural changes in your source system
- Schema inference gone awry
- Destination tables out of sync with upstream logic
Bonus: It automatically tracks which downstream resources (like materializations) need updating - no manual selection required.
How to Use It
- Go to your Capture in the Estuary Flow web app.
- Click Edit.
- Select Backfill.
- The default backfill mode will now trigger a Dataflow Reset.
That’s it - your pipeline is reset and refreshed in one action.
Prefer Fine-Grained Control?
You can still choose from advanced backfill options:
Incremental Backfill
Reprocess only the source data while keeping the existing destination intact.Materialization-Only Backfill
Rebuild destination tables from current collection data - no need to touch the source.
These modes are perfect for more targeted recovery and testing.
Known Limitation
Avoid using Dataflow Reset with Dekaf materializations (Estuary’s Kafka-compatible interface). This combination is currently unsupported.
Learn More
Want a deeper dive into backfilling options, use cases, and caveats? Check out the Estuary docs:
👉 https://docs.estuary.dev/reference/backfilling-data/
TL;DR
- Dataflow Reset is a full-pipeline refresh: source -> schema -> derivation -> destination
- Automatically recalculates schema to avoid issues caused by bad or outdated data
- Easy to trigger and safer than ever to run
- Still supports advanced, partial backfill modes
- Avoid using with Dekaf (for now)
Make your next backfill a breeze with Estuary.
Top comments (0)