DEV Community: Estuary

Snowflake Openflow Limitations: 6 Production Trade-offs

Sourabh Gupta — Tue, 27 Oct 2026 09:53:19 +0000

Short answer: Snowflake Openflow is Snowflake's managed data integration service, built on Apache NiFi. It works in production, and Snowflake ships updates often. Teams running database CDC with it, however, report six recurring trade-offs: runtime sizing that is fixed at creation, costs spread across several billing meters, naming decisions that are hard to change later, strict rules on source schema changes, failures that need deliberate alerting, and a Gen 1 to Gen 2 transition that is only partly complete.

Trade-off	What happens	Applies to	What to do
1. Runtime sizing	CDC connectors run on one node, and a runtime's size can't be changed after creation	PostgreSQL, SQL Server, Oracle CDC	Size for peak volume; test the resize path
2. Several cost meters	Runtime compute, warehouse merges, ingestion, and telemetry are billed separately	All deployments; BYOC adds a cloud bill	Model each meter during the proof of concept
3. Naming decisions	Destination names default to source names and are set before the first snapshot	Database CDC connectors	Set the Destination Schema Pattern up front
4. Schema-change rules	Primary key and numeric precision changes stop replication for that table	PostgreSQL, SQL Server	Plan source DDL; keep a re-snapshot runbook
5. Failure handling	Failed tables and retry storms need configured alerts to surface	CDC connectors	Set up staleness alerts before go-live
6. Gen 1 to Gen 2 transition	Gen 2 runtimes are GA; Gen 2 connectors are in preview for two sources	All new deployments	Check which connectors are Gen 2 today

Each trade-off below states what happens and when it applies, as documented on October 26, 2026. The guide ends with when Openflow is a good fit, a proof-of-concept checklist, and alternatives.

What is Snowflake Openflow?

Snowflake Openflow is Snowflake's data integration service, built on Apache NiFi. It came from Snowflake's acquisition of Datavolo, a company founded by NiFi's original creators. Openflow reached general availability on May 30, 2025, with some connectors still in preview at launch.

Openflow runs in two deployment models:

BYOC (bring your own cloud): runtimes run in your own cloud account, and you pay your cloud provider for the infrastructure in addition to Snowflake.
Snowflake Deployment: runtimes run on Snowpark Container Services inside your Snowflake account. This option entered preview on September 17, 2025.

Openflow also has two generations. Gen 1 objects are managed mainly through the NiFi canvas. Gen 2 turns deployments, runtimes, and connectors into Snowflake SQL objects with standard role-based access control. Gen 2 deployment and runtime objects became generally available on AWS and Azure on September 8, 2026 and on GCP on September 11. Gen 2 connector configuration, including the setup wizard and the PostgreSQL and MySQL/MariaDB CDC connectors, remains in public preview.

Openflow's packaged connectors focus on database CDC (PostgreSQL, MySQL, SQL Server, Oracle, MongoDB) and SaaS sources such as Salesforce. Most limitations below come from that CDC use case, because it is where most teams evaluate Openflow.

Limitation 1: Runtime sizing is fixed, and CDC connectors don't scale out

Openflow's database CDC connectors run on a single node, and a runtime's size can't be changed after it is created. Capacity planning therefore happens up front, not as volume grows.

Single node only. The PostgreSQL, SQL Server, and Oracle connectors don't support multi-node runtimes; min and max nodes must both be 1.
Medium minimum. The SQL Server and Oracle connectors require a Medium or Large runtime, with Large recommended for high throughput or wide rows.
Size is fixed at creation. To resize, you run the connector on a different runtime: either install a fresh connector, which means re-snapshotting tables, or migrate the existing connector to keep its progress.
Shared capacity. Current documentation allows several SQL Server connector instances of the same type on one runtime. Those connectors share one node's CPU and memory, so a heavy workload on one can slow the others.

What this means in practice: Scaling an Openflow CDC pipeline isn't always as simple as adding more compute nodes. For connectors restricted to single-node runtimes, teams may need to monitor resource usage, choose a larger runtime size, and adjust their deployment as workloads grow. This adds capacity-planning work and can make sudden increases in data volume harder to handle.

Limitation 2: Cost is spread across several meters

An Openflow CDC pipeline is not one line item. Which charges apply depends on the deployment type and how telemetry is configured, but a typical database CDC pipeline incurs several of these:

Cost meter	What it charges for	Applies to
Openflow runtime compute	vCPUs used by runtimes (BYOC), or SPCS compute pool instances billed per second with a 5-minute minimum (Snowflake Deployment)	All
Control pool	A control pool node that runs alongside active runtimes' compute pools	Snowflake Deployment
Warehouse	Database CDC connectors need a warehouse for the initial snapshot and incremental merges	All CDC connectors
Ingestion	Snowpipe or Snowpipe Streaming, based on data volume	All
Telemetry	Logs and metrics sent to your event table, billed per GB at the rate in Snowflake's Service Consumption Table (source)	All
Cloud infrastructure	Compute, storage, and networking in your own cloud account	BYOC only

Even the smallest Snowflake-hosted setup runs two instances. Snowflake's example of one small runtime active for one hour totals 2 CPU_X64_S instance-hours: one control pool node and one compute node.

Cost tracking has a gap too. The OPENFLOW_USAGE_HISTORY view does not yet contain records for Snowflake Deployments, so teams on that model must assemble costs from other views.

What this means in practice: Teams need to account for more than connector runtime costs when estimating their monthly bill. Increased CDC volume, frequent warehouse merges, and additional infrastructure can all affect total spending. This makes cost forecasting more complicated, especially as new sources and pipelines are added.

Limitation 3: Naming decisions are set early and follow the source

Openflow lets you control destination schema names, but defaults come from the source, and the choice is made before the first snapshot.

Configurable schema pattern. The SQL Server, PostgreSQL, MySQL, and Oracle connectors expose a Destination Schema Pattern parameter. A fixed name sends every table to one schema; variables build names from the source database and schema.
Defaults mirror the source. By default, SQL Server uses <source_database>_<source_schema> and PostgreSQL uses the source schema name.
Table names follow the source. Snowflake's MySQL setup guide notes that destination objects are based on source schema and table names, so those names are not under your control and can change when the source changes. Snowflake also recommends a dedicated destination database per connector to avoid collisions.

What this means in practice: Teams with established Snowflake naming conventions need to plan their destination mappings before starting replication. Changing those mappings later may require additional configuration or migration work, potentially disrupting downstream queries and dashboards.

Limitation 4: Strict rules on schema changes, keys, and sources

Openflow's CDC connectors stop replicating a table when certain source changes happen. These rules are documented, but teams often discover them only in production.

Limitation	Connectors affected	Source
Primary key changes and changes to numeric precision or scale are not supported; replication stops for the affected table	PostgreSQL, SQL Server	PostgreSQL docs, SQL Server CDC docs
Tables without a supported identity key replicate INSERTs only; UPDATEs and DELETEs need a key	PostgreSQL	PostgreSQL docs
Only username and password authentication to the source	SQL Server CDC	SQL Server CDC docs
CDC unavailable on SQL Server Express and Web editions, including Web edition on AWS RDS	SQL Server CDC	SQL Server CDC docs
Tables with data beyond Snowflake's type limits are not replicated (out-of-range dates excepted)	PostgreSQL	PostgreSQL docs

SQL Server also has two separate Openflow connectors, and the choice affects what you capture. The Change Tracking connector polls for changed rows, so multiple updates to one row between polls arrive as a single change. The CDC connector captures every row-level change from the transaction log, but it requires SQL Server Agent and the Standard or Enterprise edition.

In fairness, Openflow handles common changes such as adding and dropping columns, and recent releases improved how failures surface. For example, the SQL Server CDC connector now moves tables to FAILED state on unsupported DDL instead of retrying indefinitely. Fixing a failed table, though, typically means removing it from replication and re-snapshotting it.

What this means in practice: Routine source changes, such as altering a primary key or widening a numeric column, can stop replication for a table. Coordinate DDL changes with the source database team, test them in a staging pipeline first, and keep a documented re-snapshot procedure ready for tables that fail.

Limitation 5: Failures need deliberate alerting

When something outside the connector breaks, Openflow pipelines can fail quietly. Snowflake's PostgreSQL documentation notes that the runtime canvas shows only current table status, not status changes; changes are recorded in logs.

A post-mortem published on r/snowflake shows how this can play out. The figures below are as reported by the team:

A team upgraded its Openflow runtime from Small to Medium but kept a 25-credit monthly resource monitor on the connector's warehouse.
The monitor suspended the warehouse on a Friday. The PostgreSQL CDC connector kept retrying its merges, up to its default of 10,000 retries per table.
Over roughly 39 hours, more than a million rejected queries ran up cloud services charges that the team said far exceeded the monitor's whole quota.
The tables moved to FAILED status without an alert. The team found four days of missing data on Monday, from stale dashboards.
Recovery required a full re-snapshot of every affected table, which took several hours.

The trigger was the team's own undersized resource monitor, which they acknowledged. The lesson is that retry behavior plus unconfigured alerting turned a configuration mistake into a multi-day data outage. An Openflow product manager replied that Snowflake is changing how connectors behave when the warehouse is unavailable, so check the version history for that fix before treating this as current behavior.

Openflow does provide alerts for data staleness and queue buildup through its observability dashboard. The takeaway is to configure them, and to use resource monitors that notify rather than suspend for critical CDC warehouses.

What this means in practice: Without properly configured alerts, a failed replication task may go unnoticed until someone discovers stale data in a dashboard or report. Teams need to monitor pipeline health, configure failure and freshness alerts, and establish recovery procedures to prevent small issues from becoming extended outages.

Limitation 6: NiFi complexity and two generations to manage

The most common complaint about Openflow is that it exposes Apache NiFi concepts that many data teams have never used. In Gen 1, connectors are installed and configured on a NiFi canvas of processor groups, controller services, and parameter contexts. Users across r/snowflake and r/dataengineering describe the UI as confusing, say variables must be managed processor by processor, and note that people experienced with NiFi are harder to hire than dbt or Spark engineers.

Gen 2 addresses much of this, but the transition is only partly complete:

Mixed maturity. Gen 2 deployment and runtime objects are generally available on AWS, Azure, and GCP. Gen 2 connector configuration, including the setup wizard and the PostgreSQL and MySQL/MariaDB CDC connectors, is still in public preview.
Gen 1 is being phased out. Snowflake retired Gen 1 deployment creation on September 9, 2026, and migration of existing Gen 1 deployments to Gen 2 became generally available on October 1, 2026.
Canvas-based connectors remain. Gen 2 runtimes run both Gen 1 and Gen 2 connectors. Connectors without a Gen 2 version yet, such as SQL Server and Oracle, are still configured as Gen 1 connectors.
Two sets of objects. Gen 1 and Gen 2 use different SHOW commands, so a Gen 2 runtime does not appear in Gen 1 listings. Several users hit this when testing Gen 2.
Object limits. Snowflake notes that GRANT OWNERSHIP on Gen 2 objects can break underlying functionality, and that CLONE and CREATE OR REPLACE aren't supported.

This is also Snowflake's second approach to PostgreSQL ingestion. Its earlier native connector's documentation now says Snowflake is focused on a next-generation solution, Openflow. Teams adopting Openflow today should plan for further migration work as Gen 2 matures.

What this means in practice: Teams using Gen 1 may need to learn NiFi-specific concepts to troubleshoot or customize pipelines. Moving to Gen 2 can simplify routine setup, but teams still need to understand which connectors and capabilities are supported before planning a migration. This adds evaluation and operational work during the transition.

What Openflow Gen 2 fixes

Gen 2 removes the canvas from routine setup for supported connectors and makes Openflow manageable as code. The generally available parts are deployment and runtime SQL objects. The connector-level features below are in public preview, according to Snowflake's Gen 2 release notes and the Openflow product team:

Connector configuration is versioned, so teams can commit, roll back, and promote changes across environments with Git.
Supported connectors use a guided setup wizard instead of the canvas, and Snowflake's Cortex Code assistant can configure them conversationally.
Connectors can be created and managed with SQL, which makes automation in CI/CD pipelines practical.

Some users report a smooth upgrade. The release history also shows steady work on Openflow, with updates through 2026 including SQL Server CDC reaching general availability in March 2026. Claims that Openflow has been abandoned are not supported by the release record.

Gen 2 does not change the underlying architecture, though. Runtimes, warehouse merges, and multiple cost meters remain, and Gen 2 connectors cover two sources so far.

When Openflow is a good fit

Openflow makes the most sense for teams that are fully committed to Snowflake and want ingestion inside the same platform and bill. It suits teams that:

Need high-volume database replication where per-row pricing from other tools would be expensive. Several practitioners report that Openflow costs less than Fivetran for SQL Server at scale.
Already have NiFi expertise or need custom flows, such as decryption, protobuf decoding, or unusual transformations in flight.
Must keep data movement inside their own cloud network through BYOC.
Can adopt Gen 2 and its supported connectors.

It is a weaker fit for teams that need one pipeline to feed several destinations, want sub-minute freshness without tuning warehouse merge costs, or lack the time to manage runtimes, sizing, and alerting.

How to evaluate Openflow in a proof of concept

Test these five things with your own tables before committing to any CDC tool, Openflow included:

Runtime sizing: run peak-hour volume and confirm the runtime keeps up without a resize.
Schema changes: add a column, drop a column, and change a primary key on a test table; note which ones stop replication.
Freshness vs. cost: measure end-to-end latency at two merge schedules and record warehouse credits for each.
Failure recovery: suspend the warehouse or revoke a permission, then time how long until someone is alerted and how long recovery takes.
Monthly cost: add up every meter (runtime, warehouse, ingestion, telemetry, cloud infrastructure) for a full week and project it to a month.

Openflow vs. Estuary: side-by-side

Both tools replicate databases into Snowflake with CDC, and each has real strengths. Openflow's advantage is that it lives inside Snowflake: billing through Snowflake (plus your own cloud bill on BYOC), Snowflake governance, and NiFi's flexibility for custom flows. Estuary's advantage is that there is no infrastructure to size and loading is decoupled from capture.

	Snowflake Openflow	Estuary
Where it runs	Inside your Snowflake account (Snowpark Container Services) or your own cloud (BYOC)	Estuary's managed cloud, or private and BYOC deployments
Billing	Through Snowflake credits on your existing bill; BYOC adds your cloud provider's bill	Separate vendor bill: $0.50 per GB moved plus a per-connector-instance fee; Snowflake warehouse costs still apply
Infrastructure to manage	Runtimes you size up front; database CDC connectors are single-node, and runtime size is fixed at creation	No runtimes to size
Connector setup	NiFi canvas (Gen 1); wizard or SQL (Gen 2 preview, PostgreSQL and MySQL only)	Web dashboard or `flowctl` CLI with YAML specs
Custom in-flight processing	Full NiFi processor library for custom flows, such as decryption or protobuf decoding	Derivations for real-time transformations
Freshness vs. warehouse cost	Merge schedule controls warehouse use	Sync schedule per materialization, with faster syncs on chosen hours or days
Destination schema	Configurable schema pattern; SQL Server default is `<source_db>_<source_schema>`	Schema and table set per binding
Destinations	Built for Snowflake	Snowflake plus warehouses such as BigQuery, Databricks, Redshift, and MotherDuck from the same captured data
Failure alerting	Staleness and queue alerts available through the observability dashboard; must be set up	Built-in alerts for task failures, stalled data movement, and idle tasks
How cost grows	Practitioners report it costs less than per-row tools for large SQL Server CDC	One published rate per GB moved, plus connector fees that halve after six connectors; Enterprise plans have scaled pricing. No runtime or telemetry meters to estimate.

Comparison reflects public documentation as of October 26, 2026. Both products change often.

Alternatives to Snowflake Openflow

The right alternative depends on your sources, how fresh the data must be, and whether everything must stay on the Snowflake bill. These are the tools practitioners most often compare with Openflow:

Fivetran, including HVR: mature managed connectors. Practitioners praise HVR's real-time replication metrics, while several say usage-based pricing gets expensive as volume grows.
Estuary: managed real-time CDC that decouples capture from loading, described below.
Airbyte: a large connector catalog, available open source or as a cloud service. Self-hosting means running the infrastructure yourself.
Cloud-native services (Azure Data Factory, AWS DMS): a natural fit if you already run on that cloud. Practitioner reports on their CDC reliability are mixed.

Estuary

Estuary reads changes from the source's log, such as PostgreSQL logical replication or SQL Server CDC, and stores them in durable collections. It then writes them to Snowflake through a materialization that runs on its own schedule. Capabilities that relate to the limitations above:

No runtimes to size or scale.
Per-materialization sync schedule: sync continuously, every 15 minutes, or faster only during business hours. Estuary's docs note that slower syncs reduce destination compute costs.
Built-in alerts: Data Movement Stalled, Task Failed, and Task Idle alerts can email a person, a list, or a Slack channel. By default, a task-failure alert fires after 3 failures in 8 hours.
Destination control: you choose the Snowflake schema and table for each binding, and the same captured data can feed other destinations.

Estuary is a weaker fit if you need everything on a single Snowflake bill or rely on custom NiFi flows.

Next step: run the same PostgreSQL or SQL Server tables through both tools and compare freshness, setup effort, and total cost. Estuary's free tier includes 10 GB per month and 2 connectors. See Estuary plans and start a free trial.

2x Faster MongoDB CDC: An Engineering Deep-Dive on Performance Optimization

Sourabh Gupta — Tue, 31 Mar 2026 12:29:53 +0000

Estuary’s focus on in-house crafted connectors isn’t an accident.

It’s not about keeping secrets; we’re not a black box factory and connector source code is publicly available for anyone to review. It’s about maintaining the responsibility of ownership, starting with a high-quality base product, and refining from there.

Integrations are specifically designed to work seamlessly with Estuary, providing standard customization options and converting data to standard formats with as little waste as possible. And connectors get continuous updates to keep up with API changes or finetune performance.

Our MongoDB capture connector recently received one of these upgrades: while the connector reliably got the job done, it could fall behind in high-volume enterprise use cases. This could be especially detrimental for real-time pipelines that counted on the connector’s functionality with MongoDB’s change streams—if the connector couldn’t keep up with the data coming in, downstream systems could experience delays.

For real-time native applications, even small slowdowns have an outsized impact. Consider the route change notification for a shipment that arrives just after a driver misses the turnoff. Or a triage system that doesn't capture the latest developments in its priority calculations.

It was definitely time for some optimization work.

On the case was Mahdi Dibaiee. Based in Dublin, Ireland when not on adventures around the world, Mahdi has been a Senior Software Engineer with Estuary for nearly four years. Having worked on data planes, Estuary’s flowctl CLI, and various connectors, his deep knowledge of the platform lets him flexibly pick up whatever tasks have current top priority.

This is a behind-the-scenes look at how he analyzed the existing implementation’s limitations, researched solutions, and ended up with double the speed.

The Problem with Small Documents

“Make this integration faster,” while a laudable goal, isn’t much to go on. Why were captures falling behind? What was the expected throughput rate? And how could we find specific areas to improve?

First, start with a baseline.

The MongoDB capture connector tended towards a throughput rate of 34 MB/s when working with standard-sized documents, such as those around 20 KB apiece.

To test how the connector would react under different circumstances, Mahdi tried it out against a stream of much smaller documents, each around 250 bytes.

Something concerning happened when the connector processed these small documents. The capture’s ingestion rate dropped down to a meager 6 MB/s. While it would be unlikely to find this “tiny document” use case in the wild, 6 MB/s was still far too slow.

It also uncovered a possible path forward.

“This told us that we had a large overhead-per-document,” Mahdi explained, which resulted in the abysmal slowdown.

Essentially, all document processing would include some overhead. Changing the size of processed documents acted as a lever to quickly check just how much the overhead impacted performance: smaller documents with the same amount of overhead per document led to more overall time spent on the overhead rather than on making progress.

If he could find ways to reduce that overhead, all pipelines should speed up, not just ones with tiny documents.

But where exactly did that overhead come from? To tune the MongoDB capture’s performance, some digging would be required.

The Reason Behind the Bottleneck

To get a picture of the systems involved, Mahdi profiled a particular MongoDB capture that was struggling to keep up with its load.

First up was to rule out a couple obvious answers. He checked CPU load and memory pressure on both MongoDB’s side and the capture connector’s side. Neither indicated any issues.

Next, Mahdi wanted to see where Estuary spent the most time when ingesting data from MongoDB. He set up a detailed tracing view, dividing up the time for each data fetch and marking out network and CPU activity.

The trace exposed two areas of note: one a suspiciously empty space, and one a suspiciously long process, both related to the connector call to get more documents. In total, this caused Estuary to spend around two seconds on each batch of fetched data, which isn’t quite the millisecond latency Estuary aims for.

So, what was actually happening?

Activity trace for a MongoDB capture. ~2 seconds is highlighted, showing a noticeable gap in CPU usage before a string of activity.

600ms at the beginning of this cycle corresponded to the data fetch itself. When one batch of data finished processing, the connector sent out a request over the network for more, then started working on the new batch once it arrived.

Because of this synchronous mode of operation, the connector essentially sat around waiting for half a second each time it wanted to check for new data. When working with an end-to-end real-time system, those milliseconds in the pipeline add up. Not to mention the cumulative CPU idle time when the CPU’s doing nothing much for a full quarter of the connector’s process.

There, then, was an obvious bottleneck, but the activity following the fetch was also curious. The remaining 1.4 seconds in the cycle were spent processing documents.

By itself, emitting documents and checkpoints to Estuary shouldn’t take that long. But there was one more step in the processing phase that might: decoding MongoDB’s BSON documents in the first place.

With the possibility of optimizing document processing in the mix, there were two routes forward, two avenues to improve the connector’s performance.

Why not implement both?

From Go to Rust: An Expedient Solution

The CPU’s idle time was perhaps the more straightforward fix. Mahdi immediately identified that making the connector slightly more asynchronous would keep the CPU busy and shave those 600ms off of each batch.

To do so, he modified Estuary’s MongoDB connector to pre-fetch the next batch while still processing the first. In order to preserve both ordering and load on memory, he limited the number of fetched batches to four. With a maximum of 16 MB for each MongoDB cursor batch, this would keep the connector’s memory consumption to 64 MB.

This change alone would provide a welcome performance boost, but there was still the unsatisfyingly slow document processing time to contend with. And it was a trickier problem.

To standardize data coming from and going to a variety of different systems using a variety of different document formats and data types, Estuary translates everything to JSON as an intermediary. This makes it simple to mix and match data sources and destinations, or plug in a new connector: each connector only needs to handle its specific system and translation to or from the shared language.

Estuary translates MongoDB’s BSON documents to JSON so as to then easily translate the data to any destination format.

MongoDB documents come in BSON, or Binary JSON. This modified version of JSON generally makes for efficient storage and retrieval. It also includes a handful of additional data types, such as datetime and more specific numeric types.

This sounds like it would make for a reasonably simple conversion, but Mahdi found that Estuary’s MongoDB connector spent a lot of time decoding documents with Go’s bson package. On reflection, perhaps this wasn’t much of a surprise. Go’s reflect package, which infers data types that aren’t already known, is notoriously slow and the bson package relied heavily on reflect.

Looking for alternatives, he first performed some benchmarks on Rust’s corresponding bson crate. The results were demonstrable: Rust’s version was 2x faster than Go.

Mahdi’s meticulous research also uncovered another option. Rust’s most popular serialization/deserialization crate, serde, has a serde-transcode plugin crate. This transcoder can convert documents from one format to another without any intermediary layer, cutting down on unnecessary processing steps. With this, the BSON to JSON conversion could be 3x faster than the existing Go implementation.

serde couldn’t simply be swapped in as-is. Mahdi wrapped the out-of-the-box serializer in custom logic, extending the JSON conversion and sanitizing the data. The resulting implementation fit Estuary’s specific needs while retaining the 3x performance boost.

These changes would address both bottlenecks and refurbish the MongoDB capture connector.

End Result: Supercharged MongoDB Captures

One question remained: would these improvements hold up across various scenarios? Thorough testing commenced.

Mahdi started where it all began: the tiny documents scenario. He ran the MongoDB connector on a stream of small 250-byte documents, first using the main version before switching to use the improved branch. The measly ~6 MB/s throughput rate rose to around 17.5 MB/s, tripling throughput for the small documents use case.

Mahdi graphs out throughput results for the MongoDB connector, first using the original Go implementation, followed by the Rust transcoder.

Of course, this scenario was only ever meant as a test and example, a way to define how much overhead we were seeing as the connector processed documents.

Mahdi therefore reran the test, this time using 20 KB documents, a more standard size. The original 34 MB/s rate jumped to 57 MB/s, almost doubling throughput.

The difference when using larger documents is still substantial, even if less pronounced.

This rate was much more reasonable, allowing for around 200 GB of data ingestion per hour and ensuring the Estuary connector could keep up with higher volume use cases.

What this means in practical terms is that:

Huge initial databases would get backfilled in half the time
The platform would be able to handle twice as much data in continuous CDC mode
Which also means spikes in activity would be more easily handled: instead of choking performance, real-time events would stay real-time

After review and approval, Mahdi rolled out the changes to a select set of users first so he could closely monitor affected pipelines. He would be ready to quickly revert or revise as needed if any problems arose.

With so many use cases and interactions, one minor issue did raise its head: Rust and Go handled invalid UTF-8 characters differently. With a little more customization, Mahdi updated the connector’s leniency on invalid characters to mimic the former behavior.

Other than that, the rollout was smooth sailing, with capture throughput ticking upwards across the board.

So if you recently noticed your MongoDB capture speeding up: now you know.

What’s Next?

While 200 GB an hour is a decent clip, Mahdi noted that there is still room for further improvement. The main issue now is that the connector is relatively CPU-bound. And, after all, efficiency is one of those goals that doesn’t have a specific end.

For now, though, there are new challenges to face.

To test out the capture connector’s speed yourself, try it out in Estuary. Or set up a call to discuss how the connector could fit into your particular use case.

Or if you’re simply interested in switching to Rust for faster BSON decoding in your own code, check out Mahdi’s repo on benchmarking Rust and Go or his work in Estuary’s source code.

How to Stream OLTP Data to MotherDuck in Real Time with Estuary

Sourabh Gupta — Fri, 26 Sep 2025 05:51:23 +0000

Introduction

DuckDB is quickly becoming one of the most talked about analytical databases. It is fast, lightweight, and designed to run inside your applications, often described as SQLite for analytics. While it works great on a laptop for local analysis, production workflows need something more scalable.

That is where MotherDuck comes in. MotherDuck takes the power of DuckDB and brings it to the cloud. It adds collaboration features, secure storage, and a serverless model that lets teams use DuckDB at scale without worrying about infrastructure.

In this guide, you will learn how to stream data from an OLTP system into MotherDuck using Estuary. This approach lets you run analytical queries on fresh data without putting extra load on your production database.

🎥Prefer watching instead of reading? Check out the short walkthrough below.

Why DuckDB Is Gaining Traction

DuckDB is an open source analytical database designed with a clear goal: to make complex queries fast and simple without heavy infrastructure. Instead of being a traditional client-server database, DuckDB is embedded. It runs inside the host process, which reduces overhead and makes it easy to integrate directly into applications, notebooks, or scripts.

Several features stand out:

In-process operation: Similar to SQLite, DuckDB runs where your code runs. This avoids network calls and gives you low-latency access to data.
Columnar and vectorized execution: DuckDB is optimized for analytical queries. Its execution model speeds up heavy operations such as aggregations, filtering, and joins on large tables.
Portability and extensibility: It has a very small footprint and no external dependencies. At the same time, extensions support advanced data types and file formats, including Parquet, JSON, and geospatial data.
Seamless file access: DuckDB can query local files directly without requiring an ETL pipeline. For example, you can run SQL queries on CSV or Parquet files straight from disk.
Integration with data science tools: DuckDB connects smoothly with Python, R, and Jupyter notebooks, which makes it a favorite among data scientists. Because of this balance of speed, flexibility, and simplicity, DuckDB is increasingly used as the analytical layer in modern data pipelines, as well as for ad hoc analysis by engineers and analysts.

MotherDuck: DuckDB in the Cloud

DuckDB is excellent for local analysis, but production environments often require more than a local embedded database. Teams need collaboration, security, and scalability. That is where MotherDuck comes in.

MotherDuck is a managed cloud service built on top of DuckDB. It extends the same fast and lightweight query engine into a serverless environment while adding features that make it practical for organizations:

Serverless architecture: No servers to manage and no infrastructure overhead. MotherDuck scales automatically with your workloads.
Collaboration: Share queries, results, and datasets with teammates in real time. This makes it easier for teams to work from the same source of truth.
Secure secret storage: Manage credentials and connections safely in the cloud.
Integration with pipelines: Platforms like Estuary can write directly into MotherDuck, which means your data is always fresh and ready for analysis. In practice, MotherDuck gives teams the best of both worlds: the performance and simplicity of DuckDB combined with the scalability and ease of use of a modern cloud service.

OLTP → OLAP: The Core Use Case

Most production applications run on OLTP databases such as PostgreSQL, MySQL, or MongoDB. These systems are designed for fast inserts, updates, and deletes. They keep applications responsive but are not optimized for running heavy analytical queries.

Running aggregations, joins, or reports directly on an OLTP database can:

Slow down your application performance.
Increase operational risk by adding load to your production environment.
Limit the ability of analysts and data scientists to explore data freely.

This is why organizations separate OLTP (transactional) systems from OLAP (analytical) systems. The OLTP database handles day-to-day transactions, while an OLAP database is dedicated to complex queries and reporting.

DuckDB, and by extension MotherDuck, fits perfectly as an OLAP layer. With Estuary, you can capture real-time changes from your OLTP source and stream them into MotherDuck. This way, analysts always have up-to-date data to query without touching the production database.

Setting Up Estuary with MotherDuck

In this section, we’ll walk through the process of connecting your OLTP source to MotherDuck using Estuary. The setup is straightforward and only takes a few steps.

Step 1: Prepare Your Source in Estuary

Before you can send data to MotherDuck, you need a source system connected in Estuary. A source could be any OLTP database such as PostgreSQL, MySQL, or MongoDB. Estuary also supports SaaS applications, event streams, and file-based sources.

To prepare a source:

Go to the Captures tab in the Estuary dashboard.
Create a new capture and select the connector for your source system.
Provide the connection details (for example, host, port, database name, and credentials).
Save and publish the capture.

Once this is done, Estuary begins ingesting data from your source and continuously tracks new changes. This stream of data is stored in an internal collection, which you will later connect to MotherDuck.

Tip: If you are new to Estuary, try starting with a simple dataset (like PostgreSQL or a CSV file) before moving on to production-scale sources.

Step 2: Create a MotherDuck Materialization

With your source capture running, the next step is to set up MotherDuck as the destination for your data. In Estuary, this is called a materialization.

To create one:

Go to the Destinations tab in the Estuary dashboard.
Click New Materialization.
Search for MotherDuck in the connector catalog and select it.
Give the materialization a descriptive name so you can easily identify it later.

At this point, you will see the configuration screen for the MotherDuck connector. This is where you provide the details that allow Estuary to stage data and deliver it into your MotherDuck database.

In the next step, you’ll configure AWS S3 staging, which Estuary uses as a temporary storage location for data loads.

Step 3: Configure AWS S3 Staging

The MotherDuck connector in Estuary uses an Amazon S3 bucket as a staging area. Data is first written to S3, then loaded into MotherDuck. This design ensures high reliability and scalability for large datasets.

Here’s what you need to set up:

Create or choose an S3 bucket
- Note down the bucket name and its region.
- Optionally, you can define a prefix if you want Estuary to organize staged files under a specific folder.
Set up IAM permissions
- Create or use an IAM user that has read and write access to the S3 bucket.
- Attach a policy with at least the following actions:
  - s3:PutObject
  - s3:GetObject
  - s3:ListBucket
Generate access keys
- In the AWS console, go to the IAM user’s Security Credentials tab.
- Create an access key and secret key.
- Copy these values into the Estuary dashboard when configuring the MotherDuck connector.

At this point, Estuary knows where to stage data and has the permissions needed to write into your S3 bucket.

Tip: For production, avoid using a root account. Always generate access keys from an IAM user with the least privileges necessary.

Step 4: Set Up MotherDuck

Now that AWS S3 staging is ready, it’s time to configure the MotherDuck side of the connection. This step makes sure MotherDuck can pull the staged data into your chosen database.

Generate an access token
- Log in to your MotherDuck account.
- Open the Settings menu and go to Access Tokens.
- Create a new token and copy it into the Estuary connector configuration.
Provide AWS credentials to MotherDuck
- MotherDuck needs permission to read the staged files from your S3 bucket.
- You can provide these credentials either:
a. By running SQL statements inside MotherDuck:
```
 CREATE SECRET aws_access_key '<ACCESS_KEY>';
 CREATE SECRET aws_secret_key '<SECRET_KEY>';
```
b. Or by entering them through the MotherDuck UI.
Choose a target database
- Select an existing database in your MotherDuck account, or create a new one.
- Copy its name into the Estuary configuration.
Decide on delete behavior
- Soft deletes: Mark a record as deleted but keep it in the table for historical analysis.
- Hard deletes: Remove the record entirely.
- Choose the option that best matches your analytics or compliance needs.

Step 5: Publish and Stream Data

Once your MotherDuck materialization is configured, the final step is to publish it and start the data flow.

Select your source data
- Link an entire capture (for example, your PostgreSQL database)
- Or choose specific collections you want to replicate.
Review the configuration
- Double-check that your S3 credentials, MotherDuck token, and database name are correct.
- Make sure you selected the right delete behavior (soft or hard).
Save and publish
- Click Next, then Save & Publish.
- Estuary will immediately begin streaming data from your OLTP source into MotherDuck.

From here, data updates in your source will flow continuously into your MotherDuck database. This gives you a near real-time OLAP environment for analytics, without adding load to your production system.

Step 6: Query in MotherDuck

With the connector published, your data is now flowing into MotherDuck. The final step is to start exploring it.

Open the MotherDuck dashboard and go to Notebooks.
Select the database you configured as the destination.
Run queries using DuckDB’s familiar SQL syntax.

For example, if you replicated an orders table from your OLTP database, you could analyze top customers like this:

SELECT customer_id, COUNT(*) AS order_count
FROM orders
GROUP BY customer_id
ORDER BY order_count DESC
LIMIT 10;

Wrap-Up

By combining Estuary and MotherDuck, you can build a modern pipeline that keeps analytics separate from your production workload without adding extra complexity.

Estuary captures real-time changes from your OLTP databases.
Data is staged in S3 for reliability.
MotherDuck provides a cloud-native DuckDB environment where your team can query and collaborate.

This setup is fast to configure, easy to maintain, and scales with your needs. Instead of managing batch jobs or writing custom scripts, you can focus on analysis and insights.

✅ Key Takeaways

DuckDB is lightweight and powerful for analytics, while MotherDuck brings it to the cloud for collaboration and scalability.
Estuary makes it simple to stream data from OLTP systems into MotherDuck in real time.
AWS S3 is used as a staging layer, requiring IAM permissions and credentials.
Once published, you can query fresh data in MotherDuck notebooks using DuckDB SQL.

👉 Ready to try it yourself? Explore Estuary and see how quickly you can start streaming data into MotherDuck.

2025 Data Warehouse Benchmark: What BigQuery, Snowflake, and Others Don’t Tell You

Sourabh Gupta — Thu, 17 Jul 2025 08:11:33 +0000

We Benchmark-Tested 5 Data Warehouses. Here's What Broke.

Choosing a data warehouse shouldn’t feel like a gamble — but it often is.

Marketing sites are polished. Demos are cherry-picked. Docs are full of high-level promises. But when your data team starts moving terabytes of real data, things change fast: performance bottlenecks, cost spikes, memory errors… and sometimes complete failure.

At Estuary, we help teams build real-time data pipelines that push warehouses hard — across batch and streaming. We’ve seen the consequences of choosing the wrong warehouse. So we built the benchmark we wish existed earlier.

🔍 The Estuary 2025 Data Warehouse Benchmark

We benchmarked 5 major data warehouses under real workloads:

Google BigQuery
Snowflake
Databricks
Amazon Redshift
Microsoft Fabric

We didn’t just run canned TPCH queries — we loaded over 8TB of structured + semi-structured data, then hit each platform with real-world SQL:

Joins, window functions, filters, and nesting
Query-F (“The Frankenquery”) — a deliberately brutal query that pushes limits
Full lifecycle tracking from ingest to query via Estuary Flow
Cost-to-runtime ratios with no vendor tuning or caching games

📂 Our full methodology is open source. Clone it. Run your own tests. Contribute.

🧠 What We Learned

🔵 BigQuery

Fast — especially on nested JSON
But zero cost guardrails = high bill risk
Cost-per-minute hit $15+ under some setups

⚪ Snowflake

Stable, predictable, smart scaling
Good balance of performance and cost
Strong default choice for teams who want reliability

🟨 Databricks

Great for ML workflows
SQL under load? Needs tuning
Performance quirks at scale

🟥 Redshift & 🟩 Fabric

Memory errors, long runtimes, incomplete results
Multiple queries failed or stalled for hours
Definitely not plug-and-play ready

📉 Chart: Cost vs Runtime

This graph tracks $ per minute of query runtime across warehouses and instance sizes.

Red bands = platforms that failed under load or threw memory errors.

⚙️ Rankings That Actually Matter

We scored each platform on:

Cost-efficiency 💰
Runtime performance ⚡
Scalability 📈
Reliability under pressure 🧱
Startup-friendliness 🚀
Enterprise readiness 🏢

🎯 Some platforms were efficient at small scale but crashed under growth. Others performed well but cost 10x more than peers.

📥 Get the Full Report

If you’re:

Planning a warehouse migration
Scaling analytics or ML pipelines
Comparing Snowflake vs BigQuery vs Databricks
Or just tired of guessing…

👉 Download the full benchmark report

👨‍🔬 Built by Engineers, Not Marketers

We created this benchmark at Estuary because we work with these warehouses daily. Our product — Estuary Flow — streams real-time data from sources like PostgreSQL, Kafka, MongoDB, and SaaS apps into modern warehouses.

We’ve helped teams recover from 18-month migrations and $100k+ in wasted compute. So we’re publishing what we’ve learned.

🤝 Contribute or fork the test harness here:

🔗 GitHub Repo

🌐 Estuary GitHub

💬 Join the Discussion

Have you had similar (or better?) experiences with these platforms?

Spot something we should test next?

Drop your thoughts, logs, or horror stories in the comments. We’re all ears 👇

Refresh Smarter: How Estuary’s Dataflow Reset Makes Backfills a Breeze

Sourabh Gupta — Tue, 15 Jul 2025 04:14:10 +0000

Backfills have always been a critical - but sometimes tedious - part of managing robust data pipelines. Whether you're dealing with schema drift, outdated destination tables, or bad source data, initiating a full reset of your pipeline used to require multiple steps.

Not anymore.

With Estuary’s new Dataflow Reset feature, you can perform a clean-sweep backfill in just one step - reloading your sources, refreshing schemas, re-triggering derivations, and updating destination tables - all at once.

What Is a Dataflow Reset?

A Dataflow Reset is Estuary’s one-click solution to refresh your entire dataflow. It works from top to bottom:

Re-extracts data from the source
Re-runs all derivations
Recalculates schemas using updated data
Rebuilds destination tables

This isn't just a re-run - it's a recalibration. If your schemas previously became too broad (due to inconsistent or junk data), the reset starts fresh and reflects the true shape of your source.

When Should You Use It?

The new Dataflow Reset option is ideal for scenarios like:

Structural changes in your source system
Schema inference gone awry
Destination tables out of sync with upstream logic

Bonus: It automatically tracks which downstream resources (like materializations) need updating - no manual selection required.

How to Use It

Go to your Capture in the Estuary Flow web app.
Click Edit.
Select Backfill.
The default backfill mode will now trigger a Dataflow Reset.

That’s it - your pipeline is reset and refreshed in one action.

Prefer Fine-Grained Control?

You can still choose from advanced backfill options:

Incremental Backfill

Reprocess only the source data while keeping the existing destination intact.
Materialization-Only Backfill

Rebuild destination tables from current collection data - no need to touch the source.

These modes are perfect for more targeted recovery and testing.

Known Limitation

Avoid using Dataflow Reset with Dekaf materializations (Estuary’s Kafka-compatible interface). This combination is currently unsupported.

Learn More

Want a deeper dive into backfilling options, use cases, and caveats? Check out the Estuary docs:

👉 https://docs.estuary.dev/reference/backfilling-data/

TL;DR

Dataflow Reset is a full-pipeline refresh: source -> schema -> derivation -> destination
Automatically recalculates schema to avoid issues caused by bad or outdated data
Easy to trigger and safer than ever to run
Still supports advanced, partial backfill modes
Avoid using with Dekaf (for now)

Make your next backfill a breeze with Estuary.

How to Load Data from Amazon S3 to Snowflake in Real Time

Sourabh Gupta — Wed, 09 Jul 2025 06:39:46 +0000

Got a bunch of raw data sitting in Amazon S3 and need to get it into Snowflake for analytics — fast? You’re not alone.

Maybe it’s JSON logs, CSV exports, or event data piling up in your S3 bucket. Maybe you’ve tried batch pipelines or custom scripts but ran into delays, duplicates, or schema chaos. What you actually need is a clean, reliable way to load that S3 data to Snowflake, without spending weeks building and maintaining it.

That’s exactly what Estuary Flow is built for.

Flow makes it easy to build real-time S3 to Snowflake data pipelines with no code, no ops overhead, and no latency headaches. It connects directly to your S3 bucket, picks up new files as they arrive, and keeps your Snowflake warehouse in sync continuously.

In this walkthrough, we’ll show you how to set up an Amazon S3 to Snowflake pipeline using Estuary Flow from start to finish. You’ll go from raw files to live Snowflake tables in just a few steps.

TL;DR: If you're looking to stream data from Amazon S3 to Snowflake, you're in the right place — and Flow makes it a breeze.

Why Stream S3 Data to Snowflake in Real Time?

Let’s be honest — batch processing worked fine back when dashboards only needed to update once a day. But today, teams expect real-time answers: marketing needs up-to-the-minute campaign performance, operations teams need live inventory data, and product managers want to react to user behavior as it happens.

That’s where streaming data from S3 to Snowflake changes the game.

If you’re storing raw files — like logs, events, or exports — in Amazon S3, you’re already halfway there. The missing piece is a low-latency pipeline that gets that data into Snowflake the moment it arrives. No waiting for hourly jobs. No stale reports. Just fresh, query-ready data flowing in 24/7.

Here are a few reasons real-time sync matters:

Analytics that actually keep up – Get real-time insights instead of reading yesterday’s data.
Automation that reacts fast – Trigger workflows in Snowflake based on live S3 updates.
Simplified ops – Eliminate brittle scripts, manual backfills, and sync delays.

Note: Since Amazon S3 doesn’t support native change notifications, Flow polls your bucket every few minutes to detect new files, then streams them to Snowflake immediately. It’s batch under the hood, but real-time in effect.

Why Use Estuary Flow Instead of Traditional ETL Tools?

If you’ve tried to move data from Amazon S3 to Snowflake before, you probably know the drill: patch together an ETL tool, deal with scheduling, wrestle with schema mismatches, and hope the job doesn’t break halfway through.

The thing is, most ETL tools were built for a different era — one where “real time” meant “hourly,” and everything ran in batches. Estuary Flow flips that on its head.

Here’s how Flow makes your S3 to Snowflake pipeline way easier:

Real-Time by Default: Flow isn’t just fast — it’s built for continuous streaming. Once you connect your S3 bucket, Flow automatically picks up new files as they land and streams the data directly into Snowflake.
No Code Required: Set up everything — capture, schema, and materialization — through a clean UI. You don’t need to write Python, wrangle Airflow, or babysit cron jobs.
Schema-Aware + Smart: Flow infers the structure of your S3 data and helps you map it to Snowflake tables. You can tighten up schemas, apply transformations, and evolve structure over time without breaking your pipeline.
Exactly-Once Delivery: No duplicates. No reprocessing. Flow uses cloud-native guarantees to ensure data lands in Snowflake exactly once, even if things get weird.
Built to Scale: Whether you're syncing a few JSON files or streaming terabytes a day, Flow scales automatically without locking you into complex infrastructure.

Estuary Flow takes the friction out of real-time data integration from S3 to Snowflake, so you can focus on using the data, not moving it.

What You Need to Get Started

You don’t need much to build an Amazon S3 to Snowflake pipeline with Estuary Flow — just a few basics:

Estuary Flow Account

Amazon S3 Bucket

This is your data source. You’ll need:

Bucket name & region
Either public access or your AWS access key + secret key
(Optional) A folder path, called a “prefix”

Snowflake Account

Your destination for the data. Make sure you have:

A database, schema, and virtual warehouse
A user with access
Your account URL + login credentials
(Optional) warehouse name and role

That’s it. With these in place, you’re ready to connect the pieces and start streaming.

Step 1: Capture Data from Amazon S3

First up, you’ll connect Estuary Flow to your S3 bucket — this step is called a capture. It’s how Flow knows where to pull your data from.

Here’s how to set it up:

Log into Estuary Flow at dashboard.estuary.dev.
Click the Sources tab and select New Capture.
Choose Amazon S3 from the list of connectors.

You’ll see a form where you enter your S3 details:

Capture name – Something like myorg/s3-orders
AWS credentials – Only needed if your bucket isn’t public
Bucket name & region – From your S3 console
Prefix (optional) – To pull from a specific folder
Match keys (optional) – For filtering files, like *.json

Once you click Next, Flow will connect to your bucket and auto-generate a schema based on your data. You’ll see a preview of your Flow collection — this acts as a live copy of your S3 data inside Flow.

Click Save and Publish to finish the capture.

Behind the scenes, Flow checks your S3 bucket on a 5-minute schedule (by default) to pick up new or updated files. This is how it delivers near-real-time sync, even though S3 itself doesn’t support streaming events.

Next, let’s connect this to Snowflake.

Step 2: Materialize to Snowflake

Now that your data is flowing into Estuary, it’s time to materialize it to Snowflake — in other words, stream it directly into a Snowflake table.

Here’s how to set it up:

After saving your S3 capture, click Materialize Collections.
Choose the Snowflake connector from the destination list.

You’ll fill out a simple form with your Snowflake details:

Materialization name – e.g., myorg/s3-to-snowflake
Account URL – Like myorg-account.snowflakecomputing.com
User + Password – A Snowflake user with the right permissions
Database & Schema – Where the table will live
Warehouse – Optional, but recommended
Role – Optional if already assigned to the user

Once Flow connects, you’ll see your captured collection (from S3) listed.

From here, you can:

Rename the output table
Enable delta updates (if you want changes applied instead of full inserts)
Use Schema Inference to map your flat S3 data into Snowflake’s tabular format

To do that:

Click the Collection tab
Select Schema Inference
Review the suggested schema → Click Apply

Finally, hit Save and Publish.

✅ That’s it — you’ve now got a fully working, real-time S3 to Snowflake pipeline. Flow will continuously sync new files from your bucket straight into your Snowflake warehouse.

What’s Next? Supercharge Your S3 to Snowflake Pipeline

You now have a fully operational, real-time pipeline from Amazon S3 to Snowflake — and it runs continuously, no scripts or schedulers required.

But that’s just the beginning.

With Estuary Flow, you can take things even further:

Add Transformations (a.k.a. Derivations)

Want to clean, filter, or join your data before it lands in Snowflake? Use derivations to apply real-time transformations using SQL or TypeScript, right inside Flow.

You can enrich JSON objects, flatten nested structures, or create entirely new views.

Plug into More Systems

Need to send the same S3 data to BigQuery, Kafka, or a dashboard tool? Just add another materialization — Flow supports multi-destination sync out of the box.

Monitor + Optimize

Use Flow’s built-in observability tools or plug into OpenMetrics to monitor throughput, schema evolution, and pipeline health in real time.

Start Streaming S3 Data to Snowflake Today

The old way — batch jobs, manual scripts, clunky ETL — just can’t keep up with today’s speed of data.

With Estuary Flow, you can:

Sync Amazon S3 to Snowflake in real time
Handle schema changes effortlessly
Scale without infrastructure headaches

Ready to go from raw files to real-time insights?

Try Estuary Flow for free and build your first streaming data pipeline today.