For 5 years I maintained 2 separate sets of transformation logic for the same business domain. Batch models in dbt on Snowflake for the warehouse. A completely separate Flink application in Java for the streaming layer. Same business rules, different languages, different testing frameworks, different deploy processes, different on-call runbooks. When something broke on the streaming side, the batch team shrugged. When the warehouse was wrong, the streaming team pointed at their own dashboards and said "looks fine to us."
That split has defined data engineering careers for the better part of a decade. And it just got smaller.
Confluent shipped a free, open-source dbt adapter for Apache Flink SQL in Q2 2026. A parallel community adapter from Xebia covers self-managed Flink. Together, they let you write streaming transformations inside the same dbt project you already use for batch. Same ref(), same DAG, same dbt test. Different engine underneath.
This matters more for your career than for your architecture.
What the dbt Adapter for Apache Flink SQL Actually Does
The core idea is straightforward: you write a dbt model in SQL, and instead of compiling to Snowflake or BigQuery, it compiles to Flink SQL and runs as a continuous query against Kafka topics.
The Confluent adapter supports 3 materialization types. view creates a virtual table over a Kafka topic. streaming_table runs a continuous INSERT INTO...SELECT that writes results to a new topic with changelog semantics. streaming_source defines a connector-backed source; basically your Kafka topic declaration.
ref() works identically to batch dbt. The adapter resolves dependencies, builds the DAG, and deploys models in topological order so upstream tables exist before downstream consumers start reading. You can run dbt ls , select model_name+ for selective deployments. If you've used dbt on any warehouse, the workflow is familiar.
The genuinely clever piece is testing. When a query runs continuously and results are unbounded, how do you write a test with a deterministic pass/fail? You can't just SELECT COUNT(*) from an infinite stream and expect a stable answer. Confluent's adapter automatically switches to bounded execution mode during tests, using snapshot queries that read Kafka topics up to the current timestamp and return a finite result set. Unit tests use mock data on temporary tables. Your dbt test command works. It just works differently under the hood.
That said, data quality tests against live production streams are marked "coming soon" as of the Q2 2026 release. Unit tests with mocked inputs, yes. Asserting not_null on a stream that's been running for 3 weeks in prod, not yet.
The Career Cost of 2 Toolchains
I've been on both sides of this hiring wall. I've interviewed candidates who were excellent batch engineers and watched them stumble the moment someone asked about watermarks. I've also interviewed streaming specialists who couldn't design a slowly changing dimension to save their lives. The industry created 2 career tracks for work that, conceptually, is the same discipline.
The numbers back this up. Data engineers with Flink or Kafka production experience earn a 15-25% salary premium over equivalent batch-focused roles, with medians around $130k. That premium exists because the supply is thin; streaming has been genuinely harder to hire for because the toolchain was completely separate. Different orchestration, different monitoring, different failure modes, different interview prep.
67% of enterprises now operate both batch and streaming pipelines. The demand side is exploding. But until this year, becoming a "streaming data engineer" meant learning an entirely separate stack on top of your existing one. Kafka, Flink APIs, custom deployment scripts, bespoke testing harnesses. 2 to 3 months of ramp-up time after you already knew batch fundamentals.
dbt extending to Flink compresses that ramp. You still need to understand watermarks, windowing, state TTL, and event-time semantics. Those are concepts, and concepts are what actually matter. But you don't need to learn a new transformation framework, a new testing approach, a new CI/CD pipeline, and a new way to express lineage. You use the dbt workflow you already know and learn the streaming concepts on top of it.
Junior engineers worry about which tool to learn. Senior engineers worry about which problems to solve. Staff engineers worry about which problems to prevent. The dbt-Flink adapter is a staff-level move: it prevents the problem of maintaining 2 divergent transformation stacks.
This has interview implications too. Watermark literacy is now expected at the junior level in loops that touch streaming. 5 years ago, watermarks were specialist knowledge. In 2026, with 60% of new pipelines carrying real-time components, interviewers expect you to explain WATERMARK FOR order_time AS order_time - INTERVAL '5' SECOND and why you chose that lateness bound. The concepts haven't changed; the percentage of roles that require them has. If you want reps on windowed aggregations before your next loop, for sql interview prep, try datadriven.io where we've been expanding the streaming SQL problems to match what panels actually ask now.
Managed vs. Self-Hosted: 2 Adapters, 2 Trade-offs for Data Engineers
There are actually 2 dbt-Flink adapters, and which one you care about depends on your infrastructure.
Confluent's adapter targets Confluent Cloud exclusively. It ships with a confluent-sql Python driver (DB-API v2 compliant) that talks directly to Confluent's REST API. The upside is zero Flink cluster management; Confluent handles compute, scaling, and checkpointing. The downside is a hard lock to Kafka as your only data source. If your events live in DynamoDB, S3, or a JDBC database, they need to land in a Kafka topic first. Confluent Cloud Flink pricing runs $0.21 per CFU-hour, billed by the minute.
Xebia's adapter (originally from GetInData, released in 2023) targets self-managed Flink via the SQL Gateway. It exposes the full Flink connector ecosystem: Kafka, Elasticsearch, JDBC, S3, HDFS, Kinesis, and 50+ community connectors. You get more flexibility in exchange for operating your own Flink cluster. The trade-off is real; version 1.3.11 added session cluster lifecycle management, but the Xebia team themselves acknowledge the architecture is "lightweight and stateless" at the cost of robustness. If deployment fails between internal steps, you can lose job progress. Savepoint recovery is manual.
102 GitHub stars on the Xebia adapter after 3 years tells you something about adoption. Self-managed Flink is operationally expensive, and most teams that want streaming are gravitating toward managed offerings. But if you're already running Flink clusters and want dbt integration without a Confluent Cloud dependency, it's the only option available.
Both adapters implement bounded execution for tests. Both support ref() and DAG resolution. Both leave data quality tests on live streams incomplete. The split comes down to infrastructure.
Where the Maturity Gaps in Data Engineering Tooling Still Bite
I'd be lying if I said this was production-ready the way dbt on Snowflake is production-ready. The gaps are real, and they mostly stem from Flink SQL's architecture rather than adapter immaturity.
No snapshots. Flink SQL doesn't support MERGE or the CTE-based updates that dbt snapshots require for Type-2 slowly changing dimensions. If your warehouse workflow depends on dbt snapshot for audit trails and historical dimension tracking, that pattern doesn't exist here. You'll need a separate solution.
No incremental materialization. dbt's batch-incremental model (process only new rows since last run) doesn't map to continuous processing. You use streaming_table instead, which is a fundamentally different execution model. Existing incremental pipelines need a rewrite, not a config change.
No cluster provisioning. The adapter can't create or drop Kafka clusters. Both dbname and schema must reference pre-existing infrastructure. If you're used to Snowflake's fully declarative setup where dbt can create schemas and databases, that gap is noticeable.
10-minute session expiry on Xebia's adapter. Tables created in a Flink SQL Gateway session expire after 10 minutes by default. Run dbt test 15 minutes after dbt run and you get table-not-found errors. This kind of brittleness doesn't exist in warehouse-based dbt.
And the conceptual gap remains even when the workflow gap closes. Stateful operations in continuous streams require you to think about state TTL, changelog semantics, and event time vs. processing time ordering. These are streaming-specific concerns that batch engineers have to learn regardless of whether the command they type is still dbt run. The adapter unifies the workflow; it doesn't unify the knowledge.
As Kai Waehner at Confluent put it: "Teams should expect to work with an evolving ecosystem." That's a diplomatic way of saying it's early.
What Actually Changes
Here's my honest read. dbt running on Apache Flink is a big deal for the data engineering career path and a modest deal for architecture. Most companies still don't need streaming. Batch handles 90% of what organizations actually run. That hasn't changed.
What has changed is the cost of adding the other 10%. Previously, streaming meant a separate team, a separate stack, and a 6-figure hiring premium. Now it means learning Flink SQL concepts while keeping the dbt workflow your team already operates. The marginal cost of streaming capability dropped significantly in a single product cycle.
The Fivetran acquisition of dbt Labs (closed June 2026, combined ~$600M ARR) makes this even more interesting. The merged entity controls both ELT ingestion and SQL transformation. Adding Flink support means they're positioning to own batch and streaming transformation in one platform. That kind of consolidation has historically been great for adoption and terrible for pricing leverage, but that's a problem for next year.
For now: learn watermarks. Learn windowing. Learn event-time semantics. These are concepts that transfer across any streaming engine. The dbt adapter means you can practice them without rebuilding your entire workflow from scratch. And if you're hiring, stop splitting your DE reqs into "batch" and "streaming" tracks. The toolchain just told you those are converging.
What's your team's plan? Adopting the adapter, waiting for maturity, or still running separate batch and streaming stacks? I'm curious where people are actually landing on this.
Top comments (0)