DEV Community: DataDriven

SQL Ties Python at 58%: What It Means for Data Engineer Interviews

DataDriven — Thu, 22 Oct 2026 10:06:25 +0000

My first SQL interview question was about finding the second-highest salary. It's 2026, I've sat on both sides of something like 20+ loops since then, and someone is still asking it.

What did change this month is the data. The Stack Overflow 2026 Developer Survey came out October 6, and SQL passed HTML/CSS for the number 2 spot. It sits level with Python at 58%. The same survey says AI coding tools are now a daily habit for most of the people who use them, while very few people trust those tools with production.

My thesis for anyone prepping a Data Engineer loop right now: AI writes the first draft, and the offer goes to whoever can read that draft, find what's wrong with it, and keep it alive in production. That means hand-written SQL, debugging and pipeline reasoning, so that's where your practice hours should go.

What the Stack Overflow survey says about SQL and Python

Let's be precise, because the headline numbers get mangled fast.

The survey's language usage table has JavaScript at 62.0%, SQL at 58.4%, HTML/CSS at 58.2%, and Python at 58.0%. Bash/Shell comes next at 51.3%. That question had about 14,099 respondents, which is a lot smaller than the "30,000+" in the headline. The survey drew 30,903 responses in total, but each question has its own base.

SQL "passing" HTML/CSS comes down to 0.2 points, and it's ahead of Python by 0.4. On a sample that size, the margin of error is somewhere near plus or minus 0.8 points. That's a 3-way tie. Anyone telling you SQL "beat" Python is selling a blog post.

The tie is still the interesting part, though, and 2 cuts of the same data make it more interesting:

Among people with 16+ years of experience (n=5,296), SQL leads at 63.0% and Python drops to 53.2%, a 10-point gap in favor of SQL among the people who've been doing this the longest.
Among languages used for AI work, SQL comes last at 29.8% and Python leads at 39.0%.

So the veterans are writing SQL, and the AI work is mostly happening somewhere else. My read, which the survey doesn't measure, is that SQL is the language people still write by hand and the language of the people who get paged when something breaks. Neither of those is going anywhere.

I'll admit this matches what I've seen, which is the kind of thing you should be suspicious of me saying. Every company I've worked at, from the shop with the 1,500-line stored proc to the companies everyone's heard of, ran on SQL. The orchestrator changed and the warehouse vendor changed. The language the business logic lived in never did.

One more caveat before you build a study plan on this. The survey measures breadth of use. It tells you nothing about hiring demand, interview weight, or skill depth. It doesn't even track Airflow, Snowflake, Databricks, or dbt as tools. Snowflake and Databricks only show up in an LLM-provider chart, which says nothing about warehouse adoption. If someone shows you this survey as proof of what a data engineering tool stack looks like, they didn't read it.

The 20% number and the Data Engineer job nobody interviews for

The survey's AI data tables break down how people use AI across tasks (n=12,547). Generating code in a familiar area: 69%. Debugging, troubleshooting, or refactoring code: 63.8%. Deploying, operating, or troubleshooting production systems: 19.9%.

The rest of that production row is worth reading too. 36.0% don't use AI there and don't plan to. 22.0% say it isn't part of their workflow at all. 23.3% expect to use it more.

At the same time, the blog reports that 73% of respondents who use AI coding assistants or agents use them daily. Look at the base: that's 73% of people who already use AI tools, which isn't the whole profession. It's still a lot of daily use.

Put those side by side. People are happy to have AI write code all day. Debugging code in an editor sits only roughly 5 points behind generation, so diagnosis inside a file is clearly fine too. Once the problem lives in a running system with real data, permissions, and a business waiting on it, usage drops to around 1 in 5.

AI writes the first draft. A human still owns the pipeline at 3am. Interview for the second skill, because the first one is free now.

I've been saying this for years, so the survey just gives me better ammunition. The actual Data Engineer job is less "write a DAG" and more "figure out why this pipeline silently dropped 2M rows last Tuesday." I once debugged a job that had been quietly dropping 40% of records for 6 months before anyone noticed. No autocomplete would have found it. What found it was someone who understood the grain of the source table, the join that fanned out upstream, and the dedupe step that was "fixing" it by throwing away good rows.

The production side of the job has always been the job. What changed is that the coding side got cheap, so the gap between the 2 is easier to see.

Part of that 19.9% is probably about access too. Production systems sit behind permissions, and plenty of orgs restrict who can point an agent at prod. That's my guess; the survey doesn't connect those dots. Either way, a candidate who can reason through a broken production pipeline is scarcer than one who can produce a working query, and scarcity is what gets you paid.

Splitting your Data Engineer interview practice time

Before the plan, a warning about the evidence. Almost everything published about SQL interview formats at specific companies comes from prep vendors and candidate anecdotes. I went looking for employer-documented round structures and found contradictions. One guide says a given company runs 60 minutes with 4 SQL and 4 Python questions, another says 50 minutes with about 6, and nobody links to anything official. So I won't pretend to know exactly what your loop looks like. Ask your recruiter; they'll usually tell you. I'd still split the time like this.

SQL, by hand, with the AI turned off

Make this the biggest block. Even at companies that allow AI in interviews, your SQL fluency shows in how fast you spot what's wrong with the generated query. At companies that don't allow it, you're writing it cold.

What to practice:

Window functions, and why you'd pick ROW_NUMBER over RANK over DENSE_RANK. Lots of candidates know these functions exist. Far fewer can say out loud when ties matter and what each one does with them, and interviewers notice that difference right away.
Top-N per group, running totals and deduplication. These patterns cover most of what the job asks for every day.
Saying the grain out loud for every query: what does one row represent? Most bad joins I've seen in production came from someone who never asked that question.
Solving the same problem twice, once with a self-join and once with a window function. Interviewers love asking for the second version once you've given the first.

Debugging and pipeline troubleshooting

This is the second block, and the 19.9% figure is why. If production troubleshooting is the area AI touches least, it's the area where you set yourself apart.

Practice with broken things. Take a working query and break it on purpose: add a join that fans out, add a NULL to a key column, shift a timestamp across a timezone boundary. Then find the bug from the output alone. Practice talking through late-arriving data, schema drift, and the upstream team that renamed a column without telling anyone. The tools change every 18 months and these problems never do.

Python, scoped to what Data Engineers actually do

Python sits level with SQL in the survey, and you'll get a coding round. Stick to mediums. Do 50 and you'll be solid; few companies ask hards consistently. Focus on the data-shaped stuff: parsing messy files, dicts and sets for dedupe and grouping, generators for large inputs. If you want somewhere to practice for python coding questions, try datadriven; we built it for that category, and the problems look like the job and not like a puzzle book. (I've never solved a bit manipulation question and have absolutely no idea what I'm talking about with those. Still employed.)

Practice both with and without AI

Policies vary and they're changing. Canva's engineering team wrote publicly that they expect candidates to use AI tools in interviews for some roles. Simon Newton argued there that the old questions got solved instantly by AI and that real engineering is more about reading and improving code than writing it from scratch. That post is from 2025 and doesn't cover data engineering, so it's one company's opinion and not a trend. I agree with the opinion, though.

If your loop allows AI, the skill being tested is review. Can you read generated SQL and catch the missing partition, the wrong join type, the aggregate at the wrong grain? Practice exactly that: have a tool generate a solution, then audit it like it's a PR from a new hire.

The skills that still decide Data Engineering offers

I've been on hiring panels where we passed on strong candidates for the dumbest reasons. I've also been the candidate who got rejected after the first half of an onsite. That one hurt. The interview is arbitrary, and you should play the game anyway. Still, the things that move a panel from "lean hire" to "strong hire" have stayed pretty consistent, and the survey data lines up with them.

Data modeling is the core skill, and it doesn't depend on any tool. If you get the model wrong upstream, everything downstream is pain. Keep fact tables at grain; you can always aggregate up and you can never disaggregate down. AI will happily write you a query on a bad model, and it'll run fine, which is the problem.

Then there's explaining tradeoffs under pressure. I've watched people with 10 YOE get downleveled because they couldn't articulate their design decisions in the room. "Why batch and not streaming?" (Batch covers 90% of what companies actually run; most of y'all don't need streaming.) "Why a wide table and not a star schema?" (Storage costs around 2 cents a GB a month; the engineering overhead of maintaining a star schema costs a lot more than that.) Have these answers ready and back them with economics. Economics wins arguments that "best practices" lose.

For pipeline architecture, skip the generic system-design material about load balancers and reverse proxies. Panels want to hear how you'd handle idempotency, backfills and late data, and how you'd know a job dropped rows before the business did.

Production instincts are the 19.9% skill. Tell them about the time something broke and what you did about it. A story about a silent failure you caught is worth more than any tool on your resume.

I've lived through 3 waves of "data engineering is getting automated away," and I'm still debugging the same categories of problems. This survey reads like the 4th wave with better numbers. Developers handed AI the keyboard, and they kept the pager themselves. The people who get hired are the ones the business trusts with it.

What's your loop doing right now: are they letting you use AI in the SQL round, or making you write it cold?

PostHog Dropped ClickHouse for DuckDB: A Data Engineer's Take

DataDriven — Tue, 20 Oct 2026 10:07:09 +0000

I once watched a single analyst's 14-way join freeze a shared warehouse for a whole company on a Monday morning. Dashboards timed out, the CFO's report never showed up, and 3 teams spent the afternoon blaming each other.

That's why I read PostHog's June 29, 2026 engineering post closely. They pulled their managed warehouse off a multi-tenant ClickHouse cluster and gave every organization its own DuckDB instance. The instances sleep when idle, the data sits on S3 through DuckLake, and a Postgres wire endpoint lets ordinary tools connect. The headline sounds like "ClickHouse lost," and that's the wrong read. The post publishes zero benchmarks. PostHog's decision comes down to matching the engine to the shape of the workload, and that's the same reasoning a system design interview tests.

What PostHog changed in the warehouse, and what stayed on ClickHouse

The warehouse used to run on the same multi-tenant ClickHouse cluster PostHog uses for product analytics. That cluster serves people clicking around in a browser and expecting results quickly. Warehouse modeling jobs look nothing like that. In the post's words, "It's not atypical for a modeling query to kick off on a schedule and run for hours or even days."

A job that runs for days has a poor chance on a cluster where nodes restart, both on purpose and on their own. Every DE who's had a 9-hour backfill die at hour 8 because of a node rotation knows exactly how that ends. The post says that an unbounded number of those queries hitting their "very not elastic" cluster would be "basically untenable."

They also list ClickHouse complaints beyond the workload problem. ClickHouse has no cost-based query optimizer, and its S3, Delta Lake and Iceberg support matured slowly. Results weren't stable either: "We've seen ClickHouse change query results between releases, or based on which settings were enabled." HogQL has poor support in tools like dbt, and letting customers connect directly would make the multi-tenancy problems worse.

The hot takes skip the fact that ClickHouse stays. "This is how we use it for all of our analytics data and that isn't changing." The change covers the warehouse only. Hot, shared, low-latency event analytics keeps running on ClickHouse. Long-running, isolated modeling jobs move to DuckDB.

The new layout, per the managed warehouse docs:

Every organization gets a dedicated DuckDB worker. "There's no shared query queue, so a heavy query from another customer never competes with yours."
Workers spin up on demand and retire when idle.
Storage is DuckLake: Parquet files in object storage with a transactional SQL catalog on top, so storage and compute scale independently.
A control plane handles TLS, auth, and PostgreSQL-to-DuckDB syntax translation. Each warehouse gets its own hostname.

It's in beta, behind a waitlist, and the docs say the layout "may change during beta." Keep that in mind before you redesign your company's stack around a blog post.

PostHog split the stack by workload: a shared engine for hot queries and isolated engines for long jobs. Splitting by workload is the lesson. "DuckDB beat ClickHouse" is the part you can ignore.

The post has no benchmarks, and that's the interesting part

I've read a lot of vendor migration posts. Most of them open with a bar chart showing a 40x speedup on a query nobody runs. This one has no chart at all, and it doesn't publish throughput, data volumes, node counts, instance sizes, pricing, cold-start latency or the idle timeout. The only performance language is qualitative: queries come back "on the order of minutes," and jobs run "hours or even days."

Plenty of people will find that frustrating. I find it honest. PostHog never says DuckDB is faster. They say a shared, inelastic cluster can't safely absorb unbounded multi-hour jobs. It's an argument about blast radius and isolation, and you can evaluate it without a single benchmark.

The target customer is small. The post says the original warehouse was built for "a 20-person company where the CEO is still running their own SQL queries," and that its goal was to "delay the moment a company needs to hire their first data engineer." Per-org DuckDB works great when each tenant's data and jobs fit on one machine. The post never says where that limit is. If a tenant outgrows a single node, single-tenant compute means you scale up. DuckDB's own large-scale proof ran TPC-H on a 1.5 TB RAM, 192-core box, and that tells you which direction this design scales.

The famous benchmark measures the wrong phase. DuckDB's ClickBench lead came from hot runs, and a sleep-and-wake architecture lives or dies on cold starts. That's my inference; neither source says so. Still, quoting ClickBench to justify a scale-to-zero design is like judging a commute by how fast the car goes once it's on the highway.

Scale-to-zero has a bill, and the bill moves to the first query. Sleeping instances cost nothing until somebody opens a dashboard Monday morning after a quiet weekend and waits for the wake-up. A nightly modeling job won't notice the delay. An exec clicking a chart will. Snowflake has had this model for years, and its warehouse docs still bill "with a 60-second minimum each time the warehouse starts." Even "pay only when active" has a floor. PostHog doesn't publish theirs.

The ClickHouse optimizer complaint has a shelf life. ClickHouse 25.9 added automatic global join reordering, and the post doesn't say which version PostHog was running. Read it as a 2026 engineering judgment about their own deployment, not a permanent verdict on the engine.

The software underneath is moving too. Duckgres, PostHog's open-source Postgres wire server for DuckDB, now says in its README that as of October 8, 2026, PostHog is no longer developing it. The management plane moved to something called hogtower, and the README says the wire protocol and DuckDB query serving are unaffected. I wouldn't assume the open-source repo is what runs in production today.

I've been through enough architecture announcements to know the pattern: the diagram ships 6 months before the operational numbers do. What you can learn from right now is the reasoning.

How a data engineer should answer this in a system design interview

Swap the names out and this becomes a standard interview prompt: "You run analytics for thousands of customers on a shared cluster. Some customers want to run heavy scheduled transformations. Design it."

Weak candidates name a tool. "I'd use Snowflake." "I'd use a bigger ClickHouse cluster." I've sat on panels where that was the entire answer, and the candidate couldn't explain why when we pushed. We passed, and the tool had nothing to do with it: there was no reasoning to evaluate.

Here's how I'd structure it.

Start with the workload before you touch the architecture. How many tenants? What fraction sit idle most of the day? What's the query shape: sub-second dashboards, or jobs that run for 6 hours? Who gets hurt when one tenant's job eats the cluster? PostHog's whole decision falls out of these answers. Short, hot queries share well. Long, unbounded jobs don't.

Pick the tenancy model per component. AWS's SaaS Lens describes this vocabulary: silo (dedicated resources per tenant), pool (shared resources for economies of scale), and bridge (a mix, where "noisy neighbor attributes might steer a microservice to a silo model"). PostHog built a bridge: pooled ClickHouse for event analytics, a silo for warehouse compute. Saying "silo for the long jobs, pool for the hot path" in an interview sounds like you've actually been on call for this.

Don't overclaim about shared engines. "Shared clusters can't isolate tenants" is too strong, and a good interviewer will catch it. ClickHouse has workload scheduling. Its docs describe IO caps per workload, and they describe weights as a "best-effort guarantee from below (not a limit or quota from above)," with limits enforced "per host." CPU preemption is off by default. Merges and mutations aren't CPU-scheduled yet. So the defensible claim is that shared engines isolate partially and per host, and you should name exactly what they don't guarantee. A node restart killing a 2-day job is a failure-domain problem, and workload pools don't fix it.

Name the concurrency model; it often decides more than the tenancy model. DuckDB's concurrency docs say one process can read and write a native database file. Multi-process native writes go through Quack, which is "in beta stage as of DuckDB v1.5.2," and the docs recommend DuckLake on a PostgreSQL catalog for concurrent read-writes. One writer per file maps neatly onto one instance per org. That's my inference about why the design fits; PostHog doesn't say it. If you propose per-tenant DuckDB in an interview and can't say how writes behave, expect a rough follow-up.

Price the cold start out loud. You don't need real numbers. Name the variables: idle hours, wake latency, how often queries arrive, and the per-start billing floor. Then say which tenants come out ahead (bursty, mostly idle) and which lose (always-on dashboards). Interviewers score this kind of reasoning much higher than a confident number you made up.

Don't declare a winner without data; PostHog didn't, and you shouldn't either.

Plenty of loops still include a Spark coding round no matter what the company runs in production (and for pyspark interview questions, try datadriven.io, which we built for exactly that round). Treat that as a separate skill. The design round is where you show you can reason about fit, and nobody memorizes their way through that.

The data engineering skills this move points to

I've watched 3 waves of "this tool changes everything." This move teaches the same thing all of them did: the concepts carry over and the tools rotate.

Learn open table formats and catalogs. PostHog says data "lives in S3 independent of whatever's querying it, so we're not locked into DuckDB forever." That portability comes from the format. DuckLake keeps metadata in a SQL database and data in Parquet. The DuckLake FAQ says the only real limit is the catalog database's performance, which tells you where the commit point and the bottleneck sit. Ask "where does the commit happen?" about any lakehouse you touch. An "open" format is only as open as its write path, too: DuckDB reads Iceberg metadata directly in read-only mode, and writes go through a REST catalog.

Integrity moves up a layer. DuckLake doesn't support constraints, keys or indexes. If your duplicate detection relied on a primary key, that job now belongs to your pipeline. Get used to writing those checks yourself.

Embedded engines are real infrastructure now. DuckDB showed up at 3.3% usage in Stack Overflow's 2025 survey, which is small, but per-tenant embedded engines are a pattern you'll keep running into. Learn how they spill: DuckDB handles larger-than-memory GROUP BY, JOIN, ORDER BY and window operators, but its docs warn it can still run out of memory when operators combine, and list() and string_agg() don't spill at all. "It handles larger than memory" holds per operator, so check fit query by query.

Governance is part of the evaluation. DuckLabs announced in August 2026 that it's joining AWS, with DuckDB, DuckLake and Quack staying MIT-licensed under the DuckDB Foundation. Probably fine. Still, "who maintains this in 3 years" belongs in every architecture decision, and almost nobody asks it in a design review.

Cost reasoning beats speed reasoning. S3 GETs run $0.0004 per 1,000. My rough estimate is that request charges barely register next to compute uptime, so the economics of scale-to-zero come down to how many hours your instances are awake. Engineer time costs more than any of it. A team that stops getting paged for a dead 2-day job has saved more than any benchmark would show.

Concepts transfer across tools. Tenancy, isolation, failure domains, and cold-start economics will still matter when DuckDB is the old thing someone's migrating away from.

PostHog made a reasonable call for a specific customer, the 20-person company with a CEO still writing SQL, and they were upfront about the scope. The real test comes when one of those tenants grows into a 500-person company with a 3 TB fact table and a modeling job that runs past the weekend. I'd like to read that post.

If you've run per-tenant compute in production, what was the cold-start number that finally showed up on your dashboard?

dbt Core v2.0 Goes Apache 2.0: What Data Engineers Should Know

DataDriven — Thu, 15 Oct 2026 10:05:42 +0000

I've had the license on a tool I ran in production change under me more than once. Every time, the headline said "open" and the details lived on a page nobody on my team read until legal sent an email asking what exactly we'd deployed.

On June 1, 2026, Fivetran and dbt Labs closed their merger and announced that the Fusion engine runtime is now dbt Core v2.0, licensed under Apache 2.0. dbt's own licensing docs describe dbt v2 as proprietary. Both statements are accurate. They describe different artifacts, and if you build on dbt or interview for roles that use it, you need to know which is which.

My take: the open source runtime is real, and it's good news. The binary most people install is a separate thing with separate terms. None of it changes what you should study, because the concepts underneath dbt outlived every vendor that has owned the tooling around them.

What Fivetran and dbt Labs announced at close

The joint merger announcement confirms an all-stock deal, first announced October 13, 2025. George Fraser stays CEO and Tristan Handy becomes President. No valuation, price, or exchange ratio was disclosed. Back in October 2025 the companies said the combined business would be "approaching $600M" in ARR. The closing release doesn't repeat that number.

Then there's the "100,000 data teams" figure. In October 2025, Fivetran's release said "more than 80,000 data teams use dbt." In June 2026 the body of the release says the combined company supports "more than 100,000 data teams," and the boilerplate at the bottom says more than 100,000 teams use dbt. One of those is a combined count and the other is dbt-only, and the release never defines "data team" or says whether overlapping customers were deduplicated. Treat it as the company's own count. It doesn't tell you anything independent about adoption.

The product list at close:

dbt Core v2.0 (alpha): "the open sourcing of the dbt Fusion engine runtime," under Apache 2.0
dbt State (preview): a caching layer, with a claimed infrastructure cost cut of "30% or more"
dbt Wizard (beta): no license stated
Agents Schema: called "an open source standard for agentic context," with no status given in the release

There's also a subplot that gets less attention. In September 2025 Fivetran bought Tobiko Data, the company behind SQLMesh and SQLGlot, which is the main open source competitor to dbt. In March 2026 Fivetran donated SQLMesh to the Linux Foundation, which says the project "will remain vendor neutral." So the company that now owns dbt bought the biggest open alternative and then gave it to a foundation. The public record doesn't say why. If you're betting on SQLMesh as a hedge, wait until the governance details and roadmap are published.

Devin Pratt, an IDC analyst, told TechTarget that "the real test is keeping the DBT open-source community's trust through the transition." That's his opinion, and I agree with it. Trust gets built in license files and changelogs, and press releases don't count.

Apache 2.0 vs. the licensing docs: which dbt you're running

The dbt Core v2 launch post by Joel Labes and Grace Goheen says the shared runtime code, which dbt "previously committed to releasing publicly under the ELv2 license is now under the Apache 2.0 license as dbt Core." The dbt-core repo README backs that up: main contains the Apache 2.0 source of v2.0. The old dbt-fusion repo is archived. That part is clean, and it's a real improvement.

The same post also says the Fusion binary "contains some proprietary code," that dbt has "relicensed the Fusion binary to be more permissive than ELv2," and that some premium features need a free login or a paid dbt platform account.

Now the dbt licensing docs, last updated September 16, 2026: "dbt v2 is proprietary to dbt Labs, made available under the dbt Product Licensing Agreement." The Apache 2.0 distribution is called dbt OSS. dbt v1 remains fully Apache 2.0.

So in practice there are 2 distributions:

dbt OSS (dbt-oss package): the Apache 2.0 runtime. You can read, fork and ship it.
dbt (the recommended dbt package): bundles proprietary code under the Product Licensing Agreement, which bars reverse engineering or deriving source code. Local production use is free without an account. Some features sit behind a login.

The naming makes this harder to follow. Across dbt's own pages, the same lineage of software has been called Fusion, dbt Core v2, dbt v2, dbt OSS, and plain dbt. The licensing FAQ changelog shows the June 1 update "Removed ELv2 references," and a September 15 entry renamed "dbt Fusion engine" to "dbt" and "dbt Core v2" to "dbt OSS." The press release uses one vocabulary and the docs use another.

The license has moved before. In May 2025, when Fusion launched under ELv2, dbt Labs wrote, "The license for dbt Core is not changing." That was true when they wrote it. A year later the runtime moved to Apache 2.0 and the full binary became proprietary under a separate agreement. Usage rights got broader and the remaining closed code stayed closed. Licenses on the same product can change in both directions within 12 months.

"Fusion is Apache 2.0" is wrong. "The dbt v2 runtime is Apache 2.0 as dbt OSS, and the full binary is proprietary" is correct. Say the second one in your interview.

Status is the other moving target. At the merger, v2 was labeled alpha. By the September 16 dbt Summit, dbt Labs called v2 and dbt State "Now GA". dbt's upgrade docs, updated October 8, still show alpha, beta, and preview labels at the adapter level and don't give a GA date. So "GA" may depend on which warehouse you run. Check your adapter before you plan a migration around it.

This week, open your project's requirements file or lockfile and find out which package name is in it. Then find out who on your team approved it. In my experience the answer is usually "whoever set up the repo 2 years ago," and you have about the same odds of that being documented as of a 1,500-line stored proc having accurate comments.

The 30% savings claim and Agents Schema

The dbt State savings figure has been worded 3 different ways in 3 months:

June 1 press release: "reduce underlying infrastructure costs by 30% or more"
Fivetran's June blog: "more than 30% on average across production runs" of dbt-generated compute
September dbt Summit post: "an average 15-30%+ reduction in warehouse compute"

"Infrastructure costs" and "dbt-generated compute" measure different things. None of these sources publishes a sample, a baseline, or a methodology. The customer numbers that do appear vary widely: CarGurus reported a 9% compute reduction in dbt's own State post, Virgin Media O2 reported 25%, and RxBenefits reported 59%. Each of those is a single company's self-reported result.

My read is that caching state and skipping models whose inputs haven't changed is good engineering, and I've been writing hand-rolled versions of it since before dbt existed. Whether you save 9% or 59% depends on how much of your DAG was rebuilding unchanged data every night. If your whole project is incremental models with tight predicates, the savings are small. If someone scheduled a full refresh of 400 models every hour because "it was easier," you'll save a lot, and you also have a bigger problem.

Engineer time costs more than warehouse compute. A 30% cut on a small Snowflake bill won't pay for a migration that takes 2 engineers a quarter. Measure your own baseline before you trust anyone's average, including mine.

Agents Schema is the AI part of the announcement. The idea is a single AGENTS schema in your warehouse holding tables like AGENTS.DBT_MODEL and AGENTS.LOOKML_VIEW, so any agent that can run SQL can read your semantic context. The GitHub repo is MIT licensed with "Copyright (c) 2026 Fivetran Inc." while sitting under the dbt-labs org. When I looked it had 145 commits, but nobody had cut a release tag or published a spec, a versioning policy or a governance body. The managed product built on it, Fivetran Context Layer, is in preview and limited to Snowflake, BigQuery, or Databricks in North America.

Fivetran's own Context Layer docs say, "We recommend using Agent Context MCP because it can be more accurate and token-efficient than Agents Schema." So the open standard ships alongside a recommendation to use the proprietary service. An open table layout is useful, but whether you can move your context elsewhere depends on whatever engine fills those tables. Ask about that before you believe "without lock-in."

The v2 question in your next data engineer interview

I've sat on enough hiring panels to know how this will go. Someone will ask, "We're on dbt. Would you upgrade to v2?" Most candidates will either recite the press release or panic. Both of those lose.

A stronger answer starts by naming the artifact: spec, runtime, binary or hosted platform. Each one has different terms, and that one sentence shows you understand vendor risk better than most of the people interviewing you. Then separate the vendor's claims from your measurements: "The vendor says 15 to 30%. I'd run it against our own baseline on a staging environment for 2 weeks before committing."

After that, talk about exit cost. Your SQL is fairly portable, since SQLGlot is MIT licensed and handles dialect translation. Your tests, lineage, state and deployment workflow are harder to move, so that's where the lock-in is. Finally, bring it back to the model. If your grain is right and your models are idempotent, the transformation tool is replaceable. If they aren't, no license will help you.

That last point is the one I keep repeating. The topics that come up in nearly every DE interview loop are DAG dependencies, incremental logic, dialect differences across warehouses, and "what happens if this job runs twice?" Those are concepts, and every one of them applies to dbt v1, dbt v2, SQLMesh, Dataform, or a cron job someone wrote in 2014.

I did around 20 loops in a single job search once. I was never asked about a license. I was asked about modeling all the time, and some of those times I gave answers I'd rather forget. So don't spend your prep weekend memorizing Fusion feature flags. Spend it on fact table grain, slowly changing dimensions, and explaining why a model is incremental. If you want practice on the part that survives every vendor reshuffle, data warehouse design is covered on datadriven.io. We built it for that kind of prep, and it's where we send people who ask what to study after the tool news cycle settles down.

For DEs already running dbt in production, the checklist is short:

Find out which package you install, dbt-oss or dbt, and write it down.
If you redistribute dbt to anyone (consultancies, platform teams, managed services), read the Product Licensing Agreement in full. Sections 3 and 4 govern redistribution.
Treat the 30% number as a hypothesis to test.
Treat Agents Schema as an experiment until it has a release tag and a governance doc.

I've been through enough of these big-announcement cycles to know the pattern. The press release comes out in June, the naming gets cleaned up in September, and in 2 years everyone's debugging the same late-arriving data and broken upstream contracts with a different binary. I've watched the tooling turn over about every 18 months while those problems stayed put.

Which dbt package is in your production requirements file right now, and did anyone check its license before it got there?

Entry-Level Data Engineer Jobs: Stanford's 19% Gap vs BLS's 35%

DataDriven — Tue, 13 Oct 2026 10:05:42 +0000

2 numbers landed on my feed within a few weeks of each other this fall, and they look like they belong to 2 different industries.

The first says young workers in AI-exposed jobs are falling behind. The second says data science employment is about to grow 35% in a decade. If you're trying to land an entry-level data engineering job right now, you've probably seen both headlines and felt whiplash.

Here's my take after more interview loops than I'd like to admit, on both sides of the table: both numbers are mostly right, and they measure different things. Read together, they say the field is growing, the door into it got narrower, and the people getting through that door can do something specific. The rest of this piece covers what that is.

Stanford's 19% gap and entry-level data engineering

In August, Stanford's Digital Economy Lab reported that employment for workers aged 22 to 25 in highly AI-exposed occupations sits about 19% below where it would be if it had kept pace with same-age workers in less-exposed jobs. At the July 2025 data vintage, that gap was 15%. Experienced workers show no comparable gap.

That's a scary number. Before you panic, look at what it measures.

It's a counterfactual gap. The 19% compares exposed young workers to a benchmark built from their less-exposed peers. It doesn't count jobs lost. In raw levels, employment for 22 to 25 year olds in the 2 most exposed quintiles fell about 11% from November 2022 to June 2026, while the 3 least-exposed quintiles grew about 10%.
It comes from ADP payroll data, a balanced panel of 25,000 firms. Stanford says the sample represents neither the whole U.S. labor market nor every company that uses ADP.
The authors won't call it causal. In their words, "These are descriptive patterns, not causal estimates of the effect of AI."
The number has moved around. The original 2025 paper said 13%. A February 2026 note had fixed-effects estimates roughly 16%. Now it's 19%. Part of that is new data; part of it, as the authors admit, is that the estimates have become "more sensitive to specification choices."

I respect that last admission. I've spent enough years debugging pipelines to know that when the headline number changes every time someone touches the methodology, you treat it as a direction rather than a measurement.

The direction still matters, though. The release says the adjustment runs mainly through reduced hiring of young workers, not increased separations. The Dallas Fed and a Census Bureau working paper using administrative data found the same pattern: fewer people getting in, with no wave of layoffs pushing people out.

So juniors aren't losing jobs they already have. They're competing for fewer seats at the entrance, which means the thing to prepare for is the hiring process itself.

2 more caveats from Stanford's own release. The gap shrinks when you control for education. It also persists when you exclude tech firms and computer occupations entirely. That second point kills the "it's just Big Tech layoffs" explanation. Whatever's happening is broader than a few companies over-hiring in 2021.

The 19% says fewer juniors are getting hired into AI-exposed roles. It doesn't say the roles are disappearing. Plan for a narrower door into a room that keeps getting bigger.

BLS projects 35% growth for data scientists and never mentions data engineers

Now the optimistic number. The Bureau of Labor Statistics projects data scientist employment to grow 35% from 2025 to 2035, from 275,600 jobs to 371,000. Median pay was $120,230 in May 2025. They project about 24,800 openings per year.

For context, BLS projects total U.S. employment to grow 3.5% over the same period. Computer and mathematical occupations as a group grow 7.3%. Data scientists beat both by a mile.

The BLS data scientist page doesn't mention data engineers at all, either in the description or in the "similar occupations" list (which includes actuaries and survey researchers, but somehow not the people who build the tables the data scientists query). As far as the Occupational Outlook Handbook is concerned, the DE role sits somewhere between "data scientist" and "software developer," and you have to guess.

I've spent my whole career in a job title that the government's main career guide skips. Welcome to data engineering.

Do the math on the openings, too. 95,400 net new jobs over 10 years is about 9,500 a year. The other ~15,000 of those 24,800 annual openings come from replacement: people retiring, switching careers, or getting promoted out. Net growth explains only about 38% of annual openings.

That changes how you should read the headline. A big chunk of the opportunity is backfill, and backfill hiring managers want someone who can take over an existing system on week 2. Experienced people win that fight more often than juniors do.

The BLS projection also can't confirm or refute the Stanford gap, because it's an aggregate across all experience levels with no age breakdown. A field can grow 35% overall while hiring fewer 23 year olds; those 2 things coexist easily if the growth goes to people with 3+ years behind them.

Which is exactly what I'd expect, frankly. I've said for years that data engineering isn't an entry-level job. It combines business context, analytics, infrastructure, software engineering, and SRE work. Companies were already reluctant to hire juniors into that. AI handing them a plausible excuse to hire 1 senior instead of 2 juniors just made the reluctance louder.

Stack Overflow's SQL numbers and the debugging gap in AI-assisted juniors

The 2026 Stack Overflow Developer Survey is the source I find most useful here. It put SQL second among all languages at 58.4%, behind JavaScript at 62.0%, and SQL passed HTML/CSS for the number 2 spot. Among respondents with 16+ years of experience, SQL ranks first at 63.0%.

The more senior the developer, the more likely they're writing SQL. The language people kept calling "legacy" for 15 years is what the veterans use most.

My inference (the survey doesn't say this, I do): SQL demand isn't what's squeezing juniors. If anything, SQL's standing comes from experienced people who use it to understand data, debug it, and explain it. That's the skill gap. Juniors usually can write a query; the hard part is reasoning about what the query returned and why it's wrong.

The same survey shows AI usage is everywhere: 73% of developers who use AI coding tools use them daily. But trust is conditional. In Stack Overflow's May pulse survey, 60% of respondents said they block agents from making unapproved system changes. Developers let AI write code in familiar territory and keep it on a short leash near production.

Now combine that with one study I keep thinking about. Anthropic ran a randomized trial with 52 engineers, mostly junior, learning an unfamiliar Python library. The AI-assisted group averaged 50% on comprehension quizzes versus 67% for the group that coded by hand, and the biggest gap was on debugging questions. The speed advantage from AI was roughly 2 minutes and wasn't statistically significant.

Within the AI group, how people used it mattered most. Participants who delegated code generation to the AI scored below 40%, while participants who used it to ask conceptual questions scored 65% or higher.

That study measured learning a library, not interview performance, so applying it to hiring is my own reading. It lines up uncomfortably well with what I see in interviews, though. The candidate who can't explain why their join fanned out to 3x the expected rows used to be rare. Now I see a version of them every week.

The actual job is less "write a DAG" and more "figure out why this pipeline silently dropped 2M rows last Tuesday." AI can generate the DAG. It's much worse at telling you that the upstream team changed a timestamp from UTC to local time and your incremental load has been skipping a window every night since. That takes someone who understands the data, the grain, and the business.

So when a hiring manager has budget for 1 seat and AI can handle the boilerplate, they want the person who's good at the part AI is bad at. Juniors who prove they can debug and explain get hired. Juniors who can only produce code are competing with a tool that produces code for free.

The interview side has the same problem. I've argued that AI makes classic coding interviews nearly useless for evaluating skill: if a model can solve a medium problem on demand, solving it in front of me tells me very little. Some companies are adapting; Meta was reported in 2025 to be testing AI-enabled coding interviews. Most aren't adapting yet, so you'll still get the leetcode-style round and you should prepare for it. But expect the SQL and debugging rounds to carry more weight in the final decision, because those are the rounds where an interviewer still learns something a model can't fake for you.

Put junior practice time into SQL and debugging first

I did somewhere roughly 20 interview loops in a single job search once. I got rejected after the first half of an onsite. I did 8 rounds at one company, was told I'd passed, and then the headcount vanished. I'm telling you this so you'll believe me about the next part: none of the loops I lost came down to not knowing a tool. They came down to not explaining my reasoning clearly under pressure.

If you're early-career and staring at that 19%, here's where I'd put your time.

Learn SQL past the syntax. Window functions, handling NULLs in joins, deduplication when there's no clean key, grain mismatches, late-arriving data. Every interviewer I've sat with eventually asks some version of "this number looks wrong, why?" We built the sql coding questions on datadriven.io for exactly that kind of problem; they push you past "does it run" into "is it right."

Practice debugging out loud. Take a query that returns the wrong count and narrate the diagnosis as you check the grain, then duplicates on the join key, then the filters. Interviewers score the process more than the answer; a candidate who systematically narrows the problem beats one who guesses right.

Use AI, but be able to explain what it helped you build. Everybody uses it, and pretending otherwise is silly. Use it the way the high scorers in the Anthropic study did: ask it conceptual questions, then write and own the code yourself. If you can't explain every line of a project on your resume, an interviewer will find the line you can't explain.

Learn data modeling before you go shopping for tools. Which orchestrator you pick matters much less than whether you keep fact tables at grain, understand why you'd go wide and denormalized, and know what happens when a dimension changes, because that knowledge carries over to any company on any stack.

For the DS&A round, do mediums and then stop. Around 50 of them and you'll be solid, since few companies consistently ask hards. Spend the rest of your time on the SQL, debugging and modeling work above.

And a word on mindset, because the data can mess with your head. The Stanford gap is about hiring volume. It says nothing about your ceiling once you're in. Experienced workers show no comparable gap, which tells you the people who get through the door and build 2 or 3 years of real production experience are doing fine. The field is still growing, and the narrow part is the entrance.

Treat prep like a job. I'm not the smartest engineer I know; I just outworked the version of myself that wanted to quit after the 6th rejection. Give yourself a week to feel bad about the headlines, then get back to it.

The juniors who come out of this stretch well will be the ones who can open a broken query at 2am and say why it's broken. That skill paid well before AI, and I expect it to pay even better now.

So here's my question for the folks hiring right now: when you passed on a junior DE candidate this year, was it the coding, the SQL, or their inability to explain what they'd built?

TRM Labs' 400 TB Migration: Lessons for Data Engineer Interviews

DataDriven — Thu, 08 Oct 2026 10:05:38 +0000

I've been asked some version of "tell me about a large migration you led" in more interview loops than I can count. I've given good answers and I've given answers so bad that I watched the interviewer quietly stop taking notes. The bad ones all had the same problem: I told a story about moving data. The good ones covered how I proved the plan, how I found out it was wrong, and what I changed.

That's why I've been sending people the post TRM Labs published on June 30, 2026. It's a candid write-up of moving roughly 400 TB of Postgres onto StarRocks on Apache Iceberg, and it includes the slipped deadline and the latency regressions that reached customers. Almost nobody publishes that part, which makes it one of the best public case studies a Data Engineer can use to prepare for a migration question in a system design or behavioral interview.

I think interviewers barely care about your cutover. They want to see how you de-risk, how you report bad news, and whether you can back your numbers. The TRM post covers all 3, including the uncomfortable parts.

What TRM Labs migrated, and what the headline leaves out

The facts, straight from TRM's engineering post by Amit Plaha. The source was 2 Postgres clusters in a blue/green setup, around 400 TB covering 70+ blockchains. The target is StarRocks on Apache Iceberg, served through NGQF (NextGen Query Federation). The scope was 30+ data-intensive API routes, with business logic moved out of Postgres functions and into a TypeScript layer. The table that started it all is Address Transfers, which shows how a blockchain address has sent and received funds; it was the first table they ever created, and it was timing out on Postgres.

The post claims "annualized cloud savings north of USD 1.2 million." The schedule was a full quarter of proof work and about 1.5 quarters of route migration, and the target moved from July 30 to the first week of September.

That's a nice résumé bullet. Now read it the way a skeptical interviewer will, because they'll poke at exactly these spots.

Start with the target, which is a hybrid. Real-time data now lives in a managed AlloyDB deployment behind NGQF, and StarRocks-on-Iceberg serves batch. If you summarize this as "they replaced Postgres with a lakehouse," you've told an interviewer you skim. Every migration I've ever worked on ended up as a hybrid in some way. Something always stays behind, and someone always maintains it forever.

Then there's the $1.2M. The post says "a GCP contract renegotiation tied to a committed-use discount (CUD) purchase locked the cost savings to a calendar date." So part of that number comes from contract terms, and the post doesn't split it out or explain how it was calculated. That's a fine choice for an engineering blog. In an interview, an unbacked dollar figure is where the follow-up questions start.

The scale numbers also depend on what's being measured. A companion post on partitioning puts the table at over 500 billion rows and more than 42 TB of compressed Parquet, on sharded Postgres that cost over $100,000 a month in SSD storage alone. 400 TB and 42 TB can both be accurate; they describe different things. If you say "400 TB" in an interview, be ready to explain what that 400 measures.

If you can't explain where your migration's headline number came from, the interviewer will assume you didn't calculate it.

I learned this one the hard way. In one loop I said a migration "cut costs by about half," and the interviewer asked half of what, whether compute, storage or licensing, and over what period. I had no answer and didn't get the offer, which was fair enough.

TRM spent a full quarter proving Iceberg before moving a single route

This section is the one to study. Before they moved a single route, TRM's platform team "spent a full quarter de-risking the execution" and "ran 9+ experiments answering specific questions about Iceberg's behavior under TRM's query patterns." The post names the question areas:

Could the joins the APIs actually run hit single-digit-second latency?
How do partitioning and bucket sizing affect tail latency?
How much throughput does compaction cost?
Does it hold up at customer-facing concurrency?

Look at how specific those are. They test this workload on this engine at this concurrency, which is exactly the shape a pipeline-architecture interview answer should have. Skip the generic system design playbook; DEs don't care about load balancers. When someone asks you to design a migration, the strongest move is to say: "Before I commit, here are the 4 questions that would kill this plan, and here's the experiment for each one."

The best part of the proof phase is what it found. Andrew Fisher's companion post, Why We Stopped Partitioning by Time and What We Did Instead, says a daily-partitioned Iceberg table "broke at every point on our distribution." For the heaviest address keys, the daily layout touched about 25,000 files. Bucketing by address got that down to about 300.

The cause is skew. Around 97% of addresses hold 8% of the volume, and about 10,000 heavy-tail addresses hold 42%. Partitioning event data by time is everyone's default instinct, mine included. It falls apart when your read pattern is "give me everything for this one address," and a handful of addresses are monsters. Fisher's conclusion, paraphrased, is that file fan-out dominated read performance and partition granularity mattered much less.

The fix came from data modeling: match the physical layout to the access pattern, and plan for the skew that breaks averages. Iceberg happened to be the table format TRM applied it in. The same reasoning works in Delta, Hudi, BigQuery clustering, or whatever we're all migrating to in 2029.

The post-cutover numbers from the Fisher post: 7 days of production traffic, 280,000+ paginated queries, P50 around 1.0s and P95 around 2.5s, against a 3.5s P95 target. That's a clean result, measured on real traffic, against a target set ahead of time. Notice the structure, because you'll copy it in a minute.

The slip, the regressions, and your behavioral interview

Now the part most companies would edit out.

Start with the slip. "The original target was July 30; but that timeline shifted to the first week of September." In the middle of the project, the real-time layer behind NGQF moved from Postgres to managed AlloyDB, and that work landed on the same 30+ routes. Plaha writes that "the conventional answer was migrate routes twice," and argues against it: doubled customer-facing risk, real-time data pushed out by months, doubled engineering cost. So they migrated each route once and took the later date.

The post reports the AlloyDB decision and the date change together, and the author judged that absorbing the scope beat doing 2 passes. The case for that judgment is the argument Plaha lays out, since the post compares the 2 paths in reasoning and gives numbers for neither. In an interview, present it as a judgment call made under uncertainty, because that's what it was. Interviewers will trust that more than an invented counterfactual.

Then this line: "We re-baselined visibly with our CTO." I've watched people hide a slip until the deadline passed, and I've done it myself early in my career, and it never goes well. Plaha puts it better than I can: "Early bad news is actionable; late bad news is only reportable." Put that sentence in your behavioral prep and keep it there.

Then the regressions. "Our pre-migration benchmarks did not always predict real-world load and query shapes." And: "Latency regressions on specific routes reached customers before we caught them." A handful of routes also saw brief errors during cutover.

So a full quarter of careful proof still missed production query shapes. I don't count that against the team. A proof phase answers the questions you thought to ask, and production traffic asks the rest. What matters is what came next: "We have since invested in synthetic load testing and shadow-traffic tooling," plus staged rollouts behind feature flags and "automatic rollback triggers as standard practice." On top of that, "the engineers running the migration had the authority to call a rollback without escalating."

Compare that to Stripe's write-up on online migrations, where they used GitHub's Scientist to read from both the old and new tables and got alerted "as soon as a single piece of data was inconsistent in production." Stripe was checking correctness on production reads before switching. TRM's problem was latency on production routes. Those are different failure modes, and a strong migration answer accounts for both. If an interviewer asks "how would you validate this cutover?", the answer they're hoping for is dual reads plus shadow traffic at real concurrency, with rollback criteria written down before day 1.

Here's how I'd turn all of this into a behavioral answer from your own work. Don't claim TRM's story as yours. Cite it as "a public TRM write-up" if it helps you explain how you think, but your behavioral examples have to come from your own career.

Open with context in 2 sentences covering what moved, how big, and why. Have a number for scale and be able to say what it measures.
Walk through the proof, meaning the 3 or 4 questions that could have killed the plan and how you tested each one.
Own the miss, the thing that broke anyway, and your part in it. Interviewers can tell when you're blaming the upstream team without saying so.
Explain the re-baseline: who you told, when, and what changed.
End on the process change, the specific thing you do differently now. TRM's is shadow traffic and automatic rollback triggers. Yours needs to be just as concrete.

I've sat on panels where a candidate described a perfect migration with zero problems, and every interviewer in the room wrote "not credible" in their feedback. I've also seen candidates get strong hires for a story that included a customer-facing incident, because they could explain exactly why it happened and what they built so it couldn't happen again. Plaha has a line for the opposite pattern: "Re-running a broken approach with cosmetic tweaks is how most migrations stall." That applies to interview prep too.

How a data engineer should practice the migration question

Every migration interview question I've been asked boils down to 3 skills: modeling the data for the new access pattern, designing experiments that could prove you wrong, and communicating when the plan changes. The tools in the story barely matter. Swap StarRocks for Snowflake and AlloyDB for Aurora and the answer stays the same.

So here's the study plan.

Start with the modeling. Take a skewed dataset and explain out loud why time partitioning fails on a per-entity read pattern, and what you'd do about it. If you can't explain the 25,000-files-versus-300 result from first principles, start there. When I want structured practice on that side of the loop, data modeling interview prep is what datadriven.io is good for, and the bucketing-versus-partitioning tradeoff is exactly the kind of problem worth drilling until the explanation comes out cleanly.

For proof design, pick any migration you've lived through and write the 4 questions you wish you'd tested first, then the experiment for each one. That becomes your system design answer.

Know every figure in your story: the baseline, the method, and what's included. If part of your savings came from a contract renegotiation, say so before the interviewer asks.

Last, rehearse the bad news. Tell your slip story out loud. Who did you tell? How early? If the honest answer is "late," say that and explain what you do now. Plenty of people have a slip in their history; the ones who get hired can explain theirs without flinching.

I've been through enough platform migrations to know the lakehouse won't be the last one any of us run. The engine will change, the table format will change, and the vendor will rename everything twice. Skew, unpredictable production query shapes, and deadlines tied to a finance contract will still be there in 10 years.

So, the question for the comments: what's the worst thing your proof phase failed to catch, and did it come up in your next interview loop?

DuckLake 1.0 and Iceberg 1.11: What a Data Engineer Should Know

DataDriven — Tue, 06 Oct 2026 10:07:03 +0000

The first time I lost a table to a metadata problem, the data was fine. Every Parquet file was sitting right there in the bucket. The pointer to the current snapshot was wrong, and for around 4 hours the table reported zero rows. Nobody looked at the data; everybody looked at the dashboard.

That's the part of the data lakehouse most people skip when they study. They learn the format names, memorize "Iceberg supports time travel," and move on. In 2026 the argument stopped being about which table format wins. It's now about where the metadata should live: as files on object storage, or as rows in a SQL database. That's what you'll be asked to reason about in your next interview, and you can answer it if you understand how commits, snapshots and catalogs work. Vendor names won't get you there.

DuckLake 1.0 and the case for metadata in a database

DuckLake started as a manifesto. In May 2025, Mark Raasveldt and Hannes Mühleisen (the DuckDB folks) published the original DuckLake post. Its core claim is that metadata is a database problem and doesn't belong in the file system. Less than a year later, on April 13, 2026, DuckLake 1.0 shipped in DuckDB 1.5.2 with a production-ready spec, a reference implementation, and guaranteed backward compatibility.

The design is simple enough to explain on a whiteboard in 60 seconds, so practice doing exactly that. Schemas, file locations, snapshots and the transaction log all live in a catalog database: Postgres if you're self-hosting, SQLite or DuckDB locally, MotherDuck in the cloud. Data stays as Parquet on object storage. To find the files for a query, you run one SQL query with a WHERE clause. MotherDuck's architecture write-up compares that to Iceberg's 3 layers of metadata files, each a sequential read of roughly 100ms.

The headline feature is data inlining. Small inserts and deletes (the default threshold is 10 rows) skip Parquet entirely and get written straight into the catalog database. A background process flushes them to object storage later. That goes straight at the small files problem, which has hurt every streaming-into-the-lake project I've touched.

DataLakeHouse Hub put a number on that problem: streaming 100 rows per second into an Iceberg table, with 100 active partitions and 60-second commits, produces about 144,000 files a day. So you schedule compaction, compaction starts fighting with your writers, and eventually someone pages you.

DuckLake's data inlining post claims big numbers against Iceberg on a 100-second streaming workload: 105x faster inserts, 923x faster aggregation, 189x faster checkpointing. These are DuckLake's own benchmarks on a workload chosen to show off inlining, so read them as a demonstration of the design and don't treat them as an independent bake-off. MotherDuck also cites roughly 100 transactions per second for DuckLake versus about 1 TPS for file-based formats, and a 2026 research paper on SQL-backed catalogs measured median commit latency of 8.4ms against 180ms for Iceberg. The direction is consistent. The exact multipliers depend on who's running the test.

The 1.0 release also included 68 reliability and correctness PRs, sorted tables for range queries, Iceberg-compatible bucket partitioning, GEOMETRY and VARIANT types, and experimental Puffin-based deletion vectors. That last one matters, because it shows DuckLake hedging toward Iceberg V3 compatibility instead of trying to fight it.

It makes me laugh that I spent years watching the industry move metadata out of the Hive metastore, which was a relational database, and onto object storage because the database was "the bottleneck." Now the most-discussed new format puts metadata back into Postgres. I've been through enough cycles to stop being surprised; the hot new thing is usually the old thing with better engineering and a nicer logo.

The economics hold up, though. A managed Postgres instance costs less per month than one afternoon of a senior engineer babysitting compaction jobs. If your workload is lots of small writes, paying for a database to absorb them is cheap next to the engineer time it takes to clean up 144,000 files a day.

Iceberg 1.11, the V3 spec, and what changes for data engineers

Iceberg didn't sit still. Apache Iceberg 1.11.0 landed in May 2026; sources put the date anywhere from May 15 to May 27 depending on whether they mean the vote, the artifacts or the blog post. The Google Open Source announcement covers the main points:

Java 17 is the new floor. Java 11 is gone.
Spark 4.1 and Flink 2.1 are the default build targets. Spark 3.4 is deprecated and scheduled for removal in 1.12; Flink 1.19 support is already removed.
Remote scan planning lets the REST catalog server do the metadata work and stream back file scan tasks.
Built-in table encryption uses envelope encryption and supports Google KMS.

The bigger story is the V3 spec, ratified in June 2025 and now production-mature in 1.11.

Deletion vectors come first. In V2, frequent deletes generated piles of positional delete files, so readers had to merge them at query time. V3 replaces them with Roaring bitmaps attached to each data file and stored in Puffin files, Iceberg's binary blob container. Multiple deletion vectors share one physical file. AWS benchmarked this on EMR and measured 55% faster deletes, delete files shrinking from 1,801 bytes to 475 bytes, and full scans 28.5% faster. For CDC and MERGE-heavy tables, that's a real improvement.

Variant is a single column that holds values of arbitrary, evolving shape. It's the native replacement for the "dump the JSON into a string column and parse it at query time" workaround all of us have shipped at least once. Shredding (storing Variant fields in columnar form) currently only works with Parquet.

Row lineage adds _row_id and _last_updated_sequence_number, which give every row an identity and a change marker. That's useful for CDC and audit.

The V2 to V3 upgrade is one-way. The ALTER TABLE itself is metadata-only and rewrites no data, so it feels harmless. Once a table writes V3 metadata, though, any engine that only understands V2 can't read it. I've lived through a version bump that broke a reader nobody remembered existed: a quarterly finance extract running on an old cluster, owned by a person who'd left 2 years earlier. We found it when finance called.

Before you upgrade anything:

Inventory every reader and writer: every engine, every version, every scheduled job. Trace them through query logs, because the docs will be wrong.
Get Java 17 everywhere first. Iceberg didn't make this choice lightly; maintainers moved after contributors on Java 21 kept hitting build failures against modules pinned to 11. Your Spark runtime, your Flink jobs and your random ingestion service all need it.
Upgrade a table for a feature, and only for a feature. If you need deletion vectors, Variant or encryption, upgrade that table. If you don't, V2 tables keep working.
Expect governance to be the hard part. As one DataLakeHouse Hub piece put it, catalog migrations "fail rarely on the format and almost always on governance." Access control and permissions are where the weekend goes.

The catalog is where vendors are fighting now

Look at the business news this year and the pattern is hard to miss: everyone is putting a database back in the middle of the lake.

SAP is buying Dremio. The deal was announced May 4, 2026, with close expected in Q3. SAP says it's "fully committed to continuing to invest in and prioritize" Iceberg, Polaris and Arrow, and plans to build Business Data Cloud into an Iceberg-native lakehouse with an open catalog on Polaris and the Iceberg REST API. Dremio has been one of Iceberg's main stewards. When a company that size buys a core contributor, ask what happens 3 years from now, after the press release has been forgotten.

The safety net is governance. Apache Polaris graduated to a top-level Apache project on February 15, 2026, after 18 months of incubation, 6 releases and over 2,800 merged PRs. The vote passed with 27 binding +1s and no objections, and contributors include Snowflake, AWS, Google Cloud, Azure, dbt Labs, Stripe and others. With that many companies committing code, one acquirer can't take the project over without the others noticing. If you're on Dremio, the Iceberg REST Catalog API is your exit door. Keep it open.

Databricks is putting Postgres inside the lakehouse. It bought Neon for about $1 billion in May 2025, citing that over 80% of Neon databases were being created by AI agents. In August 2026, Databricks reported Lakebase above $100M revenue run-rate, growing twice as fast as its Lakehouse product. In May, Databricks also made Catalog Commits generally available for Unity Catalog managed Delta tables. That routes commits through the catalog so external engines can't write straight to storage and leave the catalog and the files disagreeing.

Snowflake did its own version. It bought Crunchy Data for $250 million and open-sourced pg_lake in November 2025, which lets Postgres create and query Iceberg tables directly.

Iceberg won broad engine support; Delta and Hudi both added Iceberg compatibility. The fight has moved one layer up, to the catalog, because whoever owns the commit owns the table. File-based metadata and database metadata are 2 answers to the same question: who gets to say what the current version of a table is, and how fast can they say it?

Formats are converging. Catalogs are where the lock-in lives now. Pick your catalog like you're going to be stuck with it, because you probably are.

Explaining table formats in a data engineering interview

Interviewing is a separate skill from the job. I've been on panels where a candidate who'd run Iceberg in production for 2 years fumbled "what does the catalog actually do?" because they'd never had to say it out loud. That's what costs people the level. They have the experience; they just can't put it into words under pressure.

These are the concepts that keep showing up, and how I'd answer each.

"What's the difference between Parquet and Iceberg?" Parquet is a file format; it describes how bytes are laid out in one file. Iceberg is a table format: snapshots, atomic commits, schema evolution, partition evolution, and the metadata that tells an engine which files make up the table right now. If you say "Iceberg is Parquet with metadata," a good interviewer will keep pushing until you fall over.

"What does a catalog do?" It answers one question: where is the current metadata for this table? A commit is an atomic swap of that pointer. Two writers race and one wins; the loser retries against the new snapshot. Every catalog design, including Polaris, Unity, DuckLake's Postgres and Glue, is a different way of making that swap fast and safe.

"What isolation do you get?" Snapshot isolation. Readers never block writers, and writers never corrupt an in-progress read. Candidates regularly call this serializable. It isn't, and saying so correctly tells a senior interviewer you've thought about concurrent writers. Bonus points for knowing Iceberg tracks snapshots through manifest lists, Delta through a sequential JSON commit log, and Hudi through a timeline.

"Why would anyone put metadata in a database?" Small, frequent writes. Under CDC or sub-second commits, the bottleneck is metadata overhead, and data I/O stops mattering. Object storage charges around 100ms per request; a database answers in milliseconds and handles concurrency well. That's the reason DuckLake exists, and the reason LSM-based designs like Paimon exist.

"Which format would you pick?" Never answer with a benchmark from a blog post. Name the axes: which engines need to read and write the table, what your team can actually operate, and how much vendor independence you need. For a DuckDB-heavy shop with a modest team, DuckLake on Postgres is operationally light. For 4 engines across 3 teams with real governance requirements, Iceberg behind a REST catalog is the default for a reason.

Then add the opinion that separates seniors from everyone else: most of y'all don't need 100 transactions per second. Batch still covers most of what companies run. The ability to stream into the lake is nice to have; making a 6-hour batch job reliable is what keeps you employed. Say that in the interview, with a reason, and you sound like someone who's carried the pager.

You don't need a vendor certification for any of this. Every concept above (atomic commits, isolation levels, write amplification, metadata lookup cost) existed before these products and will outlive them. A DuckLake catalog is literally tables in Postgres you can query, so SQL fluency pays off twice. If you need to sharpen it, for sql practice problems, DataDriven is great, and we built it as interview prep for the SQL half of the loop. Use it to free up your head for the architecture questions, which is where these conversations get decided.

The tools in this article will look different in 18 months. Somebody will be acquired, somebody will fork something, and Iceberg V4 will promise single-file commits. Commits, snapshots and the catalog pointer will still be what you're debugging at 3am.

So, for those of you already running a lakehouse in production: where does your metadata live today, and has a catalog ever lied to you about what's in a table?

Data Engineering Failures Cost Enterprises $3M a Month

DataDriven — Thu, 01 Oct 2026 10:06:50 +0000

Years ago I inherited a job that had been silently dropping around 40% of its records for 6 months. It didn't crash or alert, and the logs were clean. Someone in finance finally asked why the revenue numbers looked "a little light," and that's how we found out. The fix took an afternoon. Figuring out what had broken took 2 weeks, and the trust took a lot longer to rebuild.

I bring this up because Fivetran just put a price tag on that kind of failure, and it's a big one. Their 2026 enterprise benchmark says pipeline failures expose large companies to roughly $3M a month. In the same stretch of 2026, companies cited AI as the reason for a growing share of tech layoffs. The work that prevents multimillion-dollar outages is the same work getting squeezed, and if you work in data engineering, that tension is also your best career lever right now.

What Fivetran's 2026 benchmark found, and who it surveyed

Fivetran's Enterprise Data Infrastructure Benchmark Report 2026 surveyed 500 senior data and technology leaders in Q4 2025. Every respondent works at an organization with 5,000+ employees, across the US, UK, EMEA, and APAC. The industry mix leans toward financial services (23%), manufacturing (20%), and tech (20%), followed by retail/CPG, healthcare, and a little hospitality. Fivetran reports 95% confidence with a ±4.4% margin of error.

Here are the headline numbers:

4.7 pipeline breaks per month on average, rising to 8.3 at the largest enterprises
60.4 hours of downtime per month, with each incident taking nearly 13 hours to resolve
53% of engineering capacity spent on maintenance instead of new work
$2.2M per year per enterprise in maintenance labor alone
An average shop runs 328 pipelines with 35 full-time engineers
97% of leaders said pipeline failures have slowed their analytics or AI programs

The math holds together, which I appreciate. 4.7 breaks at roughly 13 hours each comes to roughly 61 hours, which matches the 60.4. Fivetran's press release puts business exposure at $49,600 per hour of downtime. Multiply that by 60.4 hours and you land right around $3M a month. A single incident can reach $1.4M.

Now the part a vendor's marketing team won't put in bold: Fivetran sells managed ELT. The report says legacy and DIY systems break 30% to 47% more often than managed ones, and that managed-platform adopters are "nearly 2x as likely" to beat ROI expectations. That might be true. It's also exactly the finding you'd expect from a company whose business is managed pipelines. Treat the breakage and downtime counts as decent signal and the "buy managed and your problems go away" conclusion as a sales pitch.

I've migrated legacy warehouses at 3am. Managed connectors do remove a whole category of pain, mostly API pagination, auth token rotation, and schema changes in SaaS sources. They don't fix a bad data model, an upstream team renaming a column without telling anyone, or a join that quietly fans out and doubles your revenue.

The number I keep coming back to is the 53%. Of 35 engineers, around 18 spend their week keeping existing pipelines alive. Every senior DE I know would call that number low.

Pipeline failures and layoffs in the same month

Now set that next to the layoff data.

The Challenger, Gray & Christmas May 2026 report has AI cited in about 22% of all US job cuts through May. In May alone it was 40%. Later 2026 coverage of Challenger's numbers puts the year-to-date share around 24% through July, with AI as the most-cited reason for several months in a row. The cumulative counts in secondary write-ups don't fully agree with each other, so I won't pretend to know the exact figure, but the direction is clear. Anyone still quoting "roughly a fifth" is out of date, because the share rose through mid-year.

TechCrunch has kept a running list of large tech layoffs where employers cited AI, including Amazon, Oracle, Microsoft, Meta, and PayPal. Andy Challenger summed it up this way: "Tech remains the center of gravity for this year's cuts, and AI is still the reason companies give."

Keep the phrase "the reason companies give" in mind. What a company says in a press release about a layoff and what actually caused it are often unrelated. I've survived multiple layoff waves. The official reason has always been a story told to the board, and this year the story is AI.

The economics don't work, and I mean that literally. A typical enterprise in Fivetran's sample has $3M a month of downtime exposure, and the people preventing it are a line item in the maintenance budget. Cut 5 of those engineers to save about $1M a year in salary, and you only need a few extra 13-hour incidents to give all of it back. You're also teaching the business that the dashboards can't be trusted, which costs more than any of it.

The labor that keeps $3M a month of downtime contained is the easiest line item to cut and the most expensive one to lose.

I'll be fair to the other side. The same research also shows data engineering postings growing, with demand moving toward people who can handle infrastructure, governance, and cost control. The squeeze lands hardest on generic and lower-tier roles. So the DE role isn't disappearing. It's being repriced, and the people doing the repricing don't fully understand what the job involves.

Where the maintenance hours go

Monte Carlo's 2026 State of Data Quality survey answers most of it. 68% of teams need 4+ hours just to detect an incident, up from 62% in 2022. Average resolution time is 15 hours. And the stat that should embarrass everyone in this industry: 74% of data issues are found by business stakeholders first, up from 47% in 2022.

So for 3 out of 4 issues, the monitoring system is a VP in a Slack channel asking why the numbers look weird. That was my 40% record-drop job exactly, except today it happens at 3x the scale.

That's where the 53% goes. Most of it is detective work: finding which of the 328 pipelines broke, which upstream source changed, which backfill needs to rerun, and which downstream tables are now wrong. The actual fix is usually the shortest part of the incident.

The other surveys in the research fill in the rest:

dbt's 2026 State of Analytics Engineering report found 72% of data teams prioritize AI coding, while only 24% prioritize AI-assisted pipeline testing and observability. So we're speeding up how fast people write pipelines and barely investing in knowing when they break.
Atlassian's DX report for Q2 2026 found AI saves engineers 4 to 6 hours a week, yet the innovation ratio barely moved, staying roughly 57% to 58%. The freed-up time gets absorbed by the maintenance backlog.
Teams with automated observability reportedly resolve incidents about 4x faster. That claim is secondhand, so hold it a bit loosely, but every incident I've worked agrees with the direction.

The tooling is catching up, for what it's worth. Apache Airflow 3.3 shipped AIP-103, a first-class task and asset state store. It persists key-value state across worker crashes and retries, which replaces the XCom hacks that never survived a retry. Apache Iceberg 1.12 replaces position delete files with deletion vectors, compact bitmaps that Dremio says cut read latency by 50% to 80% for high-frequency deletes. That matters a lot for CDC and GDPR workflows.

Both are good releases. Neither changes the fundamentals. Iceberg still needs compaction, snapshot expiration, orphan cleanup, and manifest optimization running in a coordinated way, and if you get orphan retention wrong a mid-write failure can silently corrupt your table. A state store won't help if you don't understand idempotency, because you'll just persist the wrong state faster.

This is the concepts-over-tools argument again. Idempotency, late-arriving data, schema contracts, grain, and backfill strategy work the same in Airflow 2, Airflow 3.3, Dagster, or a cron job someone wrote in 2014. Tools change every 18 months, and the reasons pipelines fail don't.

Your next data engineering interview

Compensation hasn't caught up with reliability work yet. Indeed Hiring Lab's September 2026 analysis found advertised pay in highly AI-exposed occupations up roughly 46% since 2021, compared with 25% for less-exposed fields. The AI premium on identical job titles is about 4.7%, and much larger at senior levels. Mid-level DE base pay sits about $139K median. Meanwhile the people preventing $3M a month in downtime get paid like plumbers. (Plumbers, to be fair, are often doing fine.)

I don't think that lasts, and the hiring signals already show it shifting. Recruiting guides for 2026 now treat "reliability signals" as the minimum: SLAs, monitoring, alerting, backfills, retries, tests, incident response, and postmortems. Senior job postings name observability architecture as a core responsibility, and some staff-level postings run past $260K.

Here's how to use that.

Rewrite your resume around incidents. "Built ETL pipelines using Airflow and Snowflake" is a tool list. "Cut detection time on a revenue pipeline from next-day to 20 minutes by adding row-count and freshness checks" is a story. Don't tell me you "ensured data reliability across a wide array of mission-critical workflows." If I read one more bullet like that I'm putting my fist through drywall.

Have a postmortem ready. Interviewers increasingly ask you to walk through a real incident: how you detected it, what the blast radius was, how you fixed it, and what you changed so it couldn't happen again. Pick your best one and rehearse it until you can tell it in 3 minutes without rambling. If you don't have one, you haven't been doing this long enough, or you were lucky. Either way, go break something in a side project and fix it.

Prep for pipeline architecture. I've watched people with 10 YOE get downleveled because they couldn't explain why their design would survive a late-arriving partition. DEs don't need to whiteboard load balancers. You need to explain how your pipeline handles retries without double-writing, how you'd backfill 90 days without taking down the warehouse, and where your quality checks go. That's the gap we built DataDriven to cover, so for pipeline design practice, try DataDriven: it drills exactly those failure-mode questions, which is where loops separate seniors from everyone else.

Ask about the 53% in your interviews. Ask every hiring manager how many incidents they had last quarter and who found them first. If the answer is "the business usually tells us," you're interviewing for a firefighting job, so price the offer accordingly. If they actually know their numbers, that's a team worth joining.

If you're worried about layoffs, become the person who knows why things break. Every layoff list I've seen spared the engineer who held the context on the scary pipelines. It comes down to what happens when the CFO asks, "who can tell me why the numbers are wrong?" and only one name comes up.

I've been through 3 waves of "data engineering is getting automated away." I'm still here and still debugging the same categories of problems. The Fivetran numbers just put a dollar figure on what every on-call DE already knew: the hard part of the job is keeping things working. The companies cutting that work now will rehire for it at a premium after their first $1.4M incident, and the engineers who can tell a good postmortem story will get those offers.

So here's my question: in your last pipeline incident, who noticed first, your monitoring or someone from the business?

The Real Data Behind the Data Engineering Hiring Panic

DataDriven — Tue, 29 Sep 2026 10:08:43 +0000

Last month 3 different people sent me the same screenshot. It was a slick graphic claiming tech lost 150K jobs this year while data engineering grew 414%, and entry-level postings "collapsed 67%." Each of them asked me whether they should panic, pivot, or quit.

I went looking for where those numbers come from. I couldn't find a dataset behind any of them.

Meanwhile, 3 real sources published numbers this year that you can check yourself: Stanford's payroll research, LinkedIn's own ranking of fast-growing jobs, and the Bureau of Labor Statistics. What they show is narrower and more specific than the viral numbers, and for anyone planning a data engineering career in 2026 it's the more useful picture. Early-career hiring got squeezed hard. Experienced people are doing fine. The field itself is growing slowly and steadily.

Where the viral data engineering hiring numbers come from

I've been in this industry long enough to see 3 waves of "data engineering is getting automated away." Each wave came with its own scary chart, and each chart had the same flaw: nobody could tell you where the numbers came from.

The current batch fits that pattern. The posts pushing "414% growth" and "67% collapse" (some of them right here on DEV) point to "industry reports" and "Gartner" in general terms. They don't link anything, name a dataset, or give a sample size. The closest thing I found to a source for the 67% figure is a job-postings analytics vendor's blog that says entry-level DE postings fell 67% between October 2023 and November 2024. It cites proprietary scraped posting data that you can't audit.

That doesn't prove the number is wrong. What it does mean is that nobody can check it, and when a number can't be checked, you shouldn't be making career decisions off it.

A statistic with no dataset behind it is a vibe with a percent sign.

What bugs me most is that these numbers travel. They get screenshotted, stripped of any context, and dropped into career advice threads, where someone with 2 years of experience reads them at 11pm and decides to drop out of their job search. I've watched it happen, and I've been that person at 11pm myself. I did 20-some loops in one job search. I didn't need a fake statistic telling me the market was over when the real market was already beating me up just fine.

So let's look at the numbers you can actually verify.

Stanford's payroll data on early-career hiring

The Stanford Digital Economy Lab worked with ADP to track real payroll records: millions of actual paychecks across tens of thousands of firms, instead of job postings or surveys. The research is called "Canaries in the Coal Mine", and its headline finding is that employment of software developers aged 22 to 25 fell about 20% from its late 2022 peak.

Over the same period, developers aged 30 and older at the same kinds of firms, in the same roles, grew employment by 6 to 12%. So it's an age effect inside a single occupation. Juniors lost ground while experienced people gained it, and total demand for developers didn't fall off a cliff.

The August 2026 update extends the finding past developers. Workers aged 22 to 25 across all highly AI-exposed occupations now sit 19% below where they'd be if they'd kept pace with less-exposed peers. In July 2025 that gap was 15%, so it's getting wider.

The detail I think matters most is how the decline happens. Stanford found that the adjustment runs "primarily through reduced hiring of young workers rather than increased separations." Separation rates actually fell, including for young workers in exposed fields. Firms aren't firing juniors in large numbers. They've just stopped opening the door for new ones.

That fits what I'm hearing from hiring managers. None of them describe a "fire the juniors" program. What happens instead is that a req for a junior slot quietly gets converted to a senior one, or backfills get put off until the team decides it can live without the role.

The caveats Stanford's own critics raise

This research has weak spots and you should know about them.

The Fed started its most aggressive rate-hiking cycle in 40 years in March 2022. Job postings in AI-exposed sectors started dropping around then, about 8 months before ChatGPT launched. Some economists, including a group writing in ProMarket, argue the data fits an interest-rate hiring freeze about as well as it fits an AI story. Stanford's side points out that an occupation's AI exposure doesn't correlate with how sensitive it is to interest rates, which suggests these are 2 separate channels.

My read is that it's probably both, and nobody can cleanly separate them yet. You'll also see some secondhand write-ups quoting different figures for the developer decline, so go to Stanford's own dashboard instead of trusting the reposts.

The other caveat matters a lot for our field. Stanford measures "software developers," and "data engineers" aren't a separate category in it. Applying this finding to DE specifically is an inference. I think it's a reasonable one, because junior DE work looks a lot like junior SWE work, but it's still an extrapolation.

LinkedIn and BLS on data engineering growth

Next is the "414% growth" claim.

LinkedIn publishes a yearly list called Jobs on the Rise, which ranks the fastest-growing roles using its own members' job transitions. The 2026 edition looked at jobs started between January 2023 and July 2025. AI Engineer is #1, AI Consultant/Strategist is #2, New Home Sales Specialist is #3 (really), and Data Annotator is #4.

Data engineer doesn't show up anywhere in the top 25. It wasn't in 2025's top 25 or 2024's either.

If DE had grown 414%, LinkedIn would have noticed, because tracking this is the whole point of the list. So a role supposedly growing 4x shows up 0 times in 3 years of the one ranking built to catch fast growth. I'll let you decide which of those numbers to believe.

Missing from a fastest-growing list doesn't mean shrinking, though. It means DE is a mature, established role growing at a normal pace, and the explosive growth went to titles that barely existed 3 years ago. I'd honestly be more nervous if DE were on the list, since hype-driven titles tend to crater just as quickly.

The BLS baseline

The federal government doesn't track "data engineer" at all. The Standard Occupational Classification system groups workers by the work they do, not by job title, so data engineers get spread across database architects, software developers, and data scientists depending on what their job looks like day to day. After more than a decade of the title being everywhere in industry, it still doesn't have its own code.

The closest proxy is Database Administrators and Architects, and the BLS projects 4% growth for it over the next decade, "about as fast as the average for all occupations." That's roughly 7,300 openings a year, mostly from people retiring or leaving.

The 4% average hides a split inside the category. Traditional database administrators are projected at 0% growth. Database architects are projected at 9%. Software developers, where plenty of DEs get counted, are projected at 10%. Data scientists are at 35%.

Data engineering lands somewhere between 4% and 10%, depending on how you count, and no official number pins it down more tightly than that. That growth is boring and positive, which is how a healthy, established profession looks.

414% was never real. 4 to 10% is. Boring growth is still growth, and it compounds for people who stay in the game.

What this means for your data engineering interview strategy

Put the 3 sources together and here's what I think they show, with the caveat that this is my interpretation and no single dataset proves it. The door is narrower at the bottom: fewer junior seats are opening, and whoever does get hired has to show more on day one. The middle and top are holding up, with experienced people in the same roles gaining employment. And teams aren't being gutted. Separations are low, so the pressure falls on who gets hired next, and current employees aren't being pushed out.

Your strategy depends on which side of that line you're on.

If you're early-career

This is the hard group, and I won't pretend it isn't. A 20% drop in early-career developer employment is a real hit, and the numbers back it up. If you've been getting rejected over and over, a big part of that is the market. It says little about how capable you are.

Still, "fewer junior seats" means you have to look like you've already done the job. Companies are cutting the roles where someone gets paid to learn, so show that you've learned already. Build something that breaks. Run a pipeline on a schedule for 3 months and write down every time it failed and what you did about it. Skip the tutorial DAG. The actual job is mostly debugging, things like working out why 2M rows silently disappeared last Tuesday, and that's the story that gets a junior noticed.

Also, drop the pure-SWE framing. Data modeling is where junior DE candidates get separated from everyone else. An AI tool can write a medium LeetCode solution in a few seconds, which makes that a weak signal, and interviewers increasingly know it. What still tells an interviewer something is whether you can take a messy business process and design a model for it: define the grain, handle late-arriving data, and explain why you picked one approach over another. The prep I keep sending people to is the same one I'd use myself: for data modeling interview questions, try datadriven, since its problems are built around real pipelines and schemas instead of puzzle trivia.

If you've got a few years in

You're on the side of the data that's growing, so spend less time on doom threads and more on pricing yourself properly.

The interview is where experienced people lose money. I've watched engineers with 10 years of experience get downleveled because they couldn't explain their own design decisions when put on the spot. They did the work; they just couldn't tell the story of it in 45 minutes. That's a skill you can practice, completely separate from being good at the job.

Get your war stories straight. For every big project, know the grain of the model, what broke, what it cost the business, and what you'd change now. Put the vague resume language in the trash. "Leveraged modern data stack to drive stakeholder value" tells me nothing. "Migrated 400 tables with zero downtime and cut the nightly batch from 6 hours to 90 minutes" tells me you're senior.

If you're hiring

This is aimed at the people on my side of the table. The Stanford researchers raise the uncomfortable question here: if nobody hires juniors today, who are the seniors in 2032? Every senior DE I know started as someone who broke production at least once while someone else paid their salary. Cutting junior hiring saves money this quarter, and the cost shows up years later as a senior shortage you can't hire your way out of.

Reading the job market like a data engineer

We're supposed to be the people who check the data. We trace a number back to its source table before we put it on a dashboard. We don't trust a metric until we know its grain, its filters, and who calculated it.

Do the same with job market content. When you see a scary number, ask the questions you'd ask about a broken pipeline: what's the source, what's the population, what's the time window, and is it measuring postings, hires, or actual paychecks? Postings are noisy, hires are better, and payroll records are the closest thing we have to ground truth.

When I ran that check on this year's data, a narrower story came out than the viral posts tell. Early-career hiring really is tight. Experienced engineers are gaining ground. The field is growing at a steady, unremarkable pace that no government agency can measure precisely, because government agencies still don't know what we're called.

I've been through enough hype cycles to recognize this one. In a few years someone will need to debug why the new AI pipeline is silently dropping records, and I'd put money on it being a data engineer.

So here's my question: when did you last change a career decision because of a job-market statistic, and did you ever check where that number came from?

dbt's 2026 Survey: Data Engineers Chase AI, Skip the Testing

DataDriven — Thu, 24 Sep 2026 10:08:01 +0000

The worst bug I ever inherited was a job that silently dropped 40% of records for 6 months before anyone noticed. No test caught it and no alert fired. Nobody owned the target table, so nobody looked. The code ran green every night and did the wrong thing every night.

I think about that job a lot lately, because the industry has decided to build a lot more of them and build them faster.

dbt Labs' 2026 State of Analytics Engineering report surveyed 363 practitioners and leaders between December 2025 and February 2026. 72% of teams now prioritize AI-assisted coding. Only 24% prioritize AI-assisted pipeline management, meaning testing and observability. So 3 times as many teams are working on shipping code faster as are working on checking it.

The code is the cheap part now, and checking it is where the value is. Teams that spend on generation and skip the guardrails are running up debt that someone like me will get paid to clean up in 2028.

The numbers from dbt's 2026 survey

Trust in data as a priority went from 66% to 83% in a single year. According to dbt Labs' own writeup, that's the steepest single-year increase in the survey's history.
Speed as a priority rose from 50% to 71%.
Cost reduction barely moved, from 48% to 53%.
71% worry about hallucinated or incorrect outputs reaching stakeholders.
41% say ambiguous data ownership is still a problem, roughly flat from last year.
57% report higher warehouse and compute spend, while only 36% report bigger team budgets.
77% of managers emphasize AI for productivity.

Put those in a row and you get teams that want to go faster, say they care much more about correctness, and are putting their effort into the speed half.

To be fair to the respondents, wanting speed and trust together is reasonable. Every stakeholder I've ever had wanted both. What bugs me is the ratio. Speed moved 21 points, trust moved 17, and the tooling investment went overwhelmingly to speed.

The report tells you trust jumped, but it doesn't say why, and it names no single incident or regulation behind it. My guess, and it's only a guess, is that enough teams shipped an AI-generated model that quietly produced a wrong number in front of an executive. Once that happens, trust becomes very important very fast.

Data trust jumped 17 points and nobody bought tests

Trust is a priority for 83% of respondents, but only 24% put resources into the machinery that produces it.

I've been in those planning meetings, and the gap comes down to economics.

AI-assisted coding pays off right away and is easy to see. An engineer turns a 2-day model into a 2-hour model, their manager sees a burndown chart drop, and everyone feels productive. Data observability pays off in incidents that never happen. Nobody holds a retro for the dashboard that stayed correct. You can't put "nothing broke" on a slide and get more budget.

The compute numbers make it worse. 57% of teams are spending more on the warehouse, while only 36% got more money for people. Compute bills per query and scales on demand. Headcount needs approval, a req, a hiring loop, and 3 months of ramp. When you're squeezed, you let the warehouse absorb the inefficiency and let the backlog absorb the testing.

AI made writing a dbt model nearly free. Nobody made verifying one free. Whatever you ship without a test turns into a liability with a slow fuse.

I've been through 3 waves of "data engineering is getting automated away." Each time the tools changed and the failure categories didn't. You still get schema drift, late-arriving data, upstream teams breaking contracts without telling you, and joins that fan out and silently double your revenue. AI gets you to those failures faster. It doesn't prevent them.

What worries me most about AI-generated SQL is what it does to review. When a junior engineer writes a bad join, it usually looks bad. The CTE names are weird, the formatting is off, and something makes a reviewer squint. AI-generated SQL looks clean and confident whether it's right or wrong. Review has always relied partly on the author's code looking unsure, and that signal is gone.

If you're on a team in the 72% but not the 24%, you can do most of this yourself without a platform initiative.

Test at the grain. Every model gets a uniqueness test on its declared grain. If you can't state the grain in a sentence, the model isn't done.
Test row counts across boundaries. Rows in versus rows out on every join that shouldn't change cardinality. That's the test that would have caught my 40% drop in week 1 instead of month 6.
Put freshness checks on anything a human looks at. Stale data that looks fresh does more damage than an obviously broken dashboard.
Make AI-generated models carry tests before merge. If the model wrote the SQL, it can write the tests too. Review the tests harder than the SQL.

None of this is new, and dbt has made most of it trivial for years. The survey says most teams still aren't prioritizing it.

The ownership problem data engineering keeps dodging

The number I keep coming back to is the 41% with ambiguous ownership, because it didn't move. According to dbt Labs, technical integration barriers dropped from 35% to 27% year over year, so the tooling got better. Ownership stayed stuck.

Other surveys point the same way. Joe Reis' 2026 State of Data Engineering survey of more than 1,000 data engineers found 59% citing pressure to move fast and 51% citing lack of ownership as top pain points. Only 11% said their data modeling was going well.

Then there's the result I'd put on a billboard. The Practical Data Community's April 2026 data modeling survey asked 334 people what would most improve their modeling. Training came first at 28.1%, then clearer business requirements at 24.6%, more time at 21.6%, and dedicated ownership at 21%. Better tooling got 4.8%.

Vendors keep selling tools to an industry that has told them, in writing, that tools aren't the bottleneck.

The same survey found 42.5% of data models are owned by whoever built the pipeline, 19.2% have a dedicated modeler or architect, and 7.8% have no formal owner at all. I've worked in all 3 setups. The "whoever built it owns it" model works until that person leaves, gets reorged, or gets laid off. After that the table becomes folklore.

Now add AI to that. When an engineer writes a model by hand, they at least know what they meant. When an agent generates it from a prompt, the "owner" is whoever typed the prompt, and they may not be able to explain the join logic in the output. The result is an untested model that no human wrote and nobody is named as owning, feeding an executive dashboard. It's the setup where hallucinated numbers reach stakeholders, and 71% of respondents already worry about exactly that.

Pooja Crahen, a senior manager of analytics engineering at Okta, said in the dbt Labs release that you can't get both speed and trust without discipline in modeling, validation, and ownership, and that the discipline has to be a requirement rather than a best practice. That's her opinion, and I agree with it. I'd add that it has to be a requirement someone is accountable for, with a name next to it.

AI governance frameworks won't fix this on their own. I've seen plenty of governance programs that were thorough on paper and did nothing in practice. What fixes it is a person who can say "no, this doesn't ship without a grain test" and be listened to.

Using the trust gap in your next data engineer interview

The gap is a career opportunity if you position yourself right.

I've been on both sides of the hiring table. Everyone now has AI coding assistants, so writing SQL quickly no longer sets you apart. I've done 20+ loops in a single job search, and I've sat on panels where we passed on strong candidates for dumb reasons. The candidates who stood out most, even before this survey, were the ones who could explain how they knew their pipeline was right.

Senior interviews are already testing for this through behavioral questions. "Tell me about a time you caught a data issue before anyone else noticed." "Tell me about a pipeline you improved without being asked." Those are ownership questions. They check whether you treat data quality as your job or as something downstream.

Most candidates answer them badly. They talk about the pipeline they built. The strong answer covers what broke, how they found it, what the blast radius was, and what they put in place so it can't happen again. The actual job is debugging, and the interviewers who've done the job know it.

So here's the prep plan.

Get a debugging war story and tell it with numbers. Say "Found a fan-out join inflating revenue 12% for 3 weeks; added grain tests to 40 models," not "improved data quality."

Own the modeling round. Know grain, slowly changing dimensions, and fact table design well enough to explain why a model breaks as well as how to build it. Data modeling is the skill AI is worst at faking, because it depends on business context the model doesn't have. We're biased because it's ours, but for snowflake interview questions, try datadriven.io. We built it around grain, dedup, and "why is this number wrong" style problems.

Talk about observability like someone who's been paged. Know freshness, volume and schema checks, and know which alerts get looked at and which ones get muted by week 2. Interviewers can tell whether you've been on call or only read about it.

Talk about AI without being breathless or dismissive. "I use it to draft models and I make it write tests I review harder than the SQL" is a senior answer. "AI writes all my code" and "I don't trust AI" are both junior answers.

This is where the economics favor you. Teams are spending more on compute than on people, generating code faster than they can review it, and 83% of them say trust is suddenly a top priority. The engineer who can turn "we care about trust" into tests, owners, and alerts is solving the exact problem those teams have admitted to in writing.

So I'm curious where your team lands: are you in the 24% actually investing in testing and data observability, or are you shipping AI-generated models and hoping the dashboards stay honest?

Spark 4.x Is Stable. The Data Engineering Upgrade Tax.

DataDriven — Tue, 22 Sep 2026 10:10:55 +0000

Spark 4.2.0 shipped July 14, 2026. If you're still running 3.x in production, the grace period is over.

I've been through enough platform migrations to know the pattern. The release drops. Everyone says "we'll get to it next quarter." Then 18 months later you're running an unsupported runtime with 3 engineers who know how to keep it alive and a Confluence page titled "DO NOT TOUCH" as your only documentation. The Apache Spark 3.x to 4.x jump is that migration, happening right now, and the breaking changes are worse than most teams realize.

Here's what the spark upgrade actually costs.

ANSI Mode: The Silent Pipeline Killer

The biggest breaking change in Spark 4.0 is a default that flipped.

spark.sql.ansi.enabled is now true. In Spark 3.x, this was false. That single boolean changes the error semantics of every arithmetic operation, every type cast, and every array access in your codebase.

What used to return NULL now throws an exception.

CAST('a' AS INT)? In 3.x, you got NULL. In 4.x, you get [CAST_INVALID_INPUT]. 2147483647 + 1? In 3.x, it silently wrapped to a negative number. In 4.x, [ARITHMETIC_OVERFLOW]. Division by zero, out-of-range array access; same story across the board.

This is the category of change that passes every smoke test and blows up on real data. Your CSV ingestion pipeline that casts dirty string fields to integers? Dead. Your financial rollup that divides by a denominator that's occasionally zero and falls back to NULL? Dead.

Databricks called this "one of the most significant shifts in Spark 4.0." They're right, but "significant shift" undersells it. It's a runtime behavior change with no compile-time warning and no deprecation notice in 3.x. You find out when the job fails at 2am and the on-call page wakes up someone who has no idea what ANSI mode even is.

The escape hatch exists: set spark.sql.ansi.enabled=false and keep the old behavior. But that's a stopgap, not a strategy. The Pandas API on Spark already defaults to compute.ansi_mode_support=True in 4.1+. The window for globally disabling ANSI mode is narrowing with every minor release.

The real fix is surgical. Spark 4.x ships try_add(), try_multiply(), try_cast(). Per-expression functions that return NULL instead of throwing. Audit your pipelines for every implicit cast and every arithmetic operation that could overflow. Wrap the ones that need NULL semantics in the try_* variant. Leave everything else strict. You want the errors; they're telling you about data quality problems you've been silently shipping for years.

ANSI mode doesn't break your pipelines. It reveals the bugs your pipelines have been hiding.

Java 17, the Dependency Audit, and Everything Else That Moved

Java 17 is the minimum runtime for Spark 4.x. JDK 8 and 11 support dropped entirely. This one's visible in CI; your build fails, you catch it early. The harder problem is everything downstream.

The dependency versions jumped across the board. Guava 14 to 33. Jackson 2.15 to 2.18. Hadoop 3.3.4 to 3.4.1. Arrow 12.0.1 to 18.1.0. If you're shading any of these into a fat JAR (and you probably are), your shading rules need a full audit. A library using Guava's Multimap in version 14 has a different serialized form than version 33. Deserializing old data with new classes can silently succeed with wrong behavior. I spent a full day once debugging a deserialization issue that traced back to exactly this kind of version skew in a shaded dependency. That's silent data corruption; the worst category of bug, and the one you don't find until someone at finance asks why the board deck numbers don't add up.

The javax.servlet to jakarta.servlet migration is mandatory. Any custom REST endpoints or transitive dependencies that import javax.servlet fail at runtime. Scala 2.12 is gone; you're recompiling against 2.13 with its overhauled collections API. Scalafix handles most of it mechanically, but "most" and "all" are different words.

Then there's Java 17's module system. Spark's NIO code touches JDK internals, which means , add-opens flags for java.base/sun.nio.ch, java.base/jdk.internal.misc, and a growing list of others. I've seen production configs with 6 , add-opens flags just to keep module-access warnings quiet. It works, but it's configuration debt you'll carry forward to every deploy.

The ecosystem isn't fully ready either. OpenSearch Hadoop's Spark 4.0 support issue was filed after launch and remains open. If you depend on connectors outside the core ecosystem, check compatibility before you start. Finding out your connector is blocked after you've done all the Java 17 work is a special kind of frustrating.

The upside: what shipped alongside these breaking changes is genuinely useful. The VARIANT type (GA in 4.1) gives you native semi-structured data with shredding optimization. SQL UDFs are transparent to the Catalyst optimizer. Pipe syntax (|>) lets you write SQL that reads top-to-bottom instead of inside-out. Arrow-optimized Python UDFs are on by default in 4.2, delivering faster execution with zero code changes. The upgrade has a real cost, but it's buying you real capabilities.

PySpark at 1.5 MB: What Spark Connect Actually Changes for Data Engineering

The new pyspark-client package is 1.5 MB on PyPI. No JVM. No JARs. Pure Python. pip install pyspark-connect and you're talking to a remote Spark cluster over gRPC.

For 15 years, PySpark meant bundling a local JVM. Your CI container, your notebook environment, your local dev setup; all needed Java installed. Spark Connect removes that constraint for any workflow running against a remote cluster. AWS launched Spark Connect on EMR Serverless in June 2026. Databricks Runtime 19 GA'd 2 days after Spark 4.2. The platform vendors are converging hard on this architecture.

The tradeoffs matter, though. Spark Connect sends unresolved logical plans to the server. Schema analysis happens at execution time, not at DataFrame construction. In classic PySpark, df.select("nonexistent_column") fails immediately. In Spark Connect, it fails at .show() or .collect(). Microsoft's Azure Databricks documentation spells this out with code examples. If your test suite catches column-name typos at plan construction, those tests now silently pass when they shouldn't. That's a foot-gun for teams migrating without careful integration testing.

The bigger constraint: RDD operations don't work through Spark Connect. SparkContext, RDD, anything touching private JVM methods; none of it crosses the gRPC boundary. Legacy RDD pipelines either get ported to DataFrames or stay on classic PySpark. There's no halfway option.

And production stability is still rough. Practitioner reports describe Connect servers needing daily preventive restarts and struggling with long-running, resource-intensive jobs that destabilize the shared server. The 1.5 MB client is elegant; the server side needs more operational hardening before you trust it with your most critical batch workloads.

For greenfield projects, Spark Connect is the clear direction. For existing codebases with RDD usage, this migration has a higher cost than the 1.5 MB headline suggests.

The Spark 4 Support Window Is Closing

Spark 3.5's extended LTS runs through November 2027, but "extended LTS" means security fixes only. No features, no performance backports, no bug fixes that aren't CVEs. The Apache Foundation called this extension out explicitly as pressure relief for migration. It's a deadline with a grace period, not indefinite support.

The managed-service timelines make planning harder. AWS EMR 7.5 (Spark 3.5.6) hits end of standard support November 2026. Databricks Runtime 15.4 LTS carries through August 2027. Google Dataproc hasn't published a Spark 3.5 EOL date at all. Multi-cloud shops can't coordinate a single cutover; you're staggering upgrades per platform or running mixed major versions in parallel. Both cost money and engineer time that nobody budgeted for.

Here's the career angle. Spark 4 literacy is becoming a baseline expectation in data engineering interviews. ANSI mode semantics, Spark Connect architecture, the try_* function family; these are the kinds of questions that separate candidates who've done the work from candidates who've read the bullet points. If you're prepping right now, understanding what breaks in the 3.x to 4.x transition matters more than memorizing another API call. Concepts transfer; syntax doesn't. But the concepts here are specific: why does ANSI mode throw on overflow, what does lazy schema analysis mean for testing, when do you reach for try_cast() vs. fixing the upstream data. That kind of spark practice is what DataDriven is good for, and we designed the prep around exactly these conceptual shifts because they compound everywhere you look.

The tools change every 18 months. The problems don't. Schema drift, type coercion bugs, upstream teams breaking contracts without telling you. Spark 4.x just made some of those problems louder. I'll take loud failures over silent ones every single time.

What's the gnarliest thing ANSI mode broke in your pipelines?

dbt Now Runs on Flink. What Changes for Data Engineers

DataDriven — Thu, 17 Sep 2026 10:10:17 +0000

For 5 years I maintained 2 separate sets of transformation logic for the same business domain. Batch models in dbt on Snowflake for the warehouse. A completely separate Flink application in Java for the streaming layer. Same business rules, different languages, different testing frameworks, different deploy processes, different on-call runbooks. When something broke on the streaming side, the batch team shrugged. When the warehouse was wrong, the streaming team pointed at their own dashboards and said "looks fine to us."

That split has defined data engineering careers for the better part of a decade. And it just got smaller.

Confluent shipped a free, open-source dbt adapter for Apache Flink SQL in Q2 2026. A parallel community adapter from Xebia covers self-managed Flink. Together, they let you write streaming transformations inside the same dbt project you already use for batch. Same ref(), same DAG, same dbt test. Different engine underneath.

This matters more for your career than for your architecture.

What the dbt Adapter for Apache Flink SQL Actually Does

The core idea is straightforward: you write a dbt model in SQL, and instead of compiling to Snowflake or BigQuery, it compiles to Flink SQL and runs as a continuous query against Kafka topics.

The Confluent adapter supports 3 materialization types. view creates a virtual table over a Kafka topic. streaming_table runs a continuous INSERT INTO...SELECT that writes results to a new topic with changelog semantics. streaming_source defines a connector-backed source; basically your Kafka topic declaration.

ref() works identically to batch dbt. The adapter resolves dependencies, builds the DAG, and deploys models in topological order so upstream tables exist before downstream consumers start reading. You can run dbt ls , select model_name+ for selective deployments. If you've used dbt on any warehouse, the workflow is familiar.

The genuinely clever piece is testing. When a query runs continuously and results are unbounded, how do you write a test with a deterministic pass/fail? You can't just SELECT COUNT(*) from an infinite stream and expect a stable answer. Confluent's adapter automatically switches to bounded execution mode during tests, using snapshot queries that read Kafka topics up to the current timestamp and return a finite result set. Unit tests use mock data on temporary tables. Your dbt test command works. It just works differently under the hood.

That said, data quality tests against live production streams are marked "coming soon" as of the Q2 2026 release. Unit tests with mocked inputs, yes. Asserting not_null on a stream that's been running for 3 weeks in prod, not yet.

The Career Cost of 2 Toolchains

I've been on both sides of this hiring wall. I've interviewed candidates who were excellent batch engineers and watched them stumble the moment someone asked about watermarks. I've also interviewed streaming specialists who couldn't design a slowly changing dimension to save their lives. The industry created 2 career tracks for work that, conceptually, is the same discipline.

The numbers back this up. Data engineers with Flink or Kafka production experience earn a 15-25% salary premium over equivalent batch-focused roles, with medians around $130k. That premium exists because the supply is thin; streaming has been genuinely harder to hire for because the toolchain was completely separate. Different orchestration, different monitoring, different failure modes, different interview prep.

67% of enterprises now operate both batch and streaming pipelines. The demand side is exploding. But until this year, becoming a "streaming data engineer" meant learning an entirely separate stack on top of your existing one. Kafka, Flink APIs, custom deployment scripts, bespoke testing harnesses. 2 to 3 months of ramp-up time after you already knew batch fundamentals.

dbt extending to Flink compresses that ramp. You still need to understand watermarks, windowing, state TTL, and event-time semantics. Those are concepts, and concepts are what actually matter. But you don't need to learn a new transformation framework, a new testing approach, a new CI/CD pipeline, and a new way to express lineage. You use the dbt workflow you already know and learn the streaming concepts on top of it.

Junior engineers worry about which tool to learn. Senior engineers worry about which problems to solve. Staff engineers worry about which problems to prevent. The dbt-Flink adapter is a staff-level move: it prevents the problem of maintaining 2 divergent transformation stacks.

This has interview implications too. Watermark literacy is now expected at the junior level in loops that touch streaming. 5 years ago, watermarks were specialist knowledge. In 2026, with 60% of new pipelines carrying real-time components, interviewers expect you to explain WATERMARK FOR order_time AS order_time - INTERVAL '5' SECOND and why you chose that lateness bound. The concepts haven't changed; the percentage of roles that require them has. If you want reps on windowed aggregations before your next loop, for sql interview prep, try datadriven.io where we've been expanding the streaming SQL problems to match what panels actually ask now.

Managed vs. Self-Hosted: 2 Adapters, 2 Trade-offs for Data Engineers

There are actually 2 dbt-Flink adapters, and which one you care about depends on your infrastructure.

Confluent's adapter targets Confluent Cloud exclusively. It ships with a confluent-sql Python driver (DB-API v2 compliant) that talks directly to Confluent's REST API. The upside is zero Flink cluster management; Confluent handles compute, scaling, and checkpointing. The downside is a hard lock to Kafka as your only data source. If your events live in DynamoDB, S3, or a JDBC database, they need to land in a Kafka topic first. Confluent Cloud Flink pricing runs $0.21 per CFU-hour, billed by the minute.

Xebia's adapter (originally from GetInData, released in 2023) targets self-managed Flink via the SQL Gateway. It exposes the full Flink connector ecosystem: Kafka, Elasticsearch, JDBC, S3, HDFS, Kinesis, and 50+ community connectors. You get more flexibility in exchange for operating your own Flink cluster. The trade-off is real; version 1.3.11 added session cluster lifecycle management, but the Xebia team themselves acknowledge the architecture is "lightweight and stateless" at the cost of robustness. If deployment fails between internal steps, you can lose job progress. Savepoint recovery is manual.

102 GitHub stars on the Xebia adapter after 3 years tells you something about adoption. Self-managed Flink is operationally expensive, and most teams that want streaming are gravitating toward managed offerings. But if you're already running Flink clusters and want dbt integration without a Confluent Cloud dependency, it's the only option available.

Both adapters implement bounded execution for tests. Both support ref() and DAG resolution. Both leave data quality tests on live streams incomplete. The split comes down to infrastructure.

Where the Maturity Gaps in Data Engineering Tooling Still Bite

I'd be lying if I said this was production-ready the way dbt on Snowflake is production-ready. The gaps are real, and they mostly stem from Flink SQL's architecture rather than adapter immaturity.

No snapshots. Flink SQL doesn't support MERGE or the CTE-based updates that dbt snapshots require for Type-2 slowly changing dimensions. If your warehouse workflow depends on dbt snapshot for audit trails and historical dimension tracking, that pattern doesn't exist here. You'll need a separate solution.

No incremental materialization. dbt's batch-incremental model (process only new rows since last run) doesn't map to continuous processing. You use streaming_table instead, which is a fundamentally different execution model. Existing incremental pipelines need a rewrite, not a config change.

No cluster provisioning. The adapter can't create or drop Kafka clusters. Both dbname and schema must reference pre-existing infrastructure. If you're used to Snowflake's fully declarative setup where dbt can create schemas and databases, that gap is noticeable.

10-minute session expiry on Xebia's adapter. Tables created in a Flink SQL Gateway session expire after 10 minutes by default. Run dbt test 15 minutes after dbt run and you get table-not-found errors. This kind of brittleness doesn't exist in warehouse-based dbt.

And the conceptual gap remains even when the workflow gap closes. Stateful operations in continuous streams require you to think about state TTL, changelog semantics, and event time vs. processing time ordering. These are streaming-specific concerns that batch engineers have to learn regardless of whether the command they type is still dbt run. The adapter unifies the workflow; it doesn't unify the knowledge.

As Kai Waehner at Confluent put it: "Teams should expect to work with an evolving ecosystem." That's a diplomatic way of saying it's early.

What Actually Changes

Here's my honest read. dbt running on Apache Flink is a big deal for the data engineering career path and a modest deal for architecture. Most companies still don't need streaming. Batch handles 90% of what organizations actually run. That hasn't changed.

What has changed is the cost of adding the other 10%. Previously, streaming meant a separate team, a separate stack, and a 6-figure hiring premium. Now it means learning Flink SQL concepts while keeping the dbt workflow your team already operates. The marginal cost of streaming capability dropped significantly in a single product cycle.

The Fivetran acquisition of dbt Labs (closed June 2026, combined ~$600M ARR) makes this even more interesting. The merged entity controls both ELT ingestion and SQL transformation. Adding Flink support means they're positioning to own batch and streaming transformation in one platform. That kind of consolidation has historically been great for adoption and terrible for pricing leverage, but that's a problem for next year.

For now: learn watermarks. Learn windowing. Learn event-time semantics. These are concepts that transfer across any streaming engine. The dbt adapter means you can practice them without rebuilding your entire workflow from scratch. And if you're hiring, stop splitting your DE reqs into "batch" and "streaming" tracks. The toolchain just told you those are converging.

What's your team's plan? Adopting the adapter, waiting for maturity, or still running separate batch and streaming stacks? I'm curious where people are actually landing on this.

5,800 Data Engineers on What Is Actually in Production in 2026

DataDriven — Wed, 16 Sep 2026 18:01:21 +0000

Every year, someone publishes a "State of Data Engineering" survey with 200 respondents, mostly from their own Slack community, and calls it representative. I ignore most of them. When Astronomer dropped their State of Apache Airflow 2026 report with 5,818 respondents across 122 countries, I paid attention. That's actual sample size.

I've spent years on both sides of the data engineering interview table, and I've watched candidates prep for tools that their target companies don't even use. The numbers in this survey confirm a few things I've seen firsthand, contradict some of the loudest takes on LinkedIn, and surface a gap that should make every data team uncomfortable.

What 5,800 Practitioners Are Actually Running in Production

The survey ran September 15 to November 20, 2025. 50 questions. 5,818 practitioners. 122 countries. Astronomer calls it the largest data engineering survey ever conducted. The 2025 edition had 5,250 respondents from 116 countries, so this is roughly 10% growth year over year. Steady accumulation.

One number jumped out immediately: Apache Airflow now has over 3,600 unique contributors. More than Spark's 2,000+. More than Kafka's 1,530. That's 135% more contributors than Kafka and 80% more than Spark. 3,600 humans who committed code to the project.

The contributor gap tells you something about the orchestration layer specifically. Airflow's ecosystem of provider packages (cloud connectors, custom operators, executor integrations) has lowered the friction for first-time contributors in a way that Spark's heavier core or Kafka's narrower domain focus hasn't matched. Contributor velocity is the best leading indicator of whether an open-source project will keep pace with the ecosystem around it. Right now, Airflow is outpacing both of its most obvious peers, and that matters more than any vendor's marketing slide.

84% Plan the Airflow 3 Migration. 26% Have Done It.

This gap is the most telling number in the entire report.

Airflow 3 shipped in April 2025. Less than a year later, 26% of respondents have completed the migration. 84% of those still on 2.x say they're planning it. That 58-point spread between planning and doing reflects engineering teams staring at a list of breaking changes and budgeting real project time.

The breaking changes aren't cosmetic. Airflow 3 removes SubDAGs entirely. It eliminates context variables like execution_date, prev_ds, and next_ds. Workers can no longer directly access the metadata database; everything routes through the REST API. xcom_pull(key=) stops searching upstream tasks. Each of these forces full DAG rewrites. If you've got 300 DAGs in production and half of them use execution_date for partition logic, you're looking at weeks of refactoring before you even touch your CI pipeline. That's why the gap exists.

Astronomer's own customer base tells a different story: 48% already run Airflow 3, and among their largest enterprise customers (50,000+ employees), 60% have deployed it. The gap between managed-platform customers and the broader community is a resource story. These companies have dedicated platform teams that can absorb migration work while the rest of the org keeps shipping. The 4-person data team at a Series B startup running 150 DAGs? They're going to plan the migration for Q3 and push it to Q4. Then Q1.

And the clock is ticking. Airflow 2 hit end of life in April 2026. No more security patches. No more bug fixes. No more provider package updates. If you're running Snowflake, Databricks, or BigQuery providers on Airflow 2, you're on borrowed time. Those vendors will start dropping 2.x compatibility in their own provider packages, and that second wave of pressure arrives 6 to 12 months after the official EOL. SOC 2, HIPAA, PCI-DSS; any certification that requires supported software makes this a compliance conversation on top of a technical one.

84% of teams know they need to migrate. 26% have done it. The gap is the actual cost of rewriting production DAGs while keeping the lights on.

If you're interviewing right now, expect questions on both Airflow 2 and 3 for at least another 12 months. The industry hasn't turned over yet.

The GenAI Production Gap

Here's where the survey gets uncomfortable.

Among all Airflow users, 32% report running GenAI or MLOps workloads in production. Among Astronomer's managed-platform customers, 62%. Among organizations that have been Astronomer customers for 2+ years, 83%.

51 percentage points between the general population and long-tenured platform customers. Read that again.

This has nothing to do with ambition. It has everything to do with infrastructure maturity compounding over time. Teams that spent years building out observability, error handling, schema governance, and pipeline idempotency can bolt on GenAI workloads because the plumbing already exists. Teams still figuring out how to make their batch jobs reliable aren't ready to orchestrate inference pipelines; they know it, and they're right to wait.

Separate research from K2view's 2026 enterprise survey backs this up: 76% of organizations cite data quality and consistency as their top barrier to production GenAI, and 62% say their enterprise data simply isn't ready. The bottleneck is the data platform underneath the model. MIT research referenced in the same analysis found 95% of enterprise AI pilots deliver zero P&L impact, which tells you the pilot-to-production gap is structural. You can't buy your way past it with a fancier model.

Meanwhile, 80%+ of the Astronomer survey respondents say they use AI tools to write pipelines but report that those tools hallucinate, lack context, and generate outdated syntax. I've been saying this for years: AI is making coding interviews increasingly pointless as a signal of engineering ability, but AI as a pipeline developer is nowhere close to replacing the engineer who knows why the pipeline broke at 3am last Tuesday and how to prevent it from happening again.

89% of Airflow users expect to expand orchestration into revenue-generating, external-facing solutions in 2026. The orchestration layer is moving from back-office utility to core product infrastructure. Your job as a data engineer just got a lot more visible to the business. That's both good and terrifying.

The Platform Split and What It Means for Your Next Data Engineering Interview

The conventional wisdom on LinkedIn says Databricks is running away with the market. The survey says: slow down.

Snowflake: 36.6%. Databricks: 34.7%. BigQuery: 27.8%. The total spread between first and third is under 9 percentage points. Snowflake leads Databricks by less than 2. Calling a winner here is irresponsible.

22% of Airflow users run 2 or more of the 3 major cloud data platforms. 1 in 5 teams operates a heterogeneous stack by choice. Intentional multi-platform architecture, not tech debt.

For interviews, this matters. A lot.

Snowflake loops emphasize the 3-layer architecture (storage, compute, cloud services), warehouse cost control, clustering strategies, and Snowpipe ingestion patterns. Databricks loops focus on Spark internals (shuffle semantics, partitioning), Delta Lake ACID transactions, and distributed systems thinking under concurrency pressure. These are fundamentally different prep surfaces. A single "data engineering interview" study plan that doesn't branch by platform will teach you the wrong material for half the roles you apply to.

Check the job posting. If it says Snowflake, drill Snowflake. If it says Databricks, drill Databricks. If it says both, they probably don't know what they want yet; ask in the recruiter screen. Don't waste 3 weeks studying the wrong platform because some influencer told you "Databricks is winning."

The concept-over-tool argument still holds, though. Data modeling, query optimization, understanding why things break; these transfer across all 3 platforms. The engineers who understand grain, normal forms, and late-arriving data will pass interviews on any platform after a week of syntax review. The ones who memorized COPY INTO but can't explain a slowly changing dimension will struggle regardless of which logo shows up in the job description. And if the loop includes a Python round, which most of them do now, python interview questions are what DataDriven is good for; we built the problem sets around the patterns that actually show up in production-focused loops, not LeetCode puzzles divorced from the job.

What This Survey Doesn't Tell You

Astronomer sponsors this survey and operates the platform it measures. The report blends community responses with "trends from Astro customer usage data," which means the 5,818 figure mixes voluntary respondents with inferred signals from their own product telemetry. Response rate and confidence intervals aren't published. The raw data isn't public. When the company running the survey also sells the managed version of the tool, numbers showing managed customers outperforming on every metric deserve some scrutiny.

Does that invalidate the findings? No. 5,800+ respondents across 122 countries is still the largest data engineering sample I've seen. The platform split, the migration gap, the GenAI adoption curve; these are directionally useful even with selection bias baked in. Use them as a compass, not a GPS coordinate.

The tools will keep changing every 18 months, like they always have. The problems won't. Schema drift, late-arriving data, upstream teams breaking contracts without telling you. Those are eternal. This survey confirms what I've watched play out for years: the industry is growing, the orchestration layer is becoming more strategic, and the gap between "I'm planning to do this" and "I've actually shipped it" is where careers get made.

What's your team actually running in production right now? Does it match what the survey says, or are you living in a completely different reality?