DEV Community: DataDriven

Fivetran Owns dbt Now: What Data Engineers Should Do Next

DataDriven — Thu, 12 Nov 2026 10:06:54 +0000

I've watched 3 vendor mergers from inside a data team. Every press release said nothing would change for customers. Pricing changed within 18 months each time, and so did the roadmap.

So when Fivetran and dbt Labs closed their all-stock merger on June 1, 2026, I didn't care much about the announcement wording. I wanted to know what was in the license text, which features need a login, and what it means for the next person walking into a Data Engineering interview with "dbt" on their resume.

My position: dbt is still worth knowing, and the merger doesn't change that. What it changes is how much of your career you should tie to one vendor's ingestion and transformation layer. That number should be lower than it was last year. The skills that hold their value are the ones that work with any tool: modeling, debugging, and knowing when a vendor's number doesn't add up.

The June 1 releases from Fivetran and dbt

The facts first. Fivetran's completion release names George Fraser as CEO and Tristan Handy as President, and says the combined company serves "more than 100,000 data teams." The release title is about "Trusted AI Agents," because of course it is. Every 2026 press release legally has to mention agents.

That 100,000 figure needs one caveat. Fivetran describes it as a community "across analytics, data engineering, and AI initiatives," while dbt's boilerplate says 100,000 teams "use dbt." Those cover different things, and I couldn't find an independent count. Treat it as marketing.

The more interesting part is dbt Core v2.0. It shipped as an alpha on launch day and is built on the open-sourced Fusion runtime. Read the details, though, because the "dbt Core v2 is here" post describes 2 separate distributions:

A default binary that dbt describes as "containing some proprietary code"
A build "made up entirely of the Apache 2 open source code in the dbt-core repo"

dbt's own recommendation is "Almost certainly Fusion," which means the binary with proprietary code. The same post says "some features will only work if you've authenticated, either with a free login or a paid dbt platform account." It doesn't list which ones.

The install docs fill in part of the gap. The Apache-only build leaves out SQL comprehension, full LSP, dbt lint, and VS Code integration. dbt says that build is "for organizations that require the Apache 2.0 codebase specifically," so the open version exists, but dbt doesn't expect most people to run it.

To be fair to dbt Labs, the license has moved in the open direction. Fusion launched under Elastic License 2.0 in 2025, and the licensing FAQ's change log shows code "relicensed from ELv2 to Apache 2.0" on June 1, 2026. Handy's merger-day post says "Our commitment to open source isn't changing." I believe him about the license. The license isn't the part I'm worried about.

Apache 2.0 tells you who can read the code. The login gate decides who controls your workflow. Watch the gate.

The parse-speed claim is thinner still. The v2 post's only performance statement is "Significant parse time improvements, especially on the largest dbt projects." It gives no numbers, benchmarks, or test setup. It might well be faster; Rust parsers usually are. But "significant" with no number attached is a vibe, and I can't put a vibe in a capacity plan.

dbt State and the 30% number

The merger release also introduced dbt State, described as "a caching layer for data pipelines" that helps companies "reduce underlying infrastructure costs by 30% or more."

dbt's own product page for dbt State says something different: "Reduce warehouse compute by 15-30% on average." So the product page puts 30% at the top of the range, and the press release treats it as the minimum. Both come from the same company, and they don't agree.

I went looking for an independent benchmark and didn't find one. Every 30% figure I could trace leads back to dbt or one of its customers. The headline case study is Obie Insurance, where a senior data engineer said they were "saving at least 30% on compute costs." During the same period, that team was more than 90% through a Data Vault 2.0 migration and had moved to the Fusion engine. When you change your data architecture, your engine, and your caching layer at the same time, you can't say which one saved the money. Any DE who has done a migration knows that.

Other things to know before you put 30% in a budget deck:

Python models always rebuild, and so do custom materializations. If your project leans heavily on Python, the cache helps you less.
The quoted savings are gross. Pricing is per "daily active target table," and I found no published figure that subtracts that fee from the savings.
Change metadata (timestamps and SQL hashes) goes to a single US multi-tenant instance, which your compliance team will want to know about.
The status keeps changing. The release called it a preview, a September 2026 post declared it GA, and the docs still mark the state:* selectors as Beta.

I'm not saying it doesn't work. Skipping unchanged models is a good idea; most of us have hand-built a worse version of it with a last_modified check and some prayer. My point is economic. Engineer time costs more than compute, so before you spend a quarter adopting a caching layer, measure your own warehouse bill before and after on a project that isn't also being migrated. That measurement is the only one I'd trust.

Lock-in is the part that matters for your career

Before the merger, a typical modern stack had ingestion from one vendor and transformation from another. If Fivetran raised prices, you could move ingestion and leave your dbt models alone. If something went wrong with dbt, your connectors kept running. Each vendor knew you could swap it out without touching the other.

Now one company owns both layers. That isn't automatically bad. Integrated tools are often nicer to use, and I've spent enough nights gluing 2 vendors together through a webhook to see why people want it. But the merger-day posts don't say anything about pricing for existing customers, contracts, or connector changes. You'll see plenty of loud anecdotes online about Fivetran price increases; I couldn't verify any of them against a primary source, so I won't repeat the numbers. You don't need them anyway, because the missing pricing statement tells you enough.

Fivetran bought Tobiko Data, the company behind SQLMesh, in 2025, then donated SQLMesh to the Linux Foundation in March 2026. So the most obvious alternative transformation framework is now under neutral governance, and the company that owns dbt paid for it. Before the acquisition, Tobiko's founders argued that analytics engineers need an open standard "so their careers don't depend on one company." That was a competitor's opinion when they wrote it. Now it's partly the acquirer's donation strategy. Funny industry.

As a working DE, I'd start by finding your exit cost. The license won't trap you; your Jinja will. Count your macros. Count the models that use dbt-specific behavior you couldn't easily rewrite in plain SQL. That number is your real switching cost, and most teams have never measured it.

Keep ingestion and transformation separable on paper, even if both come from the same vendor. Land raw data in tables you own, with schemas you've documented. If you ever have to replace one layer, you want to replace one layer, not both.

Expect the upgrade itself to be strict. The v1.x support windows are already pushing teams to upgrade. dbt's upgrade guide says "All deprecation warnings must be resolved before upgrading," and the recommended path is v1.12 first. In v2, , models and -m now error, misspelled configs and unknown YAML keys error, and nonexistent macros fail at parse. Your project probably has a few of those. Every project I've inherited did. According to dbt's versions page, v1.11 leaves critical support on December 18, 2026, which is about 5 weeks away.

Then check your adapter. The documented v2 adapters are BigQuery, Databricks, Redshift, and Snowflake, with ClickHouse in private beta and Spark and DuckDB as CLI-only. Postgres, Trino, and Athena were described as "likely" before GA back in May 2025 and still aren't on the supported list. If you run on Postgres, v2 isn't ready for you yet.

The v2 post also says: "You don't have to move to v2.x today, tomorrow, or ever." That's true for now. Then Handy writes "we will simply have one engine: dbt." My guess, and it's only a guess, is that the Apache-only build ends up as a side option over time. Plan as if the free tier will lag behind.

What changes in the data engineering interview loop

On the hiring side, the evidence is thinner than people want. I looked for a primary dataset showing which dbt skills interviewers test and couldn't find one. What exists is prep-site content and vendor surveys, and dbt Labs ran its own State of Analytics Engineering survey, which now belongs to the merged company. So everything below is my opinion, based on running loops and sitting through a lot of them.

"I know dbt" was never the signal on any panel I've been on. Nobody cared whether you could write {{ ref() }}; you can learn that in an afternoon. We cared about whether you could explain why the model was incremental, what happens when data arrives 3 days late, and how you'd find out a test was passing while the numbers were wrong. Those questions were the same in the Informatica days, the stored procedure days, and the Airflow-plus-SQL-files days.

Expect questions about consolidation, because interviewers like to ask about whatever's in the news. If a team has just been through a 6-figure Fivetran renewal, someone on the panel will ask how you'd evaluate building your own ingestion versus buying it. Have an answer based on cost: engineer hours, connector maintenance, schema drift handling, and what happens when the vendor raises prices. "We used Fivetran because it was there" is a weak answer, and you should have a better one ready.

The strictness of v2 also gives you interview material. If you've done the 1.x to v2 upgrade, you have a real story about deprecated flags, configs that failed silently, and broken macros that suddenly showed up. That's a debugging story, and debugging is most of the job. Tell it in the STAR format and it'll beat any answer about features.

Testing is underrated, and you can use that. dbt's own survey found 72% of respondents prioritize AI-assisted coding, but only 24% prioritize AI-assisted pipeline management, the testing and observability side. The sample is 363 people and the survey is run by the vendor, so don't overread it. But it matches what I see: everyone wants to write code faster, and fewer people want to prove the code is correct. The candidate who can explain how they caught a pipeline silently dropping rows stands out from the 10 others who list "dbt tests" as a bullet point.

Indeed found 45% of data and analytics postings mention AI terms as of December 2025. That's a keyword count of posting text, and Indeed's own numbers say it's still a minority, so don't let it pull your prep away from the basics. Job postings change their vocabulary faster than interviewers change their questions. Your loop will still test SQL, modeling, and pipeline architecture.

If I were prepping now, I'd learn dimensional and wide-table modeling until I could argue for either one based on cost. I'd practice designing incremental loads with late-arriving data, and explaining a pipeline from source to dashboard on a whiteboard, including where it fails. For the architecture part, pipeline design practice is what DataDriven is good for; we wrote our problems to sit as close as we could get to the questions panels actually ask, without the vendor trivia.

If you only know dbt, learn the SQL dbt generates. If you only know Fivetran, build one ingestion job by hand with retries, schema changes, and alerting. You'll understand what you're paying for, and you'll have something to say when the interviewer asks what happens if the vendor goes away.

The tools have changed owners, but the work hasn't: schema drift, late data, and upstream teams breaking contracts without telling you.

So, for those of you running dbt in production: are you upgrading to v2 this quarter, staying on 1.x as long as the support windows allow, or quietly testing SQLMesh in a branch nobody knows about?

Snowflake's AI Agents Fix Pipelines: Data Engineer Impact

DataDriven — Tue, 10 Nov 2026 10:06:22 +0000

The first time I trusted an automated "fix" to a pipeline, it fixed the error, and the fix was wrong. The job went green. The row counts didn't. It took 3 weeks and an angry finance analyst before anyone noticed that the dedupe logic had been "simplified" right out of existence.

So when Snowflake announced on Nov 3 that AI agents would now investigate failed pipelines and rewrite Spark jobs for you, I didn't panic. I did what I always do: I read the release notes, the docs, and the footnotes. My take is that this launch makes the Data Engineer who can review a change and own it worth more than before, while the engineer whose whole value is typing out the fix gets cheaper. Your next interview loop will start to reflect that.

What Snowflake shipped, and what is still in preview

Marketing pages blur everything together, so here's the status of each piece:

CoCo Cloud Agents: generally available.
CoCo Automations: GA as of Nov 2, but only in Snowsight and the CLI, and only in commercial AWS, Azure, and GCP regions. Cloud automations in CoCo Desktop are still preview. Government, FedRAMP, and China deployments don't get it.
Pipeline Builder: public preview, and it has been since Oct 22. The Nov 3 release just restated it.
CoCo Spark-to-Snowflake rewrites: marked GA in the launch press release footnotes. I couldn't find a standalone release note for it, so the GA status comes from one source.

This status moved fast. Automations were preview on Aug 21 and GA by Nov 2. Anything you read over the summer probably has the status wrong.

The headline pitch is in Snowflake's Nov 3 press release: when a scheduled Automations run "finds a problem in a pipeline, CoCo can trace the likely cause and propose a fix for engineers to review before they decide what goes to production."

Now read the mechanics. Per the docs, automations run on a schedule (hourly at most), with no event triggers, and run history is kept for 2 months. Each run runs as the user who created it, and you get an audit trail of what CoCo checked, found, and did. You pay standard task billing plus CoCo tokens for every run. The Automations blog describes all of that. What I couldn't find described anywhere is an approval gate that physically stops the agent from applying a change before a human signs off.

The review step is only a real control if the agent can't do the thing without you, and without that gate the guardrail comes down to role grants and scope. If your automation runs as a user with write access to prod tables, then the "human review" is whatever that role allows. Your security model is the actual reviewer, and you designed it, so check it.

Snowflake's own engineering team has been running an internal version of this. Their Oct 19 post on fixing pipeline failures with AI reports under 16 minutes on average from detection to PR, 100+ merged PRs, and 500+ pipelines improved over 5 months. Their framing: "The agent proposes the repair. A person still decides what ships."

Those are good throughput numbers. The post doesn't report an error rate, a rollback rate, or how many PRs a human rejected for being subtly wrong. Speed to PR shows the agent is fast and says nothing about whether it's right. My job-went-green-and-the-numbers-were-wrong story would have scored great on time to PR.

Data teams already distrust the part Snowflake automated

The vendors are selling hard. The practitioners are hedging.

dbt Labs' State of Analytics Engineering 2026 surveyed 363 data people between Dec 2025 and Feb 2026. 72% prioritize AI-assisted coding. Only 24% prioritize AI-assisted pipeline management, which includes testing and observability. And 71% cite incorrect or hallucinated outputs reaching stakeholders as a top concern.

Some caveats, because I'd want them if I were reading this. These are priority shares, so they don't measure adoption, and nobody's claiming 72% of teams run AI coding in prod today. The 71% measures concern; the survey reports no hallucination rate. It's also a vendor survey with a sample of 363, so treat it as a direction, not gospel.

Even with those caveats, the direction is pretty clear. Teams are happy to let AI write the code. They don't trust it to watch the code. Snowflake just shipped a product aimed right at the part people trust least.

The same report has the stat I find most interesting. The share saying trust in data and data teams is important went from 66% in 2025 to 83% in 2026. Meanwhile 57% report higher warehouse and compute spend, versus 36% reporting bigger team budgets. When spend climbs faster than headcount money and trust tops the worry list, an agent gets bought, and somebody has to be accountable for what it does.

The agent can open the PR in 16 minutes. Someone still has to know the PR is wrong, and that someone is the job.

I've been through 3 waves of "data engineering is getting automated away." Every wave automated the typing and left behind the judgment calls: what's the grain, who owns this table, is this late data or missing data, did the upstream team change the contract again. Schema drift, late-arriving data, and silent row drops don't care what tool you're on. The agent will hit those same problems and propose fixes that look plausible, which is exactly what makes them dangerous.

There's also a review-queue problem. LinearB's 2026 benchmarks, drawn from 8.1M+ PRs (vendor data, so salt accordingly), found AI-generated PRs wait 4.6x longer to be picked up, then get reviewed 2x faster once someone opens them. I read that as reviewers avoiding the PR, then rubber-stamping it. If Automations starts dropping proposed fixes into your thread every hour, your team's bottleneck becomes reviewer attention. Stack 20 agent proposals on a Monday and watch how carefully anyone reads number 17.

The fix is boring and old. Write-Audit-Publish was in a Netflix data quality deck back in 2017: write to staging, run audit checks, publish only when they pass. Snowflake doesn't ship WAP as a feature, and its docs say you can't create Iceberg branches on the Horizon external-engine path. So you build it yourself: a staging table, audit queries that check row counts, null rates, and key uniqueness against the last good run, and a swap you control. The agent proposes the change, the audit queries decide whether it can publish, and a human approves the swap, so nobody can skip the review.

Writing those audit checks is your job, and it's the skill that matters here.

Spark rewrites and the silent bugs data engineers will own

The Spark migration piece is where I'd be most careful, because it's where the bugs hide best.

Snowflake's CoCo Spark Migration skill "assesses compatibility, converts code and validates the changes," and the docs target the PySpark 3.5.3 Spark Connect DataFrame API. Some things just don't come over: RDD and low-level APIs (sc.broadcast, sc.accumulator, foreach), Structured Streaming, and writer methods like bucketBy and insertInto. Those fail loudly, which is fine, because you find them on day one.

The silent stuff is what worries me. Straight from Snowflake's own known-differences page:

Struct field order: Snowflake alphabetizes struct fields, while Spark preserves definition order. If your SCD Type 2 logic hashes a struct to detect changes, every hash changes. Congrats, every row in your dimension is now "updated."
Integer overflow: Spark wraps on overflow; Snowflake doesn't unless you set handleIntegralOverflow. Same code, different numbers.
Pre-Gregorian dates: may be "corrupted silently" when Parquet rebase modes aren't supported. The docs actually say "silently."

None of these throw an error. The job runs and the dashboard loads, but the numbers are wrong in a way only someone who understands the data model would catch. A rewrite that compiles and passes a row count check can still blow up your slowly changing dimension. Reviewing these rewrites means reviewing semantics. If you don't know why your SCD2 logic hashes what it hashes, you can't review the agent's version of it.

Then there's cost. Snowflake's customer story says Klarent cut TCO by 60% versus Amazon EMR, and their longest pipeline went from 10 hours 13 minutes to under 3. That's one customer, quoted by the vendor, with no time period, dollar figures, or methodology published. Snowflake's own engineering blog is more honest about it: "TCO therefore remains an estimate until the port is far enough along to compare."

So baseline before you touch anything. Get a full month of runtime and cost on the current platform, on a fixed workload. Otherwise you'll be "proving" savings against a number nobody wrote down. I've watched a migration get declared a win because nobody remembered what the old bill was. (Somebody got promoted for that one.)

This is also a competitive story. Databricks announced in August that it crossed a $7B revenue run-rate with >80% YoY growth. Snowflake making it easy to bring Spark over is an obvious move in that fight. Fine. Vendors fighting over your workloads is good for your leverage. It doesn't change who's on call when the rewritten job double-counts revenue.

What this means for the data engineer interview loop

A warning first: I found no primary source on what DE interviewers are testing right now. What follows is my opinion, from sitting on both sides of lots of loops.

What I can source is the shape of hiring. Indeed Hiring Lab found senior US postings up 14.7% year over year as of May 2026, with entry-level down 7.5%. In tech, the senior share of postings rose 9 points from 2019 to 2025, mostly at the expense of mid-level roles. That's all postings, not DE specifically, so don't stretch it too far. One detail stood out to me: Indeed labels seniority by task content, like ownership and decision rights, rather than by title. "Senior" in those postings looks like "the person who decides what ships."

That lines up exactly with what an agent like CoCo needs sitting next to it. Here's where I think loops drift over the next year or two:

Review questions will replace some build questions. I've said for a while that if AI can solve a medium coding problem, asking it tells me very little about you. Expect more "here's a PR, what's wrong with it" questions. Give a candidate an agent-proposed fix for a failed job and see if they spot that it drops late-arriving records. That's a far better signal than reversing a linked list.

Data modeling will get more weight, because you can't review a fix if you don't know the grain. You can't catch the struct-hash SCD2 bug if you don't understand how change detection works. Keep fact tables at grain; you can aggregate up, never down. Interviewers who've been burned by a bad automated change will probe for exactly this.

Ownership and permissions will show up in pipeline architecture rounds: "Your agent runs as a service role on an hourly schedule. What can it touch? Where's the gate? How do you roll back?" If you can talk through a write-audit-publish flow, role scoping, and a rollback plan, you sound like the senior person in those Indeed postings.

Cost judgment gets tested too. "Vendor says 60% cheaper. How do you check?" The right answer starts with a baseline and a fixed workload, not with enthusiasm.

To prep, practice reading code you didn't write. Take a PySpark job, port it in your head, and list what changes behavior without breaking. If you want structured practice on that, for spark practice, datadriven.io is great; we built it for working through the transformation problems that matter. Keep doing your 50 mediums for the loops that still ask them. Play the game. But spend real time practicing explanations: why this grain, why this audit, why you'd reject this PR.

The job was always mostly debugging. Less "write a DAG," more "figure out why this pipeline silently dropped 2M rows last Tuesday." Snowflake just built a tool that writes the first draft of the debugging. Somebody still has to know whether the draft is right, and companies will pay well for that person.

So here's what I want to know from people who've actually turned these on: what's the first agent-proposed fix you rejected, and what would have broken if you'd merged it?

AI in Data Engineer Interviews: Banned, Allowed, or Graded?

DataDriven — Tue, 03 Nov 2026 10:06:51 +0000

I sat on a hiring panel recently where 4 senior engineers spent 40 minutes arguing about whether a candidate had cheated. The candidate used an AI assistant on a take-home SQL exercise. Nobody had written down whether that was allowed. Half the room called it fraud; the other half pointed out that all 4 of us had used AI tools that morning to write the code we'd shipped before lunch.

That argument is happening at every company right now, and the result is a mess for candidates. Some Data Engineer loops ban AI, some allow it, and a growing number grade how you use it. With no shared rule across hiring teams, you can't prepare for one policy, so you have to prepare for both. The skill that works under either one is verification: proving your SQL is correct, quickly and out loud, whether you typed it or a model did.

Data Engineer interviews have no shared AI rule

Start with the job itself. Stack Overflow's 2026 Developer Survey found that 73% of respondents who use AI coding assistants or agents use them daily. Read the denominator carefully. That figure covers people who already use AI tools, so it doesn't mean 73% of all developers. Even so, among people who've adopted these tools, daily use is normal. The survey's breakdown of what people use AI for reads like a DE's week: generating code in a familiar area (69.3%), debugging and refactoring (63.8%), writing tests (58.1%).

Now look at the interview. Karat's research page on 2026 interview trends reports that 62% of organizations still prohibit AI use in technical interviews, based on 400 engineering leaders across the U.S., India, and China. Policy also changes a lot by region. On the same page, 38% of U.S. organizations allow AI in live interviews, compared with 68% in China. If you hear that "most companies ban AI," that's true in aggregate and misleading for any single loop.

Karat's own December 2025 release on its 2025-2026 AI Workforce Transformation Report shows the gap more sharply. Nearly 70% of engineering leaders plan to strengthen their AI capabilities through hiring, and less than 30% are updating assessments and training interviewers to identify AI-ready talent.

So leaders want to hire people who are good with AI tools, and most of them haven't changed the test that's supposed to find those people. I've watched every hype cycle get a hiring mandate before it gets an evaluation process.

A few caveats. Karat sells an AI-enabled interview product, so it benefits when companies worry about this question. Its pages report different sample sizes (400, 150, and 157 depending on the edition), and the release doesn't publish its methodology. Stack Overflow says nothing about interview policy at all. "Interviews lag behind daily practice" is my argument from putting those 2 data sets side by side. Neither source measures that lag directly.

I'm still confident in the argument, because I've been on both sides of these loops. Karat also quotes tech leaders who estimate that over half of candidates use AI despite being told not to. That's an estimate, but it matches what I've seen on panels, where the bans leak.

You can't control which AI policy you'll get. You can control whether your SQL holds up under either one.

AI-allowed rounds grade whether you checked the output

Candidates often assume AI-allowed rounds lowered the bar, when they actually moved it somewhere else.

Karat publishes its Human + AI technical interview rubrics, and they're blunt. The rubrics assess "when and why AI is used, how prompts are scoped, and whether outputs are checked." They also state that candidates "should neither be penalized nor implicitly rewarded for AI-outputs." Karat adds an AI Proficiency competency with 2 parts: Prompting AI Code and Evaluating AI-Generated Code. Its stated reason is one every DE who has been paged at 2am will recognize: unchecked AI-generated code introduces operational, security, and reliability risks.

Canva's engineering blog says nearly the same thing in an interviewer's guide by Karl Hörnlund. AI collaboration is one of 3 graded areas, next to problem-solving and technical depth. Strong candidates clarify requirements and state assumptions before they write code. They use AI for skeleton code and review the complex suggestions. The weak patterns have names, and they're good ones: "AI Showcase," "Feature Marathon," and "Hands-Off."

I've seen all 3 patterns in real loops.

The first pattern is the candidate who spends 10 minutes crafting the perfect prompt and can't tell you why the join is a LEFT JOIN. Feature Marathon looks like 6 extra CTEs nobody asked for, generated because the assistant made them cheap. Hands-Off is the one who pastes the output, watches it run, sees rows come back, and says "looks good." Rows coming back proves the query runs, not that the numbers are right.

Neither rubric ranks speed. My read is that an unchecked fast answer should score below a slower one you can verify and explain. Both Karat and Canva point that way, even though nobody has published evidence that these rubrics predict job performance. Karat's rubric page says "research shows" human-plus-AI interviews produce the best outcomes and gives no figures. Treat that as the vendor's position.

The bigger point is that AI-allowed rounds are finally testing the actual job. I've said it for years: the job is less "write a DAG" and more "figure out why this pipeline silently dropped 2M rows last Tuesday." I once spent a week on a job that had been dropping 40% of records for 6 months. The code was clean, the job succeeded every night, and nobody had checked the row counts. An AI assistant would have written that same clean code, and someone would still have had to check the counts.

The Stack Overflow numbers support this too. 93% of respondents rate source attribution as at least somewhat important when verifying AI output, and 48% trust AI output when they can easily verify it. Interviewers who allow AI are grading whether you do that checking.

Practice verification on real postmortems

Verification is hard to practice in the abstract. "Check your edge cases" is advice everyone nods at and nobody does. The most useful practice I've found is taking real incidents apart, because they give you concrete stories to tell in pipeline design and behavioral rounds under either policy.

Take Zendesk. Its postmortem for the June 26, 2026 Sales incident attributes the outage to "an accidental command during maintenance that unintentionally deleted seven important tables in the Sales database." The tables were restored from a recent backup, followed by a controlled reconciliation, and the final update on July 1 says remediation was complete. (The summary window ends June 28, but the dated updates run to July 1. Cite the updates.)

Look at their preventive actions: remove unnecessary Drop privileges, set environment-specific access controls, "review and regulate activities scheduled on Fridays" (every DE who has ever deployed on a Friday just felt that one), and add safeguards that block harmful commands in production.

Every one of those fixes is a permission or a guardrail; none of them relies on people being more careful. The postmortem never mentions AI tools, so don't tell it as a story about an assistant gone wrong. It's useful precisely because the fix works whether a human or an assistant types the command. In a banned-AI loop, it's a story about access control. In an AI-allowed loop, it answers "how would you let an agent near production?" You'd let it near production with the same safeguards you'd use for a tired engineer at 5pm on a Friday.

CircleCI's July 2, 2026 post-incident report teaches a different lesson. A maintenance data deletion job, split into several hundred tasks, started at 13:30 UTC. Customer project deletion requests, unrelated to that job, started at 14:40. Pipelines failed to start between 15:12 and 16:44, and some customers couldn't log in. CircleCI named the root cause as connections between the API layer and the workflows data service that weren't configured to time out slow responses under load. The deletion jobs supplied the load; the missing timeout was the bug.

Most candidates blame whatever job happened to be running. Strong candidates separate what started the incident from the guardrail that was missing. If you can explain that distinction clearly in an interview, you're showing the same judgment a rubric is looking for when it asks whether you checked the AI's output.

For practice, pick a postmortem and write 3 sentences: the trigger, the missing guardrail and the fix. Then explain how you'd catch the same failure class in a pipeline you own. Do that 10 times and you'll have a library of stories that works in any loop.

Prepare for both policies from one shared core

Before every round, ask the recruiter in writing whether AI is allowed, which tools, and whether the round is live or take-home. I once did 8 rounds at a company, was told an offer had been sent, never received it, and was then told I'd declined it. Verbal policies at companies like that change between the phone screen and the onsite.

Then build a shared core that works under either policy, and practice it without an assistant. In SQL, that means window functions, deduplication, incremental loads, NULL handling, and joins that fan out when keys are duplicated. In pipeline design, say the grain, the late-arriving data plan and the backfill strategy before you draw any boxes. For failure handling, know retries, idempotency, and what happens when an upstream team changes a contract without telling you (they will).

These concepts transfer across policies the same way they transfer across tools, and the syntax is the easy part.

For banned-AI loops, drill from memory with a timer. Mediums are enough; do 50 and you'll be solid. Practice explaining your query plan out loud, because silence on a whiteboard reads as being stuck. Keep a short syntax cheat sheet and rebuild it from memory until you don't need it. That's the old game, and the arbitrary rules still apply.

For AI-allowed loops, the main risk is accepting generated code you can't defend. Practice reviewing AI output specifically for bad joins, missing NULL checks, timezone errors, and late-arriving records. Narrate each check: "I'm verifying the grain before I trust this count." Clarify requirements before prompting, scope prompts tightly, and turn down suggestions you don't need. Turning down a suggestion and explaining why is one of the strongest signals you can send in that room.

There's research behind practicing without AI too. In a randomized trial, Anthropic found 52 mostly junior engineers averaged 50% on a comprehension quiz after coding with AI, compared with 67% for those who coded by hand. The speed gain was around 2 minutes and wasn't significant. The sample is small and the quiz was taken immediately after the task, so don't overread it. It still matches what I see on panels: candidates who learned SQL with an assistant doing the thinking struggle to verify anything, because they never built the mental model to check against.

That's why I'd put most of your practice time into unassisted SQL, then add AI-review practice on top. If you're looking for timed, unassisted practice, for sql coding questions, datadriven.io is great. It's our site, so weigh that, but it's where we send people who ask where to drill window functions before an onsite. Once your fundamentals hold up without help, run the same problems with an assistant and practice catching where it's wrong. It will be wrong. In my experience, it's most often wrong on the edge cases interviewers love.

The candidate I'd hire in 2027 can do both: write a correct dedup query from memory in a banned loop, then catch the assistant's fan-out join in an allowed one, and explain both in plain English. Policies will keep moving for a while, and in my experience the pattern of companies demanding a skill before they know how to test it lasts around 18 months. The verification skill will still matter after that settles.

So which policy have you run into in your last few Data Engineer loops: banned, allowed, or graded? And did anyone tell you before the round started?

DuckDB 2.0 for Data Engineers: Upgrade Risks and What to Know

DataDriven — Thu, 29 Oct 2026 10:06:33 +0000

Every database upgrade I've run has failed in one of 2 ways. The first kind crashes at 3am, pages somebody, and is fixed by breakfast. The second kind runs clean through every smoke test and returns a slightly different number for 3 weeks until finance asks why the dashboard moved. I'll take the crash every time.

DuckDB 2.0.0 LTS ("Cinnamon Teal") shipped on October 21, 2026, and it can fail both ways. It adds a client-server mode, async I/O for object storage, a VARIANT type, triggers, and a new parser. It also writes a file format older versions can't open, and it changes some SQL behavior in ways that give new answers without raising an error. If DuckDB sits anywhere in your pipelines, you have an upgrade decision to make this month.

My position is that the 2.0 additions are nice, but the upgrade plan is where the engineering happens. The concepts behind that plan are compatibility, rollback, write coordination, and verifying results. Those are also what a good data engineering interview should test, whatever tool is on the whiteboard.

19 days left on DuckDB 1.4 LTS

Start with the dates. The official 2.0 announcement says 1.4 LTS "is still supported until November 17, 2026." Today is October 29, so that leaves 19 days. 2.0 is supported "until at least October 2027."

Read the fine print on "LTS." DuckDB's release cycle docs say "Currently LTS releases will receive 1 year of support," and feature releases land about every 6 months. Enterprise people hear "LTS" and picture 5 years of security patches. Here you get 1 year.

Then there's the odd part. 1.5 was never labeled LTS, but its support got extended to March 2027, "effectively also turning it into an LTS release." So the non-LTS line outlives the previous LTS by roughly 4 months. If you're on 1.4 and need more time, moving to 1.5 is the cheap option, because it keeps the old file format and the old parser and stays supported into spring.

Also watch when deprecations land. Single-arrow lambdas and osx_amd64 builds are both removed in 2.1. If the 6-month cadence holds, 2.1 shows up around April 2027. That's my math, not a DuckDB date. Either way, "2.0 is supported through 2027" and "my 2.0 SQL keeps working through 2027" are separate promises, and only the first was made.

The DuckDB team's own advice is to test 2.0.0 on a shadow system first, and for mission-critical workloads, to wait for the first patch release, expected in November. Take that advice.

An LTS label tells you when the vendor stops fixing bugs. It doesn't tell you when your rollback plan stops working; that date is the first time a 2.0 process writes a file.

SQL changes in DuckDB 2.0 that won't throw an error

Deal with the file format first, because it decides whether you can roll back. According to the 2.0 migration guide, 2.0 writes storage version 69 by default. Point a 1.5.6 binary at that file and you get:

"Trying to read a database file with version number 69, but we can only read versions between 64 and 68."

2.0 can read and write your existing files, and it doesn't upgrade them automatically, so those files stay readable by older versions. New files are a different story. Once a 2.0 process creates one, the only way back to 1.x is to export and re-import. Reinstalling the old binary won't help.

If you need a way back, pin the storage version from day 1:

,  Create/attach a database file that 1.5.x readers can still open
ATTACH 'shared.duckdb' AS shared (STORAGE_VERSION 'v1.5.6');

,  Or set it for the session before creating new files
SET storage_compatibility_version = 'v1.5.6';

On the CLI, the equivalent is duckdb -storage-version v1.5.6 shared.duckdb.

Every reader and writer of a shared file has to move together. That includes the analyst's notebook pinned to 1.4, the cron job nobody owns, and the Lambda someone deployed in 2025 and forgot about. The Python client now needs Python 3.11 or newer, so some of those consumers are blocked before they ever open the file. Build a dependency map by tracing every process that opens every .duckdb file. You'll find at least one you didn't know about; I always do.

Next is the PEG parser, which replaces the PostgreSQL-derived parser. Usually you won't notice. The migration guide lists a few behavior changes that will change results quietly:

,  2.0 default: ~ and !~ do partial regex matching
SELECT 'senior data engineer' ~ 'engineer';

,  Restore the pre-2.0 full-match semantics if your filters depend on it
SET regex_match_operator_semantics = 'full';

,  Operator precedence changed: this returns 12 in 2.0
SELECT 2 * 3!;

X'…' and B'…' literals now produce BLOB and BITSTRING instead of VARCHAR. A regex filter in a dedup step that used to match whole strings can start matching substrings, and nothing crashes while your row counts drift.

Some changes are loud, which is the kind you want:

// and % by zero now raise an error instead of returning NULL.
A MERGE INTO that touches the same target row more than once now errors.
ICU moved out of core, so timezones, calendars, and collations need the icu extension. If your containers can't download extensions at runtime, check this before cutover.
S3 URL parameters are disabled by default.
Building from source needs a C++17 compiler now.

Then lambdas. x -> x + 1 is disabled by default. You can turn it back on:

SET lambda_syntax = 'enable_single_arrow';

Both the flag and the syntax disappear in 2.1, so this flag is a delay, not a fix. Use it to get through the upgrade, then grep your codebase and rewrite them. If the new parser breaks something you can't fix in time, the legacy_parser extension with SET enable_legacy_parser = true; gets you the old parser, but treat that as an emergency brake you pull once.

What works is running your production queries on 1.x and 2.0 against the same data and diffing the outputs. "It ran" isn't a pass. Compare row counts, checksums, and the aggregates finance looks at, and trust that diff over the release notes.

Client-server mode turns DuckDB into a shared database

For most of its life, DuckDB's pitch was what the "Why DuckDB" page still says: "there is no DBMS server software to install, update and maintain." 2.0 ships a polished version of Quack, the client-server protocol that first appeared in 1.5.3 in May 2026. Now there can be a server.

The Quack design post describes it as built "straight on the venerable HTTP," using request-response with a single round trip per query once connected. The defaults are reasonable: it binds to localhost only, listens on port 9494, and generates a random token at startup. 2.0 also adds CONNECT and DISCONNECT statements, and for PostgreSQL targets the remote pushdown optimizer sends SQL to the server instead of pulling whole tables over the wire.

The benchmark numbers in that post look great: 60M TPC-H lineitem rows moved in around 4.94 s with Quack, 17.40 s with Arrow Flight SQL via GizmoSQL, and 158.37 s with PostgreSQL. Read the methodology, though. DuckDB ran it on its own setup: m8g.2xlarge instances in the same availability zone with a 0.28 ms ping. Your cross-region setup will look different.

The security defaults are thin. In DuckDB's own words, the default authorization function "just says 'yes' to everything." The token is the entire access control story out of the box. The team's advice is a TLS-terminating reverse proxy, and they say, "We do not recommend opening up a DuckDB Quack endpoint directly to the Internet." Somebody will do it anyway, so put the proxy in place before the first remote query.

Concurrency is better than it was and still not fully solved. The same post says multiple processes can write in parallel, and it also admits "we hit a current limitation of DuckDB itself in concurrent insertions per second into the same table," with PostgreSQL scaling better on that workload. One more catch: attaching a remote catalog is all-or-nothing, so one table definition that can't bind fails the whole attachment.

Async I/O is the other big change. Pedro Holanda's pre-release design post shows TPC-H Q6 at SF100 against S3 dropping from 8.230 s on 1.5.5 to 2.844 s on a 2.0 dev build, with network throughput rising from around 5 Gbit/s to near the 25 Gbit/s limit. An 80.89 GB remote CSV went from 877.563 s to 45.264 s. On a local SSD the gain was roughly 1.5×. Most of the headline speedup comes from the network, so if your data sits on local disk, adjust your expectations.

Check these before you put "3× faster" in a planning doc:

Row group count matters. In a single-file test, 1 row group took 25.26 s, 10 took 3.69 s, and 306 took 2.11 s. Check how your Parquet writer sizes row groups.
The memory limit is soft. With an 8 GB limit, the benchmark peaked at 11.5 GB RSS, because read-ahead memory isn't tracked by DuckDB's memory manager.
Async covers Parquet, uncompressed seekable UTF-8 CSV, and the native format. JSON and compressed CSV still use synchronous I/O. All the benchmarks came from dev builds on a single EC2 configuration.

What a data engineer should explain about DuckDB in an interview

If DuckDB comes up in your next loop, and it will more often now, the interviewer doesn't care whether you've memorized the CONNECT syntax. They want to know whether you can reason about tradeoffs. Here's what I'd want a candidate to explain.

Start with when DuckDB beats a warehouse. The usual citation is Jordan Tigani's "Big Data is Dead", which argues 90% of queries processed under 100 MB among BigQuery customers spending over $1,000 a year. Know the caveats too: Tigani says his charts are "hand-drawn based on memory," the data can't be shared, and the post lives on a DuckDB company's blog. I agree with the direction of it; most companies have medium data and big egos. A strong candidate cites the claim and its limits in the same breath.

2.0 changes that question. Before, the conversation was about data size. Now it's about write coordination. Multi-process writes to the native format go through Quack, or you can use DuckLake with a PostgreSQL catalog (DuckLake 1.1 shipped alongside 2.0 and is intended for production use). So the questions become who writes, how many writers there are, how they coordinate, and how you upgrade all of them together. That last one is a pipeline architecture question, and it has nothing to do with load balancers.

VARIANT now "shreds" common structure, but only in row groups with at least 30,000 rows, so small micro-batches get no benefit. Object keys come back sorted, which breaks any pipeline that depends on key order or byte-level equality. DuckDB published no performance numbers for it, so a skeptical answer gets credit here.

Triggers versus idempotent upserts is my favorite question because it catches people. Triggers are new in 2.0, and the obvious use case is audit logging. On a table with UPDATE or DELETE triggers, though, INSERT ... ON CONFLICT, INSERT OR REPLACE, and MERGE INTO all fail. If your pipeline's idempotency depends on upserts, adding audit triggers means redesigning it. A candidate who can explain that tradeoff on the spot has done this work before.

Agent mode is a good test of whether someone measures the right thing. The 2.0 CLI has an agent mode for tools like Claude Code. DuckDB's own agent mode post reports output tokens dropping 59% across 132 TPC-H runs, while total cost and runtime barely moved, because the saved tokens were about 0.5% of total input. Agent mode actually took more turns (237 vs. 224). If you can tell that story, you've shown you know the difference between optimizing a metric and optimizing an outcome.

Puzzle screens won't ask about any of this. The SQL round still happens, and it'll still be window functions, dedup, and gaps-and-islands; for sql coding questions, try DataDriven, the practice set we run for exactly that kind of problem. Spend the rest of your prep time being able to talk through a rollback plan out loud, because that's the job.

(I'll admit I still can't say "Cinnamon Teal" without sounding like I'm ordering a latte.)

2.1 is maybe 6 months out. File compatibility, silent semantic drift, writers stepping on each other, and the one consumer nobody inventoried will come up again then, the same as in every engine upgrade.

What's the oldest DuckDB version still pinned somewhere in your stack, and does anyone actually know who owns it?

1% of Layoffs Blamed on AI: What Gallup Means for Data Engineers

DataDriven — Tue, 27 Oct 2026 10:06:13 +0000

The last time I got laid off, the email said "realignment" 4 times and "AI" zero times. My manager read from a script, and the script said budget. 2 quarters later the same org posted an "AI platform data engineer" role. It was my old job with a new paragraph bolted to the top.

So when Gallup published a survey saying only 1% of laid-off workers blame AI, I laughed. Then I read the rest of it. The part that matters for data engineers is further down: tech workers who rarely used AI were about 3 times as likely to have been laid off.

Here's my read, and I'll show the work. The 1% tells you what people were told. The 3x tells you who was standing closest to the exit. Neither number proves AI took anyone's job. Together they tell you how to prepare for your next interview loop: get fluent with AI tooling, get very good at checking what it gives you, and keep practicing the fundamentals these tools still get wrong.

What Gallup measured, and what it says about data engineering layoffs

Start with the source, because half the takes I've seen never opened it. Gallup's Q1 2026 workforce study, published June 17, surveyed 23,717 employed US adults from its panel between February 4 and 19, 2026. The margin of error on the full sample is ±0.9 points, which is about as solid as survey data gets.

The headline numbers:

1% of currently laid-off workers named AI or automation as the primary reason they lost their job, described in their own words.
62% of laid-off workers are AI non-users (once a year or less), compared with 50% of employed workers.
13% of laid-off workers came from tech, compared with 6% of the employed workforce. Roughly double.
Tech workers who used AI less than monthly in their role were 3 times as likely to have been laid off as tech workers who used it at least monthly.
21% of employees say their employer is letting people go; 34% say it's hiring.

Gallup says the non-user pattern holds after controlling for age, education, industry, and time since layoff. The article doesn't say whether those controls also apply to the tech-only comparison, and it gives no absolute rates for that 3x figure. You'll see "6% vs. 18%" in the coverage. I couldn't find those numbers in Gallup's published article, so I'm not using them, and you shouldn't post them in Slack like they're settled either.

A skeptic should raise a few more caveats. The study is cross-sectional, and AI usage is a snapshot taken when the survey ran. Someone who lost their job may also have lost the employer-paid copilot license, so part of the gap could be a symptom of the layoff. The tech subgroup is much smaller than 23,717 people, so its interval is wider than ±0.9. And neither Gallup nor anyone else I could find breaks out Data Engineering specifically.

Gallup itself describes the AI-use link as an association, not a cause, one that "may reflect skills differences, role type or other unmeasured factors." Take them at their word.

Why employers say "AI" and workers say "restructuring"

Now look at the employer side. Challenger, Gray & Christmas counts announced layoffs along with the reasons companies give for them. Their July 2026 report counted 33,429 announced cuts, and AI was cited for 10,970 of them, about 33%. That was the 5th straight month AI led all reasons.

Then it cooled. AI dropped to 4th in August with 3,462 cuts. In September it was 5th, with 3,961 out of 43,281. Year to date, Challenger has AI cited in 120,136 cuts, about 21%, and it's still the top reason overall. Technology leads all sectors with 165,925 cuts this year.

So employers pin roughly a fifth of announced cuts on AI, while laid-off workers name AI 1% of the time. Both can be true because they measure different things. Challenger counts what goes in the press release. Gallup counts what the laid-off person says happened to them. The timing, categories, and denominators are all different.

Challenger's July release explains part of the gap better than I could: "Naming AI in a layoff announcement can win over investors while pushing current and prospective employees away." In plain English, the investor deck says AI and the HR script says restructuring. It's one decision told to 2 audiences.

I've sat in enough planning meetings to know how this goes. Leadership sells the board an AI story. Finance needs headcount down by Q3. Somebody draws boxes on a slide and 3 of them disappear. The person in one of those boxes hears "your role was eliminated as part of a reorganization," which is technically true.

Gallup's authors put it more carefully: the 1% "may understate AI's indirect influence through restructuring and cost-cutting decisions." They also say the data "do not support the narrative that AI is directly displacing large numbers of workers." I believe both sentences.

The press release is written for investors. The termination script is written for lawyers. Neither one is written for you.

Whether AI "took" jobs is a debate for economists. For your career, the more useful question is what the people who kept their jobs were doing differently, and Gallup gives a partial answer.

The 3x gap and the AI skills data engineers need

I've been through 3 waves of "data engineering is getting automated away." I'm still employed, and I'm still debugging the same categories of problems: schema drift, late-arriving data, and upstream teams changing a column type on a Friday without telling anyone. The tools changed about every 18 months in that time, and that list of problems didn't.

So I read the 3x number skeptically at first, and then I agreed with it. This next part is my opinion, not Gallup's finding. I think the tech workers who use AI monthly or more are disproportionately the curious ones: they pick up new tooling, and they tend to sit in roles close to where the company is investing. Those people were always harder to cut. AI usage is a decent proxy for "this person moves toward the work."

You can't fake being that person, but you can become one, and the bar is low: Gallup's threshold is monthly use. If you're a working DE in 2026 and haven't touched an AI assistant in 30 days, you're in the group that showed up 3 times as often among the laid off. Fix that this week.

The rest of the industry is far past that bar. In the Stack Overflow 2026 Developer Survey, 80% of respondents said they use AI at least an hour a day. The more interesting number is trust: 48.0% trust AI output "when I can easily verify the output," and only 6.6% trust it for important work decisions.

That gap is where a Data Engineer earns their money. AI will happily write you a MERGE statement that double-counts every late-arriving record. It'll partition a ROW_NUMBER dedup on the wrong key and look completely confident doing it. I once watched a pipeline silently drop 40% of records for 6 months because of a join nobody questioned; an LLM can now write that bug in 4 seconds. Your job is to know it's wrong before it hits prod.

These are the AI skills that matter in data engineering:

Reading generated SQL closely enough to spot the grain problem, then writing the row-count and uniqueness checks that prove it's correct.
Knowing your own model. The assistant has no idea your orders table is at line-item grain. You do, or you'd better.
Debugging past the traceback. Pasting it in is fine, but you still need to know why the Spark job spilled to disk or why the backfill created duplicates.
Knowing which upstream team breaks contracts and which metric the CFO actually reads. No assistant has that context.

That's the same list I'd have given you 10 years ago. AI is one more tool, and the concepts carry over to it the way they carried over to every tool before it. A DE who understands grain, idempotency, and late data gets 10x out of an assistant. A DE who doesn't just gets bugs that look more confident.

A word on money, since everyone asks. PwC's 2026 Global AI Jobs Barometer press release reports a 62% average wage premium for AI skills in job postings. That's what employers advertise across postings, so don't treat it as the raise you'll get for putting "LLM" on your resume. If your resume has a bullet that says "generative AI" and names no table, model or number, I'm closing it. Tell me you used an assistant to draft 200 dbt models for a migration, then caught the 11 with broken grain before they shipped. That's a story.

How AI shows up in your next data engineering interview loop

AI fluency now shows up in hiring loops, though maybe not the way you'd expect.

Datadog's published interview AI guidelines, last updated May 2026, are a good example of where a lot of companies are heading. In live coding, no AI "unless explicitly stated otherwise." In take-homes, AI is allowed for structure and research, but "the work must reflect your own conclusions," and you should expect live questions about it. In rounds that permit AI, you "should be ready to explain your approach, including how and why you used AI." And "failure to disclose" its use can get you disqualified.

Read that closely. The skill they're screening for is explanation. Can you defend every line? Can you say why you threw out half of what the assistant gave you?

Here's how I'd prep for a loop right now:

Ask the recruiter which rounds allow AI. Every company does it differently and half of them are still figuring it out. Asking shows you're current, and it costs nothing.
Practice the live rounds without AI. Do SQL and Python mediums by hand. Stick to mediums; do 50 and you'll be solid. Cover window functions, gaps-and-islands, dedup at grain, and sessionization. If AI can solve a medium, the round tells an interviewer very little, but companies still run it, so you still have to clear it.
Practice narrating AI output. Take a generated query, find what's wrong with it, and explain the fix out loud. One exercise covers both the take-home follow-up and the AI-permitted round.
Build a real story about using AI on the job. Be specific: which tool, which task, what it got wrong, how you caught it, and what you shipped. Interviewers will ask, and "I use Copilot sometimes" is fog.
Prep pipeline architecture instead of generic system design: backfills, idempotency, late data and SLAs. DE interviewers rarely ask about reverse proxies.

For the SQL and modeling rounds, we built datadriven.io for data engineer interview practice, and we wrote the problems to look like the job itself.

Interviewing is a separate skill from the job. I did somewhere around 20 loops in one search. I got rejected after the first half of an onsite once, gave myself a week to feel sorry for myself, and went back to grinding. You don't control the reason on your layoff letter, but you do control how much you practice for the loop.

Keep the bigger picture in mind too. Tech is overrepresented among the laid off, 13% vs. 6%, but 34% of employees in the same Gallup survey say their employer is hiring. Companies are still building data platforms, every AI initiative needs modeled data it can trust, and somebody has to build that layer. In my opinion, the person getting hired next year is the DE who works with an assistant open and catches its mistakes before anyone else sees them.

So, for the DEs reading this: has an interview loop actually asked you to use AI live yet, or are you still writing window functions on a whiteboard with a half-dead marker?

SQL Ties Python at 58%: What It Means for Data Engineer Interviews

DataDriven — Thu, 22 Oct 2026 10:06:25 +0000

My first SQL interview question was about finding the second-highest salary. It's 2026, I've sat on both sides of something like 20+ loops since then, and someone is still asking it.

What did change this month is the data. The Stack Overflow 2026 Developer Survey came out October 6, and SQL passed HTML/CSS for the number 2 spot. It sits level with Python at 58%. The same survey says AI coding tools are now a daily habit for most of the people who use them, while very few people trust those tools with production.

My thesis for anyone prepping a Data Engineer loop right now: AI writes the first draft, and the offer goes to whoever can read that draft, find what's wrong with it, and keep it alive in production. That means hand-written SQL, debugging and pipeline reasoning, so that's where your practice hours should go.

What the Stack Overflow survey says about SQL and Python

Let's be precise, because the headline numbers get mangled fast.

The survey's language usage table has JavaScript at 62.0%, SQL at 58.4%, HTML/CSS at 58.2%, and Python at 58.0%. Bash/Shell comes next at 51.3%. That question had about 14,099 respondents, which is a lot smaller than the "30,000+" in the headline. The survey drew 30,903 responses in total, but each question has its own base.

SQL "passing" HTML/CSS comes down to 0.2 points, and it's ahead of Python by 0.4. On a sample that size, the margin of error is somewhere near plus or minus 0.8 points. That's a 3-way tie. Anyone telling you SQL "beat" Python is selling a blog post.

The tie is still the interesting part, though, and 2 cuts of the same data make it more interesting:

Among people with 16+ years of experience (n=5,296), SQL leads at 63.0% and Python drops to 53.2%, a 10-point gap in favor of SQL among the people who've been doing this the longest.
Among languages used for AI work, SQL comes last at 29.8% and Python leads at 39.0%.

So the veterans are writing SQL, and the AI work is mostly happening somewhere else. My read, which the survey doesn't measure, is that SQL is the language people still write by hand and the language of the people who get paged when something breaks. Neither of those is going anywhere.

I'll admit this matches what I've seen, which is the kind of thing you should be suspicious of me saying. Every company I've worked at, from the shop with the 1,500-line stored proc to the companies everyone's heard of, ran on SQL. The orchestrator changed and the warehouse vendor changed. The language the business logic lived in never did.

One more caveat before you build a study plan on this. The survey measures breadth of use. It tells you nothing about hiring demand, interview weight, or skill depth. It doesn't even track Airflow, Snowflake, Databricks, or dbt as tools. Snowflake and Databricks only show up in an LLM-provider chart, which says nothing about warehouse adoption. If someone shows you this survey as proof of what a data engineering tool stack looks like, they didn't read it.

The 20% number and the Data Engineer job nobody interviews for

The survey's AI data tables break down how people use AI across tasks (n=12,547). Generating code in a familiar area: 69%. Debugging, troubleshooting, or refactoring code: 63.8%. Deploying, operating, or troubleshooting production systems: 19.9%.

The rest of that production row is worth reading too. 36.0% don't use AI there and don't plan to. 22.0% say it isn't part of their workflow at all. 23.3% expect to use it more.

At the same time, the blog reports that 73% of respondents who use AI coding assistants or agents use them daily. Look at the base: that's 73% of people who already use AI tools, which isn't the whole profession. It's still a lot of daily use.

Put those side by side. People are happy to have AI write code all day. Debugging code in an editor sits only roughly 5 points behind generation, so diagnosis inside a file is clearly fine too. Once the problem lives in a running system with real data, permissions, and a business waiting on it, usage drops to around 1 in 5.

AI writes the first draft. A human still owns the pipeline at 3am. Interview for the second skill, because the first one is free now.

I've been saying this for years, so the survey just gives me better ammunition. The actual Data Engineer job is less "write a DAG" and more "figure out why this pipeline silently dropped 2M rows last Tuesday." I once debugged a job that had been quietly dropping 40% of records for 6 months before anyone noticed. No autocomplete would have found it. What found it was someone who understood the grain of the source table, the join that fanned out upstream, and the dedupe step that was "fixing" it by throwing away good rows.

The production side of the job has always been the job. What changed is that the coding side got cheap, so the gap between the 2 is easier to see.

Part of that 19.9% is probably about access too. Production systems sit behind permissions, and plenty of orgs restrict who can point an agent at prod. That's my guess; the survey doesn't connect those dots. Either way, a candidate who can reason through a broken production pipeline is scarcer than one who can produce a working query, and scarcity is what gets you paid.

Splitting your Data Engineer interview practice time

Before the plan, a warning about the evidence. Almost everything published about SQL interview formats at specific companies comes from prep vendors and candidate anecdotes. I went looking for employer-documented round structures and found contradictions. One guide says a given company runs 60 minutes with 4 SQL and 4 Python questions, another says 50 minutes with about 6, and nobody links to anything official. So I won't pretend to know exactly what your loop looks like. Ask your recruiter; they'll usually tell you. I'd still split the time like this.

SQL, by hand, with the AI turned off

Make this the biggest block. Even at companies that allow AI in interviews, your SQL fluency shows in how fast you spot what's wrong with the generated query. At companies that don't allow it, you're writing it cold.

What to practice:

Window functions, and why you'd pick ROW_NUMBER over RANK over DENSE_RANK. Lots of candidates know these functions exist. Far fewer can say out loud when ties matter and what each one does with them, and interviewers notice that difference right away.
Top-N per group, running totals and deduplication. These patterns cover most of what the job asks for every day.
Saying the grain out loud for every query: what does one row represent? Most bad joins I've seen in production came from someone who never asked that question.
Solving the same problem twice, once with a self-join and once with a window function. Interviewers love asking for the second version once you've given the first.

Debugging and pipeline troubleshooting

This is the second block, and the 19.9% figure is why. If production troubleshooting is the area AI touches least, it's the area where you set yourself apart.

Practice with broken things. Take a working query and break it on purpose: add a join that fans out, add a NULL to a key column, shift a timestamp across a timezone boundary. Then find the bug from the output alone. Practice talking through late-arriving data, schema drift, and the upstream team that renamed a column without telling anyone. The tools change every 18 months and these problems never do.

Python, scoped to what Data Engineers actually do

Python sits level with SQL in the survey, and you'll get a coding round. Stick to mediums. Do 50 and you'll be solid; few companies ask hards consistently. Focus on the data-shaped stuff: parsing messy files, dicts and sets for dedupe and grouping, generators for large inputs. If you want somewhere to practice for python coding questions, try datadriven; we built it for that category, and the problems look like the job and not like a puzzle book. (I've never solved a bit manipulation question and have absolutely no idea what I'm talking about with those. Still employed.)

Practice both with and without AI

Policies vary and they're changing. Canva's engineering team wrote publicly that they expect candidates to use AI tools in interviews for some roles. Simon Newton argued there that the old questions got solved instantly by AI and that real engineering is more about reading and improving code than writing it from scratch. That post is from 2025 and doesn't cover data engineering, so it's one company's opinion and not a trend. I agree with the opinion, though.

If your loop allows AI, the skill being tested is review. Can you read generated SQL and catch the missing partition, the wrong join type, the aggregate at the wrong grain? Practice exactly that: have a tool generate a solution, then audit it like it's a PR from a new hire.

The skills that still decide Data Engineering offers

I've been on hiring panels where we passed on strong candidates for the dumbest reasons. I've also been the candidate who got rejected after the first half of an onsite. That one hurt. The interview is arbitrary, and you should play the game anyway. Still, the things that move a panel from "lean hire" to "strong hire" have stayed pretty consistent, and the survey data lines up with them.

Data modeling is the core skill, and it doesn't depend on any tool. If you get the model wrong upstream, everything downstream is pain. Keep fact tables at grain; you can always aggregate up and you can never disaggregate down. AI will happily write you a query on a bad model, and it'll run fine, which is the problem.

Then there's explaining tradeoffs under pressure. I've watched people with 10 YOE get downleveled because they couldn't articulate their design decisions in the room. "Why batch and not streaming?" (Batch covers 90% of what companies actually run; most of y'all don't need streaming.) "Why a wide table and not a star schema?" (Storage costs around 2 cents a GB a month; the engineering overhead of maintaining a star schema costs a lot more than that.) Have these answers ready and back them with economics. Economics wins arguments that "best practices" lose.

For pipeline architecture, skip the generic system-design material about load balancers and reverse proxies. Panels want to hear how you'd handle idempotency, backfills and late data, and how you'd know a job dropped rows before the business did.

Production instincts are the 19.9% skill. Tell them about the time something broke and what you did about it. A story about a silent failure you caught is worth more than any tool on your resume.

I've lived through 3 waves of "data engineering is getting automated away," and I'm still debugging the same categories of problems. This survey reads like the 4th wave with better numbers. Developers handed AI the keyboard, and they kept the pager themselves. The people who get hired are the ones the business trusts with it.

What's your loop doing right now: are they letting you use AI in the SQL round, or making you write it cold?

PostHog Dropped ClickHouse for DuckDB: A Data Engineer's Take

DataDriven — Tue, 20 Oct 2026 10:07:09 +0000

I once watched a single analyst's 14-way join freeze a shared warehouse for a whole company on a Monday morning. Dashboards timed out, the CFO's report never showed up, and 3 teams spent the afternoon blaming each other.

That's why I read PostHog's June 29, 2026 engineering post closely. They pulled their managed warehouse off a multi-tenant ClickHouse cluster and gave every organization its own DuckDB instance. The instances sleep when idle, the data sits on S3 through DuckLake, and a Postgres wire endpoint lets ordinary tools connect. The headline sounds like "ClickHouse lost," and that's the wrong read. The post publishes zero benchmarks. PostHog's decision comes down to matching the engine to the shape of the workload, and that's the same reasoning a system design interview tests.

What PostHog changed in the warehouse, and what stayed on ClickHouse

The warehouse used to run on the same multi-tenant ClickHouse cluster PostHog uses for product analytics. That cluster serves people clicking around in a browser and expecting results quickly. Warehouse modeling jobs look nothing like that. In the post's words, "It's not atypical for a modeling query to kick off on a schedule and run for hours or even days."

A job that runs for days has a poor chance on a cluster where nodes restart, both on purpose and on their own. Every DE who's had a 9-hour backfill die at hour 8 because of a node rotation knows exactly how that ends. The post says that an unbounded number of those queries hitting their "very not elastic" cluster would be "basically untenable."

They also list ClickHouse complaints beyond the workload problem. ClickHouse has no cost-based query optimizer, and its S3, Delta Lake and Iceberg support matured slowly. Results weren't stable either: "We've seen ClickHouse change query results between releases, or based on which settings were enabled." HogQL has poor support in tools like dbt, and letting customers connect directly would make the multi-tenancy problems worse.

The hot takes skip the fact that ClickHouse stays. "This is how we use it for all of our analytics data and that isn't changing." The change covers the warehouse only. Hot, shared, low-latency event analytics keeps running on ClickHouse. Long-running, isolated modeling jobs move to DuckDB.

The new layout, per the managed warehouse docs:

Every organization gets a dedicated DuckDB worker. "There's no shared query queue, so a heavy query from another customer never competes with yours."
Workers spin up on demand and retire when idle.
Storage is DuckLake: Parquet files in object storage with a transactional SQL catalog on top, so storage and compute scale independently.
A control plane handles TLS, auth, and PostgreSQL-to-DuckDB syntax translation. Each warehouse gets its own hostname.

It's in beta, behind a waitlist, and the docs say the layout "may change during beta." Keep that in mind before you redesign your company's stack around a blog post.

PostHog split the stack by workload: a shared engine for hot queries and isolated engines for long jobs. Splitting by workload is the lesson. "DuckDB beat ClickHouse" is the part you can ignore.

The post has no benchmarks, and that's the interesting part

I've read a lot of vendor migration posts. Most of them open with a bar chart showing a 40x speedup on a query nobody runs. This one has no chart at all, and it doesn't publish throughput, data volumes, node counts, instance sizes, pricing, cold-start latency or the idle timeout. The only performance language is qualitative: queries come back "on the order of minutes," and jobs run "hours or even days."

Plenty of people will find that frustrating. I find it honest. PostHog never says DuckDB is faster. They say a shared, inelastic cluster can't safely absorb unbounded multi-hour jobs. It's an argument about blast radius and isolation, and you can evaluate it without a single benchmark.

The target customer is small. The post says the original warehouse was built for "a 20-person company where the CEO is still running their own SQL queries," and that its goal was to "delay the moment a company needs to hire their first data engineer." Per-org DuckDB works great when each tenant's data and jobs fit on one machine. The post never says where that limit is. If a tenant outgrows a single node, single-tenant compute means you scale up. DuckDB's own large-scale proof ran TPC-H on a 1.5 TB RAM, 192-core box, and that tells you which direction this design scales.

The famous benchmark measures the wrong phase. DuckDB's ClickBench lead came from hot runs, and a sleep-and-wake architecture lives or dies on cold starts. That's my inference; neither source says so. Still, quoting ClickBench to justify a scale-to-zero design is like judging a commute by how fast the car goes once it's on the highway.

Scale-to-zero has a bill, and the bill moves to the first query. Sleeping instances cost nothing until somebody opens a dashboard Monday morning after a quiet weekend and waits for the wake-up. A nightly modeling job won't notice the delay. An exec clicking a chart will. Snowflake has had this model for years, and its warehouse docs still bill "with a 60-second minimum each time the warehouse starts." Even "pay only when active" has a floor. PostHog doesn't publish theirs.

The ClickHouse optimizer complaint has a shelf life. ClickHouse 25.9 added automatic global join reordering, and the post doesn't say which version PostHog was running. Read it as a 2026 engineering judgment about their own deployment, not a permanent verdict on the engine.

The software underneath is moving too. Duckgres, PostHog's open-source Postgres wire server for DuckDB, now says in its README that as of October 8, 2026, PostHog is no longer developing it. The management plane moved to something called hogtower, and the README says the wire protocol and DuckDB query serving are unaffected. I wouldn't assume the open-source repo is what runs in production today.

I've been through enough architecture announcements to know the pattern: the diagram ships 6 months before the operational numbers do. What you can learn from right now is the reasoning.

How a data engineer should answer this in a system design interview

Swap the names out and this becomes a standard interview prompt: "You run analytics for thousands of customers on a shared cluster. Some customers want to run heavy scheduled transformations. Design it."

Weak candidates name a tool. "I'd use Snowflake." "I'd use a bigger ClickHouse cluster." I've sat on panels where that was the entire answer, and the candidate couldn't explain why when we pushed. We passed, and the tool had nothing to do with it: there was no reasoning to evaluate.

Here's how I'd structure it.

Start with the workload before you touch the architecture. How many tenants? What fraction sit idle most of the day? What's the query shape: sub-second dashboards, or jobs that run for 6 hours? Who gets hurt when one tenant's job eats the cluster? PostHog's whole decision falls out of these answers. Short, hot queries share well. Long, unbounded jobs don't.

Pick the tenancy model per component. AWS's SaaS Lens describes this vocabulary: silo (dedicated resources per tenant), pool (shared resources for economies of scale), and bridge (a mix, where "noisy neighbor attributes might steer a microservice to a silo model"). PostHog built a bridge: pooled ClickHouse for event analytics, a silo for warehouse compute. Saying "silo for the long jobs, pool for the hot path" in an interview sounds like you've actually been on call for this.

Don't overclaim about shared engines. "Shared clusters can't isolate tenants" is too strong, and a good interviewer will catch it. ClickHouse has workload scheduling. Its docs describe IO caps per workload, and they describe weights as a "best-effort guarantee from below (not a limit or quota from above)," with limits enforced "per host." CPU preemption is off by default. Merges and mutations aren't CPU-scheduled yet. So the defensible claim is that shared engines isolate partially and per host, and you should name exactly what they don't guarantee. A node restart killing a 2-day job is a failure-domain problem, and workload pools don't fix it.

Name the concurrency model; it often decides more than the tenancy model. DuckDB's concurrency docs say one process can read and write a native database file. Multi-process native writes go through Quack, which is "in beta stage as of DuckDB v1.5.2," and the docs recommend DuckLake on a PostgreSQL catalog for concurrent read-writes. One writer per file maps neatly onto one instance per org. That's my inference about why the design fits; PostHog doesn't say it. If you propose per-tenant DuckDB in an interview and can't say how writes behave, expect a rough follow-up.

Price the cold start out loud. You don't need real numbers. Name the variables: idle hours, wake latency, how often queries arrive, and the per-start billing floor. Then say which tenants come out ahead (bursty, mostly idle) and which lose (always-on dashboards). Interviewers score this kind of reasoning much higher than a confident number you made up.

Don't declare a winner without data; PostHog didn't, and you shouldn't either.

Plenty of loops still include a Spark coding round no matter what the company runs in production (and for pyspark interview questions, try datadriven.io, which we built for exactly that round). Treat that as a separate skill. The design round is where you show you can reason about fit, and nobody memorizes their way through that.

The data engineering skills this move points to

I've watched 3 waves of "this tool changes everything." This move teaches the same thing all of them did: the concepts carry over and the tools rotate.

Learn open table formats and catalogs. PostHog says data "lives in S3 independent of whatever's querying it, so we're not locked into DuckDB forever." That portability comes from the format. DuckLake keeps metadata in a SQL database and data in Parquet. The DuckLake FAQ says the only real limit is the catalog database's performance, which tells you where the commit point and the bottleneck sit. Ask "where does the commit happen?" about any lakehouse you touch. An "open" format is only as open as its write path, too: DuckDB reads Iceberg metadata directly in read-only mode, and writes go through a REST catalog.

Integrity moves up a layer. DuckLake doesn't support constraints, keys or indexes. If your duplicate detection relied on a primary key, that job now belongs to your pipeline. Get used to writing those checks yourself.

Embedded engines are real infrastructure now. DuckDB showed up at 3.3% usage in Stack Overflow's 2025 survey, which is small, but per-tenant embedded engines are a pattern you'll keep running into. Learn how they spill: DuckDB handles larger-than-memory GROUP BY, JOIN, ORDER BY and window operators, but its docs warn it can still run out of memory when operators combine, and list() and string_agg() don't spill at all. "It handles larger than memory" holds per operator, so check fit query by query.

Governance is part of the evaluation. DuckLabs announced in August 2026 that it's joining AWS, with DuckDB, DuckLake and Quack staying MIT-licensed under the DuckDB Foundation. Probably fine. Still, "who maintains this in 3 years" belongs in every architecture decision, and almost nobody asks it in a design review.

Cost reasoning beats speed reasoning. S3 GETs run $0.0004 per 1,000. My rough estimate is that request charges barely register next to compute uptime, so the economics of scale-to-zero come down to how many hours your instances are awake. Engineer time costs more than any of it. A team that stops getting paged for a dead 2-day job has saved more than any benchmark would show.

Concepts transfer across tools. Tenancy, isolation, failure domains, and cold-start economics will still matter when DuckDB is the old thing someone's migrating away from.

PostHog made a reasonable call for a specific customer, the 20-person company with a CEO still writing SQL, and they were upfront about the scope. The real test comes when one of those tenants grows into a 500-person company with a 3 TB fact table and a modeling job that runs past the weekend. I'd like to read that post.

If you've run per-tenant compute in production, what was the cold-start number that finally showed up on your dashboard?

dbt Core v2.0 Goes Apache 2.0: What Data Engineers Should Know

DataDriven — Thu, 15 Oct 2026 10:05:42 +0000

I've had the license on a tool I ran in production change under me more than once. Every time, the headline said "open" and the details lived on a page nobody on my team read until legal sent an email asking what exactly we'd deployed.

On June 1, 2026, Fivetran and dbt Labs closed their merger and announced that the Fusion engine runtime is now dbt Core v2.0, licensed under Apache 2.0. dbt's own licensing docs describe dbt v2 as proprietary. Both statements are accurate. They describe different artifacts, and if you build on dbt or interview for roles that use it, you need to know which is which.

My take: the open source runtime is real, and it's good news. The binary most people install is a separate thing with separate terms. None of it changes what you should study, because the concepts underneath dbt outlived every vendor that has owned the tooling around them.

What Fivetran and dbt Labs announced at close

The joint merger announcement confirms an all-stock deal, first announced October 13, 2025. George Fraser stays CEO and Tristan Handy becomes President. No valuation, price, or exchange ratio was disclosed. Back in October 2025 the companies said the combined business would be "approaching $600M" in ARR. The closing release doesn't repeat that number.

Then there's the "100,000 data teams" figure. In October 2025, Fivetran's release said "more than 80,000 data teams use dbt." In June 2026 the body of the release says the combined company supports "more than 100,000 data teams," and the boilerplate at the bottom says more than 100,000 teams use dbt. One of those is a combined count and the other is dbt-only, and the release never defines "data team" or says whether overlapping customers were deduplicated. Treat it as the company's own count. It doesn't tell you anything independent about adoption.

The product list at close:

dbt Core v2.0 (alpha): "the open sourcing of the dbt Fusion engine runtime," under Apache 2.0
dbt State (preview): a caching layer, with a claimed infrastructure cost cut of "30% or more"
dbt Wizard (beta): no license stated
Agents Schema: called "an open source standard for agentic context," with no status given in the release

There's also a subplot that gets less attention. In September 2025 Fivetran bought Tobiko Data, the company behind SQLMesh and SQLGlot, which is the main open source competitor to dbt. In March 2026 Fivetran donated SQLMesh to the Linux Foundation, which says the project "will remain vendor neutral." So the company that now owns dbt bought the biggest open alternative and then gave it to a foundation. The public record doesn't say why. If you're betting on SQLMesh as a hedge, wait until the governance details and roadmap are published.

Devin Pratt, an IDC analyst, told TechTarget that "the real test is keeping the DBT open-source community's trust through the transition." That's his opinion, and I agree with it. Trust gets built in license files and changelogs, and press releases don't count.

Apache 2.0 vs. the licensing docs: which dbt you're running

The dbt Core v2 launch post by Joel Labes and Grace Goheen says the shared runtime code, which dbt "previously committed to releasing publicly under the ELv2 license is now under the Apache 2.0 license as dbt Core." The dbt-core repo README backs that up: main contains the Apache 2.0 source of v2.0. The old dbt-fusion repo is archived. That part is clean, and it's a real improvement.

The same post also says the Fusion binary "contains some proprietary code," that dbt has "relicensed the Fusion binary to be more permissive than ELv2," and that some premium features need a free login or a paid dbt platform account.

Now the dbt licensing docs, last updated September 16, 2026: "dbt v2 is proprietary to dbt Labs, made available under the dbt Product Licensing Agreement." The Apache 2.0 distribution is called dbt OSS. dbt v1 remains fully Apache 2.0.

So in practice there are 2 distributions:

dbt OSS (dbt-oss package): the Apache 2.0 runtime. You can read, fork and ship it.
dbt (the recommended dbt package): bundles proprietary code under the Product Licensing Agreement, which bars reverse engineering or deriving source code. Local production use is free without an account. Some features sit behind a login.

The naming makes this harder to follow. Across dbt's own pages, the same lineage of software has been called Fusion, dbt Core v2, dbt v2, dbt OSS, and plain dbt. The licensing FAQ changelog shows the June 1 update "Removed ELv2 references," and a September 15 entry renamed "dbt Fusion engine" to "dbt" and "dbt Core v2" to "dbt OSS." The press release uses one vocabulary and the docs use another.

The license has moved before. In May 2025, when Fusion launched under ELv2, dbt Labs wrote, "The license for dbt Core is not changing." That was true when they wrote it. A year later the runtime moved to Apache 2.0 and the full binary became proprietary under a separate agreement. Usage rights got broader and the remaining closed code stayed closed. Licenses on the same product can change in both directions within 12 months.

"Fusion is Apache 2.0" is wrong. "The dbt v2 runtime is Apache 2.0 as dbt OSS, and the full binary is proprietary" is correct. Say the second one in your interview.

Status is the other moving target. At the merger, v2 was labeled alpha. By the September 16 dbt Summit, dbt Labs called v2 and dbt State "Now GA". dbt's upgrade docs, updated October 8, still show alpha, beta, and preview labels at the adapter level and don't give a GA date. So "GA" may depend on which warehouse you run. Check your adapter before you plan a migration around it.

This week, open your project's requirements file or lockfile and find out which package name is in it. Then find out who on your team approved it. In my experience the answer is usually "whoever set up the repo 2 years ago," and you have about the same odds of that being documented as of a 1,500-line stored proc having accurate comments.

The 30% savings claim and Agents Schema

The dbt State savings figure has been worded 3 different ways in 3 months:

June 1 press release: "reduce underlying infrastructure costs by 30% or more"
Fivetran's June blog: "more than 30% on average across production runs" of dbt-generated compute
September dbt Summit post: "an average 15-30%+ reduction in warehouse compute"

"Infrastructure costs" and "dbt-generated compute" measure different things. None of these sources publishes a sample, a baseline, or a methodology. The customer numbers that do appear vary widely: CarGurus reported a 9% compute reduction in dbt's own State post, Virgin Media O2 reported 25%, and RxBenefits reported 59%. Each of those is a single company's self-reported result.

My read is that caching state and skipping models whose inputs haven't changed is good engineering, and I've been writing hand-rolled versions of it since before dbt existed. Whether you save 9% or 59% depends on how much of your DAG was rebuilding unchanged data every night. If your whole project is incremental models with tight predicates, the savings are small. If someone scheduled a full refresh of 400 models every hour because "it was easier," you'll save a lot, and you also have a bigger problem.

Engineer time costs more than warehouse compute. A 30% cut on a small Snowflake bill won't pay for a migration that takes 2 engineers a quarter. Measure your own baseline before you trust anyone's average, including mine.

Agents Schema is the AI part of the announcement. The idea is a single AGENTS schema in your warehouse holding tables like AGENTS.DBT_MODEL and AGENTS.LOOKML_VIEW, so any agent that can run SQL can read your semantic context. The GitHub repo is MIT licensed with "Copyright (c) 2026 Fivetran Inc." while sitting under the dbt-labs org. When I looked it had 145 commits, but nobody had cut a release tag or published a spec, a versioning policy or a governance body. The managed product built on it, Fivetran Context Layer, is in preview and limited to Snowflake, BigQuery, or Databricks in North America.

Fivetran's own Context Layer docs say, "We recommend using Agent Context MCP because it can be more accurate and token-efficient than Agents Schema." So the open standard ships alongside a recommendation to use the proprietary service. An open table layout is useful, but whether you can move your context elsewhere depends on whatever engine fills those tables. Ask about that before you believe "without lock-in."

The v2 question in your next data engineer interview

I've sat on enough hiring panels to know how this will go. Someone will ask, "We're on dbt. Would you upgrade to v2?" Most candidates will either recite the press release or panic. Both of those lose.

A stronger answer starts by naming the artifact: spec, runtime, binary or hosted platform. Each one has different terms, and that one sentence shows you understand vendor risk better than most of the people interviewing you. Then separate the vendor's claims from your measurements: "The vendor says 15 to 30%. I'd run it against our own baseline on a staging environment for 2 weeks before committing."

After that, talk about exit cost. Your SQL is fairly portable, since SQLGlot is MIT licensed and handles dialect translation. Your tests, lineage, state and deployment workflow are harder to move, so that's where the lock-in is. Finally, bring it back to the model. If your grain is right and your models are idempotent, the transformation tool is replaceable. If they aren't, no license will help you.

That last point is the one I keep repeating. The topics that come up in nearly every DE interview loop are DAG dependencies, incremental logic, dialect differences across warehouses, and "what happens if this job runs twice?" Those are concepts, and every one of them applies to dbt v1, dbt v2, SQLMesh, Dataform, or a cron job someone wrote in 2014.

I did around 20 loops in a single job search once. I was never asked about a license. I was asked about modeling all the time, and some of those times I gave answers I'd rather forget. So don't spend your prep weekend memorizing Fusion feature flags. Spend it on fact table grain, slowly changing dimensions, and explaining why a model is incremental. If you want practice on the part that survives every vendor reshuffle, data warehouse design is covered on datadriven.io. We built it for that kind of prep, and it's where we send people who ask what to study after the tool news cycle settles down.

For DEs already running dbt in production, the checklist is short:

Find out which package you install, dbt-oss or dbt, and write it down.
If you redistribute dbt to anyone (consultancies, platform teams, managed services), read the Product Licensing Agreement in full. Sections 3 and 4 govern redistribution.
Treat the 30% number as a hypothesis to test.
Treat Agents Schema as an experiment until it has a release tag and a governance doc.

I've been through enough of these big-announcement cycles to know the pattern. The press release comes out in June, the naming gets cleaned up in September, and in 2 years everyone's debugging the same late-arriving data and broken upstream contracts with a different binary. I've watched the tooling turn over about every 18 months while those problems stayed put.

Which dbt package is in your production requirements file right now, and did anyone check its license before it got there?

Entry-Level Data Engineer Jobs: Stanford's 19% Gap vs BLS's 35%

DataDriven — Tue, 13 Oct 2026 10:05:42 +0000

2 numbers landed on my feed within a few weeks of each other this fall, and they look like they belong to 2 different industries.

The first says young workers in AI-exposed jobs are falling behind. The second says data science employment is about to grow 35% in a decade. If you're trying to land an entry-level data engineering job right now, you've probably seen both headlines and felt whiplash.

Here's my take after more interview loops than I'd like to admit, on both sides of the table: both numbers are mostly right, and they measure different things. Read together, they say the field is growing, the door into it got narrower, and the people getting through that door can do something specific. The rest of this piece covers what that is.

Stanford's 19% gap and entry-level data engineering

In August, Stanford's Digital Economy Lab reported that employment for workers aged 22 to 25 in highly AI-exposed occupations sits about 19% below where it would be if it had kept pace with same-age workers in less-exposed jobs. At the July 2025 data vintage, that gap was 15%. Experienced workers show no comparable gap.

That's a scary number. Before you panic, look at what it measures.

It's a counterfactual gap. The 19% compares exposed young workers to a benchmark built from their less-exposed peers. It doesn't count jobs lost. In raw levels, employment for 22 to 25 year olds in the 2 most exposed quintiles fell about 11% from November 2022 to June 2026, while the 3 least-exposed quintiles grew about 10%.
It comes from ADP payroll data, a balanced panel of 25,000 firms. Stanford says the sample represents neither the whole U.S. labor market nor every company that uses ADP.
The authors won't call it causal. In their words, "These are descriptive patterns, not causal estimates of the effect of AI."
The number has moved around. The original 2025 paper said 13%. A February 2026 note had fixed-effects estimates roughly 16%. Now it's 19%. Part of that is new data; part of it, as the authors admit, is that the estimates have become "more sensitive to specification choices."

I respect that last admission. I've spent enough years debugging pipelines to know that when the headline number changes every time someone touches the methodology, you treat it as a direction rather than a measurement.

The direction still matters, though. The release says the adjustment runs mainly through reduced hiring of young workers, not increased separations. The Dallas Fed and a Census Bureau working paper using administrative data found the same pattern: fewer people getting in, with no wave of layoffs pushing people out.

So juniors aren't losing jobs they already have. They're competing for fewer seats at the entrance, which means the thing to prepare for is the hiring process itself.

2 more caveats from Stanford's own release. The gap shrinks when you control for education. It also persists when you exclude tech firms and computer occupations entirely. That second point kills the "it's just Big Tech layoffs" explanation. Whatever's happening is broader than a few companies over-hiring in 2021.

The 19% says fewer juniors are getting hired into AI-exposed roles. It doesn't say the roles are disappearing. Plan for a narrower door into a room that keeps getting bigger.

BLS projects 35% growth for data scientists and never mentions data engineers

Now the optimistic number. The Bureau of Labor Statistics projects data scientist employment to grow 35% from 2025 to 2035, from 275,600 jobs to 371,000. Median pay was $120,230 in May 2025. They project about 24,800 openings per year.

For context, BLS projects total U.S. employment to grow 3.5% over the same period. Computer and mathematical occupations as a group grow 7.3%. Data scientists beat both by a mile.

The BLS data scientist page doesn't mention data engineers at all, either in the description or in the "similar occupations" list (which includes actuaries and survey researchers, but somehow not the people who build the tables the data scientists query). As far as the Occupational Outlook Handbook is concerned, the DE role sits somewhere between "data scientist" and "software developer," and you have to guess.

I've spent my whole career in a job title that the government's main career guide skips. Welcome to data engineering.

Do the math on the openings, too. 95,400 net new jobs over 10 years is about 9,500 a year. The other ~15,000 of those 24,800 annual openings come from replacement: people retiring, switching careers, or getting promoted out. Net growth explains only about 38% of annual openings.

That changes how you should read the headline. A big chunk of the opportunity is backfill, and backfill hiring managers want someone who can take over an existing system on week 2. Experienced people win that fight more often than juniors do.

The BLS projection also can't confirm or refute the Stanford gap, because it's an aggregate across all experience levels with no age breakdown. A field can grow 35% overall while hiring fewer 23 year olds; those 2 things coexist easily if the growth goes to people with 3+ years behind them.

Which is exactly what I'd expect, frankly. I've said for years that data engineering isn't an entry-level job. It combines business context, analytics, infrastructure, software engineering, and SRE work. Companies were already reluctant to hire juniors into that. AI handing them a plausible excuse to hire 1 senior instead of 2 juniors just made the reluctance louder.

Stack Overflow's SQL numbers and the debugging gap in AI-assisted juniors

The 2026 Stack Overflow Developer Survey is the source I find most useful here. It put SQL second among all languages at 58.4%, behind JavaScript at 62.0%, and SQL passed HTML/CSS for the number 2 spot. Among respondents with 16+ years of experience, SQL ranks first at 63.0%.

The more senior the developer, the more likely they're writing SQL. The language people kept calling "legacy" for 15 years is what the veterans use most.

My inference (the survey doesn't say this, I do): SQL demand isn't what's squeezing juniors. If anything, SQL's standing comes from experienced people who use it to understand data, debug it, and explain it. That's the skill gap. Juniors usually can write a query; the hard part is reasoning about what the query returned and why it's wrong.

The same survey shows AI usage is everywhere: 73% of developers who use AI coding tools use them daily. But trust is conditional. In Stack Overflow's May pulse survey, 60% of respondents said they block agents from making unapproved system changes. Developers let AI write code in familiar territory and keep it on a short leash near production.

Now combine that with one study I keep thinking about. Anthropic ran a randomized trial with 52 engineers, mostly junior, learning an unfamiliar Python library. The AI-assisted group averaged 50% on comprehension quizzes versus 67% for the group that coded by hand, and the biggest gap was on debugging questions. The speed advantage from AI was roughly 2 minutes and wasn't statistically significant.

Within the AI group, how people used it mattered most. Participants who delegated code generation to the AI scored below 40%, while participants who used it to ask conceptual questions scored 65% or higher.

That study measured learning a library, not interview performance, so applying it to hiring is my own reading. It lines up uncomfortably well with what I see in interviews, though. The candidate who can't explain why their join fanned out to 3x the expected rows used to be rare. Now I see a version of them every week.

The actual job is less "write a DAG" and more "figure out why this pipeline silently dropped 2M rows last Tuesday." AI can generate the DAG. It's much worse at telling you that the upstream team changed a timestamp from UTC to local time and your incremental load has been skipping a window every night since. That takes someone who understands the data, the grain, and the business.

So when a hiring manager has budget for 1 seat and AI can handle the boilerplate, they want the person who's good at the part AI is bad at. Juniors who prove they can debug and explain get hired. Juniors who can only produce code are competing with a tool that produces code for free.

The interview side has the same problem. I've argued that AI makes classic coding interviews nearly useless for evaluating skill: if a model can solve a medium problem on demand, solving it in front of me tells me very little. Some companies are adapting; Meta was reported in 2025 to be testing AI-enabled coding interviews. Most aren't adapting yet, so you'll still get the leetcode-style round and you should prepare for it. But expect the SQL and debugging rounds to carry more weight in the final decision, because those are the rounds where an interviewer still learns something a model can't fake for you.

Put junior practice time into SQL and debugging first

I did somewhere roughly 20 interview loops in a single job search once. I got rejected after the first half of an onsite. I did 8 rounds at one company, was told I'd passed, and then the headcount vanished. I'm telling you this so you'll believe me about the next part: none of the loops I lost came down to not knowing a tool. They came down to not explaining my reasoning clearly under pressure.

If you're early-career and staring at that 19%, here's where I'd put your time.

Learn SQL past the syntax. Window functions, handling NULLs in joins, deduplication when there's no clean key, grain mismatches, late-arriving data. Every interviewer I've sat with eventually asks some version of "this number looks wrong, why?" We built the sql coding questions on datadriven.io for exactly that kind of problem; they push you past "does it run" into "is it right."

Practice debugging out loud. Take a query that returns the wrong count and narrate the diagnosis as you check the grain, then duplicates on the join key, then the filters. Interviewers score the process more than the answer; a candidate who systematically narrows the problem beats one who guesses right.

Use AI, but be able to explain what it helped you build. Everybody uses it, and pretending otherwise is silly. Use it the way the high scorers in the Anthropic study did: ask it conceptual questions, then write and own the code yourself. If you can't explain every line of a project on your resume, an interviewer will find the line you can't explain.

Learn data modeling before you go shopping for tools. Which orchestrator you pick matters much less than whether you keep fact tables at grain, understand why you'd go wide and denormalized, and know what happens when a dimension changes, because that knowledge carries over to any company on any stack.

For the DS&A round, do mediums and then stop. Around 50 of them and you'll be solid, since few companies consistently ask hards. Spend the rest of your time on the SQL, debugging and modeling work above.

And a word on mindset, because the data can mess with your head. The Stanford gap is about hiring volume. It says nothing about your ceiling once you're in. Experienced workers show no comparable gap, which tells you the people who get through the door and build 2 or 3 years of real production experience are doing fine. The field is still growing, and the narrow part is the entrance.

Treat prep like a job. I'm not the smartest engineer I know; I just outworked the version of myself that wanted to quit after the 6th rejection. Give yourself a week to feel bad about the headlines, then get back to it.

The juniors who come out of this stretch well will be the ones who can open a broken query at 2am and say why it's broken. That skill paid well before AI, and I expect it to pay even better now.

So here's my question for the folks hiring right now: when you passed on a junior DE candidate this year, was it the coding, the SQL, or their inability to explain what they'd built?

TRM Labs' 400 TB Migration: Lessons for Data Engineer Interviews

DataDriven — Thu, 08 Oct 2026 10:05:38 +0000

I've been asked some version of "tell me about a large migration you led" in more interview loops than I can count. I've given good answers and I've given answers so bad that I watched the interviewer quietly stop taking notes. The bad ones all had the same problem: I told a story about moving data. The good ones covered how I proved the plan, how I found out it was wrong, and what I changed.

That's why I've been sending people the post TRM Labs published on June 30, 2026. It's a candid write-up of moving roughly 400 TB of Postgres onto StarRocks on Apache Iceberg, and it includes the slipped deadline and the latency regressions that reached customers. Almost nobody publishes that part, which makes it one of the best public case studies a Data Engineer can use to prepare for a migration question in a system design or behavioral interview.

I think interviewers barely care about your cutover. They want to see how you de-risk, how you report bad news, and whether you can back your numbers. The TRM post covers all 3, including the uncomfortable parts.

What TRM Labs migrated, and what the headline leaves out

The facts, straight from TRM's engineering post by Amit Plaha. The source was 2 Postgres clusters in a blue/green setup, around 400 TB covering 70+ blockchains. The target is StarRocks on Apache Iceberg, served through NGQF (NextGen Query Federation). The scope was 30+ data-intensive API routes, with business logic moved out of Postgres functions and into a TypeScript layer. The table that started it all is Address Transfers, which shows how a blockchain address has sent and received funds; it was the first table they ever created, and it was timing out on Postgres.

The post claims "annualized cloud savings north of USD 1.2 million." The schedule was a full quarter of proof work and about 1.5 quarters of route migration, and the target moved from July 30 to the first week of September.

That's a nice résumé bullet. Now read it the way a skeptical interviewer will, because they'll poke at exactly these spots.

Start with the target, which is a hybrid. Real-time data now lives in a managed AlloyDB deployment behind NGQF, and StarRocks-on-Iceberg serves batch. If you summarize this as "they replaced Postgres with a lakehouse," you've told an interviewer you skim. Every migration I've ever worked on ended up as a hybrid in some way. Something always stays behind, and someone always maintains it forever.

Then there's the $1.2M. The post says "a GCP contract renegotiation tied to a committed-use discount (CUD) purchase locked the cost savings to a calendar date." So part of that number comes from contract terms, and the post doesn't split it out or explain how it was calculated. That's a fine choice for an engineering blog. In an interview, an unbacked dollar figure is where the follow-up questions start.

The scale numbers also depend on what's being measured. A companion post on partitioning puts the table at over 500 billion rows and more than 42 TB of compressed Parquet, on sharded Postgres that cost over $100,000 a month in SSD storage alone. 400 TB and 42 TB can both be accurate; they describe different things. If you say "400 TB" in an interview, be ready to explain what that 400 measures.

If you can't explain where your migration's headline number came from, the interviewer will assume you didn't calculate it.

I learned this one the hard way. In one loop I said a migration "cut costs by about half," and the interviewer asked half of what, whether compute, storage or licensing, and over what period. I had no answer and didn't get the offer, which was fair enough.

TRM spent a full quarter proving Iceberg before moving a single route

This section is the one to study. Before they moved a single route, TRM's platform team "spent a full quarter de-risking the execution" and "ran 9+ experiments answering specific questions about Iceberg's behavior under TRM's query patterns." The post names the question areas:

Could the joins the APIs actually run hit single-digit-second latency?
How do partitioning and bucket sizing affect tail latency?
How much throughput does compaction cost?
Does it hold up at customer-facing concurrency?

Look at how specific those are. They test this workload on this engine at this concurrency, which is exactly the shape a pipeline-architecture interview answer should have. Skip the generic system design playbook; DEs don't care about load balancers. When someone asks you to design a migration, the strongest move is to say: "Before I commit, here are the 4 questions that would kill this plan, and here's the experiment for each one."

The best part of the proof phase is what it found. Andrew Fisher's companion post, Why We Stopped Partitioning by Time and What We Did Instead, says a daily-partitioned Iceberg table "broke at every point on our distribution." For the heaviest address keys, the daily layout touched about 25,000 files. Bucketing by address got that down to about 300.

The cause is skew. Around 97% of addresses hold 8% of the volume, and about 10,000 heavy-tail addresses hold 42%. Partitioning event data by time is everyone's default instinct, mine included. It falls apart when your read pattern is "give me everything for this one address," and a handful of addresses are monsters. Fisher's conclusion, paraphrased, is that file fan-out dominated read performance and partition granularity mattered much less.

The fix came from data modeling: match the physical layout to the access pattern, and plan for the skew that breaks averages. Iceberg happened to be the table format TRM applied it in. The same reasoning works in Delta, Hudi, BigQuery clustering, or whatever we're all migrating to in 2029.

The post-cutover numbers from the Fisher post: 7 days of production traffic, 280,000+ paginated queries, P50 around 1.0s and P95 around 2.5s, against a 3.5s P95 target. That's a clean result, measured on real traffic, against a target set ahead of time. Notice the structure, because you'll copy it in a minute.

The slip, the regressions, and your behavioral interview

Now the part most companies would edit out.

Start with the slip. "The original target was July 30; but that timeline shifted to the first week of September." In the middle of the project, the real-time layer behind NGQF moved from Postgres to managed AlloyDB, and that work landed on the same 30+ routes. Plaha writes that "the conventional answer was migrate routes twice," and argues against it: doubled customer-facing risk, real-time data pushed out by months, doubled engineering cost. So they migrated each route once and took the later date.

The post reports the AlloyDB decision and the date change together, and the author judged that absorbing the scope beat doing 2 passes. The case for that judgment is the argument Plaha lays out, since the post compares the 2 paths in reasoning and gives numbers for neither. In an interview, present it as a judgment call made under uncertainty, because that's what it was. Interviewers will trust that more than an invented counterfactual.

Then this line: "We re-baselined visibly with our CTO." I've watched people hide a slip until the deadline passed, and I've done it myself early in my career, and it never goes well. Plaha puts it better than I can: "Early bad news is actionable; late bad news is only reportable." Put that sentence in your behavioral prep and keep it there.

Then the regressions. "Our pre-migration benchmarks did not always predict real-world load and query shapes." And: "Latency regressions on specific routes reached customers before we caught them." A handful of routes also saw brief errors during cutover.

So a full quarter of careful proof still missed production query shapes. I don't count that against the team. A proof phase answers the questions you thought to ask, and production traffic asks the rest. What matters is what came next: "We have since invested in synthetic load testing and shadow-traffic tooling," plus staged rollouts behind feature flags and "automatic rollback triggers as standard practice." On top of that, "the engineers running the migration had the authority to call a rollback without escalating."

Compare that to Stripe's write-up on online migrations, where they used GitHub's Scientist to read from both the old and new tables and got alerted "as soon as a single piece of data was inconsistent in production." Stripe was checking correctness on production reads before switching. TRM's problem was latency on production routes. Those are different failure modes, and a strong migration answer accounts for both. If an interviewer asks "how would you validate this cutover?", the answer they're hoping for is dual reads plus shadow traffic at real concurrency, with rollback criteria written down before day 1.

Here's how I'd turn all of this into a behavioral answer from your own work. Don't claim TRM's story as yours. Cite it as "a public TRM write-up" if it helps you explain how you think, but your behavioral examples have to come from your own career.

Open with context in 2 sentences covering what moved, how big, and why. Have a number for scale and be able to say what it measures.
Walk through the proof, meaning the 3 or 4 questions that could have killed the plan and how you tested each one.
Own the miss, the thing that broke anyway, and your part in it. Interviewers can tell when you're blaming the upstream team without saying so.
Explain the re-baseline: who you told, when, and what changed.
End on the process change, the specific thing you do differently now. TRM's is shadow traffic and automatic rollback triggers. Yours needs to be just as concrete.

I've sat on panels where a candidate described a perfect migration with zero problems, and every interviewer in the room wrote "not credible" in their feedback. I've also seen candidates get strong hires for a story that included a customer-facing incident, because they could explain exactly why it happened and what they built so it couldn't happen again. Plaha has a line for the opposite pattern: "Re-running a broken approach with cosmetic tweaks is how most migrations stall." That applies to interview prep too.

How a data engineer should practice the migration question

Every migration interview question I've been asked boils down to 3 skills: modeling the data for the new access pattern, designing experiments that could prove you wrong, and communicating when the plan changes. The tools in the story barely matter. Swap StarRocks for Snowflake and AlloyDB for Aurora and the answer stays the same.

So here's the study plan.

Start with the modeling. Take a skewed dataset and explain out loud why time partitioning fails on a per-entity read pattern, and what you'd do about it. If you can't explain the 25,000-files-versus-300 result from first principles, start there. When I want structured practice on that side of the loop, data modeling interview prep is what datadriven.io is good for, and the bucketing-versus-partitioning tradeoff is exactly the kind of problem worth drilling until the explanation comes out cleanly.

For proof design, pick any migration you've lived through and write the 4 questions you wish you'd tested first, then the experiment for each one. That becomes your system design answer.

Know every figure in your story: the baseline, the method, and what's included. If part of your savings came from a contract renegotiation, say so before the interviewer asks.

Last, rehearse the bad news. Tell your slip story out loud. Who did you tell? How early? If the honest answer is "late," say that and explain what you do now. Plenty of people have a slip in their history; the ones who get hired can explain theirs without flinching.

I've been through enough platform migrations to know the lakehouse won't be the last one any of us run. The engine will change, the table format will change, and the vendor will rename everything twice. Skew, unpredictable production query shapes, and deadlines tied to a finance contract will still be there in 10 years.

So, the question for the comments: what's the worst thing your proof phase failed to catch, and did it come up in your next interview loop?

DuckLake 1.0 and Iceberg 1.11: What a Data Engineer Should Know

DataDriven — Tue, 06 Oct 2026 10:07:03 +0000

The first time I lost a table to a metadata problem, the data was fine. Every Parquet file was sitting right there in the bucket. The pointer to the current snapshot was wrong, and for around 4 hours the table reported zero rows. Nobody looked at the data; everybody looked at the dashboard.

That's the part of the data lakehouse most people skip when they study. They learn the format names, memorize "Iceberg supports time travel," and move on. In 2026 the argument stopped being about which table format wins. It's now about where the metadata should live: as files on object storage, or as rows in a SQL database. That's what you'll be asked to reason about in your next interview, and you can answer it if you understand how commits, snapshots and catalogs work. Vendor names won't get you there.

DuckLake 1.0 and the case for metadata in a database

DuckLake started as a manifesto. In May 2025, Mark Raasveldt and Hannes Mühleisen (the DuckDB folks) published the original DuckLake post. Its core claim is that metadata is a database problem and doesn't belong in the file system. Less than a year later, on April 13, 2026, DuckLake 1.0 shipped in DuckDB 1.5.2 with a production-ready spec, a reference implementation, and guaranteed backward compatibility.

The design is simple enough to explain on a whiteboard in 60 seconds, so practice doing exactly that. Schemas, file locations, snapshots and the transaction log all live in a catalog database: Postgres if you're self-hosting, SQLite or DuckDB locally, MotherDuck in the cloud. Data stays as Parquet on object storage. To find the files for a query, you run one SQL query with a WHERE clause. MotherDuck's architecture write-up compares that to Iceberg's 3 layers of metadata files, each a sequential read of roughly 100ms.

The headline feature is data inlining. Small inserts and deletes (the default threshold is 10 rows) skip Parquet entirely and get written straight into the catalog database. A background process flushes them to object storage later. That goes straight at the small files problem, which has hurt every streaming-into-the-lake project I've touched.

DataLakeHouse Hub put a number on that problem: streaming 100 rows per second into an Iceberg table, with 100 active partitions and 60-second commits, produces about 144,000 files a day. So you schedule compaction, compaction starts fighting with your writers, and eventually someone pages you.

DuckLake's data inlining post claims big numbers against Iceberg on a 100-second streaming workload: 105x faster inserts, 923x faster aggregation, 189x faster checkpointing. These are DuckLake's own benchmarks on a workload chosen to show off inlining, so read them as a demonstration of the design and don't treat them as an independent bake-off. MotherDuck also cites roughly 100 transactions per second for DuckLake versus about 1 TPS for file-based formats, and a 2026 research paper on SQL-backed catalogs measured median commit latency of 8.4ms against 180ms for Iceberg. The direction is consistent. The exact multipliers depend on who's running the test.

The 1.0 release also included 68 reliability and correctness PRs, sorted tables for range queries, Iceberg-compatible bucket partitioning, GEOMETRY and VARIANT types, and experimental Puffin-based deletion vectors. That last one matters, because it shows DuckLake hedging toward Iceberg V3 compatibility instead of trying to fight it.

It makes me laugh that I spent years watching the industry move metadata out of the Hive metastore, which was a relational database, and onto object storage because the database was "the bottleneck." Now the most-discussed new format puts metadata back into Postgres. I've been through enough cycles to stop being surprised; the hot new thing is usually the old thing with better engineering and a nicer logo.

The economics hold up, though. A managed Postgres instance costs less per month than one afternoon of a senior engineer babysitting compaction jobs. If your workload is lots of small writes, paying for a database to absorb them is cheap next to the engineer time it takes to clean up 144,000 files a day.

Iceberg 1.11, the V3 spec, and what changes for data engineers

Iceberg didn't sit still. Apache Iceberg 1.11.0 landed in May 2026; sources put the date anywhere from May 15 to May 27 depending on whether they mean the vote, the artifacts or the blog post. The Google Open Source announcement covers the main points:

Java 17 is the new floor. Java 11 is gone.
Spark 4.1 and Flink 2.1 are the default build targets. Spark 3.4 is deprecated and scheduled for removal in 1.12; Flink 1.19 support is already removed.
Remote scan planning lets the REST catalog server do the metadata work and stream back file scan tasks.
Built-in table encryption uses envelope encryption and supports Google KMS.

The bigger story is the V3 spec, ratified in June 2025 and now production-mature in 1.11.

Deletion vectors come first. In V2, frequent deletes generated piles of positional delete files, so readers had to merge them at query time. V3 replaces them with Roaring bitmaps attached to each data file and stored in Puffin files, Iceberg's binary blob container. Multiple deletion vectors share one physical file. AWS benchmarked this on EMR and measured 55% faster deletes, delete files shrinking from 1,801 bytes to 475 bytes, and full scans 28.5% faster. For CDC and MERGE-heavy tables, that's a real improvement.

Variant is a single column that holds values of arbitrary, evolving shape. It's the native replacement for the "dump the JSON into a string column and parse it at query time" workaround all of us have shipped at least once. Shredding (storing Variant fields in columnar form) currently only works with Parquet.

Row lineage adds _row_id and _last_updated_sequence_number, which give every row an identity and a change marker. That's useful for CDC and audit.

The V2 to V3 upgrade is one-way. The ALTER TABLE itself is metadata-only and rewrites no data, so it feels harmless. Once a table writes V3 metadata, though, any engine that only understands V2 can't read it. I've lived through a version bump that broke a reader nobody remembered existed: a quarterly finance extract running on an old cluster, owned by a person who'd left 2 years earlier. We found it when finance called.

Before you upgrade anything:

Inventory every reader and writer: every engine, every version, every scheduled job. Trace them through query logs, because the docs will be wrong.
Get Java 17 everywhere first. Iceberg didn't make this choice lightly; maintainers moved after contributors on Java 21 kept hitting build failures against modules pinned to 11. Your Spark runtime, your Flink jobs and your random ingestion service all need it.
Upgrade a table for a feature, and only for a feature. If you need deletion vectors, Variant or encryption, upgrade that table. If you don't, V2 tables keep working.
Expect governance to be the hard part. As one DataLakeHouse Hub piece put it, catalog migrations "fail rarely on the format and almost always on governance." Access control and permissions are where the weekend goes.

The catalog is where vendors are fighting now

Look at the business news this year and the pattern is hard to miss: everyone is putting a database back in the middle of the lake.

SAP is buying Dremio. The deal was announced May 4, 2026, with close expected in Q3. SAP says it's "fully committed to continuing to invest in and prioritize" Iceberg, Polaris and Arrow, and plans to build Business Data Cloud into an Iceberg-native lakehouse with an open catalog on Polaris and the Iceberg REST API. Dremio has been one of Iceberg's main stewards. When a company that size buys a core contributor, ask what happens 3 years from now, after the press release has been forgotten.

The safety net is governance. Apache Polaris graduated to a top-level Apache project on February 15, 2026, after 18 months of incubation, 6 releases and over 2,800 merged PRs. The vote passed with 27 binding +1s and no objections, and contributors include Snowflake, AWS, Google Cloud, Azure, dbt Labs, Stripe and others. With that many companies committing code, one acquirer can't take the project over without the others noticing. If you're on Dremio, the Iceberg REST Catalog API is your exit door. Keep it open.

Databricks is putting Postgres inside the lakehouse. It bought Neon for about $1 billion in May 2025, citing that over 80% of Neon databases were being created by AI agents. In August 2026, Databricks reported Lakebase above $100M revenue run-rate, growing twice as fast as its Lakehouse product. In May, Databricks also made Catalog Commits generally available for Unity Catalog managed Delta tables. That routes commits through the catalog so external engines can't write straight to storage and leave the catalog and the files disagreeing.

Snowflake did its own version. It bought Crunchy Data for $250 million and open-sourced pg_lake in November 2025, which lets Postgres create and query Iceberg tables directly.

Iceberg won broad engine support; Delta and Hudi both added Iceberg compatibility. The fight has moved one layer up, to the catalog, because whoever owns the commit owns the table. File-based metadata and database metadata are 2 answers to the same question: who gets to say what the current version of a table is, and how fast can they say it?

Formats are converging. Catalogs are where the lock-in lives now. Pick your catalog like you're going to be stuck with it, because you probably are.

Explaining table formats in a data engineering interview

Interviewing is a separate skill from the job. I've been on panels where a candidate who'd run Iceberg in production for 2 years fumbled "what does the catalog actually do?" because they'd never had to say it out loud. That's what costs people the level. They have the experience; they just can't put it into words under pressure.

These are the concepts that keep showing up, and how I'd answer each.

"What's the difference between Parquet and Iceberg?" Parquet is a file format; it describes how bytes are laid out in one file. Iceberg is a table format: snapshots, atomic commits, schema evolution, partition evolution, and the metadata that tells an engine which files make up the table right now. If you say "Iceberg is Parquet with metadata," a good interviewer will keep pushing until you fall over.

"What does a catalog do?" It answers one question: where is the current metadata for this table? A commit is an atomic swap of that pointer. Two writers race and one wins; the loser retries against the new snapshot. Every catalog design, including Polaris, Unity, DuckLake's Postgres and Glue, is a different way of making that swap fast and safe.

"What isolation do you get?" Snapshot isolation. Readers never block writers, and writers never corrupt an in-progress read. Candidates regularly call this serializable. It isn't, and saying so correctly tells a senior interviewer you've thought about concurrent writers. Bonus points for knowing Iceberg tracks snapshots through manifest lists, Delta through a sequential JSON commit log, and Hudi through a timeline.

"Why would anyone put metadata in a database?" Small, frequent writes. Under CDC or sub-second commits, the bottleneck is metadata overhead, and data I/O stops mattering. Object storage charges around 100ms per request; a database answers in milliseconds and handles concurrency well. That's the reason DuckLake exists, and the reason LSM-based designs like Paimon exist.

"Which format would you pick?" Never answer with a benchmark from a blog post. Name the axes: which engines need to read and write the table, what your team can actually operate, and how much vendor independence you need. For a DuckDB-heavy shop with a modest team, DuckLake on Postgres is operationally light. For 4 engines across 3 teams with real governance requirements, Iceberg behind a REST catalog is the default for a reason.

Then add the opinion that separates seniors from everyone else: most of y'all don't need 100 transactions per second. Batch still covers most of what companies run. The ability to stream into the lake is nice to have; making a 6-hour batch job reliable is what keeps you employed. Say that in the interview, with a reason, and you sound like someone who's carried the pager.

You don't need a vendor certification for any of this. Every concept above (atomic commits, isolation levels, write amplification, metadata lookup cost) existed before these products and will outlive them. A DuckLake catalog is literally tables in Postgres you can query, so SQL fluency pays off twice. If you need to sharpen it, for sql practice problems, DataDriven is great, and we built it as interview prep for the SQL half of the loop. Use it to free up your head for the architecture questions, which is where these conversations get decided.

The tools in this article will look different in 18 months. Somebody will be acquired, somebody will fork something, and Iceberg V4 will promise single-file commits. Commits, snapshots and the catalog pointer will still be what you're debugging at 3am.

So, for those of you already running a lakehouse in production: where does your metadata live today, and has a catalog ever lied to you about what's in a table?

Data Engineering Failures Cost Enterprises $3M a Month

DataDriven — Thu, 01 Oct 2026 10:06:50 +0000

Years ago I inherited a job that had been silently dropping around 40% of its records for 6 months. It didn't crash or alert, and the logs were clean. Someone in finance finally asked why the revenue numbers looked "a little light," and that's how we found out. The fix took an afternoon. Figuring out what had broken took 2 weeks, and the trust took a lot longer to rebuild.

I bring this up because Fivetran just put a price tag on that kind of failure, and it's a big one. Their 2026 enterprise benchmark says pipeline failures expose large companies to roughly $3M a month. In the same stretch of 2026, companies cited AI as the reason for a growing share of tech layoffs. The work that prevents multimillion-dollar outages is the same work getting squeezed, and if you work in data engineering, that tension is also your best career lever right now.

What Fivetran's 2026 benchmark found, and who it surveyed

Fivetran's Enterprise Data Infrastructure Benchmark Report 2026 surveyed 500 senior data and technology leaders in Q4 2025. Every respondent works at an organization with 5,000+ employees, across the US, UK, EMEA, and APAC. The industry mix leans toward financial services (23%), manufacturing (20%), and tech (20%), followed by retail/CPG, healthcare, and a little hospitality. Fivetran reports 95% confidence with a ±4.4% margin of error.

Here are the headline numbers:

4.7 pipeline breaks per month on average, rising to 8.3 at the largest enterprises
60.4 hours of downtime per month, with each incident taking nearly 13 hours to resolve
53% of engineering capacity spent on maintenance instead of new work
$2.2M per year per enterprise in maintenance labor alone
An average shop runs 328 pipelines with 35 full-time engineers
97% of leaders said pipeline failures have slowed their analytics or AI programs

The math holds together, which I appreciate. 4.7 breaks at roughly 13 hours each comes to roughly 61 hours, which matches the 60.4. Fivetran's press release puts business exposure at $49,600 per hour of downtime. Multiply that by 60.4 hours and you land right around $3M a month. A single incident can reach $1.4M.

Now the part a vendor's marketing team won't put in bold: Fivetran sells managed ELT. The report says legacy and DIY systems break 30% to 47% more often than managed ones, and that managed-platform adopters are "nearly 2x as likely" to beat ROI expectations. That might be true. It's also exactly the finding you'd expect from a company whose business is managed pipelines. Treat the breakage and downtime counts as decent signal and the "buy managed and your problems go away" conclusion as a sales pitch.

I've migrated legacy warehouses at 3am. Managed connectors do remove a whole category of pain, mostly API pagination, auth token rotation, and schema changes in SaaS sources. They don't fix a bad data model, an upstream team renaming a column without telling anyone, or a join that quietly fans out and doubles your revenue.

The number I keep coming back to is the 53%. Of 35 engineers, around 18 spend their week keeping existing pipelines alive. Every senior DE I know would call that number low.

Pipeline failures and layoffs in the same month

Now set that next to the layoff data.

The Challenger, Gray & Christmas May 2026 report has AI cited in about 22% of all US job cuts through May. In May alone it was 40%. Later 2026 coverage of Challenger's numbers puts the year-to-date share around 24% through July, with AI as the most-cited reason for several months in a row. The cumulative counts in secondary write-ups don't fully agree with each other, so I won't pretend to know the exact figure, but the direction is clear. Anyone still quoting "roughly a fifth" is out of date, because the share rose through mid-year.

TechCrunch has kept a running list of large tech layoffs where employers cited AI, including Amazon, Oracle, Microsoft, Meta, and PayPal. Andy Challenger summed it up this way: "Tech remains the center of gravity for this year's cuts, and AI is still the reason companies give."

Keep the phrase "the reason companies give" in mind. What a company says in a press release about a layoff and what actually caused it are often unrelated. I've survived multiple layoff waves. The official reason has always been a story told to the board, and this year the story is AI.

The economics don't work, and I mean that literally. A typical enterprise in Fivetran's sample has $3M a month of downtime exposure, and the people preventing it are a line item in the maintenance budget. Cut 5 of those engineers to save about $1M a year in salary, and you only need a few extra 13-hour incidents to give all of it back. You're also teaching the business that the dashboards can't be trusted, which costs more than any of it.

The labor that keeps $3M a month of downtime contained is the easiest line item to cut and the most expensive one to lose.

I'll be fair to the other side. The same research also shows data engineering postings growing, with demand moving toward people who can handle infrastructure, governance, and cost control. The squeeze lands hardest on generic and lower-tier roles. So the DE role isn't disappearing. It's being repriced, and the people doing the repricing don't fully understand what the job involves.

Where the maintenance hours go

Monte Carlo's 2026 State of Data Quality survey answers most of it. 68% of teams need 4+ hours just to detect an incident, up from 62% in 2022. Average resolution time is 15 hours. And the stat that should embarrass everyone in this industry: 74% of data issues are found by business stakeholders first, up from 47% in 2022.

So for 3 out of 4 issues, the monitoring system is a VP in a Slack channel asking why the numbers look weird. That was my 40% record-drop job exactly, except today it happens at 3x the scale.

That's where the 53% goes. Most of it is detective work: finding which of the 328 pipelines broke, which upstream source changed, which backfill needs to rerun, and which downstream tables are now wrong. The actual fix is usually the shortest part of the incident.

The other surveys in the research fill in the rest:

dbt's 2026 State of Analytics Engineering report found 72% of data teams prioritize AI coding, while only 24% prioritize AI-assisted pipeline testing and observability. So we're speeding up how fast people write pipelines and barely investing in knowing when they break.
Atlassian's DX report for Q2 2026 found AI saves engineers 4 to 6 hours a week, yet the innovation ratio barely moved, staying roughly 57% to 58%. The freed-up time gets absorbed by the maintenance backlog.
Teams with automated observability reportedly resolve incidents about 4x faster. That claim is secondhand, so hold it a bit loosely, but every incident I've worked agrees with the direction.

The tooling is catching up, for what it's worth. Apache Airflow 3.3 shipped AIP-103, a first-class task and asset state store. It persists key-value state across worker crashes and retries, which replaces the XCom hacks that never survived a retry. Apache Iceberg 1.12 replaces position delete files with deletion vectors, compact bitmaps that Dremio says cut read latency by 50% to 80% for high-frequency deletes. That matters a lot for CDC and GDPR workflows.

Both are good releases. Neither changes the fundamentals. Iceberg still needs compaction, snapshot expiration, orphan cleanup, and manifest optimization running in a coordinated way, and if you get orphan retention wrong a mid-write failure can silently corrupt your table. A state store won't help if you don't understand idempotency, because you'll just persist the wrong state faster.

This is the concepts-over-tools argument again. Idempotency, late-arriving data, schema contracts, grain, and backfill strategy work the same in Airflow 2, Airflow 3.3, Dagster, or a cron job someone wrote in 2014. Tools change every 18 months, and the reasons pipelines fail don't.

Your next data engineering interview

Compensation hasn't caught up with reliability work yet. Indeed Hiring Lab's September 2026 analysis found advertised pay in highly AI-exposed occupations up roughly 46% since 2021, compared with 25% for less-exposed fields. The AI premium on identical job titles is about 4.7%, and much larger at senior levels. Mid-level DE base pay sits about $139K median. Meanwhile the people preventing $3M a month in downtime get paid like plumbers. (Plumbers, to be fair, are often doing fine.)

I don't think that lasts, and the hiring signals already show it shifting. Recruiting guides for 2026 now treat "reliability signals" as the minimum: SLAs, monitoring, alerting, backfills, retries, tests, incident response, and postmortems. Senior job postings name observability architecture as a core responsibility, and some staff-level postings run past $260K.

Here's how to use that.

Rewrite your resume around incidents. "Built ETL pipelines using Airflow and Snowflake" is a tool list. "Cut detection time on a revenue pipeline from next-day to 20 minutes by adding row-count and freshness checks" is a story. Don't tell me you "ensured data reliability across a wide array of mission-critical workflows." If I read one more bullet like that I'm putting my fist through drywall.

Have a postmortem ready. Interviewers increasingly ask you to walk through a real incident: how you detected it, what the blast radius was, how you fixed it, and what you changed so it couldn't happen again. Pick your best one and rehearse it until you can tell it in 3 minutes without rambling. If you don't have one, you haven't been doing this long enough, or you were lucky. Either way, go break something in a side project and fix it.

Prep for pipeline architecture. I've watched people with 10 YOE get downleveled because they couldn't explain why their design would survive a late-arriving partition. DEs don't need to whiteboard load balancers. You need to explain how your pipeline handles retries without double-writing, how you'd backfill 90 days without taking down the warehouse, and where your quality checks go. That's the gap we built DataDriven to cover, so for pipeline design practice, try DataDriven: it drills exactly those failure-mode questions, which is where loops separate seniors from everyone else.

Ask about the 53% in your interviews. Ask every hiring manager how many incidents they had last quarter and who found them first. If the answer is "the business usually tells us," you're interviewing for a firefighting job, so price the offer accordingly. If they actually know their numbers, that's a team worth joining.

If you're worried about layoffs, become the person who knows why things break. Every layoff list I've seen spared the engineer who held the context on the scary pipelines. It comes down to what happens when the CFO asks, "who can tell me why the numbers are wrong?" and only one name comes up.

I've been through 3 waves of "data engineering is getting automated away." I'm still here and still debugging the same categories of problems. The Fivetran numbers just put a dollar figure on what every on-call DE already knew: the hard part of the job is keeping things working. The companies cutting that work now will rehire for it at a premium after their first $1.4M incident, and the engineers who can tell a good postmortem story will get those offers.

So here's my question: in your last pipeline incident, who noticed first, your monitoring or someone from the business?