DEV Community: Rahman

SQL Interview Question: Find App Store Power Purchasers with Two-Level GROUP BY

Rahman — Mon, 21 Sep 2026 06:30:32 +0000

Some SQL problems look simple until you read the rules twice. This one is tagged Apple and rated hard, and it teaches a useful pattern: grouping the results of a grouping.

Let's solve it step by step.

The Problem in Plain English

What is a power purchaser?

A power purchaser is a customer who made at least 3 in-app purchases in each of these months:

April 2023
May 2023
June 2023

Miss the mark in even one month, and the customer is out. No purchases in a month also means out.

What should the query return?

For every power purchaser, return:

user_id
email
total_amount_spent: the sum of all their purchases from April 1 to June 30, 2023, rounded to 2 decimal places

Sort by total_amount_spent from high to low. If two users tie, the smaller user_id comes first.

Rules that trip people up

A purchase with a NULL amount still counts as a purchase. It adds 0 to the total.
A purchase with 0.00 also counts as a purchase and adds 0.
The total includes every purchase in the window, not only the first 3 in each month.

The Tables

users

Column	Type
user_id	integer
join_date	date
email	string

purchases

Column	Type
purchase_id	integer
user_id	integer
purchase_date	date
amount	decimal

A Quick Example

users

user_id	join_date	email
601	2022-03-14	j.alvarado@icloud.com
602	2021-08-02	s.moreau@icloud.com

purchases

purchase_id	user_id	purchase_date	amount
1	601	2023-04-03	0.99
2	601	2023-04-03	1.99
3	601	2023-04-01	0.00
4	601	2023-05-01	2.99
5	601	2023-05-02	1.99
6	601	2023-05-03	4.99
7	601	2023-05-04	0.99
8	601	2023-06-01	1.99
9	601	2023-06-02	2.99
10	601	2023-06-03	9.99
11	602	2023-04-03	1.99
12	602	2023-04-03	2.99
13	602	2023-04-01	0.99
14	602	2023-05-01	0.99
15	602	2023-05-02	1.99
16	602	2023-05-03	2.99
17	602	2023-06-01	1.99
18	602	2023-06-02	2.99

User 601 made 3 purchases in April (one was 0.00), 4 in May, and 3 in June. Every month reaches 3, so this user qualifies. The total across all 10 purchases is 28.91.

User 602 made 3 in April, 3 in May, but only 2 in June. That is one month short, so this user is out.

Expected output:

user_id	email	total_amount_spent
601	j.alvarado@icloud.com	28.91

How to Think About It

The trick is that we need two rounds of grouping:

Group by user and month to find which months have 3 or more purchases.
Group those good months by user to find who has all 3 months.

A regular WHERE can't do this, because we are filtering on counts. That is the job of HAVING.

Step-by-Step Solution

Step 1: Count purchases per user, per month

First, keep only the April to June 2023 rows. Then group by user and month.

SELECT
  user_id,
  EXTRACT(MONTH FROM purchase_date) AS purchase_month,
  COUNT(*) AS purchase_count
FROM purchases
WHERE purchase_date BETWEEN '2023-04-01' AND '2023-06-30'
GROUP BY user_id, EXTRACT(MONTH FROM purchase_date);

For our example, this gives:

user_id	purchase_month	purchase_count
601	4	3
601	5	4
601	6	3
602	4	3
602	5	3
602	6	2

COUNT(*) counts rows, so NULL amounts are counted too. That is what the problem wants.

Step 2: Keep only months with 3 or more purchases

Add a HAVING clause to drop weak months.

HAVING COUNT(*) >= 3

Now user 602's June row (count 2) is gone.

Step 3: Find users who passed in all 3 months

Below, monthly_counts stands for the result of Steps 1 and 2. In the final query, we wrap it in a CTE with that name.

Each remaining row is one qualifying month for one user. So if a user has exactly 3 rows, they passed April, May, and June.

SELECT user_id
FROM monthly_counts
GROUP BY user_id
HAVING COUNT(*) = 3;

A user with no purchases in a month never appears for that month, so they can't reach 3 rows. The "missing month" rule is handled automatically.

Step 4: Add up all spending and sort

Now join back to purchases and users. Joining to the list of power users keeps only qualifying customers. Then sum every purchase in the window, not just the ones that passed the test.

Final Query

WITH monthly_counts AS (
  SELECT
    user_id,
    EXTRACT(MONTH FROM purchase_date) AS purchase_month,
    COUNT(*) AS purchase_count
  FROM purchases
  WHERE purchase_date BETWEEN '2023-04-01' AND '2023-06-30'
  GROUP BY user_id, EXTRACT(MONTH FROM purchase_date)
  HAVING COUNT(*) >= 3
),
power_users AS (
  SELECT user_id
  FROM monthly_counts
  GROUP BY user_id
  HAVING COUNT(*) = 3
)
SELECT
  u.user_id,
  u.email,
  CAST(COALESCE(SUM(p.amount), 0) AS DECIMAL(10,2)) AS total_amount_spent
FROM purchases p
JOIN users u        ON u.user_id = p.user_id
JOIN power_users pu ON pu.user_id = p.user_id
WHERE p.purchase_date BETWEEN '2023-04-01' AND '2023-06-30'
GROUP BY u.user_id, u.email
ORDER BY total_amount_spent DESC, u.user_id ASC;

Two small notes on this version:

SUM skips NULL values, so they add 0 as required. COALESCE is a safety net in case a user's amounts are all NULL.
The CTEs (the WITH blocks) give each step a name, so the query reads from top to bottom.

Common Mistakes

Using WHERE COUNT(*) >= 3. Aggregates can't go in WHERE. Use HAVING.
Summing only the qualifying purchases. The total must cover every purchase in the window.
Dropping NULL or 0.00 rows. They count toward the 3-purchase minimum.
Checking the total count instead of each month. A user with 9 purchases in April alone should not pass.

Small Things to Watch

EXTRACT(MONTH ...) works in PostgreSQL, MySQL, and DuckDB. In SQL Server, use MONTH(purchase_date).
If your data covered more than one year, group by the year as well. Otherwise April 2022 and April 2023 would be mixed together.
If purchase_date were a timestamp, BETWEEN ... AND '2023-06-30' would miss most of June 30. Use < '2023-07-01' instead.

Wrap-Up

The pattern here is worth remembering: group, filter with HAVING, then group again. It shows up in many interview questions about streaks, "every month" rules, and consistency checks.

Try changing the rules and see if you can adapt the query. For example, what if a user only needed 3 purchases in any two of the three months?

Meta's "Combined Reaction Volume" SQL Question, Explained Simply

Rahman — Sat, 19 Sep 2026 14:30:09 +0000

Your Feed post gets turned into a Reel. Somebody watches it and taps like. Does the database record that as one like, or two?

That messy little detail is the whole point of a Meta SQL interview question called Combined Reaction Volume. It looks like a basic counting problem. It isn't. And if you rush it, you'll get a number that's technically correct and completely wrong at the same time.

The setup

Meta keeps reactions to Feed posts and Reels in two separate tables, because the two products are built by separate teams:

post_reactions — likes, loves, and other reactions on regular Feed posts
reel_reactions — the same kinds of reactions, but on Reels

Here's the catch. A Reel can get cross-posted to the Feed. When that happens, one single tap of the like button can end up logged in both tables. On top of that, retry logic sometimes fires twice and inserts the exact same row into one table by mistake.

So the real question isn't "how many rows are there." It's "how many real reactions happened," once you strip out the copies.

What counts as "the same reaction"

The question defines a reaction event as a match on three things: user_id, content_id, and reaction_date. If those three line up between two rows — even across the two different tables, even if one row has a NULL reaction type — you treat them as one event, not two.

The task: for each user, count their distinct reaction events between January 1 and March 31, 2024, and only keep users with at least 5.

Where people go wrong

The instinct is to smash both tables together and count. Something like this:

-- Looks right. Silently double-counts.
SELECT user_id, COUNT(*) AS reaction_count
FROM (
    SELECT user_id, content_id, reaction_date FROM post_reactions
    UNION ALL
    SELECT user_id, content_id, reaction_date FROM reel_reactions
) combined
GROUP BY user_id;

UNION ALL stacks every row from both tables on top of each other, duplicates included. Run this against the sample data and user 10 comes back with 6 reactions. The real number is 5. That one cross-posted like got counted twice, and the query has no idea it did anything wrong.

Solving it, step by step

Step 1 — Cut each table down to just what defines an event.
Drop reaction_id and reaction_type, they're not part of what makes an event unique. Keep only user_id, content_id, and reaction_date, and filter both tables to the Q1 2024 range.

SELECT user_id, content_id, reaction_date
FROM post_reactions
WHERE reaction_date BETWEEN '2024-01-01' AND '2024-03-31'

Step 2 — Combine the two tables with UNION, not UNION ALL.
This is the actual trick. Plain UNION automatically drops exact duplicate rows, whether they came from the same table twice or from two different tables. Swap UNION ALL for UNION and the double-counted like disappears on its own.

...
UNION
SELECT user_id, content_id, reaction_date
FROM reel_reactions
WHERE reaction_date BETWEEN '2024-01-01' AND '2024-03-31'

Step 3 — Count what's left, per user.
Wrap the combined result in a subquery and group by user_id.

SELECT user_id, COUNT(*) AS distinct_reaction_count
FROM (...) combined_reactions
GROUP BY user_id

Step 4 — Apply the 5-event minimum, then sort.
HAVING filters on the group total, not on individual rows, so it comes after GROUP BY. Sort by count descending, and by user_id ascending to keep ties in a fixed order.

HAVING COUNT(*) >= 5
ORDER BY distinct_reaction_count DESC, user_id ASC

Put it all together:

SELECT
    user_id,
    COUNT(*) AS distinct_reaction_count
FROM (
    SELECT user_id, content_id, reaction_date
    FROM post_reactions
    WHERE reaction_date BETWEEN '2024-01-01' AND '2024-03-31'
    UNION
    SELECT user_id, content_id, reaction_date
    FROM reel_reactions
    WHERE reaction_date BETWEEN '2024-01-01' AND '2024-03-31'
) combined_reactions
GROUP BY user_id
HAVING COUNT(*) >= 5
ORDER BY distinct_reaction_count DESC, user_id ASC;

Why this one matters

Most people learn UNION as "the thing that stacks two tables together" and UNION ALL as "the faster version of the same thing." This question flips that around. UNION isn't just a stacking tool, it's a deduplication tool, and knowing when you actually want duplicates removed versus when you don't is the real skill being tested here, not the syntax.

Practice these types of questions on DataCurlew.

I built a 100% private WhatsApp chat analyzer (Zero servers, zero tracking)

Rahman — Sat, 19 Sep 2026 08:11:31 +0000

Most chat analysis tools ask you to upload private conversation archives to a remote server.

Even if the site claims "we delete logs immediately," handing unencrypted personal chats to an unfamiliar API is a major privacy compromise.

I built PurpleChats to solve this with one rule: zero backend.

Every calculation—from unzipping .zip exports to generating interactive Spotify Wrapped-style cards—runs strictly in the user's browser.

What It Does

🎁 WhatsApp Wrapped Mode: 3D interactive story cards showing chat streaks, late-night chatter, and response time percentiles.
📊 Group Dynamics: Response latency rankings, conversation starter badges, and interactive interaction graphs.
🔒 100% Client-Side Privacy: No databases, no telemetry, and zero network calls. When you close the tab, the data is gone.

How It Works Under the Hood

Parsing tens of thousands of messages on the main browser thread causes frame drops.

To keep the UI running at a clean 60fps:

In-Memory Unzipping: Uses fflate to unpack .zip archives directly into RAM without saving temporary files.
Web Workers: All regex matching, date normalization, and metric calculations run in a background worker. A 30,000-message chat processes in under 180ms.
Canvas Card Export: Uses html-to-image to let users export shareable metric cards as PNGs directly from the DOM.

Tech Stack

Framework: React 18 + TypeScript + Vite
Styling: Tailwind CSS (editorial warm-paper design)
Processing: Web Workers + fflate
Hosting: Cloudflare Pages (free static CDN)

Try It Out

🌐 Live App: https://purplechats.pages.dev
💻 GitHub: https://github.com/Rahmman001/PurpleChats

The project is fully open source under the MIT license. Feedback and contributions are welcome!

Why I Put PostgreSQL in WebAssembly to Fix SQL Interview Prep

Rahman — Thu, 17 Sep 2026 10:17:20 +0000

A few months ago, a backend engineer friend was preparing for a Meta data engineering screen. He asked me to review his SQL prep routine.

He was using the usual suspects: two popular coding websites and a paid course. But watching him practice was painful.

Every single query submission triggered a loading spinner that hung for three to six seconds while a remote backend spun up a container, connected to an ephemeral database, ran the query, and shipped back a JSON blob. Worse, whenever he tried using PostgreSQL-specific functions—like DATE_TRUNC('month', ts) or window functions with specific frame clauses—half the platforms choked. Why? Because under the hood, many "SQL interview" platforms run SQLite with light string-replacement hacks rather than real Postgres.

Day-to-day SQL is usually simple: an ORM query, a basic GROUP BY, or a quick dashboard filter.

FAANG technical interviews are completely different. Companies like Meta, Google, and Amazon test edge-case relational logic: 7-day rolling user retention across discontinuous date gaps, dense ranking ties with multi-column partitions, and recursive hierarchy trees.

Practicing those problems shouldn't feel like waiting on dial-up internet.

I wanted something faster. No servers. No container spin-ups. Real PostgreSQL. Instant grading.

Here is how I built DataCurlew using PostgreSQL 16 compiled to WebAssembly.

The Architecture: Why Run Postgres in the Browser?

Traditionally, an online code execution platform looks like this:

Browser (Monaco/CodeMirror) 
  → API Gateway 
  → Queue (RabbitMQ / Redis) 
  → Sandbox Worker (Docker / Firecracker) 
  → Ephemeral DB 
  → Response (3–5 seconds total)

Running that architecture has two major problems:

Latency: Developers hate feedback delays. When you are debugging a nested CTE, waiting 4 seconds per run destroys your mental flow.
Infrastructure Cost: Hosting thousands of isolated Docker containers executing arbitrary SQL costs real money, which gets passed on to users through steep monthly subscriptions.

Enter PGlite and WebAssembly

ElectricSQL recently open-sourced PGlite—a lightweight build of official PostgreSQL packaged into a WebAssembly binary. It isn't a mock or an emulator. It is the real C source of PostgreSQL compiled via Emscripten down to roughly 3MB gzipped.

That shifted the entire architecture:

Browser Tab (React 18)
  ├── CodeMirror 6 (Editor)
  ├── PGlite WASM (In-Memory PostgreSQL 16 Engine)
  └── Grading Engine (Deterministic Table Diffing)

Total Execution Latency: ~1.4ms
Server Load: Zero

When you hit Run Query on DataCurlew, the query doesn't leave your machine. It executes in-memory inside the browser's WebAssembly runtime.

The query runs in 1.4 milliseconds.

Because it runs real PostgreSQL 16, every single Postgres feature works out of the box:

Window functions with custom frame specifications (ROWS BETWEEN 1 PRECEDING AND CURRENT ROW)
Common Table Expressions (including WITH RECURSIVE)
Full datetime math (INTERVAL '7 days', DATE_TRUNC)
Native JSONB manipulation (jsonb_array_elements)
Statistical aggregates (PERCENTILE_CONT, FILTER (WHERE ...))

Two Hard Problems We Had to Solve

Moving the database into the client sounds simple until you actually grade user submissions.

1. Fast, Deterministic Sandbox Resets

In an interview, candidates write destructive or state-altering statements (DROP TABLE, DELETE, UPDATE), or they introduce infinite loops in recursive CTEs.

If the user mutates an input table, how do you prevent that mutation from corrupting the hidden test cases?

Instead of re-downloading schemas over the network for every run, we store compiled DDL blueprints in client memory. Before evaluating a candidate's solution against hidden datasets:

The engine spins an isolated in-memory PGlite instance.
Tables are seeded with test fixtures via raw SQL buffers.
The user query executes against that isolated session.
A deterministic diff compares the resulting row matrix against the expected ground-truth dataset.

This entire sequence completes in under 20 milliseconds.

2. Table Comparison is Harder Than It Looks

In standard algorithmic problems, checking output === expected works. In SQL, two queries can be mathematically identical while producing subtle tabular differences:

Type Coercion: One query returns 42 as an integer; another returns '42' as text, or 42.00 as a numeric type.
Floating-Point Rounding: Financial or retention aggregations can have microscopic rounding variances across WASM platforms.
NULL Semantics: In SQL, NULL = NULL evaluates to unknown, not true.
Row Ordering: Unless an explicit ORDER BY is specified in the problem statement, relational sets have no guaranteed order.

We built a custom normalization and diffing layer (compareResults.ts) with strict unit test coverage that reconciles column types, evaluates unordered row hashes when order is arbitrary, and checks exact sorting rules when the problem demands strict rank order.

Breaking Down Real FAANG Questions: The Playbooks

Beyond the sandbox, what engineers actually need is strategic intuition.

Most platforms just post the problem and a short solution. But when candidates freeze in an interview, it's almost never because they forgot the syntax of JOIN. It's because they couldn't decompose the problem statement into relational steps.

We built dedicated Company Playbooks for companies like Amazon, Google, and Meta:

-- Example: Meta Interview Question
-- Finding monthly retention with window functions
WITH monthly_activity AS (
  SELECT 
    user_id, 
    DATE_TRUNC('month', stream_date) AS active_month
  FROM streams 
  GROUP BY 1, 2
)
SELECT 
  active_month,
  DENSE_RANK() OVER (ORDER BY COUNT(*) DESC) AS rank
FROM monthly_activity 
GROUP BY active_month;

Each problem breakdown includes:

Interactive Multi-Table Schemas: Candidates can toggle between multiple input tables, explore edge-case data fixtures, and see the expected output matrix before typing code.
Relational Logic Walkthroughs: Step-by-step breakdowns showing why a CTE or self-join was chosen over a subquery.
Immediate Sandbox Handoff: One click opens the problem in the in-browser sandbox with the exact schema pre-loaded.

Takeaways from Shipping WebAssembly to Production

If you are considering compiling heavy C/C++ runtimes into WebAssembly for developer tools:

The browser is an operating system now. Don't default to server-side workers for code execution if a client-side WASM runtime exists. The latency win changes the product feel completely.
Cold start matters. We pre-warm the PGlite instance as soon as the candidate navigates toward a problem. By the time they finish reading the task statement, the engine is warm and responsive.
Keep it accessible. You don't need an account, a credit card, or a Docker daemon running to practice. You just open the page and write SQL.

Check It Out

If you have an upcoming SQL interview—or just want to test how fast PostgreSQL runs inside your browser tab—try it out:

🔗 DataCurlew (datacurlew.com)

💻 Free SQL Learning Path

🏢 Meta SQL Playbook

I would love to hear feedback from other engineers: what SQL edge cases do you find hardest in technical interviews, and what features would make your practice workflow smoother?

The Data Modeling Concepts Nobody Mentions After Star Schema 101

Rahman — Wed, 16 Sep 2026 14:22:38 +0000

Most data modeling tutorials stop at the same place: here's a fact table, here's a dimension table, join them, done. And that's genuinely enough to get a warehouse working. It's also enough to make you think you're finished learning.

Then two things happen in the same month. A sales deal gets credit split between three reps instead of one, and your fact table — built for one rep per deal — has no clean way to hold that. And finance asks for "account balance on the last day of every month," and you realize your fact table only stores balance-change events, not balances. Neither of these is a bug. They're just the next layer of modeling that nobody mentioned in the "intro to star schemas" post.

Here's that next layer, in plain words, with a running example: an online furniture store.

When a dimension needs its own dimensions: snowflake schema

In a basic star schema, dim_product might hold everything about a product in one flat row — including its category name and department name, repeated for every single product in that category.

-- dim_product (star schema — flat, some repetition)
product_key | product_name | category_name | department_name
201         | Oak Desk     | Desks         | Furniture
202         | Pine Desk    | Desks         | Furniture

If "Desks" ever gets renamed to "Work Desks," you're updating that string in every row that mentions it. A snowflake schema splits the dimension further, so category and department live in their own tables:

-- dim_product (snowflaked)
product_key | product_name | category_key
201         | Oak Desk     | 55

-- dim_category
category_key | category_name | department_key
55           | Desks          | 9

Now a rename touches one row. The cost is that a query which used to need one join now needs two or three. Neither version is "correct" — a snowflake schema wins when a dimension's attributes actually change independently and often; a flat star schema wins when the query speed matters more than that theoretical tidiness. Most teams default to star and only snowflake the one or two dimensions that genuinely need it.

Not every fact table looks the same

The sales-and-refunds fact table from a basic tutorial — one row per event — is called a transaction fact table. It's the most common type, but it's not the only shape a fact table can take.

A periodic snapshot fact table captures a measurement at a regular interval, whether or not anything happened. Think of a warehouse's inventory count, taken every night at midnight:

-- fact_inventory_daily
snapshot_date | product_key | warehouse_key | units_on_hand
2025-09-14    | 201         | 3             | 42
2025-09-15    | 201         | 3             | 39

Nobody "did" anything to generate the September 15th row — it's just a photograph of the state of things that day. This is the shape you want for "what was our balance on the last day of the month," because a transaction table only tells you about changes, not states.

An accumulating snapshot fact table is for tracking something with a defined start and end, where you update the same row as it moves through stages — an order going from placed to packed to shipped to delivered:

-- fact_order_fulfillment
order_id | placed_date | packed_date | shipped_date | delivered_date | days_to_deliver
5001     | 2025-09-01  | 2025-09-01  | 2025-09-02   | 2025-09-05      | 4
5002     | 2025-09-03  | 2025-09-03  | NULL         | NULL            | NULL

Instead of one row per status change, there's one row per order, and columns get filled in as milestones happen. This is the natural shape for pipeline and fulfillment reporting — you can instantly see how many orders are stuck between "packed" and "shipped" without scanning an event log.

When there's nothing to measure: factless fact tables

A fact table doesn't have to have a number in it. Say you want to track which students attended which class sessions. There's no "amount," no "quantity" — just the fact that an event happened.

-- fact_attendance
student_key | class_session_key | date_key
77          | 1204               | 20250915

You get your measure from COUNT(*), not from a column. This shows up more than people expect — eligibility events, promotions applied, security check-ins. If you catch yourself adding a fake count = 1 column just so the table "feels like" a proper fact table, that's usually a sign it's a factless fact table and that's fine.

Too many tiny dimensions: junk dimensions

Say your orders have a handful of small yes/no or short-code attributes: is it gift-wrapped, what payment method was used, was it a rush order. Giving each one its own dimension table is technically correct and practically annoying — five tiny tables, five extra keys cluttering the fact table.

A junk dimension bundles them into one table, with one row per real combination that shows up in your data:

-- dim_order_attributes (junk dimension)
attributes_key | is_gift_wrapped | payment_method | is_rush_order
1              | true             | credit_card    | false
2              | false            | paypal         | true
3              | true             | credit_card    | true

The fact table holds one foreign key instead of three or four. It's not elegant in the textbook sense — it's a deliberate tradeoff of purity for a cleaner schema.

Metadata that doesn't need a dimension: degenerate dimensions

Every order has an order number. It's not really "context" the way a customer or product is — it doesn't have its own attributes, it's just an identifier from the source system. Rather than build a dim_order table with one column in it, you store the order number directly in the fact table:

-- fact_sales
order_line_id | order_number | date_key | customer_key | revenue
9001          | ORD-58291    | 20250915 | 55           | 39.98

That's a degenerate dimension — a dimension-like attribute that lives in the fact table because building a separate table for it would be pure overhead.

When one fact needs many dimension values: bridge tables

Here's the many-to-many problem from the start of this post. A deal in your CRM can have three sales reps sharing credit for it. A fact table row expects one key per dimension, so fact_deals can't just hold three rep_key columns without breaking the model the moment a fourth rep gets added.

A bridge table sits between the fact and the dimension, and carries a weighting factor so the numbers don't double- or triple-count:

-- bridge_deal_reps
deal_id | rep_key | credit_weight
5001    | 12      | 0.5
5001    | 19      | 0.3
5001    | 24      | 0.2

Multiply revenue by credit_weight when reporting by rep, and the totals still add up correctly across the team even though three people are attached to one deal. This same pattern handles patients with multiple diagnoses, students with multiple majors — anywhere the real world refuses to be one-to-one.

Keeping fact tables speaking the same language: conformed dimensions

Once you have more than one fact table — fact_sales and fact_support_tickets, say — someone will eventually ask "which customers bought furniture and also filed a support ticket last quarter?" That question only works cleanly if both fact tables point to the same dim_customer and dim_date tables, with the same keys and the same definitions.

That shared table is a conformed dimension. It's less a technique and more a rule: build dim_date once, build dim_customer once, and make every fact table in the warehouse reference those same tables instead of each team quietly building their own slightly-different version. Skip this, and you're back to the finance-vs-marketing mismatch problem, just one layer deeper.

One step further: Data Vault, briefly

If you ever land somewhere with heavy audit or compliance requirements — banking, insurance, healthcare — you might run into Data Vault modeling instead of star schemas. It splits everything into three table types: hubs (just the business keys, like customer IDs), links (relationships between hubs), and satellites (the descriptive attributes, each with a timestamp). It's more rigid and more verbose than a star schema, but it makes "what did we know, and when did we know it" fully traceable. You probably won't build one as an early-career engineer, but knowing the name and the shape means you won't be lost the first time you see one in a legacy system.

The actual lesson here

None of these are things to use everywhere, all the time. A junk dimension on a table with one boolean column is overkill. A bridge table where the relationship really is one-to-one is overkill in the other direction. The skill isn't memorizing all seven patterns — it's recognizing which real-world shape you're looking at (a snapshot, a many-to-many, an event with no number attached) and reaching for the model built for that shape instead of forcing everything into the one you learned first.

Which one of these have you had to reach for on the job, and what broke that made you go looking for it?

Solving the Classic SQL "Gaps and Islands" Problem: 3 Modern Approaches

Rahman — Tue, 15 Sep 2026 03:15:39 +0000

If you've ever needed to find consecutive streaks in data — days a user logged in back to back, uninterrupted stretches of sensor readings, runs of matching status codes — you've run into the "gaps and islands" problem. The "islands" are the consecutive runs. The "gaps" are the breaks between them. SQL doesn't have a built-in FIND_STREAKS() function, so you build it with window functions instead.

Here's the sample data we'll use throughout — a table of user login dates:

user_id	login_date
1	2026-01-01
1	2026-01-02
1	2026-01-03
1	2026-01-05
1	2026-01-06

User 1 logged in three days straight, skipped the 4th, then logged in two more days. That's two islands — a 3-day streak and a 2-day streak — separated by one gap. Every approach below should output exactly that:

user_id	streak_start	streak_end	streak_length
1	2026-01-01	2026-01-03	3
1	2026-01-05	2026-01-06	2

Quick refresher: the window functions doing the work

Before the approaches, three building blocks, briefly:

ROW_NUMBER() OVER (PARTITION BY ... ORDER BY ...) — numbers rows 1, 2, 3... within each group, in the order you specify. Doesn't skip numbers, doesn't care about ties.
LAG(col) OVER (...) — looks at the previous row's value for that column, for the current row. LEAD() is the same idea but looks forward instead.
SUM(col) OVER (PARTITION BY ... ORDER BY ...) — a running total, recalculated row by row, instead of collapsing everything into one number the way a normal SUM() in a GROUP BY would.

All three approaches below are really just different combinations of these three ideas.

Approach 1: The row-number trick

This is the classic, and the cleverest one to understand. Here's the logic: if a run of dates has zero gaps, then subtracting a simple counter (1, 2, 3...) from each date should always land on the same result — because both the dates and the counter are increasing by exactly one each row. The moment there's a gap, that arithmetic breaks, and the result shifts to a new value. That shift is exactly what makes it usable as a group ID — rows in the same island always land on the same computed value, and rows in a different island always land on a different one.

WITH numbered AS (
  SELECT
    user_id,
    login_date,
    ROW_NUMBER() OVER (PARTITION BY user_id ORDER BY login_date) AS rn
  FROM login_days
)
SELECT
  user_id,
  MIN(login_date) AS streak_start,
  MAX(login_date) AS streak_end,
  COUNT(*) AS streak_length
FROM numbered
GROUP BY user_id, login_date - (rn * INTERVAL '1 day')
ORDER BY user_id, streak_start;

Walking through it: rn gives each row a position (1, 2, 3, 4, 5). Subtracting rn days from login_date gives Jan 1, Jan 1, Jan 1, Jan 2, Jan 2 — the first three rows collapse to the same value because there's no gap between them, and the last two collapse to a different shared value because the gap on Jan 4 threw the counter out of sync. Group by that computed value, and each group is one island.

It's elegant once it clicks, but it only works cleanly for evenly-spaced values like whole days or plain integers. Try it on timestamps or irregular intervals and the subtraction stops producing clean matches.

Approach 2: Flag-then-sum

More flexible, and easier to explain to a teammate six months from now, because every step is visible instead of relying on one clever subtraction. First, compare each row to the one before it using LAG(), and flag it with a 1 whenever there's a gap — meaning this row starts a brand new island. Then run a cumulative SUM() over that flag. Since the flag only increases when a new island starts, that running total is the island number — it stays flat across a streak and steps up by one every time a new streak begins.

WITH flagged AS (
  SELECT
    user_id,
    login_date,
    CASE
      WHEN login_date - LAG(login_date) OVER (PARTITION BY user_id ORDER BY login_date) = 1
      THEN 0 ELSE 1
    END AS is_new_streak
  FROM login_days
)
SELECT
  user_id,
  MIN(login_date) AS streak_start,
  MAX(login_date) AS streak_end,
  COUNT(*) AS streak_length
FROM (
  SELECT *, SUM(is_new_streak) OVER (PARTITION BY user_id ORDER BY login_date) AS streak_id
  FROM flagged
) grouped
GROUP BY user_id, streak_id
ORDER BY user_id, streak_start;

Notice the is_new_streak flag is a single CASE expression you fully control. Right now it checks "is the gap more than 1 day," but you could just as easily change the condition to "is the gap more than 1 hour," or "did the status code change," and the rest of the query wouldn't need to change at all. That's what makes this the one worth reaching for by default.

Approach 3: Match the boundaries directly

No grouping, no aggregation step — just find every row where a streak starts (the day before it is missing from the data) and every row where a streak ends (the day after it is missing), then line up the Nth start with the Nth end.

WITH boundaries AS (
  SELECT
    user_id,
    login_date,
    CASE WHEN LAG(login_date) OVER (PARTITION BY user_id ORDER BY login_date) = login_date - 1 THEN 0 ELSE 1 END AS is_start,
    CASE WHEN LEAD(login_date) OVER (PARTITION BY user_id ORDER BY login_date) = login_date + 1 THEN 0 ELSE 1 END AS is_end
  FROM login_days
),
starts AS (
  SELECT user_id, login_date AS streak_start,
         ROW_NUMBER() OVER (PARTITION BY user_id ORDER BY login_date) AS rn
  FROM boundaries WHERE is_start = 1
),
ends AS (
  SELECT user_id, login_date AS streak_end,
         ROW_NUMBER() OVER (PARTITION BY user_id ORDER BY login_date) AS rn
  FROM boundaries WHERE is_end = 1
)
SELECT s.user_id, s.streak_start, e.streak_end
FROM starts s
JOIN ends e ON s.user_id = e.user_id AND s.rn = e.rn
ORDER BY s.user_id, s.streak_start;

is_start checks whether the previous row is one day back — if not (or if there is no previous row), this row is the beginning of an island. is_end does the mirror check looking forward. Filtering down to just the start rows and just the end rows gives two short lists — the first start pairs with the first end, the second with the second, and so on, because islands can't overlap or interleave.

This skips the group-by step entirely, which can matter on very large tables where aggregation is the expensive part of the query plan. The tradeoff is it takes a moment longer to read for anyone seeing it for the first time — there's more setup before the payoff.

Which one to reach for

Approach 2 is the safest default — readable, flexible, handles unusual gap definitions without a rewrite, and reads almost like plain English once you know what LAG and the running SUM() are doing. Reach for Approach 1 when your data is simple integers or daily dates and you want the shortest possible query, and you're confident nobody will need to change the gap logic later. Reach for Approach 3 when you're working at a scale where skipping the aggregation step actually shows up in your query time — profile first, don't guess.

All three lean on the same two window-function ideas: comparing a row to its neighbor, and turning that comparison into a group. Once that clicks, gaps and islands stops being a "classic hard problem" and starts being a five-minute pattern you reach for on autopilot.

Nobody Warns You the Data Engineering Interview Isn't About Data Engineering

Rahman — Mon, 14 Sep 2026 03:59:31 +0000

I've sat on both sides of the interview table more times than I can count. Here's the thing nobody tells you going in: the interview almost never tests what it claims to.

You think it's testing SQL. It's testing whether you panic.

The whiteboard SQL problem

Somewhere along the way, "write a window function while a stranger watches" became a proxy for "will you be good at this job." It isn't. I've worked with people who nailed a ROW_NUMBER() OVER (PARTITION BY...) from memory, then shipped a pipeline that silently dropped 4% of records because nobody asked what "duplicate" meant to the business. I've also watched someone fumble join syntax live, apologize twice, and turn out to be exactly who you want paged at 2am when a DAG falls over.

The instinct that predicts job performance isn't recall. It's asking "what happens when this data is wrong, and who notices first?" Almost nobody tests for that directly.

What actually separates candidates

Two things, from what I've seen over and over:

Whether they narrate their reasoning instead of going silent and typing
Whether they ask a clarifying question before writing a line of code

That's most of the signal. Not cleverness — just, do they think like someone maintaining this in six months, or someone racing to produce an answer?

I once watched a candidate pick a "wrong" schema — normalized where I'd have gone semi-denormalized — but she named the exact tradeoff and the metric she'd watch to know if she'd guessed wrong. We hired her. The "correct" answer was never really the point.

Where I contradict myself

And yet I still ask SQL questions. Still make people code live, which everyone hates, me included. Because there's a floor you can't skip — if someone can't reason about a GROUP BY at all, no amount of narrated thinking saves that. The theater is annoying. It just isn't pure noise.

If you're early career

Stop sounding like you already know everything. Experienced interviewers can smell a memorized answer — the explanation stops adapting the moment someone pushes on it.

Instead: say "let me think out loud" and actually do it. Ask about data volume before picking an approach. Admit a design's weakness instead of defending it like it's your firstborn. None of that takes more study hours. It takes being a little less afraid of looking uncertain for forty-five minutes.

The part I keep coming back to

The interview is a lossy proxy. Someone great freezes and gets filtered out; someone mediocre gets through on a rehearsed question. That's not a scandal, it's just what happens compressing "will you be good at this for two years" into forty-five minutes and a shared screen.

So prep less syntax you'll forget by Thursday. Prep the muscle of thinking out loud when you don't know the answer yet. That's the actual job.

Recursive CTEs: How SQL Secretly Learned to Loop

Rahman — Sun, 13 Sep 2026 02:06:19 +0000

Ask a SQL query to find "all employees under this manager," and things get ugly fast if you don't know how many levels deep the org chart goes. A regular join handles one level. Two joins handle two levels. Nobody's writing seven joins for seven levels of middle management.

This is exactly the problem recursive CTEs exist to solve — and most people who've written SQL for years have never touched one.

The setup: data that references itself

Picture a plain employees table where each row points to its own manager:

CREATE TABLE employees (
  id INT PRIMARY KEY,
  name TEXT,
  manager_id INT REFERENCES employees(id)
);

Simple structure. The pain shows up the moment you ask "give me this manager and everyone below them, however many layers down." A JOIN can walk exactly one level of that relationship. It has no concept of "keep going until there's nothing left."

The recursive CTE

A recursive CTE is a query that references itself, built from two parts glued together with UNION ALL:

WITH RECURSIVE org_chart AS (
  -- Anchor member: where the recursion starts
  SELECT id, name, manager_id, 1 AS depth
  FROM employees
  WHERE id = 3  -- the manager we're starting from

  UNION ALL

  -- Recursive member: joins back to the CTE itself
  SELECT e.id, e.name, e.manager_id, oc.depth + 1
  FROM employees e
  JOIN org_chart oc ON e.manager_id = oc.id
)
SELECT * FROM org_chart;

Here's what's actually happening:

The anchor member runs once and produces the starting row(s) — in this case, the one manager we care about.
The recursive member joins the real table back to the CTE's own results so far, pulling in the next level down.
The database repeats step 2 — feeding each round's output back in as input — until a round produces zero new rows, then stops.

That depth column isn't required, but it's worth keeping. It turns "an unordered pile of employees" into something you can actually order and indent to look like a real org chart.

A second use case: category trees

Org charts are the classic example, but this pattern shows up anywhere data nests: product categories, comment threads, folder structures, bill-of-materials breakdowns. Same shape, same fix:

WITH RECURSIVE category_tree AS (
  SELECT id, name, parent_id, name::TEXT AS path
  FROM categories
  WHERE parent_id IS NULL

  UNION ALL

  SELECT c.id, c.name, c.parent_id, ct.path || ' > ' || c.name
  FROM categories c
  JOIN category_tree ct ON c.parent_id = ct.id
)
SELECT * FROM category_tree;

That path column builds a breadcrumb trail (Electronics > Laptops > Gaming) for free, just by concatenating as it recurses.

Where this bites people

UNION ALL, not UNION. Swap in plain UNION and the database now has to deduplicate every intermediate round against every other round, which is expensive and usually unnecessary — you already know these are distinct rows by construction.

Cyclic data causes infinite loops. If your hierarchy has a cycle (employee A manages B, B somehow manages A — yes, this happens with bad data), a naive recursive CTE will spin forever or until it hits an engine-specific safety limit. Guard against it by tracking visited IDs in an array and checking before each recursive step, or add a hard depth cap:

WHERE depth < 50

Engine support varies. PostgreSQL, SQL Server, and MySQL 8.0+ all support WITH RECURSIVE. Older MySQL versions (pre-8.0) don't support it at all — you'd be stuck with application-level recursion or a different table design.

It's not free at scale. For very deep or very wide hierarchies queried constantly, a recursive CTE recomputes the walk every single time. If read performance matters more than write simplicity, patterns like closure tables or materialized paths trade some insert/update complexity for much faster reads.

The takeaway

Most SQL work never needs this. But the moment you're modeling anything that nests — org charts, categories, threads, parts lists — a recursive CTE replaces what would otherwise be a loop in application code, a stack of unknown-depth joins, or a recursive function call. It's one of the few places SQL quietly does something genuinely clever.

Window Functions Are the SQL Superpower Nobody Taught You

Rahman — Sat, 12 Sep 2026 05:30:45 +0000

For a long time, I avoided window functions. They looked complicated. PARTITION BY, OVER(), weird syntax that didn't look like the rest of SQL. So I did what a lot of people do — solved the same problems with clunky self-joins and subqueries, and just accepted that some queries were going to be long and slow.

Then I actually learned window functions, and it felt like discovering a whole feature had been sitting in the toolbox the entire time, unused.

The short version: OVER() lets you calculate across a group of rows without collapsing them into one. Same rows in, same rows out — just smarter.

Here's what they do, why they matter, and a few uses that come up constantly in real work.

The core idea, in plain words

A normal GROUP BY collapses rows. Ten rows go in, one summary row comes out per group. That's useful, but sometimes you want the opposite — keep every row, and just attach some extra calculated info to each one, pulled from the other rows around it.

That's a window function. Same number of rows in, same number out. Nothing collapses.

SELECT
  employee,
  department,
  salary,
  AVG(salary) OVER (PARTITION BY department) AS dept_avg
FROM employees;

Every employee keeps their own row. But now each row also shows the average salary for their department, calculated across the whole group. No GROUP BY needed, no subquery, no join.

Running totals without the pain

Want a running total of sales, day by day? Without window functions, this usually turns into a self-join or a correlated subquery. Slow. Annoying to read six months later when you have to touch it again.

SELECT
  sale_date,
  amount,
  SUM(amount) OVER (ORDER BY sale_date) AS running_total
FROM sales;

ORDER BY inside the OVER() tells the database to add up everything from the start of the data through the current row. One line. No subquery.

sale_date	amount	running_total
Jan 1	100	100
Jan 2	50	150
Jan 3	75	225

Top N per group, the way it should be done

This is the classic problem: find the top 3 highest-paid employees in each department. Doing this without window functions usually means some ugly combination of subqueries and LIMIT, applied per group, which SQL doesn't naturally support.

With RANK(), it's clean:

SELECT *
FROM (
  SELECT
    employee,
    department,
    salary,
    RANK() OVER (PARTITION BY department ORDER BY salary DESC) AS rank
  FROM employees
) ranked
WHERE rank <= 3;

PARTITION BY splits the data into departments. ORDER BY salary DESC ranks people within each department. Then you just filter down to rank <= 3. That's the whole trick.

Cheat sheet: which ranking function do you actually want?

ROW_NUMBER() — always gives unique numbers, even for ties (1, 2, 3, 4…)
RANK() — ties get the same number, but leaves a gap after (1, 1, 3, 4…)
DENSE_RANK() — ties get the same number, no gap (1, 1, 2, 3…)

Pick based on whether you care about the gap. Most of the time, DENSE_RANK() is what people actually want and RANK() is what they reach for out of habit.

I rewrote a reporting query at a past job that used two nested subqueries and a LIMIT inside a correlated join just to pull the top 5 customers per region. Forty-some lines. The RANK() version above did the same thing in nine.

Comparing a row to the one before it

LAG() grabs a value from a previous row. LEAD() grabs one from a row ahead. Trends, gaps, row-to-row comparisons — this is the pair you reach for.

SELECT
  user_id,
  order_date,
  order_date - LAG(order_date) OVER (
    PARTITION BY user_id ORDER BY order_date
  ) AS days_since_last_order
FROM orders;

This tells you how many days passed between each user's orders — no self-join required. Swap LAG for LEAD and you get the days until the next order instead. Same idea, opposite direction.

The mistake almost everyone makes at first

Forgetting ORDER BY inside OVER() when you actually need it. Without it, functions like SUM() add up the entire partition for every row, not a running total.

-- wrong: every row shows the department's full total, not a running total
SUM(amount) OVER (PARTITION BY department)

-- right: running total within each department, row by row
SUM(amount) OVER (PARTITION BY department ORDER BY sale_date)

Same function, one missing clause, completely different result. This one gets people every time.

Why this is worth learning properly

Window functions replace a whole category of clunky, slow queries with something readable and fast. Running totals, rankings, row-to-row comparisons, moving averages — all of it gets simpler once OVER() stops looking scary. It's one of those SQL features that feels intimidating for about twenty minutes, and then becomes something you reach for every single week.

Further reading:

PostgreSQL: Documentation: 18: 3.5. Window Functions

postgresql.org

The 5-Minute Trick That Landed Me 3 Job Offers

Rahman — Fri, 11 Sep 2026 04:12:01 +0000

I used to prep for interviews the normal way. Read the job post twice. Practice "tell me about yourself" in the mirror. Show up hoping for the best.

Three offers later, I do one thing differently. Five minutes, right before I log in or walk through the door.

The trick

I write down one sentence about the company's biggest problem right now. Not their mission. Not their "values." Their actual problem.

How do I find it? Quick scan. Recent news, a Glassdoor review or two, sometimes just the job post itself. If a company's hiring four people for one role, something's broken or something's growing fast. Either way, that's the problem.

I write the sentence by hand. Something like:

"They're scaling support faster than they can train people, and it's probably showing up as slow response times."

Takes two minutes, maybe three.

Then I turn it into one question. That's the other two minutes.

"I noticed you're growing the support team fast — is that mostly about response time, or are you prepping for a bigger launch?"

Why it works

I ask that question in almost every interview now. Every single time, the room shifts a little. The interviewer leans in. Sometimes they laugh and say "yeah, actually, funny you ask." Sometimes they go quiet for a second because nobody's asked them that all week.

Here's the reason. Everyone else walks in ready to talk about themselves. I walk in already thinking about their problem. That's it. That's the whole trick.

It's not a magic script. It's not a personality hack. It's five minutes of thinking about them instead of me, right before the pressure hits.

Try it

Before your next interview:

Write one sentence about their real problem.
Turn it into one question.
Watch what happens.

If You Understand These 5 AI Terms, You're Ahead of 90% of People

Rahman — Thu, 10 Sep 2026 05:22:50 +0000

A few months back I sat in a meeting where someone said, dead serious, "we should fine-tune the RAG." Nobody blinked. Everyone just nodded, the way you nod when a doctor says a Latin word and you'd rather die than ask what it means.

Thing is, that sentence didn't mean anything. Fine-tuning and RAG aren't even the same species of tool. Nobody in the room knew that. Including, I'm pretty sure, the guy who said it.

That's where we're at with AI right now. Everyone's using the words. Almost nobody knows what's underneath them. So here are five, told as stories instead of definitions, because that's honestly all these terms are once you scrape the jargon off. Read these and you'll know more than half the people who use them for a living.

1. Context Window — the whiteboard that erases itself

You're explaining something to a friend on a whiteboard. You keep writing. The board's only so big, though — so when you hit the edge, the oldest stuff at the top starts getting wiped to make room for the new.

That's a context window. How much an AI can hold in its head at once. Not forever. Just for now, and only so much of it.

It's why a chatbot feels razor-sharp for the first ten minutes of a conversation and then seems to forget something you told it three messages ago. It's not being lazy or dumb. The board filled up. The beginning got erased so the end had somewhere to go.

2. RAG — the open-book exam

Two students, same test. One memorized the whole textbook, cover to cover. The other memorized nothing — but she's allowed to bring the book in and flip to the right page.

That second student is doing RAG — Retrieval-Augmented Generation, if you want the full name, though honestly nobody says it out loud. Instead of trying to remember everything, the AI goes and grabs the actual document — a manual, your company's files, this morning's news — and reads the relevant part before it answers.

Why this matters: it's the difference between a machine that guesses and one that checks. Closed-book AI would rather invent an answer than admit it doesn't know. Open-book AI can point at the page and say, here, this is where I got it.

3. Fine-Tuning — the apprentice learning one craft

Picture someone who already knows a bit of everything. A jack-of-all-trades. Now you need them to become a master violin maker — specifically violins, nothing else. You don't hand them a new book. You put a violin in their hands and have them build it, over and over, until the way they work has actually changed.

That's fine-tuning. Not new facts. New habits. Tone, style, the specific way it handles one job, reshaped through repetition.

People mix this up with the open-book exam constantly, and it costs real money when they do. Problem is "it doesn't know something"? Hand it the book. Problem is "it knows the facts but talks wrong for the job every single time"? That's a habit, not a knowledge gap — retrain it. Two different problems. Two different fixes. Mixing them up is expensive.

4. Hallucination — the confident stranger giving directions

You know this person. You ask for directions and they answer instantly — full confidence, pointing down the street, no hesitation — and they are completely, catastrophically wrong. They didn't lie. They just couldn't stand the silence of "I don't know," so their brain filled it with something that sounded plausible instead.

That's a hallucination. An AI stating something false with the exact same tone of voice as something true. Not a glitch. Not a typo. It has no built-in sense of doubt, so when the real answer isn't there, it manufactures one that fits.

Carry this one with you: confidence is not proof. Ever. The wrong answer and the right answer come out of the machine sounding identically sure of themselves.

5. Agent — the assistant who finally leaves the office

Most AI you've touched is like a smart person stuck behind a glass window. Talks to you. Answers questions. Hands you a note through the slot. Can't actually get up and go do the thing.

An agent walks out from behind the glass. It doesn't just answer — it acts, checks what happened, decides what's next, on its own, in a loop. Book the flight. Check the confirmation. Notice the date's wrong. Fix it. Try again. Nobody's holding its hand through each step.

That loop is the whole appeal. It's also the scary part, and I don't think enough people say this out loud: a glass-window AI can only hand you a bad sentence. One that's out in the world can take a bad action first — send the email, book the wrong day, delete the wrong file — and you find out after.

So what

None of this is hard. That's kind of the unsettling part, if I'm honest. The gap between sounds like an expert and actually is one turns out to be five plain stories wide.

Next time someone in a meeting says a sentence that doesn't quite add up — you won't nod along. You'll know exactly which word they got wrong.

Why Learning Data Engineering Is Still a Smart Bet in 2026 (Even With AI Eating the Easy Parts)

Rahman — Thu, 10 Sep 2026 04:05:46 +0000

A friend messaged me last month, half-panicking: "Isn't AI going to write all the ETL pipelines soon? Why learn this now?" Fair question. Short answer: AI already ate the boilerplate. That's exactly why the people left standing are getting paid more, not less.

The market isn't shrinking. The entry point is

Hiring for data engineering grew double digits year over year going into 2026, and the global market for data engineering services is sitting north of $100 billion, headed toward roughly double that by the early 2030s. That's infrastructure money, not a niche corner of tech.

What actually changed is the junior on-ramp. Entry-level postings dropped sharply — AI now handles the staging SQL, the scaffolded Airflow DAGs, the repetitive schema mapping that used to be someone's first job.

What didn't disappear: the person who can debug why a pipeline silently dropped 4% of records overnight, or decide whether a workload belongs in Snowflake or a lakehouse. AI raised the bar on data quality and governance. Someone still has to own that.

The pay reflects it

Numbers vary by survey — normal for comp data — but the pattern holds. Mid-level engineers generally clear $120K to $150K base. Senior engineers regularly land north of $170K, with total comp at larger companies well past $200K. Even remote roles aren't cheap anymore; remote mid-level pay competes with plenty of on-site jobs outside the big hubs.

I'm not saying chase the number. I'm saying the market's voting with its budget, and it's voting for this skill set.

What's actually worth learning

Not everything under "data engineering" carries equal weight right now:

SQL, deeply — window functions, query plan reading, knowing why an index isn't getting used.
Streaming, not just batch — Kafka or Flink is the biggest differentiator between candidates today.
Orchestration and modeling — Airflow plus dbt is close to a default stack.
One cloud platform, deeply — pick AWS, GCP, or Azure and go past the surface.

Notice what's missing: knowing every tool. Nobody's hiring the longest skills section. They're hiring whoever can explain a tradeoff.

If you're starting from zero

Don't try to start in data engineering. Start adjacent — backend dev, data analysis, DevOps — and transfer in once you've got something real to point to: a pipeline you built, a migration you owned. The field has quietly stopped being a realistic first job, and pretending otherwise just gets you filtered out.

My actual take

The panic about AI replacing data engineers has it backwards. AI didn't remove the need for the skill — it removed the need for the shallow version of it. What's left is demand for people who understand systems, not syntax. Harder bar. Exactly why it's worth clearing.

The easy jobs are gone. The real ones pay better than they ever did.