DEV Community: Gowtham Potureddi

SQL MERGE / UPSERT / ON CONFLICT: Dialect-Aware Idempotent Writes

Gowtham Potureddi — Sun, 12 Jul 2026 12:12:53 +0000

sql merge upsert is the primitive every incremental load pipeline, every CDC apply-loop, every API upsert endpoint, and every dbt incremental model eventually reaches for — and it is the primitive most engineers ship with a race condition on day one because the honest "insert-if-new-else-update" contract is deceptively subtle and the dialect story is genuinely fragmented across eight engines that all spell the same idea differently. Every backend intern types SELECT ... IF NOT EXISTS THEN INSERT ELSE UPDATE in week one and only discovers the race between the SELECT and the INSERT when two concurrent workers produce a primary-key violation at 3 a.m. This guide is the honest, dialect-aware tour of what actually happens when you ask an engine to atomically merge new rows into an existing table without losing writes and without producing duplicates.

The tour walks the eight engines you have to keep straight in 2026 — the ANSI SQL:2003 MERGE INTO ... USING ... ON ... WHEN MATCHED / WHEN NOT MATCHED / WHEN NOT MATCHED BY SOURCE grammar and its three row-level branches, the OLTP-flavoured INSERT ... ON CONFLICT (col) DO UPDATE SET col = EXCLUDED.col (Postgres, SQLite) and INSERT ... ON DUPLICATE KEY UPDATE ... = new.col (MySQL) native UPSERTs that guarantee atomicity via unique-index detection instead of a two-statement dance, the warehouse-flavoured MERGE INTO t USING s ON ... on Snowflake, BigQuery, and Databricks with their per-engine storage-rewrite models (micro-partition rewrites on Snowflake, partition rewrites on BigQuery, Delta file rewrites on Databricks), the notorious SQL Server MERGE cardinality bug and the safer alternatives senior engineers use in its place, and the anti-patterns that turn any of the above into a lost-update incident under concurrency. Every section ships a teaching block followed by a Solution-Tail interview answer — code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works — so you leave with the two-line skeleton and the reason it wins.

When you want hands-on reps immediately after reading, drill the SQL optimization practice library → for merge-cost tuning, rehearse the SQL join drill room → since MERGE is a specialised join, and sharpen the broader SQL practice surface → that covers 450+ DE-focused problems on incremental loads, idempotency, and upsert patterns.

On this page

Why idempotent writes matter in 2026
SQL Standard MERGE anatomy
Postgres ON CONFLICT + MySQL ON DUPLICATE KEY + SQLite UPSERT
Snowflake + BigQuery + Databricks MERGE
Anti-patterns and dialect matrix
Cheat sheet — MERGE / UPSERT recipe list
Frequently asked questions
Practice on PipeCode

1. Why idempotent writes matter in 2026

The `sql merge upsert` mental model — idempotency as a contract, the two loading anti-patterns, and where MERGE shows up in every real DE pipeline

The one-sentence invariant: an idempotent write is a statement that, when replayed any number of times against the same target state and the same input, produces the same final state — no duplicates, no lost updates, no partial writes — and MERGE / UPSERT / ON CONFLICT are the SQL primitives that let you express this contract atomically in a single round-trip instead of a two-statement dance that races under concurrency. Every pipeline eventually retries; only pipelines built on idempotent primitives survive that retry without corruption.

Where MERGE / UPSERT actually shows up in production.

dbt incremental models. Every materialized='incremental' model with unique_key='id' compiles to a MERGE on Snowflake / BigQuery / Databricks and to an INSERT ... ON CONFLICT ... DO UPDATE on Postgres. The unique_key config is the ON condition; the entire dbt incremental pattern is one big MERGE.
CDC apply-loops. Debezium streams inserts, updates, and deletes from Postgres → Kafka → Snowflake. The apply job reads a batch of change events and MERGEs them into the target table. Without MERGE, you'd have to run separate INSERT / UPDATE / DELETE statements per row-op and risk partial application.
Event ingestion. IoT sensor readings, product analytics events, ML feature updates — all arrive as append-only streams that need to upsert by (entity_id, event_ts) when late-arriving events overwrite earlier snapshots.
API upsert endpoints. REST PUT /users/:id is semantically an upsert — create if new, update if existing. Backend code that does SELECT ... IF NOT EXISTS INSERT ELSE UPDATE has a race window; the correct implementation is a single INSERT ... ON CONFLICT ... DO UPDATE.
Dimension SCD updates. Slowly Changing Dimension Type 1 (overwrite) is a simple MERGE. SCD Type 2 (history-preserving) is a MERGE with an expiration-date trick — insert a new row, close the old one via WHEN MATCHED THEN UPDATE.
Fact-table backfills. Reload the last 24 h of clickstream from source. MERGE by (event_id) so re-runs are idempotent even when the same event flows through the pipeline twice.
Change data feeds. Delta Lake CHANGES feed, Snowflake Streams — both consume via MERGE downstream, since the produced feed contains inserts and updates.

The two loading anti-patterns.

Anti-pattern 1 — SELECT-then-INSERT / UPDATE. The junior instinct: SELECT ... IF NOT EXISTS INSERT ... ELSE UPDATE .... This has a race window between the SELECT and the write — two concurrent workers can both see the row as missing, both attempt the INSERT, and one gets a primary-key violation. On some engines, the race is even more subtle: one INSERT wins, the other returns "success" but silently drops the update semantics. Every senior review flags this the moment it appears.
Anti-pattern 2 — blind INSERT with error swallowing. INSERT ...; IF error THEN UPDATE. This works but leaves an audit-log stink (rejected INSERT logs on every duplicate), thrashes the write-ahead log, and fails on some engines that don't cleanly recover from constraint violations mid-transaction.
Anti-pattern 3 — DELETE + INSERT. The "just delete the row and re-insert" pattern. Simple, but breaks referential integrity (child tables with foreign keys), triggers on both DELETE and INSERT (double effects), and violates the "no gaps in AUTO_INCREMENT" invariant many downstream systems rely on.
Anti-pattern 4 — application-side transactions with pessimistic locks. BEGIN; SELECT FOR UPDATE; if row exists UPDATE else INSERT; COMMIT. Works, but takes an exclusive row-lock that serializes writers and creates a hotspot. Fine for low volume; disastrous at 10 k QPS.

Idempotency as a contract in five bullets.

Same input → same final state. Regardless of how many times you replay the same source batch, the target ends up in the same state. run 5 times = run 1 time.
No visible intermediate states. During the MERGE, the target is either fully-old or fully-new to any concurrent reader. Isolation-level dependent, but MERGE is atomic per-row on most engines.
No lost updates under retry. If a worker crashes mid-batch and the batch is re-driven, no rows are lost.
No duplicates under retry. No rows appear twice; the unique-index / MERGE ON condition guarantees this.
Deterministic under concurrency. Two concurrent MERGEs of the same source batch produce identical target states (though they may serialize / block each other during execution).

What senior interviewers actually probe when they open a MERGE question.

Do you know the difference between MERGE and UPSERT? MERGE is the ANSI SQL statement with three branches (MATCHED / NOT MATCHED [BY TARGET | BY SOURCE]). UPSERT is the informal term for insert-or-update; on Postgres / MySQL / SQLite it's typically spelled INSERT ... ON CONFLICT or ON DUPLICATE KEY UPDATE. MERGE is a superset; UPSERT is the common two-branch case.
Do you know the race condition in a naive UPSERT? SELECT then INSERT has a TOCTOU (time-of-check to time-of-use) race. Fix is atomic UPSERT.
Do you know the SQL Server MERGE bug? SQL Server's MERGE has a known cardinality bug where multiple source rows matching the same target row cause silent data loss. Microsoft's own docs recommend workaround patterns.
Do you know the storage cost per engine? Snowflake MERGE rewrites micro-partitions (cost per partition rewritten). BigQuery MERGE rewrites partitions (cost per partition rewritten + slot_ms). Delta MERGE rewrites Parquet files (cost per file rewritten + Photon acceleration if enabled).
Do you know how MERGE composes with SCD? SCD Type 1 = MERGE with WHEN MATCHED THEN UPDATE. SCD Type 2 = MERGE with WHEN MATCHED THEN UPDATE ... on the expiration row plus a fresh INSERT for the new version. Snowflake's MERGE ... WHEN MATCHED AND cond THEN INSERT is the modern spelling.
Do you know the race-safe API upsert pattern? REST PUT /users/:id handler that does a single INSERT ... ON CONFLICT (id) DO UPDATE SET name = EXCLUDED.name RETURNING id. Atomic, idempotent, no application-side check.

Worked example — the race condition in the naive SELECT-then-INSERT pattern

Detailed explanation. Two backend workers both receive a PUT /users/42 request at the same millisecond. Both check "does user 42 exist?" — both get NO, because the row doesn't exist yet. Both attempt the INSERT. Under READ COMMITTED isolation, one wins and the other gets a UNIQUE-constraint violation. Under snapshot isolation, both may commit and one silently loses.

Question. Given the endpoint below, describe the race window and show the fixed atomic version.

Input.

# The naive PUT handler
def put_user(user_id: int, name: str):
    with db.transaction():
        existing = db.query("SELECT id FROM users WHERE id = %s", (user_id,))
        if existing:
            db.execute("UPDATE users SET name = %s WHERE id = %s", (name, user_id))
        else:
            db.execute("INSERT INTO users (id, name) VALUES (%s, %s)", (user_id, name))

Code.

# The atomic fix — single-statement UPSERT
def put_user(user_id: int, name: str):
    db.execute(
        """
        INSERT INTO users (id, name) VALUES (%s, %s)
        ON CONFLICT (id) DO UPDATE SET name = EXCLUDED.name
        """,
        (user_id, name),
    )

Step-by-step explanation.

Two concurrent workers both call put_user(42, 'alice') and put_user(42, 'bob') at the same time.
Naive path — both open a transaction. Both run SELECT id FROM users WHERE id = 42. Under READ COMMITTED, both see no row (before either INSERT commits). Both branches choose INSERT. Both attempt INSERT INTO users (id, name) VALUES (42, ...). The one that commits first wins; the other hits a UNIQUE-constraint violation on the primary key.
Result — one worker's request fails with a database error. The client sees a 500 and retries; the row is now visible; the retry does UPDATE. Semantically correct, but with a spurious error and log noise.
Atomic path — a single INSERT ... ON CONFLICT (id) DO UPDATE statement. Postgres detects the unique-index conflict inside the write, atomically routes to UPDATE, and commits. No race window, no error, no retry.
Rule — every UPSERT must be a single statement. Never two.

Output.

Worker	Naive (race)	Atomic (fix)
Worker A	INSERT (42, 'alice')	INSERT ... ON CONFLICT (42, 'alice') → INSERT wins
Worker B	INSERT (42, 'bob') → UNIQUE violation	INSERT ... ON CONFLICT (42, 'bob') → UPDATE branch
Client A sees	201 Created (alice wins first)	201 Created
Client B sees	500 Internal Server Error	200 OK (bob overrides alice)
Final row	(42, 'alice' or 'bob') non-deterministic	(42, 'alice' or 'bob') non-deterministic but idempotent

Rule of thumb. Never write SELECT then INSERT. Every backend UPSERT is one statement. Snapshot the source, ship a single INSERT ... ON CONFLICT (Postgres), ON DUPLICATE KEY UPDATE (MySQL), or MERGE INTO ... WHEN NOT MATCHED / MATCHED (warehouses).

Worked example — idempotent replay of a Kafka batch

Detailed explanation. A common CDC pattern: consume a Kafka topic of user-profile change events, apply to a target table. If the consumer crashes mid-batch and re-drives the same offsets, the same events flow through again. The apply logic must be idempotent — replaying the batch produces the same final state.

Question. Given the Kafka batch below with three user-profile events, show the idempotent MERGE statement that produces the same final state whether the batch runs once or five times.

Input.

[
  {"op": "u", "id": 42, "name": "alice", "updated_at": "2026-07-12T10:00:00Z"},
  {"op": "u", "id": 43, "name": "bob",   "updated_at": "2026-07-12T10:01:00Z"},
  {"op": "u", "id": 42, "name": "alice-updated", "updated_at": "2026-07-12T10:02:00Z"}
]

Code.

-- Stage the batch as a temp table
CREATE TEMP TABLE batch (id INT, name TEXT, updated_at TIMESTAMPTZ);
INSERT INTO batch VALUES
  (42, 'alice',         '2026-07-12 10:00:00Z'),
  (43, 'bob',           '2026-07-12 10:01:00Z'),
  (42, 'alice-updated', '2026-07-12 10:02:00Z');

-- Idempotent MERGE
MERGE INTO users AS u
USING (
  SELECT id, name, updated_at
  FROM (
    SELECT id, name, updated_at,
           ROW_NUMBER() OVER (PARTITION BY id ORDER BY updated_at DESC) AS rn
    FROM batch
  ) x
  WHERE rn = 1
) AS b
ON u.id = b.id
WHEN MATCHED AND b.updated_at > u.updated_at
  THEN UPDATE SET name = b.name, updated_at = b.updated_at
WHEN NOT MATCHED
  THEN INSERT (id, name, updated_at) VALUES (b.id, b.name, b.updated_at);

Step-by-step explanation.

The Kafka batch has two events for id=42 — the later one (10:02) should win. Naive replay would apply both events in order; a re-drive after crash could apply them again but must not corrupt state.
The ROW_NUMBER() OVER (PARTITION BY id ORDER BY updated_at DESC) deduplicates within the batch — for id=42, only the latest event survives. Same batch → same deduped source → same MERGE result.
WHEN MATCHED AND b.updated_at > u.updated_at — guards against out-of-order arrivals. If a stale event arrives after a fresh one, it's not applied.
WHEN NOT MATCHED THEN INSERT — new IDs are inserted.
Replay safety — running the MERGE twice on the same batch is a no-op the second time because b.updated_at > u.updated_at is now false (the first run already updated u.updated_at to the batch's max).

Output.

Run	id=42 name / updated_at	id=43 name / updated_at
Empty target	—	—
After run 1	alice-updated / 10:02	bob / 10:01
After run 2 (replay)	alice-updated / 10:02	bob / 10:01
After run 5 (replay)	alice-updated / 10:02	bob / 10:01

Rule of thumb. Every CDC / event-stream apply-loop needs three properties in the MERGE — batch-level dedup, timestamp guard on WHEN MATCHED, and monotonic updated_at in the source. Together they make replay a no-op.

Worked example — a dbt incremental model that compiles to a MERGE

Detailed explanation. dbt is the most common surface for MERGE in modern DE. materialized='incremental' + unique_key='id' config compiles to an engine-specific MERGE. The template is worth memorising — every incremental model looks like this.

Question. Write a dbt model that incrementally loads recent orders into the target table, using MERGE semantics with id as the merge key.

Input.

-- models/incremental_orders.sql (dbt)
{{
  config(
    materialized='incremental',
    unique_key='id',
    incremental_strategy='merge',
    on_schema_change='append_new_columns'
  )
}}

SELECT id, customer_id, total, created_at, updated_at
FROM {{ source('raw', 'orders') }}
{% if is_incremental() %}
  WHERE updated_at > (SELECT MAX(updated_at) FROM {{ this }})
{% endif %}

Code. (dbt compiles this to different SQL per engine.)

-- Postgres compile output
INSERT INTO analytics.incremental_orders (id, customer_id, total, created_at, updated_at)
SELECT id, customer_id, total, created_at, updated_at
FROM raw.orders
WHERE updated_at > (SELECT MAX(updated_at) FROM analytics.incremental_orders)
ON CONFLICT (id) DO UPDATE SET
  customer_id = EXCLUDED.customer_id,
  total = EXCLUDED.total,
  updated_at = EXCLUDED.updated_at;

-- Snowflake compile output
MERGE INTO analytics.incremental_orders AS t
USING (
  SELECT id, customer_id, total, created_at, updated_at
  FROM raw.orders
  WHERE updated_at > (SELECT MAX(updated_at) FROM analytics.incremental_orders)
) AS s
ON t.id = s.id
WHEN MATCHED THEN UPDATE SET
  customer_id = s.customer_id,
  total = s.total,
  updated_at = s.updated_at
WHEN NOT MATCHED THEN INSERT
  (id, customer_id, total, created_at, updated_at)
  VALUES (s.id, s.customer_id, s.total, s.created_at, s.updated_at);

-- BigQuery compile output
MERGE INTO `project.analytics.incremental_orders` AS t
USING (...) AS s
ON t.id = s.id
WHEN MATCHED THEN UPDATE SET ...
WHEN NOT MATCHED BY TARGET THEN INSERT (...) VALUES (...);

Step-by-step explanation.

is_incremental() — dbt macro that returns true on subsequent runs (after the initial full-refresh). On first run, dbt does CREATE TABLE AS SELECT ....
WHERE updated_at > (SELECT MAX(updated_at) FROM {{ this }}) — the incremental filter. Only pulls source rows newer than the highwater mark on the target.
unique_key='id' — the ON condition for MERGE. dbt generates the appropriate WHEN MATCHED / WHEN NOT MATCHED branches.
on_schema_change='append_new_columns' — dbt handles schema drift by appending new columns to the target when they appear in the source.
Every engine compiles to the same conceptual MERGE, but the syntax differs — Postgres uses INSERT ... ON CONFLICT, warehouses use MERGE INTO. dbt hides the difference behind the same model config.

Output.

Engine	Statement type	Notes
Postgres	`INSERT ... ON CONFLICT`	Atomic UPSERT
Snowflake	`MERGE INTO`	Micro-partition rewrite
BigQuery	`MERGE INTO ... WHEN NOT MATCHED BY TARGET`	Partition rewrite; `--dry_run` first
Databricks	`MERGE INTO` (Delta)	Parquet file rewrite; Photon-accelerated

Rule of thumb. dbt's incremental model is the canonical MERGE surface. Every DE codebase has hundreds of these; learn the compile output on your target engine so you can debug the compiled SQL when performance regresses.

Common beginner mistakes

Writing SELECT then INSERT instead of a single-statement UPSERT.
Forgetting the WHERE updated_at > highwater incremental filter — every run reads the full source.
Forgetting the WHEN MATCHED AND b.updated_at > u.updated_at timestamp guard — out-of-order events overwrite fresh ones.
Confusing MERGE with UPSERT — MERGE is ANSI three-branch; UPSERT is the two-branch native pattern on OLTP engines.
Assuming MERGE is atomic across concurrent writers — most engines still require row-level locking to serialize.
Using MERGE for high-QPS API upserts — MERGE is designed for batch, not for 10 k QPS. Use ON CONFLICT on OLTP engines instead.

`sql merge upsert` interview question on choosing between MERGE and native UPSERT

A senior interviewer often opens with: "You're building the API layer for a two-billion-row users table. Every profile update from the mobile app hits a PUT /users/:id endpoint. Do you use MERGE or INSERT ... ON CONFLICT, and why? Then design the equivalent daily-batch backfill from a Kafka topic into the analytics warehouse — same question there."

Solution Using dual-endpoint architecture — `INSERT ... ON CONFLICT` for OLTP, `MERGE INTO` for warehouse

-- ===================================================================
-- OLTP endpoint: high QPS, single-row UPSERT on Postgres 15
-- ===================================================================
-- API handler: PUT /users/:id
-- Expected QPS: 10,000+
-- Payload: {name, email, updated_at}
INSERT INTO users (id, name, email, updated_at)
VALUES (%s, %s, %s, %s)
ON CONFLICT (id) DO UPDATE SET
  name = EXCLUDED.name,
  email = EXCLUDED.email,
  updated_at = EXCLUDED.updated_at
WHERE users.updated_at < EXCLUDED.updated_at   -- guard against stale writes
RETURNING id, name, email, updated_at;

-- ===================================================================
-- Warehouse backfill: daily batch, ~50M rows, Snowflake
-- ===================================================================
-- 1. Load source stage from S3 → Snowflake table s_users_delta
-- 2. Dedup within the batch, apply MERGE
MERGE INTO analytics.dim_users AS t
USING (
  SELECT id, name, email, updated_at
  FROM (
    SELECT id, name, email, updated_at,
           ROW_NUMBER() OVER (PARTITION BY id ORDER BY updated_at DESC) AS rn
    FROM stage.s_users_delta
    WHERE _kafka_ts BETWEEN :start_ts AND :end_ts
  ) x
  WHERE rn = 1
) AS s
ON t.id = s.id
WHEN MATCHED AND s.updated_at > t.updated_at
  THEN UPDATE SET name = s.name, email = s.email, updated_at = s.updated_at
WHEN NOT MATCHED
  THEN INSERT (id, name, email, updated_at)
       VALUES (s.id, s.name, s.email, s.updated_at);

Step-by-step trace.

Step	OLTP `INSERT ... ON CONFLICT`	Warehouse `MERGE INTO`
1	client sends `PUT /users/42` with new name	daily batch: ~50M rows land in stage
2	single-row statement, atomic on unique index	dedup batch by (id, updated_at DESC) — one row per id
3	Postgres detects conflict → routes to UPDATE branch	Snowflake plans MERGE — scans stage, matches on t.id
4	RETURNING clause gives the client the persisted row	WHEN MATCHED AND updated_at > → UPDATE; WHEN NOT MATCHED → INSERT
5	commit: single-row lock held briefly	micro-partitions rewrite; wall clock ~2 min on Medium WH
6	client sees 200 in ~1-2 ms	orchestrator sees success; audit log has row_count
7	O(log n) per call (unique index seek)	O(source + target_matched) per batch

The OLTP endpoint uses ON CONFLICT because the workload is single-row, high-QPS, and needs atomic conflict detection at the unique index. The warehouse backfill uses MERGE INTO because the workload is batch, tens of millions of rows, and benefits from the source-scan-once execution the MERGE grammar guarantees. Two different primitives, same idempotency contract.

Output:

Endpoint	Cost model	Latency per call/batch	Idempotency guarantee
OLTP `ON CONFLICT`	~50 μs CPU + 1-2 ms wall clock	1-2 ms	Atomic on unique index
Warehouse `MERGE INTO`	~500 credits (Medium WH × 2 min)	~2 min	Deterministic on batch-dedup + timestamp guard

Why this works — concept by concept:

Dual primitive for dual workload — OLTP is high-QPS single-row; native ON CONFLICT is designed for exactly that shape (atomic conflict detection, minimal locking, fast). Warehouse batch is 50 M rows; MERGE INTO scans the source once, plans MATCHED / NOT MATCHED branches, and rewrites storage in a single pass.
WHERE users.updated_at < EXCLUDED.updated_at guard — protects against stale writes overtaking fresh writes when two concurrent clients update the same row with different timestamps. The write with the older updated_at becomes a no-op.
ROW_NUMBER() OVER (PARTITION BY id ORDER BY updated_at DESC) batch dedup — the Kafka source may have 5 events for the same user in one batch. The dedup keeps only the latest per id; the MERGE never sees duplicates in the source, avoiding the SQL Server MERGE cardinality bug (see H2-5) and simplifying the WHEN MATCHED logic.
WHEN MATCHED AND s.updated_at > t.updated_at — out-of-order events across batches are guarded. If a stale batch arrives after a fresh one, MATCHED rows fail the condition and the branch is a no-op.
Cost — OLTP per-call is O(log n) (unique index seek) with a fixed ~1-2 ms overhead. Warehouse per-batch is O(source_scan + target_matched_partitions_rewrite); a 50 M-row batch on Snowflake Medium is roughly 2 minutes and ~500 credits. Total daily cost is bounded by batch cadence.

SQL
Topic — optimization
SQL optimization drills

Practice →

SQL Topic — indexing Unique index and constraint drills

Practice →

2. SQL Standard MERGE anatomy

`sql standard merge` — the ANSI SQL:2003 grammar, the three row-level branches, and how `WHEN NOT MATCHED BY SOURCE` extends the two-branch UPSERT into a three-way sync

The mental model in one line: the ANSI MERGE statement is a single-round-trip three-way sync operator that walks the source table once, joins each source row against the target via the ON condition, and applies exactly one of three actions per row — UPDATE (when a target match exists), INSERT (when the source row has no target match), or DELETE (when a target row has no source match) — turning the merge of two tables into a deterministic, atomic bulk operation. Master the grammar and you can express incremental loads, dimension updates, CDC apply-loops, and full three-way syncs in a single statement per pipeline.

Slot 1 — the six clauses of a canonical MERGE.

MERGE INTO target_table [AS t] — the write destination. Must be a base table (not a view on most engines, though some allow updatable views).
USING source_table [AS s] — the source. Can be a base table, subquery, CTE, or table function. Warehouses often materialise this as a temp table if complex.
ON t.pk = s.pk — the match condition. Determines which target rows are "matched" by each source row. Usually equality on the natural or synthetic primary key; can be composite.
WHEN MATCHED [AND condition] THEN UPDATE SET col = s.col, ... — the UPDATE branch. Optional AND condition scopes which matched rows this branch applies to.
WHEN NOT MATCHED [BY TARGET] [AND condition] THEN INSERT (cols) VALUES (s.cols) — the INSERT branch. Fires when a source row has no target match.
WHEN NOT MATCHED BY SOURCE [AND condition] THEN DELETE / UPDATE — the DELETE branch. Fires when a target row has no source match. Not universally supported; SQL Server and Snowflake support it, others do not.

Slot 2 — the three-way ANSI split.

INSERT branch (WHEN NOT MATCHED [BY TARGET]). New source rows land in the target. Guaranteed exactly once per source row that has no ON match.
UPDATE branch (WHEN MATCHED). Existing target rows are updated in place using values from the joined source row. Can conditionally UPDATE via AND condition or even DELETE the target row in some dialects.
DELETE branch (WHEN NOT MATCHED BY SOURCE). Target rows without a source match are removed. Used for full-refresh syncs where source is the authoritative snapshot.
The two-branch case (INSERT + UPDATE) is UPSERT. No DELETE branch → source can only add or update, never remove. Common for append-heavy pipelines.
The three-branch case is full sync. Source is the authoritative snapshot; target must mirror it. Common for dimension tables or config tables.

Slot 3 — deterministic evaluation.

Source is scanned once. The MERGE reads the source in a single pass. Each source row is joined against the target and routed to one branch. No per-row re-evaluation.
Target is scanned once (via join). For matched rows, the target is read (locked, in some dialects) once during the join.
Each source row triggers exactly one action. UPDATE, INSERT, or (with BY SOURCE) DELETE. The branches are mutually exclusive.
Row-level triggers fire in ANSI order. BEFORE INSERT / BEFORE UPDATE / BEFORE DELETE fire before the row-level action; AFTER equivalents fire after. Some engines have historically buggy trigger fire orders (SQL Server).
Statement-level triggers fire once. Total per-branch action counts are reported to the statement-level trigger — one AFTER MERGE trigger sees all inserts / updates / deletes.

Slot 4 — the ON condition rules.

Best case — equality on a unique key. ON t.id = s.id where t.id has a primary key or unique index. Deterministic, planner uses index seek, cost is O(source + log(target)).
Composite keys work. ON t.tenant_id = s.tenant_id AND t.entity_id = s.entity_id. Use composite unique index on (tenant_id, entity_id).
Non-unique ON conditions are dangerous. If a source row matches multiple target rows, some engines UPDATE all matched rows; others raise an error; SQL Server has the notorious cardinality bug.
Range conditions are allowed but rarely useful. ON t.start_ts <= s.event_ts AND t.end_ts > s.event_ts — a range join. Legal but expensive; often better to precompute the target row via a separate query.
Filter predicates in ON are pushed to the join. Adding AND s.tenant_id = 42 to the ON is equivalent to filtering source before MERGE. Cleaner to filter source in the USING subquery.

Slot 5 — the WHEN MATCHED variants.

WHEN MATCHED THEN UPDATE SET col = s.col — the classic. Updates all matched rows.
WHEN MATCHED AND s.updated_at > t.updated_at THEN UPDATE SET ... — conditional UPDATE. Only updates when the source is fresher. Common for CDC.
WHEN MATCHED AND s.is_deleted THEN DELETE — conditional DELETE via the MATCHED branch. Handy for tombstone events.
WHEN MATCHED THEN DO NOTHING — some dialects support explicit no-op. Rarely useful; equivalent to not writing the branch.
Multiple WHEN MATCHED clauses. Some dialects allow multiple; each row picks the first matching condition. Order matters.

Slot 6 — the WHEN NOT MATCHED variants.

WHEN NOT MATCHED [BY TARGET] THEN INSERT (...) VALUES (...) — the classic. Inserts new rows.
WHEN NOT MATCHED AND s.op = 'i' THEN INSERT — conditional INSERT. Only insert when the source-op is "insert" (CDC pattern).
WHEN NOT MATCHED THEN DO NOTHING — explicit no-op. Rare.
WHEN NOT MATCHED BY SOURCE THEN DELETE — the DELETE branch. Fires for target rows without source matches. Full-sync semantics.
WHEN NOT MATCHED BY SOURCE AND t.status = 'A' THEN UPDATE SET status = 'D' — soft-delete via UPDATE on the BY SOURCE branch. Preserves history.

Slot 7 — the row-level action decision tree.

For each source row s: Find target rows where t.pk = s.pk. Zero or more.
If exactly one target match — WHEN MATCHED branch fires. Optional condition filters. Action: UPDATE or DELETE.
If no target match — WHEN NOT MATCHED [BY TARGET] branch fires. Action: INSERT.
If multiple target matches — depends on dialect. Some UPDATE all (non-deterministic if source values differ per matched row). SQL Server crashes with cardinality error. Some engines allow it only if all matched target rows agree.
After all source rows processed — for each target row NOT matched by any source row, WHEN NOT MATCHED BY SOURCE branch fires (if written).

Slot 8 — the ANSI spec bites.

SQL Server MERGE bugs. Long list of known issues — cardinality errors, trigger fire order, plan-cache instability with parameter sniffing. Microsoft documents workarounds. Interview signal: knowing SQL Server MERGE is the buggy one and preferring the CTE-based UPDATE ... FROM ... for critical paths.
Oracle MERGE. ANSI-compliant but the WHEN NOT MATCHED BY SOURCE branch is missing. Full three-way sync requires a separate DELETE statement.
Postgres MERGE. Added in Postgres 15. Full ANSI compliance for the three branches. Postgres 17 added WHEN NOT MATCHED BY SOURCE. Most Postgres code still uses INSERT ... ON CONFLICT because it predates MERGE.
BigQuery MERGE. ANSI-compliant. WHEN NOT MATCHED BY TARGET is the explicit spelling; the implicit WHEN NOT MATCHED is also accepted. WHEN NOT MATCHED BY SOURCE supported.
Snowflake MERGE. ANSI-compliant with the full three-branch grammar plus optional AND cond on every branch.
Databricks MERGE. ANSI-compliant on Delta tables. Photon-accelerated. WHEN MATCHED AND cond THEN DELETE is idiomatic for tombstones.

Common beginner mistakes

Writing MERGE with a non-unique ON condition — cardinality bug or silent data loss.
Forgetting WHEN NOT MATCHED [BY TARGET] — some engines assume TARGET, some require the explicit spelling.
Writing INSERT ... VALUES (t.id, ...) — you meant s.id. Common typo.
Multiple WHEN MATCHED clauses without understanding the order — first matching condition wins per row.
Using MERGE for high-QPS API upserts — designed for batch. Use ON CONFLICT on OLTP.
Assuming DELETE branch (BY SOURCE) exists on all engines — Oracle and MySQL do not have it.

Worked example — the two-branch MERGE (UPSERT) on Snowflake

Detailed explanation. The most common MERGE pattern: dbt-style incremental UPSERT. Source is a batch of new + updated rows; target is the existing table; INSERT new IDs, UPDATE existing.

Question. Write a Snowflake MERGE that upserts new orders from the stage_orders table into analytics.orders, matching on order_id.

Input.

order_id	customer_id	total	updated_at
100	42	25.00	2026-07-12 09:00
101	43	40.00	2026-07-12 09:10
102	44	15.00	2026-07-12 09:15

Target (analytics.orders) currently has:

order_id	customer_id	total	updated_at
100	42	20.00	2026-07-12 08:00

Code.

MERGE INTO analytics.orders AS t
USING stage_orders AS s
ON t.order_id = s.order_id
WHEN MATCHED THEN UPDATE SET
  customer_id = s.customer_id,
  total = s.total,
  updated_at = s.updated_at
WHEN NOT MATCHED THEN INSERT
  (order_id, customer_id, total, updated_at)
  VALUES (s.order_id, s.customer_id, s.total, s.updated_at);

Step-by-step explanation.

Source scan — Snowflake reads stage_orders once (3 rows).
For each source row, join against target on order_id. 100 matches, 101 and 102 don't.
100 — WHEN MATCHED branch fires. UPDATE the target row's customer_id, total, updated_at to source values.
101 and 102 — WHEN NOT MATCHED branch fires. INSERT them with the source values.
Snowflake reports rows_inserted: 2, rows_updated: 1 in the query result summary.

Output.

order_id	customer_id	total	updated_at
100	42	25.00	2026-07-12 09:00
101	43	40.00	2026-07-12 09:10
102	44	15.00	2026-07-12 09:15

Rule of thumb. Two-branch MERGE (INSERT + UPDATE) covers 80% of dbt incremental use cases. Learn this template cold; the rest of the ANSI grammar layers on top.

Worked example — the three-branch MERGE with DELETE (full sync)

Detailed explanation. Full-sync MERGE: source is the authoritative snapshot; target must exactly mirror it. Rows in target that are no longer in source must be deleted.

Question. Write a Snowflake MERGE that fully syncs the target dim_products from the source stage_products. Insert new products, update changed ones, delete products no longer in source.

Input.

product_id	name	price
1	Widget	10.00
2	Gadget	20.00
4	Sprocket	30.00

Target (dim_products) currently:

product_id	name	price
1	Widget	8.00
2	Gadget	20.00
3	Doohickey	15.00

Code.

MERGE INTO dim_products AS t
USING stage_products AS s
ON t.product_id = s.product_id
WHEN MATCHED AND (t.name <> s.name OR t.price <> s.price) THEN UPDATE SET
  name = s.name,
  price = s.price
WHEN NOT MATCHED BY TARGET THEN INSERT
  (product_id, name, price) VALUES (s.product_id, s.name, s.price)
WHEN NOT MATCHED BY SOURCE THEN DELETE;

Step-by-step explanation.

Source scan — 3 rows: (1, Widget, 10), (2, Gadget, 20), (4, Sprocket, 30).
Target rows: (1, Widget, 8), (2, Gadget, 20), (3, Doohickey, 15).
1 — matched. UPDATE branch fires with condition. (t.name <> s.name OR t.price <> s.price) — name is same, price differs (8 vs 10) → UPDATE. Set price=10.
2 — matched. Condition — same name and same price → UPDATE skipped (no-op).
3 — NOT MATCHED BY SOURCE — DELETE branch fires. Row removed.
4 — NOT MATCHED BY TARGET — INSERT branch fires. Row added.

Output.

product_id	name	price
1	Widget	10.00
2	Gadget	20.00
4	Sprocket	30.00

Rule of thumb. Three-branch MERGE (with BY SOURCE DELETE) is the full-sync pattern for dimension tables and config snapshots. Confirm the DELETE branch is intentional — a bad source query can delete production dim rows.

Worked example — conditional UPDATE using `WHEN MATCHED AND cond`

Detailed explanation. CDC apply-loops often want to update only when the source is newer than the target. The AND clause on WHEN MATCHED expresses this cleanly.

Question. Write a Snowflake MERGE that upserts CDC events into users, only updating when the event's updated_at is newer than the target's updated_at.

Input.

-- CDC batch (source)
CREATE TEMP TABLE cdc_batch AS
SELECT * FROM VALUES
  (42, 'alice-new', TIMESTAMP '2026-07-12 10:00'),
  (43, 'bob-STALE', TIMESTAMP '2026-07-12 08:00')   -- older than target
AS t(id, name, updated_at);

-- Target
CREATE TEMP TABLE users AS
SELECT * FROM VALUES
  (42, 'alice-old', TIMESTAMP '2026-07-12 09:00'),
  (43, 'bob-fresh', TIMESTAMP '2026-07-12 09:30')
AS t(id, name, updated_at);

Code.

MERGE INTO users AS t
USING cdc_batch AS s
ON t.id = s.id
WHEN MATCHED AND s.updated_at > t.updated_at THEN UPDATE SET
  name = s.name,
  updated_at = s.updated_at
WHEN NOT MATCHED THEN INSERT
  (id, name, updated_at) VALUES (s.id, s.name, s.updated_at);

Step-by-step explanation.

Source scan — 2 rows: (42, alice-new, 10:00) and (43, bob-STALE, 08:00).
Target: (42, alice-old, 09:00) and (43, bob-fresh, 09:30).
42 — matched. Condition 10:00 > 09:00 → true. UPDATE fires. name → alice-new, updated_at → 10:00.
43 — matched. Condition 08:00 > 09:30 → false. UPDATE skipped — no-op.
Result — 42 updated, 43 unchanged. Stale event correctly ignored.

Output.

id	name	updated_at
42	alice-new	2026-07-12 10:00
43	bob-fresh	2026-07-12 09:30

Rule of thumb. Every CDC apply-loop needs WHEN MATCHED AND s.updated_at > t.updated_at. Without it, stale events replay corrupt fresh state.

`sql standard merge` interview question on writing a three-branch MERGE

A senior interviewer often asks: "Write a Snowflake MERGE that syncs the stg_customers staging table into analytics.dim_customers — insert new customers, update changed customers, and soft-delete customers no longer in source by setting is_active = FALSE and deactivated_at = CURRENT_TIMESTAMP(). Do not hard-delete — retain history."

Solution Using three-branch MERGE with soft-delete on WHEN NOT MATCHED BY SOURCE

MERGE INTO analytics.dim_customers AS t
USING (
  SELECT
    id,
    name,
    email,
    country,
    _snapshot_ts AS updated_at
  FROM stg_customers
  WHERE _batch_id = :batch_id
) AS s
ON t.id = s.id
WHEN MATCHED AND (
    t.name    IS DISTINCT FROM s.name
 OR t.email   IS DISTINCT FROM s.email
 OR t.country IS DISTINCT FROM s.country
) THEN UPDATE SET
  name = s.name,
  email = s.email,
  country = s.country,
  updated_at = s.updated_at,
  is_active = TRUE,
  deactivated_at = NULL
WHEN NOT MATCHED BY TARGET THEN INSERT
  (id, name, email, country, updated_at, is_active, deactivated_at)
  VALUES (s.id, s.name, s.email, s.country, s.updated_at, TRUE, NULL)
WHEN NOT MATCHED BY SOURCE AND t.is_active = TRUE THEN UPDATE SET
  is_active = FALSE,
  deactivated_at = CURRENT_TIMESTAMP();

Step-by-step trace.

Step	Action	Rows affected
1	Source scan of `stg_customers` filtered by batch_id	~500K rows
2	Join against `dim_customers` on `id`	matched + unmatched sets
3	WHEN MATCHED AND cols DIFFER → UPDATE (name, email, country, updated_at)	~50K changed
4	WHEN NOT MATCHED BY TARGET → INSERT	~5K new
5	WHEN NOT MATCHED BY SOURCE AND is_active → UPDATE (soft-delete)	~1K deactivated
6	Report row_count summary	50K+5K+1K = 56K

Output:

Metric	Value
Rows inserted	~5,000
Rows updated (data change)	~50,000
Rows soft-deleted	~1,000
Rows re-activated (previously inactive, back in source)	included in UPDATE branch
Total wall clock (Snowflake Medium WH, 500K source × 2M target)	~90 seconds

Why this works — concept by concept:

IS DISTINCT FROM — null-safe inequality. NULL <> NULL is NULL (falsy in boolean context); NULL IS DISTINCT FROM NULL is FALSE (correct). Guards against updating rows where a column is null on both sides.
Conditional WHEN MATCHED — only UPDATE if a column actually changed. Skips writes on rows that arrived unchanged, saving micro-partition rewrite cost.
Re-activation via WHEN MATCHED UPDATE — a previously soft-deleted customer that re-appears in source flips is_active = TRUE, deactivated_at = NULL. Handled inside the MATCHED branch alongside the data update.
Soft-delete via WHEN NOT MATCHED BY SOURCE — preserves history. Analysts can still query the customer's fields; downstream reports respect is_active.
AND t.is_active = TRUE guard on the soft-delete branch — prevents re-writing already-deactivated rows on subsequent runs. Deactivation is a one-time transition.
Cost — MERGE reads source once and target once via the join. On Snowflake, cost is dominated by micro-partition rewrites for the matched+changed set. On Medium WH, 500K source × 2M target with 50K UPDATEs + 5K INSERTs + 1K soft-DELETEs is roughly 90 seconds of wall clock and ~10 credits. Batch-level idempotency: replaying the same batch is a no-op because the IS DISTINCT FROM guard filters unchanged rows.

SQL
Topic — joins
SQL join and merge drills

Practice →

SQL Topic — SQL SQL practice library

Practice →

3. Postgres ON CONFLICT + MySQL ON DUPLICATE KEY + SQLite UPSERT

The OLTP-native `sql upsert` — `postgres on conflict`, `mysql on duplicate key update`, and `sqlite upsert` — three spellings of the same atomic conflict-resolution primitive

The mental model in one line: on OLTP engines (Postgres, MySQL, SQLite), the idiomatic UPSERT is not MERGE — it is a special extension of INSERT that says "if the INSERT would violate a unique constraint, resolve the conflict this way instead of raising an error", and the resolution runs atomically inside the write path with no application-side check and no race window. Each engine spells it differently but the semantics are identical: atomic, race-free, single-round-trip.

Slot 1 — Postgres INSERT ... ON CONFLICT ... DO UPDATE / DO NOTHING.

INSERT INTO t (id, name) VALUES (1, 'a') ON CONFLICT (id) DO UPDATE SET name = EXCLUDED.name — the canonical spelling. The (id) is the conflict target — a column list matching a unique index or primary key.
ON CONFLICT ON CONSTRAINT constraint_name DO UPDATE ... — alternate spelling that names the constraint explicitly. Useful when multiple unique indexes exist and you need to specify which one.
ON CONFLICT DO NOTHING — the "insert or skip" pattern. Silently drops rows that would conflict. Common for append-only tables where duplicate detection matters more than the actual duplicate values.
EXCLUDED.col — the pseudo-table representing the row that would have been inserted. Available inside the DO UPDATE clause. SET col = EXCLUDED.col copies the incoming value; SET col = t.col + EXCLUDED.col adds them (increment counter pattern).
WHERE clause on DO UPDATE. ON CONFLICT (id) DO UPDATE SET name = EXCLUDED.name WHERE t.updated_at < EXCLUDED.updated_at. Guards against stale writes overwriting fresh writes. Common for LWW (last-writer-wins) semantics with timestamp guard.
RETURNING — returns the persisted row(s). Distinguishes INSERT vs UPDATE via a RETURNING (xmax = 0) AS was_insert trick.
INSERT ... ON CONFLICT ... is atomic. Detects the conflict inside the write path via the unique index. No race window. No application-side check.

Slot 2 — MySQL INSERT ... ON DUPLICATE KEY UPDATE.

INSERT INTO t (id, name) VALUES (1, 'a') ON DUPLICATE KEY UPDATE name = 'a' — the canonical spelling. Note: no conflict target — any unique-index conflict triggers the DUPLICATE branch.
VALUES(col) pseudo-function — pre-MySQL 8.0.20 spelling. ON DUPLICATE KEY UPDATE name = VALUES(name) — the VALUES(name) returns the incoming value.
new.col row alias (MySQL 8.0.19+). INSERT INTO t (id, name) VALUES (1, 'a') AS new ON DUPLICATE KEY UPDATE name = new.name. Cleaner than VALUES(). Preferred spelling.
No explicit conflict target. MySQL treats any unique-index conflict as a duplicate. If multiple unique indexes exist and different rows conflict on each, behaviour is dialect-specific and often surprising.
ROW_COUNT() semantics. Returns 1 for INSERT, 2 for UPDATE. Handy for detecting which branch fired.
INSERT IGNORE ... — the MySQL equivalent of ON CONFLICT DO NOTHING. Silently drops rows that would conflict. Also drops rows that would fail any insert-time error, which is dangerous — avoid.
REPLACE INTO — do not use. MySQL's REPLACE INTO is a DELETE-then-INSERT. Breaks FK constraints, fires DELETE triggers, and re-numbers AUTO_INCREMENT. Historical wart; use ON DUPLICATE KEY UPDATE instead.

Slot 3 — SQLite INSERT ... ON CONFLICT ... DO UPDATE / DO NOTHING.

INSERT INTO t (id, name) VALUES (1, 'a') ON CONFLICT (id) DO UPDATE SET name = excluded.name — the canonical spelling. Since SQLite 3.24 (2018). Grammar mirrors Postgres, with the excluded pseudo-table being lowercase (Postgres uses uppercase EXCLUDED; both accept either in practice).
Conflict target on unique column or unique index. ON CONFLICT (id) — targets the primary key or unique index on id.
WHERE clause on DO UPDATE. Same as Postgres.
DO NOTHING supported. Same semantics.
INSERT OR REPLACE INTO ... — the legacy SQLite spelling. Also DELETE-then-INSERT, same wart as MySQL's REPLACE.
INSERT OR IGNORE INTO ... — silently skip on conflict. Same as ON CONFLICT DO NOTHING but without a target.

Slot 4 — the EXCLUDED / VALUES / excluded pseudo-tables.

Postgres EXCLUDED — represents the row that would have been inserted. Read-only. Reference as EXCLUDED.col. Available inside DO UPDATE SET ... = EXCLUDED.col.
MySQL VALUES(col) — a function that returns the incoming value for col. Legacy; MySQL 8.0.20 deprecated it in favor of the new.col row alias.
MySQL new.col — the row alias set via VALUES (...) AS new. Preferred in 8.0.19+.
SQLite excluded — same as Postgres. Lowercase, but SQLite is case-insensitive.
Use case — increment counter. INSERT INTO counters (id, n) VALUES (1, 1) ON CONFLICT (id) DO UPDATE SET n = counters.n + EXCLUDED.n. Atomically increments if row exists, initializes to 1 if new.
Use case — LWW with timestamp. INSERT ... ON CONFLICT (id) DO UPDATE SET val = EXCLUDED.val WHERE EXCLUDED.updated_at > t.updated_at. Only updates when the incoming row is newer.

Slot 5 — constraint targeting.

Postgres — must name a target. ON CONFLICT (id) DO UPDATE. If you omit the target, the syntax is ON CONFLICT DO NOTHING only (no UPDATE). This is by design — DO UPDATE needs to know which columns' EXCLUDED values to reference.
Postgres — target must match a unique constraint. The column list must exactly match a unique or primary key constraint. Partial or expression-index conflicts require ON CONFLICT ON CONSTRAINT.
Postgres partial unique index. CREATE UNIQUE INDEX idx ON t (id) WHERE is_active. Conflict target must include the WHERE clause: ON CONFLICT (id) WHERE is_active DO UPDATE SET ....
MySQL — no target needed. Any unique-index conflict fires the DUPLICATE branch. This is simpler but harder to reason about with multiple unique indexes.
SQLite — target optional. Column list preferred but not required. Similar to MySQL in permissiveness.

Slot 6 — atomicity and locking.

Postgres — row-level lock during conflict resolution. The unique-index lookup and subsequent UPDATE happen under an exclusive row lock. Concurrent writers serialize on the same key; different keys go through in parallel.
MySQL — gap-locking implications. InnoDB uses gap locks and next-key locks during conflict detection. Occasionally causes surprise deadlocks under high concurrency; the default is fine for most workloads.
SQLite — full DB lock. SQLite serialises writes at the database level (single writer). Concurrency is limited but consistency is bulletproof.
All three engines — conflict detection inside the write. No two-statement race. The unique-index check happens as part of the INSERT, atomically.

Slot 7 — RETURNING and side effects.

Postgres INSERT ... ON CONFLICT ... RETURNING ... — returns the persisted row (post-INSERT or post-UPDATE). Handy for API layers that need to know the current server-side state.
Postgres RETURNING (xmax = 0) AS was_insert — distinguishes INSERT vs UPDATE. xmax = 0 when the row was inserted (no prior version); otherwise the row was updated.
MySQL — no RETURNING. MariaDB 10.5+ added RETURNING, but MySQL proper does not. Use SELECT ... FROM ... WHERE id = LAST_INSERT_ID() or SELECT ROW_COUNT().
SQLite INSERT ... ON CONFLICT ... RETURNING ... — supported since 3.35 (2021). Same semantics as Postgres.
Trigger fire order. All three engines fire triggers appropriately per branch (BEFORE INSERT + AFTER INSERT for the INSERT branch; BEFORE UPDATE + AFTER UPDATE for the UPDATE branch).

Slot 8 — interview probes on OLTP UPSERT.

"When would you use ON CONFLICT DO NOTHING?" — Append-only tables where dedup matters more than the values. Common for audit logs, event ledgers.
"How do you distinguish INSERT vs UPDATE in Postgres?" — RETURNING (xmax = 0) AS was_insert. xmax = 0 on the returned row means INSERT.
"Why is MySQL's REPLACE INTO bad?" — DELETE-then-INSERT breaks FK, fires DELETE triggers, and re-numbers AUTO_INCREMENT. Use ON DUPLICATE KEY UPDATE instead.
"What's the EXCLUDED pseudo-table?" — The row that would have been inserted, accessible in DO UPDATE. EXCLUDED.col gives you the incoming value.
"How do you write a race-safe counter increment?" — INSERT INTO counters (id, n) VALUES (1, 1) ON CONFLICT (id) DO UPDATE SET n = counters.n + 1 RETURNING n. Atomic.
"What happens if the INSERT would satisfy multiple unique indexes?" — Postgres requires you to name one target. MySQL uses whichever fires first (non-deterministic). SQLite similar to MySQL.

Common beginner mistakes

Writing ON DUPLICATE KEY UPDATE name = 'a' — hardcodes the value instead of name = new.name or name = VALUES(name).
Forgetting the WHERE clause guard on DO UPDATE — allows stale writes to overwrite fresh writes.
Using INSERT IGNORE on MySQL — silently drops errors beyond duplicate keys; masks real issues.
Using REPLACE INTO on MySQL — DELETE-then-INSERT wart.
Not naming the conflict target on Postgres DO UPDATE — Postgres requires it.
Assuming ON CONFLICT semantics work on views — they only work on base tables.

Worked example — the atomic counter increment on Postgres

Detailed explanation. A hit-counter pattern that increments a per-key counter atomically, initializing to 1 on first hit. The atomic ON CONFLICT DO UPDATE SET n = t.n + EXCLUDED.n is race-free even under thousands of concurrent writers.

Question. Write a Postgres UPSERT that increments a page-view counter for a given page_id, initializing to 1 if the row doesn't exist, returning the new count.

Input.

CREATE TABLE page_views (page_id INT PRIMARY KEY, n INT NOT NULL);

Code.

INSERT INTO page_views (page_id, n) VALUES (%s, 1)
ON CONFLICT (page_id) DO UPDATE SET n = page_views.n + EXCLUDED.n
RETURNING n;

Step-by-step explanation.

Client calls with page_id = 42. First call — no row exists. INSERT branch fires. Row (42, 1) inserted. RETURNING gives n = 1.
Second concurrent call with same page_id = 42. Row exists. ON CONFLICT branch fires. UPDATE sets n = page_views.n + 1 = 2. Under exclusive row lock — no race.
Third concurrent call — same. UPDATE sets n = 3.
Under 10,000 concurrent calls — Postgres serialises writers on the same page_id via row lock. Different pages go through in parallel.
No lost increments. Every call increments by exactly 1.

Output.

Call	page_id	Returned n
1	42	1
2	42	2
3	42	3
4	43	1
5	43	2

Rule of thumb. Atomic counter increments are the poster child for OLTP UPSERT. Never use SELECT n THEN UPDATE n = n + 1 — always INSERT ... ON CONFLICT DO UPDATE.

Worked example — LWW UPSERT with timestamp guard on Postgres

Detailed explanation. A common CDC / cache-invalidation pattern — update the cached row only if the incoming version is newer than the cached version. Guards against out-of-order arrivals.

Question. Write a Postgres UPSERT for a session cache where each row has an updated_at timestamp. Only update if the incoming version is newer.

Input.

CREATE TABLE session_cache (
  session_id UUID PRIMARY KEY,
  payload JSONB NOT NULL,
  updated_at TIMESTAMPTZ NOT NULL
);

Code.

INSERT INTO session_cache (session_id, payload, updated_at)
VALUES (%s, %s, %s)
ON CONFLICT (session_id) DO UPDATE SET
  payload = EXCLUDED.payload,
  updated_at = EXCLUDED.updated_at
WHERE EXCLUDED.updated_at > session_cache.updated_at;

Step-by-step explanation.

Client calls with session_id = X, payload = P, updated_at = 10:00. Row doesn't exist. INSERT.
Second call with same session_id, payload = P2, updated_at = 09:00 (older — network delivered out of order). ON CONFLICT branch. WHERE clause: 09:00 > 10:00 → false. UPDATE skipped. Row unchanged.
Third call: payload = P3, updated_at = 11:00. ON CONFLICT branch. WHERE: 11:00 > 10:00 → true. UPDATE fires. Row now has P3, 11:00.
Fourth call: payload = P2, updated_at = 09:00 (very late duplicate). ON CONFLICT. WHERE: 09:00 > 11:00 → false. Skipped.
Idempotent — the same duplicate calls have no effect after the first application.

Output.

Call	payload arg	updated_at arg	Row state after
1	P	10:00	(P, 10:00)
2	P2	09:00 (stale)	(P, 10:00) unchanged
3	P3	11:00	(P3, 11:00)
4	P2	09:00 (dup)	(P3, 11:00) unchanged

Rule of thumb. LWW UPSERT with timestamp guard is the standard cache-pattern. Never write ordered assumptions into your API client — the pattern handles late/duplicate arrivals correctly at the write.

Worked example — MySQL UPSERT with the row alias

Detailed explanation. MySQL 8.0.19+ ships a cleaner UPSERT syntax with row-aliasing. Prefer this over VALUES() in new code.

Question. Write a MySQL 8 UPSERT for the page_views example, using the row alias.

Input.

CREATE TABLE page_views (page_id INT PRIMARY KEY, n INT NOT NULL);

Code.

INSERT INTO page_views (page_id, n) VALUES (%s, 1) AS new
ON DUPLICATE KEY UPDATE n = page_views.n + new.n;

Step-by-step explanation.

AS new — alias the incoming row. Subsequent references to new.n return the incoming value.
If page_id doesn't exist — INSERT branch. Row (42, 1) inserted.
If page_id exists — ON DUPLICATE KEY branch. UPDATE n = page_views.n + new.n. Row goes from (42, N) to (42, N+1).
Pre-MySQL-8.0.19 spelling: ... ON DUPLICATE KEY UPDATE n = page_views.n + VALUES(n). Both work; new is cleaner.
SELECT ROW_COUNT() — returns 1 for insert, 2 for update. Distinguish INSERT vs UPDATE this way.

Output.

Call	Row state after	ROW_COUNT()
First	(42, 1)	1 (INSERT)
Second	(42, 2)	2 (UPDATE)
Third	(42, 3)	2 (UPDATE)

Rule of thumb. MySQL 8+ ships two UPSERT spellings — VALUES() (legacy) and AS new (row alias). Prefer the row alias.

`postgres on conflict` interview question on multi-column conflict target

A senior interviewer asks: "You have a orders_by_customer_month table with a composite PK (customer_id, year_month) tracking monthly order totals. Write a Postgres UPSERT that atomically adds a new order's total to the monthly counter, initializing on first order per customer per month."

Solution Using composite conflict target + atomic increment via EXCLUDED

CREATE TABLE orders_by_customer_month (
  customer_id INT NOT NULL,
  year_month CHAR(7) NOT NULL,     -- e.g. '2026-07'
  total_amount NUMERIC(12,2) NOT NULL DEFAULT 0,
  order_count INT NOT NULL DEFAULT 0,
  last_updated TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  PRIMARY KEY (customer_id, year_month)
);

-- The upsert
INSERT INTO orders_by_customer_month
  (customer_id, year_month, total_amount, order_count, last_updated)
VALUES
  (%s, TO_CHAR(%s::date, 'YYYY-MM'), %s, 1, NOW())
ON CONFLICT (customer_id, year_month) DO UPDATE SET
  total_amount = orders_by_customer_month.total_amount + EXCLUDED.total_amount,
  order_count  = orders_by_customer_month.order_count + 1,
  last_updated = NOW()
RETURNING total_amount, order_count;

Step-by-step trace.

Step	(cust=42, month=2026-07, amount=25)	State after
1	First order for cust 42 in July 2026	INSERT → (42, '2026-07', 25.00, 1)
2	Second order for cust 42 in July, amount 40	CONFLICT → UPDATE → total=65.00, count=2
3	First order for cust 43 in July, amount 30	INSERT → (43, '2026-07', 30.00, 1)
4	Third order for cust 42 in July, amount 15	CONFLICT → UPDATE → total=80.00, count=3
5	First order for cust 42 in August, amount 20	INSERT → (42, '2026-08', 20.00, 1)

Output:

customer_id	year_month	total_amount	order_count
42	2026-07	80.00	3
42	2026-08	20.00	1
43	2026-07	30.00	1

Why this works — concept by concept:

Composite conflict target — ON CONFLICT (customer_id, year_month) matches the composite PK. Row-level lock is scoped to the specific (customer, month) bucket, so different customers or different months proceed in parallel.
EXCLUDED.total_amount + running-total via reference to the target — the DO UPDATE clause reads orders_by_customer_month.total_amount (current stored value) and adds EXCLUDED.total_amount (incoming order's amount). Atomic under exclusive row lock; no read-modify-write race.
+ 1 for order_count — increment by exactly 1 per order. Race-free because the entire UPSERT runs under the row lock.
NOW() in both branches — timestamp always updated to the moment of the write, whether it was an INSERT or an UPDATE. Handy for observability.
RETURNING — the API can return {total_amount, order_count} after the write in the same round-trip. No follow-up SELECT needed.
Cost — per-call O(log n) on the composite unique index. Under 10,000 concurrent writers, throughput is bounded by lock contention on hot (customer, month) buckets; for cold buckets, throughput scales linearly with connections.

SQL
Topic — SQL
SQL practice library

Practice →

SQL Topic — indexing Composite unique index drills

Practice →

4. Snowflake + BigQuery + Databricks MERGE

Warehouse-native `snowflake merge`, `bigquery merge`, and Databricks MERGE — the same ANSI grammar, three different storage-rewrite models, three different cost surfaces

The mental model in one line: on modern warehouses, MERGE is the canonical bulk-idempotent-write primitive; Snowflake rewrites the affected micro-partitions, BigQuery rewrites the affected partitions, Databricks/Delta rewrites the affected Parquet files, and the cost surface for a given MERGE is roughly (rows_affected / rows_per_partition) × (partition_rewrite_cost) plus the source-scan cost. Understanding the storage-rewrite model per engine is the difference between a 30-second dbt incremental and a 30-minute one.

Slot 1 — Snowflake MERGE anatomy.

Grammar — full ANSI three-branch: MERGE INTO t USING s ON t.pk = s.pk WHEN MATCHED [AND cond] THEN UPDATE / DELETE WHEN NOT MATCHED [AND cond] THEN INSERT.
Storage rewrite — Snowflake rewrites the micro-partitions containing matched target rows. If your MERGE touches rows in 47 of 8,000 partitions, only those 47 are rewritten. Immutable, versioned; old micro-partitions retained per Time Travel retention.
Cost — warehouse tier × wall clock. A Medium warehouse for 2 minutes ≈ 4 credits (~$8 at $2/credit). Optimising MERGE means minimising rewritten partitions (via CLUSTER BY on the ON column) and minimising wall clock (via appropriate WH sizing).
Query Profile — MERGE shows as a plan with source scan, target scan (via join), and a Delete/Insert operator at the top. Read partition-scan and pruning % on both source and target.
Cluster keys — cluster the target on the merge key (t.pk) so target scan prunes. Cluster the source on the merge key for source-side pruning.
Concurrent MERGE handling — Snowflake serialises concurrent MERGEs on the same target table by default. Concurrent MERGEs against different partitions may parallelise.
STREAM ... ON TABLE — combines with MERGE for CDC-style incremental loads. The stream captures inserts/updates/deletes on the source; MERGE consumes.
WHEN MATCHED AND is_deleted THEN DELETE — supports tombstone-driven deletes cleanly.
WHEN NOT MATCHED BY SOURCE THEN DELETE — full-sync deletes supported.
Return value — Snowflake reports number of rows inserted / updated / deleted in the query result summary.

Slot 2 — BigQuery MERGE anatomy.

Grammar — full ANSI three-branch: MERGE INTO t USING s ON t.pk = s.pk WHEN MATCHED [AND cond] THEN UPDATE / DELETE WHEN NOT MATCHED BY TARGET [AND cond] THEN INSERT WHEN NOT MATCHED BY SOURCE [AND cond] THEN UPDATE / DELETE. The BY TARGET / BY SOURCE spellings are explicit.
Storage rewrite — BigQuery rewrites the affected partitions. Each partition is a set of columnar files; MERGE rewrites the entire partition, not individual rows. This is why partition alignment on the MERGE key is critical.
Cost — on-demand. $5/TB scanned + rewrite cost. Full-table MERGE on a 100 TB unpartitioned table scans 100 TB → $500 per MERGE.
Cost — flat-rate. slot_ms. MERGE consumes slots for source-scan + join + rewrite. A well-planned MERGE completes in tens of thousands of slot_ms; a badly-planned one in millions.
--dry_run — estimate bytes-processed before running. Free. Use in CI to gate expensive MERGEs.
Partition alignment — if the source's date range covers only July 2026, BigQuery can prune to partitions 20260701..20260731 on the target if the ON clause includes a partition column filter.
Cluster keys — cluster the target on the MERGE key for block-level pruning inside partitions.
Concurrent MERGE handling — BigQuery serialises writes to a table. Concurrent MERGEs queue.
WHEN NOT MATCHED BY SOURCE — supported. Requires a full target scan; expensive on large tables.
Return value — the job's numDmlAffectedRows metadata field reports total rows affected.

Slot 3 — Databricks / Delta Lake MERGE anatomy.

Grammar — full ANSI three-branch on Delta tables. Non-Delta tables (Parquet, CSV, etc.) do NOT support MERGE.
Storage rewrite — Delta MERGE rewrites the affected Parquet files (typically 100 MB – 1 GB per file). Reads the affected files, applies the branches, writes new files, updates the Delta transaction log to point at the new files.
Cost — DBUs. Databricks Units are the compute-time unit. MERGE cost = DBU/hr × wall clock. Photon acceleration reduces wall clock and DBU cost by 3-5× typical.
File pruning — Delta uses data-skipping via min/max per file for the MERGE key. Cluster / Z-order on the MERGE key for pruning.
Concurrent MERGE handling — Delta uses optimistic concurrency control. Two concurrent MERGEs to the same table may both start; one commits, the other detects the conflict on commit and retries (with the latest snapshot).
WHEN MATCHED AND cond THEN UPDATE / DELETE — full conditional branches.
WHEN NOT MATCHED AND cond THEN INSERT — full support.
WHEN NOT MATCHED BY SOURCE — supported since Databricks Runtime 12.
CHANGE DATA FEED — Delta's built-in CDC feed. Combine with downstream MERGE for incremental pipelines.
OPTIMIZE ... ZORDER BY — periodic maintenance to reduce fragmentation from many small MERGE writes. Rewrite small files into big ones, z-order by the MERGE key.

Slot 4 — the storage-rewrite math per engine.

Snowflake — per micro-partition. Micro-partition size is ~50-500 MB uncompressed. A MERGE touching 47 micro-partitions rewrites 47 × ~200 MB = ~9 GB of storage. Cost is proportional.
BigQuery — per partition. Partition size varies (daily partitions may be MB or TB). A MERGE touching 30 partitions rewrites all 30 in full — even if only 1 row per partition matched.
Delta — per Parquet file. File size default ~1 GB; can be tuned. A MERGE touching 100 files rewrites those 100 files = ~100 GB of storage. OPTIMIZE periodically compacts to keep files at target size.
Optimising rewrite cost. Cluster / partition / Z-order the target on the MERGE key so the affected files are localised, not spread across all files.
The "small MERGE, big cost" trap. A MERGE that updates 100 rows spread across 100 partitions rewrites 100 partitions. Same MERGE on a table clustered by the merge key rewrites 1 partition. Same cost model on all three engines.

Slot 5 — cost surface per engine.

Snowflake. Warehouse tier × wall clock. Optimise for shorter wall clock at given tier.
BigQuery on-demand. Bytes scanned × $5/TB + rewrite. Optimise for smaller bytes scanned via partition prune.
BigQuery flat-rate. Slot commitment × month. Optimise for slot efficiency.
Databricks. DBU/hr × wall clock. Photon acceleration is often the biggest lever.
Cluster / partition / z-order investments. Rewrite the target's physical layout once; benefit every subsequent MERGE.

Slot 6 — CDC apply-loops with STREAM / CHANGES.

Snowflake CREATE STREAM ON TABLE source_events — captures inserts/updates/deletes. Combined with MERGE, gives you an idempotent CDC apply-loop.
Snowflake TASK — schedule the MERGE from the stream, runs every N minutes.
BigQuery INFORMATION_SCHEMA.STREAMING_TABLES — not a native CDC feed; use Datastream or third-party.
Delta CHANGES feed — enable spark.databricks.delta.properties.defaults.enableChangeDataFeed = true. Downstream reads the CDC feed and applies via MERGE.
Common pattern — source stream → dedup by key → MERGE with timestamp guard → target dimension table.

Slot 7 — the 7 most common warehouse MERGE pathologies.

Unpartitioned target on BigQuery. MERGE scans the full table on every run. Fix — PARTITION BY DATE(created_at).
Unclustered target on Snowflake. MERGE rewrites micro-partitions across the whole table. Fix — CLUSTER BY (merge_key).
Delta table without OPTIMIZE. Many small files from frequent MERGEs. Fix — schedule OPTIMIZE t ZORDER BY (merge_key) weekly.
SELECT * in source. Reads more columns than needed. Fix — project only the merge key + updated columns.
Non-unique ON condition. Ambiguous WHEN MATCHED, silent data loss. Fix — dedup source before MERGE.
MERGE on the wrong join order. Fix — the smaller side goes on the right. Warehouses generally figure this out, but hints help.
No timestamp guard. Stale events overwrite fresh. Fix — WHEN MATCHED AND s.updated_at > t.updated_at.

Common beginner mistakes

Using MERGE on unpartitioned BigQuery tables — costs 100× more than needed.
Forgetting WHEN NOT MATCHED BY TARGET on BigQuery — the explicit spelling is required in some contexts.
Running Delta MERGE on non-Delta tables — silent failure (Parquet tables reject MERGE).
Not scheduling OPTIMIZE ZORDER BY on Delta — files fragment into thousands of tiny ones.
Assuming Snowflake serialises all MERGE writers — different partitions can parallelise.
Non-deterministic source rows — duplicate keys in source produce undefined behaviour on WHEN MATCHED across all three engines.

Worked example — dbt incremental on Snowflake, from raw batch to MERGE

Detailed explanation. The daily dbt incremental pattern on Snowflake. Loads yesterday's orders into a snapshot table, then MERGEs into the analytics dim table.

Question. Write the full Snowflake MERGE for the dbt incremental with unique_key='order_id', timestamp guard on updated_at, and a _dbt_updated_at tracker column on the target.

Input. stg_orders has ~1M rows (yesterday's batch), dim_orders has ~500M rows (all-time).

Code.

MERGE INTO analytics.dim_orders AS t
USING (
  SELECT
    order_id, customer_id, total, status, updated_at,
    CURRENT_TIMESTAMP() AS _dbt_updated_at
  FROM (
    SELECT
      order_id, customer_id, total, status, updated_at,
      ROW_NUMBER() OVER (PARTITION BY order_id ORDER BY updated_at DESC) AS rn
    FROM staging.stg_orders
    WHERE _batch_id = :batch_id
  ) x
  WHERE rn = 1
) AS s
ON t.order_id = s.order_id
WHEN MATCHED AND s.updated_at > t.updated_at THEN UPDATE SET
  customer_id = s.customer_id,
  total = s.total,
  status = s.status,
  updated_at = s.updated_at,
  _dbt_updated_at = s._dbt_updated_at
WHEN NOT MATCHED THEN INSERT
  (order_id, customer_id, total, status, updated_at, _dbt_updated_at)
  VALUES
  (s.order_id, s.customer_id, s.total, s.status, s.updated_at, s._dbt_updated_at);

Step-by-step explanation.

Source subquery dedupes the batch by order_id, keeping the latest updated_at per order.
Target scan on dim_orders uses the cluster key on order_id for pruning. Only micro-partitions containing matching order_ids are touched.
WHEN MATCHED AND timestamp — updates only when source is fresher. Prevents stale replays from overwriting.
WHEN NOT MATCHED — inserts new order_ids into the target.
_dbt_updated_at — a target-side observability column tracking when dbt last touched the row. Common in dbt patterns.

Output.

Metric	Value
Source rows scanned	1,000,000
Source rows deduped	950,000 (some order_ids had multiple events)
Target rows scanned (matched)	800,000
Rows updated	750,000 (50K stale, guard skipped)
Rows inserted	150,000 (new orders)
Wall clock on Medium WH	45 seconds
Credits used	~1.5

Rule of thumb. dbt incremental on Snowflake is the canonical warehouse MERGE. Cluster on unique_key, dedup source in the USING subquery, use timestamp guard on WHEN MATCHED.

Worked example — BigQuery MERGE with partition prune predicate

Detailed explanation. BigQuery MERGE cost is dominated by target-scan bytes. Adding a partition-prune predicate to the ON or WHERE clause is the single biggest optimisation.

Question. Write a BigQuery MERGE that upserts July 2026 orders from stage_orders into dim_orders (partitioned by DATE(created_at), clustered by order_id), with a partition-prune predicate.

Input.

-- Target
CREATE TABLE `project.analytics.dim_orders` (
  order_id STRING,
  customer_id STRING,
  total NUMERIC,
  status STRING,
  created_at TIMESTAMP,
  updated_at TIMESTAMP
)
PARTITION BY DATE(created_at)
CLUSTER BY order_id;

-- Source
CREATE TABLE `project.staging.stage_orders` AS (
  SELECT * FROM raw.orders WHERE _batch_id = 'BATCH-2026-07-12'
);

Code.

MERGE INTO `project.analytics.dim_orders` AS t
USING (
  SELECT DISTINCT
    order_id, customer_id, total, status, created_at, updated_at
  FROM `project.staging.stage_orders`
) AS s
ON t.order_id = s.order_id
   AND DATE(t.created_at) BETWEEN DATE '2026-07-01' AND DATE '2026-07-31'
   AND DATE(s.created_at) BETWEEN DATE '2026-07-01' AND DATE '2026-07-31'
WHEN MATCHED AND s.updated_at > t.updated_at THEN UPDATE SET
  customer_id = s.customer_id,
  total = s.total,
  status = s.status,
  updated_at = s.updated_at
WHEN NOT MATCHED BY TARGET THEN INSERT
  (order_id, customer_id, total, status, created_at, updated_at)
  VALUES (s.order_id, s.customer_id, s.total, s.status, s.created_at, s.updated_at);

Step-by-step explanation.

DATE(t.created_at) BETWEEN ... AND ... in the ON — the partition-prune predicate. BigQuery uses this to skip all partitions outside July 2026.
Source subquery — SELECT DISTINCT to dedup. In production, prefer ROW_NUMBER() with a timestamp guard.
WHEN MATCHED — timestamp guard.
WHEN NOT MATCHED BY TARGET — INSERT the new order.
--dry_run — verify bytes-processed. Should be roughly 31 partitions × avg_partition_size + source_size. Full-table scan without the prune would be all_partitions × avg_partition_size.

Output.

Query	Bytes processed	Cost (on-demand @ $5/TB)
Without partition prune	12 TB	$60 per MERGE
With partition prune	400 GB	$2 per MERGE
Speedup	30×	30× cost saving

Rule of thumb. Every BigQuery MERGE against a partitioned target needs a partition-prune predicate. Verify with --dry_run before running.

Worked example — Databricks Delta MERGE with tombstone deletes

Detailed explanation. CDC events often carry a soft-delete flag or op='d' marker. The Delta MERGE handles inserts / updates / deletes in one statement using conditional branches.

Question. Write a Databricks MERGE that consumes a Kafka batch of CDC events (with op in {'i', 'u', 'd'}) and applies to a Delta target table.

Input. cdc_batch table with columns id, name, op, updated_at.

Code.

MERGE INTO analytics.users_delta AS t
USING (
  SELECT id, name, op, updated_at
  FROM (
    SELECT
      id, name, op, updated_at,
      ROW_NUMBER() OVER (PARTITION BY id ORDER BY updated_at DESC) AS rn
    FROM cdc_batch
  )
  WHERE rn = 1
) AS s
ON t.id = s.id
WHEN MATCHED AND s.op = 'd' THEN DELETE
WHEN MATCHED AND s.op IN ('i', 'u') AND s.updated_at > t.updated_at THEN UPDATE SET
  name = s.name,
  updated_at = s.updated_at
WHEN NOT MATCHED AND s.op IN ('i', 'u') THEN INSERT
  (id, name, updated_at) VALUES (s.id, s.name, s.updated_at);

Step-by-step explanation.

Source dedup — keep the latest per id (by updated_at).
WHEN MATCHED AND s.op = 'd' THEN DELETE — tombstones become target deletes.
WHEN MATCHED AND s.op IN ('i', 'u') AND fresher THEN UPDATE — updates with timestamp guard.
WHEN NOT MATCHED AND s.op IN ('i', 'u') THEN INSERT — inserts new rows.
If op = 'd' and the row doesn't exist in target — no-op (WHEN NOT MATCHED with op = 'd' is not written; the branch is silently skipped). Correct behaviour: delete of a nonexistent row is a no-op.

Output.

CDC event	Target action
`(42, 'alice', 'i', ...)` new	INSERT
`(42, 'alice-2', 'u', ...)` existing, fresher	UPDATE
`(42, '', 'd', ...)` existing	DELETE
`(99, '', 'd', ...)` not in target	no-op

Rule of thumb. Multi-branch conditional MERGE is the modern CDC apply pattern. Learn it once; ship it everywhere.

`bigquery merge` interview question on cost-optimised MERGE

A senior interviewer asks: "You're on-call for a daily dbt incremental that MERGEs 5M orders into a 2B-row BigQuery table, and it's costing $500/day. Walk me through the diagnosis and the top three optimisations."

Solution Using dry-run + partition prune + cluster on merge key

-- Step 1 — Confirm cost via --dry_run
bq query --dry_run --use_legacy_sql=false '
MERGE INTO `project.analytics.dim_orders` AS t
USING (SELECT * FROM staging.stg_orders WHERE _batch_id = "current") AS s
ON t.order_id = s.order_id
WHEN MATCHED THEN UPDATE SET ...
WHEN NOT MATCHED BY TARGET THEN INSERT ...';
-- Output: This query will process 100 TB when run.

-- Step 2 — Repartition + cluster the target (one-time cost)
CREATE OR REPLACE TABLE `project.analytics.dim_orders`
PARTITION BY DATE(created_at)
CLUSTER BY order_id
AS SELECT * FROM `project.analytics.dim_orders`;

-- Step 3 — Add partition prune predicate to future MERGEs
MERGE INTO `project.analytics.dim_orders` AS t
USING (
  SELECT DISTINCT * FROM staging.stg_orders WHERE _batch_id = "current"
) AS s
ON t.order_id = s.order_id
  AND DATE(t.created_at) >= CURRENT_DATE() - 7
  AND DATE(s.created_at) >= CURRENT_DATE() - 7
WHEN MATCHED AND s.updated_at > t.updated_at THEN UPDATE SET ...
WHEN NOT MATCHED BY TARGET THEN INSERT ...;

Step-by-step trace.

Fix	Bytes processed	Cost	Ratio
Baseline (unpartitioned)	100 TB	$500	1×
Partition target	3 TB	$15	33× cheaper
+ Partition prune predicate	300 GB	$1.50	333× cheaper
+ Cluster on order_id	100 GB	$0.50	1000× cheaper
+ Materialised view for hot filter	~30 GB	$0.15	3333× cheaper

Output:

Change	Effect
Repartition target	Daily partitions align with source date range
Cluster on order_id	Blocks pruned inside partitions on the merge key
Partition prune predicate	Explicit filter in ON scoping affected partitions
Source dedup	Prevents multi-match cardinality issues
Timestamp guard	Idempotent replay

Why this works — concept by concept:

Partition target on DATE(created_at) — daily partitions. Source batches typically cover a small date window; the MERGE prunes to just those partitions. 300× cost reduction on typical dbt loads.
Cluster on order_id — the MERGE key. BigQuery uses block-level metadata to skip blocks that can't contain matching order_ids. 3-10× on top of partition prune.
Partition prune predicate in ON — explicit hint to the planner: "we only care about the last 7 days of data". Prevents accidental full-table scans when source has stale data.
--dry_run before every deploy — catches accidental full-table MERGEs in CI. Fail the PR if estimated bytes > 10 TB.
Materialised view (final step) — for dashboards that repeatedly query the MERGE output, an MV avoids re-scanning the target on read. Combined with BI Engine, dashboards are sub-second.
Cost — one-time re-partition + cluster cost is ~$500 for a 2 TB target rewrite. Ongoing per-MERGE cost drops from $500/day to <$1/day. Payback in one day.

SQL
Topic — optimization
SQL optimization drills

Practice →

SQL Topic — SQL SQL practice library

Practice →

5. Anti-patterns and dialect matrix

The `sql merge upsert` failure modes — SQL Server's cardinality bug, concurrent MERGE races, non-deterministic sources — and the 8-engine dialect matrix that ties it all together

The mental model in one line: MERGE and UPSERT are the correct primitives for idempotent writes, but they ship with a well-known set of failure modes — non-deterministic source rows, concurrent-writer lost updates, the SQL Server MERGE cardinality bug, and trigger fire-order weirdness — that every senior engineer knows to defend against with dedup, unique indexes, and (when necessary) explicit serialization. Master the failure modes and you can write MERGEs that survive concurrency, retries, and data-drift.

Slot 1 — the SQL Server MERGE cardinality bug.

The bug. When a MERGE has multiple source rows matching the same target row, SQL Server sometimes silently produces incorrect results — either running the UPDATE branch multiple times per target row (non-deterministic final value) or attempting both an UPDATE and an INSERT for the same target row (constraint violation).
Reproduction. Simple MERGE with a non-unique ON condition or a source with duplicate keys. Search for "SQL Server MERGE bug" — Microsoft's own docs acknowledge these issues.
Documented issues (Microsoft KB). Multiple Cannot insert duplicate key errors, race conditions with concurrent inserts, trigger fire-order surprises.
Workaround 1 — dedup source before MERGE. Use a CTE with ROW_NUMBER() to guarantee unique source keys.
Workaround 2 — use INSERT + UPDATE separately. Ship an INSERT ... WHERE NOT EXISTS + UPDATE ... FROM in a transaction. Verbose but bug-free.
Workaround 3 — use SERIALIZABLE isolation. Prevents the race but hurts throughput.
Community recommendation. Many senior SQL Server DBAs recommend avoiding MERGE for critical paths and using explicit INSERT / UPDATE via CTE instead. Aaron Bertrand's "Please stop using MERGE" is the canonical write-up.

Slot 2 — trigger fire order.

BEFORE INSERT vs BEFORE UPDATE per branch. MERGE fires the appropriate BEFORE trigger for each row's branch (INSERT branch → BEFORE INSERT; UPDATE branch → BEFORE UPDATE). Simple in principle.
AFTER triggers fire per statement, not per branch. After the MERGE completes, statement-level AFTER triggers fire once, seeing the union of INSERTs / UPDATEs / DELETEs.
SQL Server historical bugs. In some SQL Server versions, AFTER INSERT and AFTER UPDATE triggers on the same table both fire twice due to MERGE's compilation. Patched in later versions but worth verifying in your environment.
Postgres — clean. Row-level BEFORE triggers fire per row per branch. Statement-level AFTER triggers fire once for the entire MERGE.
Interview signal. Knowing that trigger behaviour on MERGE differs subtly across engines is a senior-level detail.

Slot 3 — concurrent MERGE races.

The setup. Two workers both start a MERGE on the same target key. Under READ COMMITTED, both see the pre-write target state. Both attempt to INSERT the same key.
What happens on Postgres. The unique index catches the second INSERT and raises a unique-violation error. If the second MERGE was inside an atomic UPSERT (INSERT ... ON CONFLICT), the error is caught internally and routed to UPDATE.
What happens on Snowflake. Snowflake serialises MERGE writes on the same table via a Cluster-Table Lock. The second MERGE blocks until the first commits, then runs on the fresh state. No race.
What happens on BigQuery. BigQuery queues concurrent DML. Second MERGE runs after first commits.
What happens on Delta. Optimistic concurrency — both start, first commits, second detects the conflict at commit and retries with the fresh snapshot.
The safe pattern. Use atomic UPSERT on OLTP (INSERT ... ON CONFLICT). Use MERGE with unique ON condition on warehouses. Rely on engine-level serialisation for the concurrent-write safety.

Slot 4 — non-deterministic source rows.

The problem. Source contains multiple rows with the same MERGE key. WHEN MATCHED fires once per source row per matched target row — the target's final state depends on the order in which source rows are processed. Order is not deterministic on parallel plans.
The signal. UPDATE runs multiple times on the same target row; final target value depends on which source row was processed last. Different runs of the same MERGE produce different final states.
Fix — dedup source before MERGE. ROW_NUMBER() OVER (PARTITION BY id ORDER BY updated_at DESC) in the USING subquery.
Fix — aggregate source before MERGE. SELECT id, MAX(updated_at) AS updated_at, ARRAY_AGG(name) FROM source GROUP BY id. Preserves determinism.
SQL Server specific. With multiple source matches, SQL Server may attempt both UPDATE and INSERT on the same target — the notorious cardinality error.
Snowflake / BigQuery. Both accept multiple matches but final target state is undefined. Effectively equivalent to dedup being your responsibility.

Slot 5 — the safe MERGE checklist.

Unique source keys. Dedup source before MERGE using ROW_NUMBER() or GROUP BY.
Unique target keys. Ensure the ON condition uses a primary key or unique index. Composite is fine.
Timestamp guard. WHEN MATCHED AND s.updated_at > t.updated_at for CDC / event workloads.
IS DISTINCT FROM for null-safe compare. When conditional-updating on column difference.
Idempotency test. Run the MERGE twice on the same batch — output should be identical after both runs.
Cardinality assertion. For SQL Server, wrap MERGE in a check that raises if source cardinality doesn't match unique keys.
Cost dry-run. For BigQuery, run --dry_run first.
Cluster target on ON key. For warehouse MERGE, cluster the target for pruning.
Rollback plan. For Time-Travel-supported engines (Snowflake, Delta), you can rollback the target if a MERGE ships bad data. Know the retention.

Slot 6 — the 8-engine dialect matrix (one-line spellings).

Engine	Native UPSERT syntax	MERGE support
Postgres 15+	`INSERT ... ON CONFLICT (col) DO UPDATE SET col = EXCLUDED.col`	Full ANSI MERGE (PG 15+); BY SOURCE in PG 17+
MySQL 8	`INSERT ... AS new ON DUPLICATE KEY UPDATE col = new.col`	No native MERGE
SQLite 3.24+	`INSERT ... ON CONFLICT (col) DO UPDATE SET col = excluded.col`	No native MERGE
SQL Server 2008+	`MERGE INTO ... USING ... WHEN MATCHED THEN UPDATE / WHEN NOT MATCHED THEN INSERT`	Full ANSI MERGE (with known bugs); prefer INSERT + UPDATE via CTE for critical paths
Oracle 9i+	`MERGE INTO ... USING ... WHEN MATCHED THEN UPDATE ... WHEN NOT MATCHED THEN INSERT`	Full ANSI MERGE (no BY SOURCE)
Snowflake	`MERGE INTO ... USING ... WHEN MATCHED [AND cond] THEN UPDATE / WHEN NOT MATCHED THEN INSERT`	Full ANSI MERGE with BY SOURCE
BigQuery	`MERGE INTO ... USING ... WHEN MATCHED THEN UPDATE / WHEN NOT MATCHED BY TARGET THEN INSERT / WHEN NOT MATCHED BY SOURCE THEN DELETE`	Full ANSI MERGE
Databricks (Delta)	`MERGE INTO delta_table USING ... WHEN MATCHED [AND cond] THEN UPDATE / DELETE / WHEN NOT MATCHED THEN INSERT / WHEN NOT MATCHED BY SOURCE THEN DELETE`	Full ANSI MERGE on Delta tables only

Slot 7 — the 5-step senior MERGE reading strategy.

Step 1 — inspect the ON condition. Is it unique on both sides? If not, dedup source.
Step 2 — inspect WHEN MATCHED guard. Is there a timestamp / condition guard? If not, replays are non-idempotent.
Step 3 — inspect source dedup. Does the USING subquery have ROW_NUMBER() or GROUP BY? If not, multiple-match risk.
Step 4 — inspect target index / cluster. Is the target clustered on the ON key? If not, MERGE reads more than needed.
Step 5 — inspect concurrency. Is this the only writer? If not, understand engine-level serialisation guarantees.

Slot 8 — when NOT to use MERGE.

High-QPS single-row API calls. Use OLTP UPSERT (INSERT ... ON CONFLICT). MERGE is batch-oriented; per-call cost is higher.
SQL Server critical paths. Use INSERT + UPDATE via CTE. Avoid the known bugs.
Append-only tables. Use INSERT ... ON CONFLICT DO NOTHING. MERGE overhead is unnecessary.
Full-refresh dimension loads. TRUNCATE + INSERT can be simpler than three-branch MERGE with WHEN NOT MATCHED BY SOURCE.
Cross-database MERGE. Not universally supported. Materialise the source into a local staging table first.

Common beginner mistakes

Using MERGE on SQL Server for critical paths — subject to known bugs.
Non-unique ON condition — non-deterministic UPDATE.
Assuming concurrent MERGEs are magically safe — engine-level serialisation varies.
Forgetting that MERGE is batch — using it for API single-row upserts.
Not deduping source — the #1 cause of MERGE bugs across all engines.
Ignoring trigger fire order — MERGE fires per-branch, not per-statement.

Worked example — SQL Server MERGE cardinality bug and the CTE workaround

Detailed explanation. Reproduce the SQL Server MERGE cardinality bug, then show the safe CTE-based rewrite.

Question. Show a SQL Server MERGE that triggers the cardinality bug and the safe rewrite using INSERT + UPDATE via CTE.

Input. Source has two rows with the same PK; target has one matching row.

Code — buggy MERGE.

-- Buggy: source has two rows with same PK
MERGE INTO dbo.users AS t
USING (VALUES (42, 'alice'), (42, 'alice-dup')) AS s(id, name)
ON t.id = s.id
WHEN MATCHED THEN UPDATE SET name = s.name
WHEN NOT MATCHED THEN INSERT (id, name) VALUES (s.id, s.name);
-- Msg 8672: The MERGE statement attempted to UPDATE or DELETE the same row more than once.

Code — safe rewrite.

-- Safe: dedup source, then explicit INSERT + UPDATE via CTE
WITH deduped AS (
  SELECT id, name,
         ROW_NUMBER() OVER (PARTITION BY id ORDER BY name DESC) AS rn
  FROM (VALUES (42, 'alice'), (42, 'alice-dup')) AS raw(id, name)
),
source AS (
  SELECT id, name FROM deduped WHERE rn = 1
)
UPDATE t SET name = s.name
FROM dbo.users t
JOIN source s ON s.id = t.id;

INSERT INTO dbo.users (id, name)
SELECT id, name FROM source
WHERE id NOT IN (SELECT id FROM dbo.users);

Step-by-step explanation.

Buggy MERGE — source has two rows for id=42. Target has one. WHEN MATCHED fires twice per source row against the same target row. SQL Server raises 8672 error.
Safe rewrite — first dedup source via ROW_NUMBER() in a CTE. Then run explicit UPDATE FROM CTE and INSERT WHERE NOT EXISTS in a transaction.
Same conceptual output; no MERGE, no bug.
Aaron Bertrand's blog "Please stop using MERGE" documents this pattern extensively.
On other engines (Snowflake, BigQuery, Databricks), the same deduped MERGE works fine — the bug is specific to SQL Server.

Output.

Approach	Result
Buggy MERGE (dup source)	Msg 8672 error
Deduped MERGE	Works fine on non-SQL-Server
CTE + separate UPDATE + INSERT	Works on all engines including SQL Server

Rule of thumb. On SQL Server, always dedup source before MERGE, and consider CTE + explicit INSERT + UPDATE for critical paths.

Worked example — race-safe concurrent UPSERT under 10K QPS

Detailed explanation. A high-QPS API upsert on Postgres. Multiple concurrent workers all hit the same key; the correct pattern is atomic INSERT ... ON CONFLICT.

Question. Show the race-safe concurrent UPSERT pattern for a page_views counter with 10 K QPS on a single key.

Input.

CREATE TABLE page_views (page_id INT PRIMARY KEY, n BIGINT NOT NULL);

Code.

INSERT INTO page_views (page_id, n) VALUES (%s, 1)
ON CONFLICT (page_id) DO UPDATE SET n = page_views.n + 1
RETURNING n;

Step-by-step explanation.

Worker A calls with page_id = 42. Row doesn't exist. INSERT. Row (42, 1). Return 1.
Worker B calls concurrently with page_id = 42. Row now exists (or is being inserted). Under exclusive row lock, worker B waits for A's commit, then sees (42, 1). UPDATE fires: n = 1 + 1 = 2. Return 2.
Under 10K QPS on same key — Postgres serialises writers on the page_id = 42 row. Throughput is bounded by row-lock hold time (~microseconds). Cluster-wide throughput ~50K QPS on modern hardware for hot keys; scales linearly for distributed keys.
No lost increments. Every call increments by exactly 1.
Comparison — MERGE for this pattern would be much slower (higher per-call overhead), and it's harder to guarantee atomicity across concurrent MERGEs. Native ON CONFLICT is the right tool.

Output.

Concurrent workers	Final n after 10,000 calls
1	10,000
10	10,000
100	10,000
1,000	10,000
10,000	10,000

Rule of thumb. For high-QPS atomic increments, always use native INSERT ... ON CONFLICT DO UPDATE on Postgres/SQLite (or ON DUPLICATE KEY UPDATE on MySQL). Never MERGE — the per-call overhead is too high for API calls.

Worked example — the SCD Type 2 pattern with MERGE

Detailed explanation. SCD Type 2 preserves history — every change to a dimension row creates a new version, with valid_from / valid_to timestamps. MERGE handles it in two passes: close the current version, insert the new version.

Question. Write the SCD Type 2 MERGE on Snowflake for a dim_customer table with history preservation.

Input.

CREATE TABLE analytics.dim_customer (
  customer_id INT NOT NULL,
  name TEXT NOT NULL,
  email TEXT NOT NULL,
  valid_from TIMESTAMPTZ NOT NULL,
  valid_to TIMESTAMPTZ NOT NULL DEFAULT '9999-12-31',
  is_current BOOLEAN NOT NULL DEFAULT TRUE
);

CREATE UNIQUE INDEX idx_current ON dim_customer(customer_id) WHERE is_current;

Code.

-- Step 1: Close current versions of rows that will get a new version
UPDATE analytics.dim_customer t
SET valid_to = CURRENT_TIMESTAMP(), is_current = FALSE
WHERE is_current
  AND EXISTS (
    SELECT 1 FROM stage.stg_customer s
    WHERE s.customer_id = t.customer_id
      AND (s.name <> t.name OR s.email <> t.email)
  );

-- Step 2: Insert new versions (including brand-new customers)
INSERT INTO analytics.dim_customer (customer_id, name, email, valid_from, valid_to, is_current)
SELECT s.customer_id, s.name, s.email, CURRENT_TIMESTAMP(), '9999-12-31', TRUE
FROM stage.stg_customer s
LEFT JOIN analytics.dim_customer t
  ON t.customer_id = s.customer_id AND t.is_current
WHERE t.customer_id IS NULL
   OR t.name <> s.name
   OR t.email <> s.email;

Step-by-step explanation.

Step 1 closes existing "current" rows that have a changed source row. Sets valid_to = NOW() and is_current = FALSE. History preserved.
Step 2 inserts new rows for (a) brand-new customers not in target, and (b) existing customers whose columns changed (they now have a closed old row and need a fresh current row).
Unique index on (customer_id) WHERE is_current ensures at most one current row per customer at any time.
Some engines allow this in a single MERGE with WHEN MATCHED THEN UPDATE ... AND then INSERT. Snowflake accepts multi-branch MERGE for this pattern.
dbt's snapshot macro implements this exact pattern with additional book-keeping.

Output.

customer_id	name	email	valid_from	valid_to	is_current
42	alice-v1	a@old.com	2026-01-01	2026-07-12 09:00	FALSE
42	alice-v2	a@new.com	2026-07-12 09:00	9999-12-31	TRUE
43	bob	b@x.com	2026-07-12 09:00	9999-12-31	TRUE

Rule of thumb. SCD Type 2 needs history preservation — never overwrite the old row. Use two-step MERGE (close-then-insert) or dbt's snapshot macro.

`mysql on duplicate key update` interview question on choosing among UPSERT, MERGE, and REPLACE

A senior interviewer asks: "You're consulting on a MySQL application that ships REPLACE INTO for its user-profile save endpoint at 5K QPS. Why is this a bad choice, and what would you change?"

Solution Using ON DUPLICATE KEY UPDATE with row alias + RETURNING via ROW_COUNT()

-- BEFORE (bad): REPLACE INTO
-- Semantics: DELETE if exists, then INSERT — breaks FK, fires DELETE triggers,
-- re-numbers AUTO_INCREMENT.
REPLACE INTO users (id, name, email, updated_at)
VALUES (?, ?, ?, ?);

-- AFTER (good): INSERT ... ON DUPLICATE KEY UPDATE with row alias
INSERT INTO users (id, name, email, updated_at) VALUES (?, ?, ?, ?) AS new
ON DUPLICATE KEY UPDATE
  name = new.name,
  email = new.email,
  updated_at = new.updated_at;

-- Bonus: distinguish INSERT vs UPDATE via ROW_COUNT()
-- 1 → insert, 2 → update, 0 → no-op (row unchanged if you added a WHERE-like guard)
SELECT ROW_COUNT();

Step-by-step trace.

Step	REPLACE (bad)	ON DUPLICATE KEY UPDATE (good)
1	Client sends PUT /users/42	Same
2	DELETE FROM users WHERE id=42 → FK cascades kick in	INSERT attempts → PK conflict on id=42
3	INSERT users (42, ...) → AUTO_INCREMENT bumps	ON DUPLICATE branch → UPDATE cols
4	Delete triggers fire (audit-log spam)	Update triggers fire (correct)
5	If FK child rows referenced user 42 → violation OR cascade	No FK impact
6	Latency: 2 SQL statements internally	Latency: 1 statement
7	Idempotent? YES but destructive to FK graph	Idempotent AND safe

Output:

Metric	REPLACE	ON DUPLICATE KEY UPDATE
FK integrity	Breaks (cascade may delete child rows)	Preserved
DELETE triggers fired	YES (audit spam)	NO
UPDATE triggers fired	NO (only DELETE + INSERT)	YES (correct semantic)
AUTO_INCREMENT bumps	YES (gap in id sequence)	NO
Wall clock per call	~2 ms	~1 ms
Correctness under FK	Breaks	Preserved

Why this works — concept by concept:

INSERT ... ON DUPLICATE KEY UPDATE preserves FK graph — the target row's PK doesn't change; foreign key references from child tables remain valid. REPLACE would DELETE the row (potentially cascading), then INSERT a fresh row with the same id but different internal identity.
Correct trigger semantics — UPDATE triggers fire on the UPDATE branch; INSERT triggers fire on the INSERT branch. REPLACE fires DELETE triggers on every "update", which is wrong.
No AUTO_INCREMENT bump — the target row keeps its id; no gap in the sequence. REPLACE bumps AUTO_INCREMENT because internally it's a fresh INSERT.
Row alias AS new — MySQL 8.0.19+ cleaner spelling. Preferred over the legacy VALUES() function.
ROW_COUNT() for distinguishing INSERT vs UPDATE — returns 1 for insert, 2 for update. Useful in application code that needs to know which branch fired.
Cost — one statement per call, ~1 ms wall clock. At 5K QPS, total DB CPU is bounded and predictable.

SQL
Topic — SQL
SQL practice library

Practice →

SQL Topic — joins/sql SQL join drills

Practice →

Cheat sheet — MERGE / UPSERT recipe list

Two primitives, one contract. MERGE (ANSI, three-branch) for batch idempotent writes; native UPSERT (INSERT ... ON CONFLICT / ON DUPLICATE KEY UPDATE) for OLTP single-row atomic upserts.
Never SELECT-then-INSERT. Two-statement pattern has a race window. Always single-statement atomic UPSERT.
Never REPLACE INTO on MySQL. DELETE-then-INSERT wart. Use ON DUPLICATE KEY UPDATE instead.
Postgres UPSERT skeleton. INSERT INTO t (id, v) VALUES (?, ?) ON CONFLICT (id) DO UPDATE SET v = EXCLUDED.v.
Postgres UPSERT with guard. ... WHERE t.updated_at < EXCLUDED.updated_at — LWW semantics.
Postgres counter increment. INSERT INTO c (id, n) VALUES (?, 1) ON CONFLICT (id) DO UPDATE SET n = c.n + 1 — atomic increment.
Postgres INSERT-or-skip. INSERT ... ON CONFLICT (id) DO NOTHING — no update on conflict.
Postgres RETURNING. Add RETURNING (xmax = 0) AS was_insert, * to distinguish INSERT vs UPDATE and return the persisted row.
MySQL UPSERT skeleton (8.0.19+). INSERT INTO t (id, v) VALUES (?, ?) AS new ON DUPLICATE KEY UPDATE v = new.v.
MySQL UPSERT skeleton (pre-8.0.19). INSERT INTO t (id, v) VALUES (?, ?) ON DUPLICATE KEY UPDATE v = VALUES(v).
MySQL branch detection. ROW_COUNT() returns 1 for INSERT, 2 for UPDATE.
SQLite UPSERT skeleton. INSERT INTO t (id, v) VALUES (?, ?) ON CONFLICT (id) DO UPDATE SET v = excluded.v — same as Postgres.
ANSI MERGE skeleton. MERGE INTO t USING s ON t.pk = s.pk WHEN MATCHED THEN UPDATE SET ... WHEN NOT MATCHED THEN INSERT ....
ANSI MERGE three-branch (full sync). Add WHEN NOT MATCHED BY SOURCE THEN DELETE.
Timestamp guard. WHEN MATCHED AND s.updated_at > t.updated_at THEN UPDATE — protects against stale-event overwrites.
Batch dedup. Wrap source in ROW_NUMBER() OVER (PARTITION BY key ORDER BY updated_at DESC) = 1 — one row per key. Every warehouse MERGE needs this.
SQL Server MERGE bugs. Cardinality error on multiple source matches, plan-cache surprises. Prefer CTE + INSERT + UPDATE for critical paths.
SQL Server safe rewrite. WITH deduped AS (...) UPDATE FROM deduped; INSERT FROM deduped WHERE NOT EXISTS.
Oracle MERGE. Full ANSI two-branch. No WHEN NOT MATCHED BY SOURCE (must separate).
Snowflake MERGE. Full ANSI three-branch. Cluster target on merge key for pruning.
Snowflake MERGE cost. Warehouse tier × wall clock. Rewrites affected micro-partitions.
BigQuery MERGE. Full ANSI three-branch. Partition + cluster target. Use --dry_run first.
BigQuery MERGE cost. On-demand: bytes billed × $5/TB. Flat-rate: slot_ms. Partition prune predicate in ON is essential.
Delta MERGE. Full ANSI three-branch on Delta tables (Parquet won't accept). Photon accelerates. Schedule OPTIMIZE ZORDER BY periodically.
Optimistic concurrency (Delta). Two concurrent MERGEs — first commits, second retries.
SCD Type 2 pattern. Two-step: close current rows, insert new versions. Preserves history.
CDC apply-loop pattern. Dedup source, WHEN MATCHED AND fresher, WHEN NOT MATCHED INSERT, WHEN MATCHED AND is_deleted DELETE.
API upsert pattern. PUT /users/:id handler → single INSERT ... ON CONFLICT DO UPDATE — no race window.
Bulk load pattern. COPY into staging → MERGE from staging → drop staging.
When OFF the beaten path. For very high QPS or cross-database, consider queue-based writes (Kafka → downstream apply job) instead of direct UPSERT / MERGE.
Return values across engines. Postgres/SQLite: RETURNING. MySQL: SELECT LAST_INSERT_ID() / ROW_COUNT(). Warehouses: query result summary (rows inserted / updated / deleted).
Idempotency test. Run MERGE twice on same batch. Same final state = correct. Different state = bug.
Rollback plan. Snowflake Time Travel, Delta version history. Retain enough history to rollback a bad MERGE.

Frequently asked questions

What's the difference between MERGE and UPSERT?

MERGE is the ANSI SQL:2003 statement with up to three branches — WHEN MATCHED THEN UPDATE / DELETE, WHEN NOT MATCHED [BY TARGET] THEN INSERT, WHEN NOT MATCHED BY SOURCE THEN UPDATE / DELETE — designed for batch idempotent writes. It scans the source once and joins against the target via the ON condition, routing each row to exactly one branch. UPSERT is the informal term for "insert if new, update if existing"; on Postgres and SQLite it's spelled INSERT ... ON CONFLICT (col) DO UPDATE SET col = EXCLUDED.col, on MySQL it's INSERT ... ON DUPLICATE KEY UPDATE col = new.col, and on warehouses it's a two-branch MERGE. UPSERT is the OLTP-native primitive optimised for single-row, high-QPS atomic writes with conflict detection inside the write path (via unique index). MERGE is the warehouse-native primitive optimised for batch idempotent loads that can affect thousands or millions of rows in a single statement. Both provide the same idempotency contract; MERGE is the superset (adds full-sync DELETE), UPSERT is the two-branch case (INSERT + UPDATE only). Rule of thumb — high-QPS API endpoints use UPSERT; batch pipelines (dbt incremental, CDC apply-loops) use MERGE.

When should I use INSERT ... ON CONFLICT vs MERGE in Postgres?

Postgres 15+ ships both, so the question is real. Use INSERT ... ON CONFLICT (col) DO UPDATE SET for single-row API upserts, high-QPS backend writes, atomic counter increments, and any workload that fits the "insert one row, resolve via unique index" pattern. It's atomic, low-latency (~1 ms), race-free, and the community-blessed idiom. Use MERGE INTO for batch loads — dbt incrementals, CDC apply-loops, bulk staging-to-analytics moves, and full-sync patterns that need WHEN NOT MATCHED BY SOURCE (delete-if-not-in-source semantics). MERGE is more expressive (three branches with conditions), but has higher per-call overhead — you don't want it for single-row API calls at 10K QPS. In practice, most Postgres codebases stick with INSERT ... ON CONFLICT for everything except full-sync scenarios, because it was the only idiomatic option pre-Postgres 15 and it's already the shared reflex across the Postgres community. New Postgres 15+ codebases can start using MERGE for batch, but ON CONFLICT remains the correct choice for OLTP.

Why does SQL Server's MERGE have known bugs?

SQL Server's MERGE has a documented history of correctness issues — cardinality errors when multiple source rows match the same target row (raising Msg 8672: The MERGE statement attempted to UPDATE or DELETE the same row more than once), trigger fire-order surprises (double-firing on both INSERT and UPDATE triggers in some patched-and-repatched versions), plan-cache instability with parameter sniffing, race conditions with concurrent inserts even under SERIALIZABLE isolation, and semantic surprises with WHEN NOT MATCHED BY SOURCE on partitioned tables. Aaron Bertrand's widely-referenced "Please stop using MERGE" post (updated across the years) catalogs the specific issues and Microsoft's Connect / Feedback item history acknowledging them. The safe pattern on SQL Server for critical paths is to dedup source in a CTE and then run explicit UPDATE FROM CTE and INSERT ... WHERE NOT EXISTS inside a transaction — verbose, but bug-free and equivalent in semantics. On other engines (Postgres, Snowflake, BigQuery, Databricks), MERGE works as specified and is the correct choice. If you're on SQL Server and you must use MERGE, always: (1) dedup the source before merging, (2) test with concurrency, (3) verify trigger fire-order in your environment, (4) prefer INSERT + UPDATE via CTE for anything mission-critical.

How do I write an idempotent MERGE in Snowflake vs BigQuery?

Snowflake — the canonical dbt-incremental pattern. Source-scan-once, cluster target on the merge key, dedup source in the USING subquery, timestamp guard on WHEN MATCHED, and let Snowflake serialise concurrent writers automatically:

MERGE INTO analytics.dim_orders AS t
USING (
  SELECT id, name, updated_at
  FROM (
    SELECT id, name, updated_at,
           ROW_NUMBER() OVER (PARTITION BY id ORDER BY updated_at DESC) AS rn
    FROM stage.orders_batch
  ) WHERE rn = 1
) AS s
ON t.id = s.id
WHEN MATCHED AND s.updated_at > t.updated_at THEN UPDATE SET name = s.name, updated_at = s.updated_at
WHEN NOT MATCHED THEN INSERT (id, name, updated_at) VALUES (s.id, s.name, s.updated_at);

BigQuery — same shape but add a partition-prune predicate in the ON clause so the target scan skips partitions outside the source's date range. Without this, BigQuery scans the full target and costs 30–100× more:

MERGE INTO `project.analytics.dim_orders` AS t
USING (SELECT DISTINCT ... FROM stage.orders_batch) AS s
ON t.id = s.id
   AND DATE(t.created_at) BETWEEN DATE '2026-07-01' AND DATE '2026-07-31'
   AND DATE(s.created_at) BETWEEN DATE '2026-07-01' AND DATE '2026-07-31'
WHEN MATCHED AND s.updated_at > t.updated_at THEN UPDATE SET ...
WHEN NOT MATCHED BY TARGET THEN INSERT ...;

Verify BigQuery costs with bq query --dry_run before running — if estimated bytes exceed expectations, add or refine the partition prune predicate. In both engines, idempotency is guaranteed by the timestamp guard + source dedup — replaying the same batch is a no-op after the first run.

What's the race condition in a naive UPSERT?

The classic race: two workers concurrently receive the same request (PUT /users/42). Both open a transaction. Both run SELECT id FROM users WHERE id = 42 and get "no row." Both decide to INSERT. Under READ COMMITTED isolation, both attempt the INSERT — one wins (commits first), the other hits a UNIQUE-constraint violation on the primary key and raises an error to the client. Symptoms — sporadic 500 errors under load, audit-log noise (Duplicate key violation in every log rotation), retry storms, and (in some engines) the loser's subsequent RETRY may silently overwrite the winner's data if the retry logic is naive. The fix — use atomic UPSERT via a single statement: INSERT INTO users (id, name) VALUES (42, 'alice') ON CONFLICT (id) DO UPDATE SET name = EXCLUDED.name. Postgres detects the unique-index conflict inside the write path (before the constraint check completes) and atomically routes execution to the UPDATE branch. No two-statement race, no error, no retry. The same applies to MySQL (ON DUPLICATE KEY UPDATE), SQLite (ON CONFLICT DO UPDATE), and warehouse MERGE INTO (though warehouses also serialize concurrent MERGE statements). Rule — every backend UPSERT must be a single statement; never a SELECT followed by an INSERT or UPDATE.

How do I handle SCD Type 2 with MERGE?

SCD Type 2 preserves history — every dimension change creates a new version, with valid_from and valid_to timestamps and an is_current flag. The MERGE pattern is two-step: first close the current version of changed rows, then insert the new versions. Snowflake and BigQuery both express this in two statements (or two MERGE branches):

-- Step 1: close current versions for changed rows
UPDATE dim_customer t
SET valid_to = CURRENT_TIMESTAMP(), is_current = FALSE
WHERE is_current AND EXISTS (
  SELECT 1 FROM stg_customer s
  WHERE s.customer_id = t.customer_id
    AND (s.name <> t.name OR s.email <> t.email)
);

-- Step 2: insert new versions (new customers + changed existing customers)
INSERT INTO dim_customer (customer_id, name, email, valid_from, valid_to, is_current)
SELECT s.customer_id, s.name, s.email, CURRENT_TIMESTAMP(), '9999-12-31', TRUE
FROM stg_customer s
LEFT JOIN dim_customer t ON t.customer_id = s.customer_id AND t.is_current
WHERE t.customer_id IS NULL
   OR t.name <> s.name
   OR t.email <> s.email;

A unique partial index (customer_id) WHERE is_current enforces "one current version per customer" at the storage layer. dbt's snapshot macro implements this pattern with additional book-keeping (dbt_valid_from, dbt_valid_to, dbt_scd_id). For queries, downstream analytics either filter by WHERE is_current (Type 1 view) or by WHERE :as_of_date BETWEEN valid_from AND valid_to (Type 2 view). The idempotency contract still holds — replaying the same batch produces the same final state because the WHERE EXISTS ... AND s.col <> t.col guard skips unchanged rows.

Practice on PipeCode

Drill the SQL practice library → — 450+ DE-focused questions covering INSERT ... ON CONFLICT, MERGE INTO, and every adjacent pattern.
Sharpen the join reflex on SQL join drills → — MERGE is fundamentally a join with row-level actions per matched / unmatched row.
Layer SQL indexing drills → — unique indexes, composite unique constraints, and partial indexes are what MERGE / UPSERT depend on for atomicity.
Push the difficulty ceiling with SQL optimization drills → — read MERGE execution plans, spot the missing partition prune, and rewrite the MERGE for 100× cost reduction.
Warm up with aggregation drills → — pre-aggregate source before MERGE, dedup with ROW_NUMBER(), use MAX() for LWW semantics.
Layer window function drills → — ROW_NUMBER() OVER (PARTITION BY key ORDER BY updated_at DESC) is the batch-dedup pattern every MERGE ships with.
For the broader SQL interview surface, take the SQL for Data Engineering course →.

Pipecode.ai is Leetcode for Data Engineering — every `sql merge upsert` recipe above ships with hands-on practice rooms where you type `INSERT ... ON CONFLICT (id) DO UPDATE SET col = EXCLUDED.col` on a live Postgres, migrate the same UPSERT to MySQL's `ON DUPLICATE KEY UPDATE col = new.col` row-alias spelling, translate it to a Snowflake three-branch `MERGE INTO ... USING ... WHEN MATCHED [AND cond] THEN UPDATE / WHEN NOT MATCHED THEN INSERT`, benchmark it against a BigQuery MERGE with partition-prune predicate and `--dry_run` cost gate, and finally dodge the SQL Server MERGE cardinality bug with the CTE + INSERT + UPDATE safe rewrite — the exact 8-engine `postgres on conflict` / `mysql on duplicate key update` / `snowflake merge` fluency that senior DE interviews probe. PipeCode pairs every UPSERT and MERGE concept with 450+ DE-focused problems and a real-time scoring engine, so you never have to wonder whether your idempotent-write answer holds up under a senior interviewer's depth probes.

Practice SQL now →
Optimization drills →

SQL EXPLAIN ANALYZE Deep Dive: Reading Execution Plans in Postgres, Snowflake, BigQuery

Gowtham Potureddi — Sat, 11 Jul 2026 16:26:34 +0000

sql explain analyze is the primitive every senior data engineer, analytics engineer, and DBA reaches for at 3 a.m. when a dashboard query that used to run in 200 milliseconds suddenly takes 40 seconds — and it is the primitive most engineers can type on day one but only a small minority can actually read on day one thousand. The gap between "I know the keyword" and "I can look at a plan artefact and name the fix out loud in one breath" is the gap between a mid-level candidate who gets nervous when the interviewer opens the plan tab and a senior candidate who takes the mouse, points at the hot node, and says "the estimate is thirty times the actual — run ANALYZE, then rewrite the join." This guide is the honest tour of what actually happens inside the planner when you prefix a query with EXPLAIN (ANALYZE, BUFFERS, VERBOSE) — how the four numbers per node compose into a plan tree, why cost= is a made-up unit while actual time= is wall clock, how Snowflake's Query Profile, BigQuery's execution graph, and SQL Server's actual execution plan translate the same ideas into different artefacts, and how the same five-step reading discipline turns any of them into a concrete fix.

The tour walks the four engines you have to keep straight in 2026 — Postgres EXPLAIN (ANALYZE, BUFFERS, VERBOSE, FORMAT TEXT) with its cost / rows / loops / actual-time-per-node grid and the BUFFERS clause that surfaces the shared-hit / shared-read / temp-read triangle where spilling hides, the Snowflake Query Profile with its operator cards, micro-partition pruning percent, and local-vs-remote disk spill panel that decides warehouse sizing, the BigQuery execution graph with its stage-based DAG, slot-milliseconds accounting, --dry_run cost estimator, and INFORMATION_SCHEMA.JOBS audit trail, and finally the SQL Server actual execution plan XML with SET STATISTICS TIME/IO ON and the graphical SSMS view. Each section ships a teaching block followed by a Solution-Tail interview answer — code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works — so you leave with the mental model, the syntax, and the senior read-strategy your next interviewer is listening for.

When you want hands-on reps immediately after reading, drill the SQL optimization practice library → for plan-reading and rewrite drills, warm up the index reflex with the SQL indexing room →, and layer the broader SQL practice surface → covering 450+ DE-focused problems that pair every plan artefact with a concrete rewrite.

On this page

Why execution plans matter in 2026
Postgres EXPLAIN ANALYZE anatomy
Snowflake QUERY_PROFILE
BigQuery execution graph
Dialect matrix + 5-step reading strategy
Cheat sheet — plan-reading recipe list
Frequently asked questions
Practice on PipeCode

1. Why execution plans matter in 2026

The `sql query optimizer` mental model — cost-based optimisation, statistics, cardinality estimation, and why every senior read-strategy starts with a plan

The one-sentence invariant: an execution plan is the planner's compiled decision tree for how to physically execute a logical SQL statement — which access method for each table, which join algorithm for each join, which order to evaluate joins in, whether to sort, whether to hash, whether to spill, whether to parallelise — and reading it is the fastest way to translate "this query is slow" into "here is the exact one-line fix". Every engine ships an optimiser; every optimiser makes decisions based on table statistics and a cost model; every cost model is imperfect; and every imperfect cost model produces a bad plan on some query somewhere in your product every week. The senior discipline is not to memorise a hundred rewrites — it is to open the plan, find the one node where estimate diverges from actual, and let that number tell you what to fix.

The planner mental model in five layers.

Parser. Turns the SQL string into an abstract syntax tree. This step never fails silently — a syntax error surfaces as a parse error. Nothing interesting to read at this layer.
Rewriter. Applies rule-based transformations — view expansion, subquery flattening, predicate pushdown into the CTE / subquery boundary, constant folding. On Postgres this is the pg_rewrite layer; on Snowflake it is folded into the optimiser; on BigQuery it happens before stage assignment.
Cost-based optimiser (CBO). Considers many equivalent plan shapes, assigns a cost to each based on table statistics (row counts, distinct values, most common values, histograms, correlation coefficient), and picks the cheapest. This is where "the planner got it wrong" usually happens — the stats are stale, the histogram is coarse, the correlation between two columns is off, or the CBO does not know about a functional dependency your data has.
Executor. Walks the chosen plan tree and returns tuples up the pipeline. Modern executors are pipelined (results stream up the tree) rather than materialised (each node writes to a temp table). The EXPLAIN ANALYZE output shows the actual row count and wall clock at each node — the executor's ground truth.
Statistics collector. A background job on every engine that samples tables, updates histograms, and feeds the CBO on the next planning cycle. On Postgres — ANALYZE or autovacuum. On Snowflake — automatic and invisible. On BigQuery — table metadata refreshed continuously.

Cost units vs actual time — the two axes of every plan node.

Cost is a made-up unit. In Postgres, cost is measured in "arbitrary planner units" — a Seq Scan on a 1-row table costs about 0.01 units; a Hash Join on a million rows might cost 250,000 units. The unit does not correspond to milliseconds, CPU cycles, or bytes. It is only meaningful relative to other plan alternatives during optimisation.
Actual time is wall clock. actual time=0.017..0.019 means "the first row from this node was ready at 0.017 ms after query start; the last row at 0.019 ms." This is real, measurable, comparable across queries. Every senior read starts here — cost is the optimiser's guess, actual time is what happened.
The estimate-vs-actual gap is the smoking gun. If a node has rows=100 loops=1 in the plan header but actual rows=100000 loops=1, the optimiser was off by a thousand times — a bad plan is almost guaranteed. Fix — run ANALYZE, check for stale stats, or add extended statistics on correlated columns.
Cost is only comparable within a single plan. You cannot compare cost 12,500 for query A against cost 8,700 for query B on the same table — the units are internal to the planner, not portable across queries.

What senior interviewers actually probe when they open the plan tab.

Do you know how to read the plan tree? Bottom-up, inner-most first. The Limit at the top is the last node executed; the Seq Scan / Index Scan at the bottom is the first. Candidates who read top-down miss the hot leaf every time.
Do you know the node types? Seq Scan = full table scan. Index Scan = seek + heap fetch. Bitmap Heap Scan = index-driven heap fetch with bitmap OR. Hash Join = build hash table on smaller side, probe with larger. Merge Join = zip two pre-sorted streams. Nested Loop = outer × inner. Sort = sort operator (may spill). Hash Aggregate = hash-based GROUP BY. Gather = parallel-worker collector.
Do you know when the plan is bad? Estimate off by 10× or more. Buffers: temp read non-zero (spilling to disk). Nested Loop on two large tables. Seq Scan where an index would work. Sort → Hash Aggregate where the Sort is doing all the work.
Do you know the fix? Run ANALYZE. Add a covering index. Rewrite a correlated subquery into a JOIN. Add a LATERAL where a scalar subquery loops. Split a WITH RECURSIVE into a materialised temp table. Increase work_mem for a specific session. Add pg_hint_plan hints in exceptional cases.
Do you know the trade-offs? Adding an index speeds reads but slows writes. Increasing work_mem speeds sorts but risks OOM under concurrency. Materialising a CTE eats memory but avoids a recompute. Hints lock the plan — great for known cases, terrible when data shifts.

The reading discipline — bottom-up, node-by-node, cost-vs-actual ratio.

Step 1 — bottom-up. Find the leaf nodes (scans, Values clauses, function-returning-set nodes). Read their actual rows and actual time. This is what the executor read from disk (or memory).
Step 2 — cost-vs-actual ratio. For each node, compare rows= (estimate) to actual rows=. A ratio within 2× is fine. 10× is worth investigating. 100× is almost always a bad plan.
Step 3 — hot node. Find the node whose actual time is the largest fraction of total wall clock. This is the node to optimise. Speeding up a node that takes 0.1 ms of a 40 s query is pointless.
Step 4 — spill / shuffle / parallelism. Check for Buffers: temp read (spill), Sort Method: external merge (bigger spill), Workers Launched: 0 (parallelism disabled), Rows Removed by Filter: 999999 (post-filter waste). Each is a specific fix.
Step 5 — the one-line rewrite. Based on the hot node's failure mode, name the fix out loud — add index, rewrite join, increase memory, run ANALYZE, add hint. Senior candidates land on the fix within thirty seconds of opening the plan.

Why interviewers open plan tabs in senior loops.

It's a fluency check on the executor mental model. Anyone can type EXPLAIN; the senior signal is being able to look at "Hash Join (cost=12500..37500 rows=1000 width=64) (actual time=45..190 rows=1000000 loops=1)" and say "estimate was a thousand rows, actual was a million — the join is spilling to disk and the plan is wrong" in one breath.
It's a fluency check on the storage model. Postgres reads pages; Snowflake reads micro-partitions; BigQuery reads columnar files. The Buffers line, the bytes scanned line, the Partitions scanned line — each surfaces the storage cost that differs per engine.
It's a fluency check on cost. On BigQuery, wrong plan = wrong bill ($5/TB scanned on-demand). On Snowflake, wrong plan = wrong warehouse size (2× per tier). On Postgres, wrong plan = wrong tail latency (100 ms → 40 s). Senior candidates can quantify the cost of a bad plan.
It's a fluency check on system design. "Design a Postgres schema to support this dashboard" — great answers include the index plan (which composite indexes, in which order), the analyse cadence, and how the cost model reacts to skew. The plan surface is the system design surface.
It's a fluency check on debugging discipline. "The query got 100× slower after a data load — walk me through your diagnosis." Every senior answer starts with EXPLAIN ANALYZE on the slow and fast versions and diffs the plan trees.

Worked example — the day-one `EXPLAIN` on a mystery-slow query

Detailed explanation. The archetype: a dashboard query that used to run in 200 ms now takes 40 seconds. The junior instinct is to add an index; the senior instinct is to open the plan and let the numbers name the fix. The single most useful EXPLAIN variant on Postgres is EXPLAIN (ANALYZE, BUFFERS, VERBOSE, FORMAT TEXT).

Question. Given the query below on a Postgres 15 database, prefix it with the right EXPLAIN variant to surface plan shape, actual timings, buffer accounting, and node targeting. Name the four things you would read first.

Input.

-- The mystery-slow query
SELECT o.customer_id, SUM(o.total) AS lifetime_value
FROM orders o
JOIN customers c ON c.id = o.customer_id
WHERE c.country = 'US'
  AND o.created_at >= NOW() - INTERVAL '30 days'
GROUP BY o.customer_id
ORDER BY lifetime_value DESC
LIMIT 100;

Code.

EXPLAIN (ANALYZE, BUFFERS, VERBOSE, FORMAT TEXT)
SELECT o.customer_id, SUM(o.total) AS lifetime_value
FROM orders o
JOIN customers c ON c.id = o.customer_id
WHERE c.country = 'US'
  AND o.created_at >= NOW() - INTERVAL '30 days'
GROUP BY o.customer_id
ORDER BY lifetime_value DESC
LIMIT 100;

Step-by-step explanation.

EXPLAIN alone prints the planned tree with estimates only — no query executes. Use this when the query is prohibitively expensive to actually run.
ANALYZE runs the query and returns real actual time and actual rows per node. This is what you want 95% of the time — the plan the executor actually walked.
BUFFERS adds the shared hit / shared read / temp read triangle per node — cache hits, cold buffer reads from disk, and temp writes (spill). Non-zero temp read is a bright-red flag.
VERBOSE adds the column list and expression detail for each node — expensive to eyeball but crucial when you need to know exactly which columns are projected at each layer.
FORMAT TEXT (the default) is the classic indented tree. FORMAT JSON is machine-readable for CI regression tests; FORMAT YAML is a middle ground.

Output (elided).

Limit  (cost=250.31..250.56 rows=100 width=16) (actual time=42.512..42.548 rows=100 loops=1)
  Buffers: shared hit=1234, read=456, temp read=42 written=42
  ->  Sort  (cost=250.31..252.81 rows=1000 width=16) (actual time=42.510..42.520 rows=100 loops=1)
        Sort Key: (sum(o.total)) DESC
        Sort Method: top-N heapsort  Memory: 33kB
        ...

Rule of thumb. For 95% of interview-flow diagnosis and on-call debugging, EXPLAIN (ANALYZE, BUFFERS) is the right variant. Add VERBOSE when you suspect a projection issue; add FORMAT JSON when you want to diff plans across two queries programmatically.

Worked example — cost units are not milliseconds (the day-two lesson)

Detailed explanation. The most common junior misconception: reading cost=12500..37500 and treating those as milliseconds. They are not. They are internal planner units used only to rank plan alternatives. The single number that translates to real time is actual time=.

Question. Given the plan fragment below, which numbers are milliseconds and which are made-up units? What is the total wall-clock time of the query?

Input.

Limit  (cost=12500.31..12500.56 rows=100 width=16) (actual time=42.512..42.548 rows=100 loops=1)
  ->  Sort  (cost=12500.31..12502.81 rows=1000 width=16) (actual time=42.510..42.520 rows=100 loops=1)
        ->  HashAggregate  (cost=12495.00..12497.50 rows=1000 width=16) (actual time=41.900..42.400 rows=850 loops=1)
              ->  Hash Join  (cost=1250.00..12480.00 rows=100000 width=12) (actual time=25.100..40.500 rows=98000 loops=1)
                    ...
Planning Time: 0.150 ms
Execution Time: 42.702 ms

Code. (No code — a plan-reading exercise.)

made-up units: cost=..
milliseconds:  actual time=..
row counts:    rows= (estimate) and actual rows= (real)
loops:         actual loops=N — number of times this node was executed
totals:        Planning Time (ms) + Execution Time (ms)

Step-by-step explanation.

cost=12500.31..12500.56 — the two numbers are the estimated startup cost and total cost, in arbitrary planner units. Startup cost is what the node pays before emitting its first row; total cost is what it pays to emit all rows.
actual time=42.512..42.548 — the two numbers are the real wall-clock times for the first and last row emitted by this node, in milliseconds. If loops > 1, these are the per-loop averages, not the total.
rows=100 — the optimiser's estimate. actual rows=100 — the real count. If these differ by 10×, the plan is likely suboptimal.
loops=1 — this node executed once. Nodes inside a nested loop, function scan, or subquery may execute many times; in that case wall-clock time = actual time (per loop) × loops.
Execution Time: 42.702 ms — the true wall clock for the query. Planning Time — how long the optimiser took to plan (usually sub-millisecond; large for many-JOIN queries).

Output. Wall-clock total = 42.702 ms. The cost numbers (12500, 12480) are not milliseconds — do not multiply them.

Rule of thumb. Read actual time for wall clock. Read cost only to compare plan alternatives when explaining why the optimiser chose plan A over plan B. Never quote cost numbers to a business stakeholder.

Worked example — bottom-up reading discipline

Detailed explanation. The plan tree in FORMAT TEXT is nested — inner-most first. Execution walks bottom-up: leaf nodes (scans) produce rows, they feed intermediate nodes (joins, aggregates), which feed the top node (Limit, Sort). Reading top-down misses which leaf is doing the work.

Question. Given the plan below, which node is the bottleneck? Which node should you optimise? Give the reasoning.

Input.

Limit  (actual time=1200.5..1200.6 rows=100 loops=1)
  ->  Sort  (actual time=1200.4..1200.5 rows=100 loops=1)
        ->  HashAggregate  (actual time=1198.3..1199.1 rows=850 loops=1)
              ->  Hash Join  (actual time=100.5..1195.4 rows=98000 loops=1)
                    ->  Seq Scan on orders o  (actual time=99.0..1150.0 rows=10000000 loops=1)
                          Filter: (created_at >= now() - '30 days'::interval)
                          Rows Removed by Filter: 90000000
                    ->  Hash  (actual time=0.5..0.5 rows=1000 loops=1)
                          ->  Seq Scan on customers c  (actual time=0.1..0.4 rows=1000 loops=1)
                                Filter: (country = 'US'::text)

Code. (No code — a plan-reading exercise.)

Step-by-step explanation.

Read bottom-up. The two Seq Scan leaves execute first. Seq Scan on customers c — 0.4 ms — cheap. Seq Scan on orders o — 1150 ms — this is the hot node.
Rows Removed by Filter: 90000000 on the orders scan — the engine read 100 M rows and threw away 90 M of them via the created_at >= NOW() - INTERVAL '30 days' filter. That means no index on created_at exists, or the filter is not sargable.
The Hash Join at 100.5..1195.4 inherits the cost of the orders scan (~1095 ms of the join time is upstream from the scan). The join itself is cheap.
HashAggregate and Sort are microseconds — they operate on 100 K joined rows, not 100 M.
The fix is at the leaf — add an index on orders(created_at). The join, aggregate, and sort are already optimal.

Output. The bottleneck is Seq Scan on orders o. Fix: CREATE INDEX ON orders(created_at). Estimated speed-up: 100 M reads → ~1 M reads = ~100× — from ~1200 ms to ~15 ms.

Rule of thumb. Bottom-up. Always. The leaf that dominates wall clock is the leaf to optimise. Top-level nodes rarely need attention unless they explicitly spill.

Common beginner mistakes

Reading the plan top-down — missing the hot leaf.
Treating cost= as milliseconds — confusing internal units for wall clock.
Ignoring Rows Removed by Filter — a filter that discards 90% of rows is telling you which index to add.
Skipping BUFFERS — the temp-read line is the smoking gun for spilling.
Confusing loops=1 with loops=N — inside a nested loop, per-loop times must be multiplied by loops.

SQL interview question on when to reach for `EXPLAIN ANALYZE`

A senior interviewer often opens with: "You're on-call and a dashboard query for the CFO went from 200 ms to 40 seconds after last night's data load. Walk me through your first ten minutes of diagnosis. Which EXPLAIN variant do you run, which numbers do you read first, and what are the top three fixes you would try in order?"

Solution Using `EXPLAIN (ANALYZE, BUFFERS)` on the slow and fast versions, diffing plan trees

-- Step 1 — capture the slow plan
EXPLAIN (ANALYZE, BUFFERS, VERBOSE, FORMAT TEXT)
SELECT customer_id, SUM(total)
FROM orders
WHERE created_at >= '2026-06-10'
GROUP BY customer_id
ORDER BY SUM(total) DESC
LIMIT 100;

-- Step 2 — capture the fast plan by rolling back one week
-- (via a read replica pinned to yesterday's snapshot, or a staging DB)
EXPLAIN (ANALYZE, BUFFERS, VERBOSE, FORMAT TEXT)
SELECT customer_id, SUM(total)
FROM orders
WHERE created_at >= '2026-06-03'
GROUP BY customer_id
ORDER BY SUM(total) DESC
LIMIT 100;

-- Step 3 — refresh statistics if the slow plan estimate is off by 10× or more
ANALYZE orders;

-- Step 4 — re-run the slow plan
EXPLAIN (ANALYZE, BUFFERS)
SELECT customer_id, SUM(total)
FROM orders
WHERE created_at >= '2026-06-10'
GROUP BY customer_id
ORDER BY SUM(total) DESC
LIMIT 100;

-- Step 5 — if still slow, check for a missing index by hunting Rows Removed by Filter
-- Fix: CREATE INDEX ON orders(created_at) or a composite (customer_id, created_at DESC)

Step-by-step trace.

Step	Action	Reads
1	Prefix slow query with `EXPLAIN (ANALYZE, BUFFERS)`	Plan tree with cost/actual/loops/buffers
2	Compare to fast-version plan on replica	Diff shows which node's cost or rows changed
3	Look for `rows=` vs `actual rows=` gap ≥ 10×	Indicates stale statistics
4	Run `ANALYZE orders` to refresh statistics	Optimiser now has fresh histograms
5	Re-`EXPLAIN` — if plan is now fast, root cause was stale stats	Fixed
6	If still slow, look for `Rows Removed by Filter: NNN`	Missing index — add one
7	Verify final plan is `Index Scan` not `Seq Scan` at the hot leaf	Ship

Output:

Diagnosis outcome	Root cause	One-line fix
Estimate rows 10× < actual	Stale statistics	`ANALYZE orders;`
`Rows Removed by Filter` in the millions	Missing index	`CREATE INDEX ON orders(created_at);`
`Buffers: temp read=NNN` non-zero	Sort or hash spill	`SET work_mem = '128MB';` (session)
`Nested Loop` on two large tables	Bad join estimate	Rewrite join, add composite index, or hint
`Seq Scan` on a large table with cheap filter	Selectivity < 1% but no index	Add partial index, or composite index

Why this works — concept by concept:

Diff two plans — a slow-and-a-fast plan on the same query shape (yesterday vs today, prod vs replica) instantly localises the regression. If Hash Join was cheap yesterday and expensive today, the stats on the join columns are stale or the data distribution shifted.
ANALYZE fixes stale statistics — after a big data load, the planner's histograms and MCV lists are out of date. Running ANALYZE table re-samples and refreshes them; the next EXPLAIN on the same query often shows a completely different, correctly-costed plan.
Rows Removed by Filter names the missing index — the executor is telling you "I read N rows, threw away most of them via this filter." That filter should be an index seek instead of a scan. Add the index and the filter becomes an Index Cond — zero rows discarded.
BUFFERS: temp read is spilling — the planner allocated a sort or hash that outgrew work_mem. Two fixes: rewrite the query to sort or aggregate less (LIMIT earlier, smaller GROUP BY), or bump work_mem for this session. Never bump it globally — under concurrency, every session gets its own copy and you run out of RAM.
Cost — EXPLAIN alone is free (planning cost only). EXPLAIN ANALYZE runs the query — expensive if the query is slow. On production, prefer EXPLAIN alone for exploratory diagnosis; use EXPLAIN ANALYZE when you can afford the extra wall clock or when the query returns quickly enough. The read discipline itself is O(depth of plan tree) — a five-node plan reads in five seconds.

SQL
Topic — optimization
SQL optimization drills

Practice →

SQL Topic — indexing SQL indexing problems

Practice →

2. Postgres EXPLAIN ANALYZE anatomy

`postgres explain analyze` — the four numbers per node, the six major node types, and how `BUFFERS` surfaces the spill triangle

The mental model in one line: every Postgres plan node emits four numbers — cost (planner units), rows (estimate), actual time (wall clock), and loops (executions) — plus a variable-length BUFFERS triangle of shared hit / shared read / temp read that reveals whether the node lived in cache, hit cold buffers, or spilled to disk. Once you memorise the four numbers per node and the six major node types, you can read almost any Postgres plan in one pass.

Slot 1 — the four numbers per node.

cost=startup..total — planner units. Startup cost is what the node pays before emitting its first row (sort must complete before emit; hash must build before probe). Total cost is the price to emit all rows. Ratio total / startup reveals whether the node is pipelined (near 1:1) or blocking (large gap).
rows=N — the planner's estimate. Comes from table statistics (pg_stats) and the selectivity of the predicates above. Skewed columns without extended statistics produce bad estimates.
actual time=first..last — wall clock (ms). First is when the first row was ready; last is when the last row was ready. If loops > 1, both are per-loop averages.
actual rows=N — the real count. If rows=100 but actual rows=100000, the optimiser was off by 1000× — almost certainly a bad plan.
loops=N — how many times the executor ran this node. loops=1 for most; loops=N for nodes inside a nested loop or a subquery per outer row. Wall-clock cost of the node = actual time × loops.

Slot 2 — the six major node types you must recognise on sight.

Seq Scan — full table scan. Reads every row in the table, applies filters row-by-row. Fast on small tables; catastrophic on big tables when a filter has good selectivity (should have been an index).
Index Scan — btree seek + heap fetch. Uses an index to locate rows, then reads the heap page. Best when selectivity is high (< 1% of table) and the projected columns are mostly in the index or need heap fetch.
Index Only Scan — btree seek without heap fetch. Available when all projected columns are covered by the index. Fastest read path — often 10× faster than Index Scan because it skips heap I/O.
Bitmap Heap Scan — index-driven with heap fetch, but the executor first builds a bitmap of matching heap pages, then reads them in sorted order. Fast on medium-selectivity filters (1–10% of table) — better than Seq Scan because it skips non-matching pages, better than plain Index Scan because it batches heap I/O.
Hash Join — build a hash table on the smaller side, probe with the larger. O(build_side_rows + probe_side_rows). Great for large tables with no useful sort order; may spill if the build side exceeds work_mem.
Merge Join — zip two pre-sorted streams. O(N + M) after sort, or O(N + M) if both sides come pre-sorted from an index. Great when input is already sorted; expensive if you have to explicitly sort first.
Nested Loop — outer × inner: for each outer row, execute the inner. O(outer × inner). Fine when outer is tiny; catastrophic when outer is millions.
Sort — external sort operator. Cheap when input fits work_mem; expensive when it spills to temp read. Sort Method: quicksort = in-memory; Sort Method: top-N heapsort = smart bounded sort for ORDER BY ... LIMIT N; Sort Method: external merge = spilled.
Hash Aggregate / Group Aggregate — hash-based or sort-based GROUP BY. Hash Aggregate builds a hash on group keys; Group Aggregate requires pre-sorted input. Hash Aggregate may spill; Group Aggregate never does but requires a Sort upstream.
Gather / Gather Merge — parallel worker collector. Gather collects rows in arbitrary order; Gather Merge preserves sort order from workers. Presence of these means the planner chose parallel execution — parallel_workers and max_parallel_workers_per_gather control this.

Slot 3 — the BUFFERS triangle: shared hit / shared read / temp read.

shared hit=N — pages served from the buffer cache (shared buffers). Fast — RAM read, no disk I/O.
shared read=N — pages read from disk into the buffer cache. Slower than hit; each read is ~100 μs on SSD, ~5 ms on spinning disk. High shared read = cold cache; may warm up on subsequent runs.
shared dirtied=N — pages modified by the query (usually only for DML). Ignore for read-only diagnosis.
temp read=N / temp written=N — pages the executor spilled to disk because a Sort, Hash Join, or Hash Aggregate overflowed work_mem. This is the spill signal. Any non-zero temp read means the query is thrashing — fix is either query rewrite (limit earlier, smaller GROUP BY) or SET work_mem = '128MB' for the session.
shared read / (shared hit + shared read) — the cache miss rate. Under 5% is fine; over 50% means the working set doesn't fit in shared_buffers or the buffer cache is being churned.

Slot 4 — reading a plan bottom-up in five moves.

Move 1 — identify the leaves. Every Seq Scan, Index Scan, Bitmap Heap Scan, Values Scan, Function Scan is a leaf. Read their actual time × loops for total per-leaf cost.
Move 2 — find the hot leaf. The leaf with the largest actual time × loops is the bottleneck. If Seq Scan on orders is 1200 ms and everything else is 5 ms, the fix is at that scan.
Move 3 — check Rows Removed by Filter. If a Seq Scan has Filter: created_at > NOW() - INTERVAL '30 days' and Rows Removed by Filter: 99000000, the filter should be an index. Add one.
Move 4 — check estimate-vs-actual. For each node, compare rows= to actual rows=. A 10× gap is worth investigating; a 100× gap almost always means a bad plan. Run ANALYZE.
Move 5 — check BUFFERS: temp read. Non-zero temp read = spill. Fix by rewriting the query or bumping work_mem.

Slot 5 — the estimate-vs-actual failure modes.

Stale statistics. After a data load, the histograms and MCV lists are out of date. Run ANALYZE table to refresh.
Skewed columns without extended stats. A column with 90% of rows having value 'X' and 10% having other values will fool the optimiser unless you create extended statistics: CREATE STATISTICS s ON col1, col2 FROM t;.
Correlated columns. If country and city are highly correlated but the planner assumes independence, it can be off by 100× on multi-column predicates. Fix — extended statistics on (country, city).
Function-wrapped columns. WHERE EXTRACT(YEAR FROM created_at) = 2026 is not sargable — the planner can't use the histogram. Rewrite: WHERE created_at >= '2026-01-01' AND created_at < '2027-01-01'.
Prepared-statement plan reuse. With plan_cache_mode = 'auto', Postgres may pick a generic plan that is bad for specific parameter values. Force plan_cache_mode = 'force_custom_plan' if the shape hurts you.

Slot 6 — the eight most common Postgres plan pathologies.

Seq Scan on a large filtered table. Missing index. Fix — CREATE INDEX ON t (filter_col); or a composite index if multiple filters combine.
Nested Loop with millions on the outer side. Very slow — should be Hash Join. Fix — check estimates; if the planner underestimated the outer, run ANALYZE. If the estimate is right and the plan is still bad, raise enable_nestloop = off (dangerous — use only after diagnosis).
Sort with external merge method. Spilling. Fix — rewrite to sort less (add LIMIT, smaller GROUP BY, pre-aggregate), or bump work_mem.
Hash Join with temp read non-zero. Spilling. Fix — swap build side (planner may have picked the wrong side), or bump work_mem.
Bitmap Heap Scan reading most of the table. The bitmap is close to full — the index isn't selective enough. Fix — either accept Seq Scan (which is what the planner would pick with better stats), or add a partial index.
Index Scan with high Rows Removed by Index Recheck. The index was used but many rows didn't match the full filter — the index only covered part of the predicate. Fix — extend the index to a covering INCLUDE index.
Parallel Seq Scan with Workers Launched: 0. Parallelism was disabled — usually because max_parallel_workers_per_gather = 0 or the table is too small (min_parallel_table_scan_size). Fix — configure appropriately.
CTE Scan on a materialised CTE. Pre-PG12 CTEs were materialised (always). PG12+ inlines by default; if a CTE is materialised on new Postgres, it's an explicit WITH ... AS MATERIALIZED — verify that was intentional.

Slot 7 — EXPLAIN variants you should know.

EXPLAIN query — plan-only, no execution. Free but based on estimates only. Useful for expensive queries or when you can't afford to run.
EXPLAIN ANALYZE query — runs the query and returns actual timings. The everyday variant.
EXPLAIN (ANALYZE, BUFFERS) — adds the BUFFERS triangle. Use always for on-call diagnosis.
EXPLAIN (ANALYZE, BUFFERS, VERBOSE) — adds column projections. Useful for projection issues.
EXPLAIN (ANALYZE, BUFFERS, FORMAT JSON) — machine-readable. Great for CI regression tests, plan-diffing tooling, and pg_stat_statements correlation.
EXPLAIN (ANALYZE, BUFFERS, WAL) — adds WAL (write-ahead log) accounting for DML. Useful when diagnosing slow INSERTs / UPDATEs.
EXPLAIN (ANALYZE, BUFFERS, SETTINGS) — adds non-default GUC settings. Useful when a query works differently on prod vs a replica.

Common beginner mistakes

Running EXPLAIN without ANALYZE on a slow query and drawing conclusions from estimates only.
Bumping work_mem globally after seeing a single spill — every backend gets its own copy and OOM under concurrency.
Adding an index without checking that the plan changes — sometimes the planner still prefers Seq Scan due to bad stats or low selectivity.
Reading top-down — missing which leaf is the actual bottleneck.
Ignoring loops — a Nested Loop with actual time=0.1ms loops=1000000 costs 100 seconds, not 0.1 ms.
Copying EXPLAIN output into Slack without formatting — a plan tree is nearly unreadable when line breaks are lost.

Worked example — the `Seq Scan` that should have been an `Index Scan`

Detailed explanation. The most common Postgres plan bug: a Seq Scan on a large table where a filter has good selectivity. The plan shows a Seq Scan with Rows Removed by Filter in the millions; the fix is a single-column index.

Question. Given the plan below on an orders table (100 M rows), identify the fix. Show the DDL that would flip the plan to Index Scan (or Bitmap Heap Scan) and reduce wall clock from ~1200 ms to under 20 ms.

Input.

QUERY PLAN
Limit  (cost=... rows=100 width=16) (actual time=1200.5..1200.6 rows=100 loops=1)
  Buffers: shared hit=1234 read=456
  ->  Sort  (cost=... rows=100 width=16) (actual time=1200.4..1200.5 rows=100 loops=1)
        ->  Seq Scan on orders o  (cost=... rows=1000000 width=12) (actual time=99.0..1150.0 rows=1000000 loops=1)
              Filter: (created_at >= now() - '30 days'::interval)
              Rows Removed by Filter: 99000000
              Buffers: shared hit=1234 read=456

Code.

-- Fix: add index on the filter column
CREATE INDEX CONCURRENTLY idx_orders_created_at
  ON orders (created_at DESC);

-- Verify the plan flip
EXPLAIN (ANALYZE, BUFFERS)
SELECT customer_id, SUM(total)
FROM orders
WHERE created_at >= NOW() - INTERVAL '30 days'
GROUP BY customer_id
ORDER BY SUM(total) DESC
LIMIT 100;

Step-by-step explanation.

The current plan reads 100 M rows (Seq Scan on orders) and discards 99 M via the created_at filter — clear signal of a missing index on created_at.
CREATE INDEX CONCURRENTLY — the CONCURRENTLY clause avoids taking an ACCESS EXCLUSIVE lock, so writes continue during index build. Takes longer (double-scan) but keeps prod live.
DESC in the index matches the query's ORDER BY SUM(total) DESC — but the sort is on the aggregate, not created_at; so DESC here doesn't help ordering. It's still fine because btree indexes can be scanned in either direction.
After the index exists, run ANALYZE orders (or wait for autovacuum). The optimiser now has fresh selectivity estimates for created_at and will pick Index Scan or Bitmap Heap Scan when the filter is selective enough.
Re-run EXPLAIN (ANALYZE, BUFFERS) — the new plan should show Bitmap Heap Scan or Index Scan on orders with actual rows=1000000 (the filter-matching rows only) and wall clock under 20 ms.

Output.

Before	After
`Seq Scan on orders`	`Bitmap Heap Scan on orders`
`actual rows=1000000 loops=1`	`actual rows=1000000 loops=1`
`Rows Removed by Filter: 99000000`	`Rows Removed by Filter: 0`
Buffers: shared read=456	Buffers: shared read=12
Wall clock ~1200 ms	Wall clock ~15 ms

Rule of thumb. Any Seq Scan with Rows Removed by Filter in the millions is a missing-index diagnosis. The DDL is one line; the wall-clock win is 80–100× typical.

Worked example — the `Sort` that spills and how `work_mem` fixes it

Detailed explanation. A common failure mode: a Sort node with Sort Method: external merge and temp read in the buffers. The engine ran out of work_mem and spilled to a temp file. Two fixes — rewrite to sort less, or increase work_mem.

Question. Given the plan below with a spilled Sort, show two fixes and their trade-offs.

Input.

Sort  (cost=... rows=10000000 width=64) (actual time=8500.0..12000.0 rows=10000000 loops=1)
  Sort Key: total DESC
  Sort Method: external merge  Disk: 640000kB
  Buffers: shared hit=5000, temp read=80000 written=80000
  ->  Seq Scan on orders  (cost=... rows=10000000 width=64) (actual time=0.5..2200.0 rows=10000000 loops=1)

Code.

-- Fix A — bump work_mem for this session only (safe, cheap)
SET work_mem = '256MB';
SELECT * FROM orders ORDER BY total DESC;
RESET work_mem;

-- Fix B — rewrite to sort less (best when you only want top-N)
SELECT * FROM orders
ORDER BY total DESC
LIMIT 100;
-- Postgres picks Sort Method: top-N heapsort — bounded memory, no spill

-- Fix C — pre-aggregate before sorting (best when downstream only needs summaries)
SELECT customer_id, SUM(total) AS total
FROM orders
GROUP BY customer_id
ORDER BY total DESC
LIMIT 100;

Step-by-step explanation.

The current plan sorts 10 M rows and spills 640 MB to disk (Sort Method: external merge Disk: 640000kB). Wall clock is 12 s.
Fix A — SET work_mem = '256MB' for the session. The Sort fits in memory, method changes to quicksort, wall clock drops to ~3 s. Never ALTER SYSTEM SET work_mem = '256MB' — under concurrency, 100 sessions × 256 MB = 25 GB of hash / sort memory.
Fix B — add LIMIT 100. The planner picks Sort Method: top-N heapsort — a bounded heap of size N. Memory usage is N × row_width, not full-table. Wall clock drops to ~2 s.
Fix C — pre-aggregate. Instead of sorting 10 M raw rows, sort 100 K aggregated rows. Hash Aggregate → Sort → Limit. Wall clock ~500 ms.
Prefer C > B > A. Rewriting the query is usually cheaper than tuning memory; increasing memory is the last resort when the query truly needs a big sort.

Output.

Fix	Sort Method	Disk spill	Wall clock
None	external merge	640 MB	12 s
A: `work_mem=256MB`	quicksort	0	3 s
B: `LIMIT 100`	top-N heapsort	0	2 s
C: pre-aggregate + LIMIT	quicksort	0	500 ms

Rule of thumb. Diagnose spill first — the Sort Method: external merge line and non-zero temp read are the smoking gun. Prefer to rewrite the query; bump work_mem only for the session, never globally.

Worked example — `EXPLAIN (ANALYZE, BUFFERS)` on a `Hash Join` bottleneck

Detailed explanation. A Hash Join between two large tables can dominate wall clock when the build side is bigger than expected. Reading BUFFERS reveals whether the hash is spilling.

Question. Given the plan below with a slow Hash Join, walk through diagnosis. Which side is the build side? What does temp read on the hash imply? What's the fix?

Input.

Hash Join  (cost=... rows=1000000 width=32) (actual time=4500.0..8500.0 rows=980000 loops=1)
  Hash Cond: (o.customer_id = c.id)
  Buffers: shared hit=1000 read=4500 temp read=25000 written=25000
  ->  Seq Scan on orders o  (cost=... rows=10000000 width=16) (actual time=0.5..3000.0 rows=10000000 loops=1)
  ->  Hash  (cost=... rows=5000000 width=20) (actual time=4200.0..4200.0 rows=5000000 loops=1)
        Buckets: 65536 Batches: 8 Memory Usage: 4096kB
        ->  Seq Scan on customers c  (cost=... rows=5000000 width=20) (actual time=0.5..2000.0 rows=5000000 loops=1)

Code.

-- Fix A — bump work_mem for the session
SET work_mem = '256MB';

-- Fix B — swap the build side by rewriting (planner picked wrong)
-- The customers table is bigger than orders in this query, but customers should be filtered first
SELECT *
FROM orders o
JOIN (SELECT id FROM customers WHERE country = 'US') c
  ON c.id = o.customer_id;
-- Now the hash is built on filtered customers (much smaller), not raw customers

-- Fix C — force the planner via a subquery / CTE materialisation
WITH us_customers AS MATERIALIZED (
  SELECT id FROM customers WHERE country = 'US'
)
SELECT o.*
FROM orders o
JOIN us_customers c ON c.id = o.customer_id;

Step-by-step explanation.

Hash is the build side — Postgres always builds the hash on the bottom child of Hash Join. Here, that's customers c (5 M rows).
Batches: 8 — the hash was partitioned into 8 batches because it didn't fit in work_mem. Batches > 1 means spilling.
Buffers: shared hit=1000 read=4500 temp read=25000 — the temp read=25000 is the smoking gun. 25 000 pages × 8 KB = 200 MB spilled.
Fix A — bump work_mem. If bumped to 256 MB, Batches drops to 1 and temp read goes to 0; wall clock roughly halves.
Fix B — swap sides. If customers is filtered by a selectivity predicate (country = 'US'), do the filter before the join. The filtered customers hash is 500 K rows (10% of 5 M), fits in work_mem, no spill.

Output.

Fix	Batches	temp read	Wall clock
None	8	200 MB	8.5 s
A: work_mem=256MB	1	0	4.5 s
B: pre-filter build side	1	0	1.5 s
C: MATERIALIZED CTE	1	0	1.5 s

Rule of thumb. Batches > 1 on a Hash node = spilling. Prefer to filter the build side down before the join; work_mem bump is the last resort.

`postgres explain analyze` interview question on a two-plan diff

A senior interviewer often asks: "Here are two EXPLAIN (ANALYZE, BUFFERS) outputs for the same query — one from prod and one from a staging DB with the same data. Prod is 30 seconds; staging is 300 ms. Walk me through the diff and name the root cause."

Solution Using bottom-up plan diffing to isolate the regressing node

-- Prod plan (slow — 30 s)
Limit  (actual time=30000..30000 rows=100 loops=1)
  Buffers: shared hit=5000 read=45000 temp read=8000
  ->  Sort  (actual time=29995..30000 rows=100 loops=1)
        Sort Method: external merge  Disk: 64000kB
        ->  HashAggregate  (actual time=28000..29500 rows=850000 loops=1)
              ->  Hash Join  (actual time=100..27000 rows=1000000 loops=1)
                    Buffers: temp read=8000
                    ->  Seq Scan on orders  (actual time=0.5..12000 rows=100000000 loops=1)
                    ->  Hash  (actual time=500..500 rows=5000000 loops=1)
                          Batches: 8  Memory Usage: 4096kB
                          ->  Seq Scan on customers  (actual time=0.5..300 rows=5000000 loops=1)

-- Staging plan (fast — 300 ms)
Limit  (actual time=290..300 rows=100 loops=1)
  Buffers: shared hit=1500 read=200
  ->  Sort  (actual time=289..300 rows=100 loops=1)
        Sort Method: top-N heapsort  Memory: 33kB
        ->  HashAggregate  (actual time=250..280 rows=850000 loops=1)
              ->  Hash Join  (actual time=50..220 rows=1000000 loops=1)
                    ->  Bitmap Heap Scan on orders  (actual time=1..80 rows=1000000 loops=1)
                          Recheck Cond: (created_at >= '2026-06-10')
                          ->  Bitmap Index Scan on idx_orders_created_at
                    ->  Hash  (actual time=100..100 rows=5000000 loops=1)
                          Batches: 1  Memory Usage: 512000kB
                          ->  Seq Scan on customers  (actual time=0.5..80 rows=5000000 loops=1)

Step-by-step trace.

Node	Prod actual time	Staging actual time	Diff
Seq/Bitmap Scan on orders	12000 ms (Seq, 100M rows)	80 ms (Bitmap, 1M rows)	Missing index on prod
Hash (customers)	500 ms, Batches: 8	100 ms, Batches: 1	work_mem too low on prod
Sort	external merge, 64 MB spill	top-N heapsort, 33 kB	Sort method differs — cascades from filter
Total wall clock	30 s	300 ms	100× gap

Prod is missing an index on orders(created_at) — the Seq Scan reads 100 M rows and the filter discards 99 M. Staging has the index; the plan flips to Bitmap Heap Scan reading only 1 M matched rows. Prod's work_mem is also too low — the customers hash spills to 8 batches; staging has bigger work_mem and fits in 1 batch. Two fixes, one root cause per — but the dominant fix is the index.

Output:

Fix priority	Change	Est. wall clock after
1	`CREATE INDEX CONCURRENTLY idx_orders_created_at ON orders(created_at);` then `ANALYZE orders;`	~500 ms
2	`ALTER SYSTEM SET work_mem = '32MB';` (was 4 MB) then restart / reload	~350 ms
3	Ensure both plans are on Postgres 15+ (top-N heapsort improvements)	~300 ms

Why this works — concept by concept:

Bottom-up diff — comparing plan trees leaf-first surfaces the storage-access divergence immediately. The prod Seq Scan vs staging Bitmap Heap Scan is a one-line diff that pinpoints the missing index.
Rows Removed by Filter — even without staging as a reference, prod's plan alone shows 99 M rows discarded by a filter. That is the missing-index tell every time.
Batches: N > 1 — the Hash node's Batches: 8 line means the hash spilled to 8 partitions on disk. Combined with Memory Usage: 4096kB and Buffers: temp read=8000, it names the spill and its size (~64 MB).
Sort Method divergence — prod uses external merge (spilled); staging uses top-N heapsort (bounded). The method differs because the row count reaching the sort differs — the filter pathology upstream cascades all the way down.
Cost — Diagnosis cost = one EXPLAIN (ANALYZE, BUFFERS) × 2 plans = 30 s + 300 ms wall clock. Fix cost = one CREATE INDEX CONCURRENTLY (typically minutes on 100 M rows, non-blocking) + ANALYZE. Downstream cost = zero — the index also benefits every other orders(created_at) filter in the codebase.

SQL
Topic — optimization
SQL optimization drills

Practice →

SQL Topic — indexing Composite index and covering-index drills

Practice →

3. Snowflake QUERY_PROFILE

`snowflake query profile` — the operator stack, micro-partition pruning percent, and how local vs remote disk spill drives warehouse sizing

The mental model in one line: Snowflake's Query Profile presents the executed query as a stack of operator cards — TableScan, Filter, Aggregate, Join, Sort, WindowFunction — each annotated with bytes scanned, partitions total vs partitions scanned (pruning percent), row counts, and a spillage panel with local disk spill and remote disk spill in bytes; reading it top-down and hunting for low pruning percent plus non-zero remote spill is the entire diagnosis discipline. Once you can navigate the Snowsight Query Profile tab and its underlying QUERY_HISTORY view, you can size warehouses, spot bad table clustering, and rewrite joins in one pass.

Slot 1 — where to find the profile.

Snowsight → Activity → Query History → click into a query → Profile tab. The graphical, interactive view. Every operator is a clickable card with bytes scanned, rows produced, and % of total execution time.
SELECT * FROM SNOWFLAKE.ACCOUNT_USAGE.QUERY_HISTORY WHERE query_id = 'abc123'; — the underlying table. Available roughly 45 minutes to 3 hours delayed. Columns include bytes_scanned, partitions_scanned, partitions_total, bytes_spilled_to_local_storage, bytes_spilled_to_remote_storage, warehouse_size.
SELECT * FROM SNOWFLAKE.INFORMATION_SCHEMA.QUERY_HISTORY WHERE query_id = 'abc123'; — near-real-time (7 days retention). Same columns as ACCOUNT_USAGE but faster availability, shorter retention.
GET_QUERY_OPERATOR_STATS(query_id) — table function returning per-operator statistics. Machine-readable; use for programmatic profile analysis.
The SYSTEM$EXPLAIN_PLAN_JSON(query_text) function — returns the planned (pre-execution) plan as JSON. Useful when you can't afford to execute.
WEB_UI shortcut — every query in Snowsight has a "Query Profile" link that opens the interactive view directly.

Slot 2 — the operator card anatomy.

TableScan — the leaf. Reads from micro-partitions. Card shows Bytes scanned, Partitions total, Partitions scanned, and Pruning %. Pruning percent is the single most important number in every profile.
Filter — applies a WHERE predicate. Card shows input row count and output row count. The pushdown-into-scan is invisible on Snowflake — good filters are absorbed into the TableScan card.
JoinFilter — a Bloom filter used to skip micro-partitions on the probe side of a hash join. Presence means Snowflake is doing dynamic partition pruning — a huge win when it applies.
Join — hash join (default), sort-merge join (rare), broadcast join (for small side). Card shows Build side rows, Probe side rows, and the join algorithm.
Aggregate — hash-based GROUP BY. Card shows input rows, groups produced, bytes spilled (if any).
Sort — order-by operator. Card shows input rows, bytes spilled, sort keys.
WindowFunction — analytic function (ROW_NUMBER, RANK, SUM() OVER (...)). Card shows partitions, bytes spilled.
Result — the final projection. Card shows total rows returned, total bytes returned to client.
Broadcast / DistributeByHash — data-distribution nodes across the warehouse's cluster nodes. Presence of Broadcast usually means the planner detected a small side.
Execution time % — every card has a per-operator percent of total execution time. Sort operators by this — the hot operator is the fix target.

Slot 3 — micro-partition pruning percent (the single most important number).

Partitions total — how many micro-partitions the table has in total. Micro-partitions are ~50–500 MB chunks Snowflake maintains internally; you don't create or manage them.
Partitions scanned — how many were actually read. Ideally a tiny fraction of total.
Pruning % = 1 - partitions_scanned / partitions_total — the % of micro-partitions skipped by min/max pruning on the WHERE clause. 99%+ is great. 50% means half your storage cost. 0% means the filter was not sargable or the table wasn't clustered on the filter column.
How Snowflake prunes. For each micro-partition, it stores min and max of each column. A WHERE created_at >= '2026-06-10' predicate can eliminate any micro-partition whose max < '2026-06-10'. Pruning happens before scan — no bytes read from pruned partitions.
When pruning fails. Function-wrapped columns (DATE(created_at)), non-clustered columns, complex predicates the optimiser can't push down. Fix — rewrite the predicate to be sargable, or CLUSTER BY the table on the filter column.

Slot 4 — local vs remote disk spill and warehouse sizing.

Local disk spill = spill onto the warehouse node's SSD. Slower than RAM but manageable. bytes_spilled_to_local_storage non-zero means the warehouse ran out of memory but is still within its SSD budget.
Remote disk spill = spill onto Snowflake's remote storage (S3-backed). Slow — 10–100× slower than local spill. bytes_spilled_to_remote_storage non-zero is the strongest signal that the warehouse is too small.
Warehouse sizing rule of thumb. If a query has non-zero remote spill, escalate one warehouse tier (X-Small → Small → Medium → Large → X-Large → 2XL → 3XL → 4XL). Each tier is roughly 2× the compute and 2× the memory. Remote spill going to zero usually happens within one or two tier bumps.
Cost trade-off. Each tier bump doubles cost per minute. If a query is remote-spilling and you bump L → XL, you pay 2× per minute but the query may finish in 1/3 the time — net savings.
Auto-suspend and multi-cluster. Auto-suspend is orthogonal; it kills the warehouse after idle. Multi-cluster is for concurrency (multiple queries at once), not for individual query size.

Slot 5 — bytes scanned vs partitions scanned.

bytes_scanned — the compressed bytes Snowflake read from storage. Columnar reads mean only projected columns are counted.
partitions_scanned — a proxy for I/O. Each micro-partition is ~50–500 MB uncompressed.
bytes_written_to_result / bytes_written — outbound data.
Cost model on Snowflake — compute is time-billed (per second on the warehouse tier), not bytes-billed. So bytes_scanned is a diagnostic, not a bill line. The bill is warehouse tier × query wall clock.
Optimising bytes scanned — pick fewer columns (SELECT a, b, c not SELECT *), push predicates to prune, use CLUSTER BY on high-filter columns.

Slot 6 — reading a Snowflake profile top-down.

Move 1 — sort operators by execution time %. The hot operator is the one with 40%+ of total time.
Move 2 — check the hot operator's card. If it's a TableScan, check pruning %. If it's a Join, check build vs probe sizes. If it's an Aggregate, check spillage.
Move 3 — check spillage panel. Non-zero remote spill = warehouse too small. Non-zero local spill = borderline.
Move 4 — check bytes scanned across TableScans. If one TableScan reads 90% of total bytes, is it necessary? Can columns be projected away or partitions pruned?
Move 5 — name the fix. Cluster the table. Rewrite the filter. Bump warehouse. Add a materialised view. Rewrite the join.

Slot 7 — the eight most common Snowflake profile pathologies.

Pruning % = 0. Full-table scan. Either the filter isn't sargable or the table isn't clustered on the filter column. Fix — rewrite the filter (DATE(created_at) = '2026-07-01' → created_at BETWEEN '2026-07-01' AND '2026-07-02') or add CLUSTER BY (created_at).
Non-zero remote disk spill. Warehouse too small. Bump one tier.
JoinFilter not present. Snowflake didn't detect an opportunity for Bloom-filter dynamic pruning. Usually because one side is huge and the other is not filtered. Rewrite the query to filter the build side first.
Broadcast join with a large "small" side. The planner broadcast a table that isn't actually small. Verify with SELECT COUNT(*) FROM small_side — if it's over ~1M rows, force a shuffle join by hinting.
Cartesian join (accidental cross join). Card shows 10× the expected row count. Root cause is a missing join key. Search the query text for a missing ON clause.
Sort with high spill. Same as Postgres — either rewrite to sort less (add LIMIT, pre-aggregate), or bump warehouse.
Window function on a huge partition. ROW_NUMBER() OVER (PARTITION BY user_id) on a table where one user has 100 M rows spills catastrophically. Fix — pre-filter to relevant users, or handle skew explicitly.
QUERY_HISTORY delay hurts on-call. ACCOUNT_USAGE.QUERY_HISTORY can be up to 3 hours delayed. For real-time on-call, use INFORMATION_SCHEMA.QUERY_HISTORY (near-real-time, 7 days).

Slot 8 — interview probes on Snowflake plan reading.

"How do you read a Snowflake query profile?" — Open Snowsight Query History, click into the query, open the Profile tab. Sort operators by execution time %. Check hot operator's card (pruning %, spillage, join sides). Name fix.
"What is micro-partition pruning?" — Snowflake maintains min/max per column per micro-partition. WHERE col < X eliminates partitions with min > X. Pruning % tells you the effectiveness of the filter against the physical layout.
"Why is my query slow at page 1M on OFFSET pagination in Snowflake?" — Micro-partition pruning helps for the initial WHERE, but OFFSET's scan-and-discard is amplified by columnar reads. Switch to keyset (previous blog).
"When would you use CLUSTER BY?" — When a table is very large (multiple TB), has a well-defined filter column (usually date), and pruning % is low. Cost — Snowflake maintains clustering in the background; cost per TB per month is documented.
"How do you size a warehouse?" — Start Medium. Watch for remote spill. If zero, drop to Small; if non-zero, bump to Large. Iterate. Multi-cluster is orthogonal (for concurrency).

Common beginner mistakes

Reading the profile bottom-up (Postgres reflex) — Snowflake profiles read top-down.
Ignoring pruning % — a full-table scan at 0% pruning is silent unless you look at that number.
Confusing local and remote spill — remote is much more expensive.
Only bumping warehouse without checking pruning — often the fix is to rewrite the filter, not scale up.
Missing that SELECT * costs more than SELECT a, b, c — Snowflake is columnar; fewer projected columns = fewer bytes scanned.

Worked example — the TableScan with 0% pruning that a `CLUSTER BY` cures

Detailed explanation. The archetype: a fact table sliced by date_col with a WHERE date_col = X filter shows pruning % of 0. The table isn't clustered on date_col; the min/max of each micro-partition covers the whole date range, so no partition can be pruned. The fix is CLUSTER BY (date_col).

Question. Given the profile card below on a events table (200 TB, 800 M micro-partitions) with WHERE event_date = '2026-07-10', name the fix. Show the DDL and the expected pruning % after.

Input.

TableScan (events)
  Partitions total:     800,000
  Partitions scanned:   800,000
  Pruning %:            0.0%
  Bytes scanned:        180 GB
  Rows produced:        4,200 M
  Execution time %:     78%

Code.

-- Fix: cluster the table by event_date
ALTER TABLE events CLUSTER BY (event_date);

-- Snowflake reclusters in the background; monitor with:
SELECT SYSTEM$CLUSTERING_INFORMATION('events', '(event_date)');

-- After clustering completes (typically hours to days on 200 TB):
-- The same WHERE event_date = '2026-07-10' should show:
-- Pruning % ≈ 99.5%
-- Bytes scanned ≈ 900 MB
-- Wall clock ≈ 10× faster

Step-by-step explanation.

Current state — 800 K partitions, 800 K scanned = 0% pruning. Each micro-partition's event_date min/max range covers so many days that no partition can be eliminated by the filter.
ALTER TABLE ... CLUSTER BY starts a background reclustering job. Snowflake rewrites micro-partitions so event_date values are co-located. Cost — measured in credits per TB; documented in Snowflake pricing.
SYSTEM$CLUSTERING_INFORMATION returns a JSON describing the current clustering depth. average_depth close to 1 means fully clustered.
After clustering, WHERE event_date = '2026-07-10' prunes down to the ~4 K micro-partitions covering that day — pruning % ≈ 99.5%.
Rule of thumb — cluster on the column your dashboard filters on. Almost always a date or timestamp. Don't cluster on high-cardinality unrelated columns.

Output.

Before	After
Pruning % 0.0%	Pruning % 99.5%
Bytes scanned 180 GB	Bytes scanned 900 MB
Partitions scanned 800,000	Partitions scanned 4,000
Execution time 78% of total	Execution time 8% of total
Wall clock ~4 min	Wall clock ~25 s

Rule of thumb. Any TableScan with pruning % under 50% on a filter column is a clustering diagnosis. Cluster once, benefit every dashboard that filters on that column.

Worked example — the remote spill that a warehouse bump cures

Detailed explanation. A Sort or Aggregate operator with non-zero bytes_spilled_to_remote_storage is the strongest single signal that the warehouse is undersized. One tier bump usually zeroes it out.

Question. Given the profile card below on a Sort operator with 220 GB remote spill on a Medium warehouse, name the fix. Show the ALTER WAREHOUSE that would cure it.

Input.

Sort
  Sort keys:                        event_date, user_id
  Input rows:                       4,200,000,000
  Output rows:                      4,200,000,000
  Bytes spilled to local storage:   80 GB
  Bytes spilled to remote storage:  220 GB
  Execution time %:                 61%
Warehouse:  MEDIUM

Code.

-- Fix A — bump the warehouse for this run
ALTER WAREHOUSE analytics_wh SET WAREHOUSE_SIZE = 'LARGE';
-- Re-run query
-- Expected: local spill ~40 GB, remote spill 0

-- Fix B — rewrite the query to sort less (LIMIT, pre-aggregate)
SELECT user_id, event_date, MAX(event_ts) AS last_event
FROM events
GROUP BY user_id, event_date
ORDER BY event_date DESC, user_id
LIMIT 1000;
-- Sort now operates on ~2M aggregated rows, not 4.2B raw rows

-- Fix C — if this is a one-off, use a larger warehouse only for this session
USE WAREHOUSE big_wh;  -- pre-created 2XL
-- run query
USE WAREHOUSE analytics_wh;

Step-by-step explanation.

Current state — Medium warehouse, 220 GB remote spill on the Sort. Remote spill dominates wall clock; each byte spilled to remote storage is 10–100× slower than in-memory.
Fix A — bump to Large. Large has 2× memory of Medium (~256 GB vs 128 GB for Sort spill headroom). Remote spill should drop to zero; local spill may remain but is manageable.
Fix B — rewrite. If the query only needs top-N or pre-aggregated data, do that first. Sorting 4.2 B rows to keep 1000 is wasteful; pre-aggregate to 2 M rows, then sort.
Fix C — separate warehouse. If this is a nightly job that hits Large only once, keep the daytime warehouse Medium and switch to a bigger warehouse only for the nightly step.
Cost trade-off — Large is 2× Medium's per-second cost, but if wall clock drops from 30 min to 5 min, net cost drops.

Output.

Fix	Remote spill	Local spill	Wall clock	Cost
None	220 GB	80 GB	30 min	30 × 1 = 30 units
A: Large warehouse	0	40 GB	8 min	8 × 2 = 16 units
B: pre-aggregate	0	0	3 min	3 × 1 = 3 units
C: dedicated 2XL for the job	0	0	4 min	4 × 4 = 16 units

Rule of thumb. Prefer B (rewrite) > A (bump warehouse) > C (dedicated warehouse). Rewriting is usually the cheapest and most durable fix.

Worked example — the Broadcast join that broke on data growth

Detailed explanation. Snowflake's planner picks broadcast join when one side is estimated small enough to distribute to all warehouse nodes. If the small side grows (say from 100 K rows to 20 M rows) but stats haven't refreshed, the broadcast may fail or become catastrophic.

Question. Given the profile card below with a Broadcast join that's dominating wall clock, name the diagnosis and the fix.

Input.

Join (INNER)
  Type:           BROADCAST
  Build side:     dim_customer (20 M rows, 1.2 GB in memory)
  Probe side:     fact_orders (4.2 B rows)
  Execution time %: 55%
  Bytes spilled to remote: 8 GB

Code.

-- Diagnosis: dim_customer isn't small anymore; broadcasting 20M rows to every node is expensive.

-- Fix A — force a hash join instead of broadcast (use hint)
SELECT /*+ NO_BROADCAST(dim_customer) */
  o.id, o.total, c.name
FROM fact_orders o
JOIN dim_customer c ON c.id = o.customer_id
WHERE o.event_date = '2026-07-10';

-- Fix B — filter dim_customer down before the join
SELECT o.id, o.total, c.name
FROM fact_orders o
JOIN (
  SELECT id, name FROM dim_customer WHERE active_flag = TRUE
) c ON c.id = o.customer_id
WHERE o.event_date = '2026-07-10';

-- Fix C — cluster fact_orders by event_date so the probe side prunes first
ALTER TABLE fact_orders CLUSTER BY (event_date);

Step-by-step explanation.

Broadcast join built a hash on dim_customer (20 M rows, 1.2 GB) and shipped it to every warehouse node. Memory pressure caused spill.
Fix A — hint NO_BROADCAST on dim_customer forces a shuffle join; both sides are hash-partitioned by the join key. No broadcast means no per-node memory footprint.
Fix B — filter the dim table to only the active customers (say 4 M rows). The broadcast becomes cheap again.
Fix C — cluster the fact table so the WHERE event_date prunes probe-side partitions first. Combined with (A) or (B), the query is near-optimal.
Interview signal — knowing when Snowflake picks broadcast vs shuffle join and how to force each is a senior-level probe.

Output.

Fix	Join type	Remote spill	Wall clock
None	Broadcast	8 GB	4 min
A: NO_BROADCAST	Hash / Shuffle	0	2 min
B: filter dim	Broadcast (smaller)	0	90 s
B + C	Broadcast + probe pruning	0	45 s

Rule of thumb. Broadcast join is optimal for genuinely small dim tables (< 10 K rows). When the "small" side grows past ~1 M rows, force shuffle join or filter the dim first.

`snowflake query profile` interview question on plan reading and warehouse sizing

A senior interviewer often asks: "A finance dashboard on Snowflake takes 8 minutes to load. You're the on-call DE. Walk me through the Query Profile — what do you look at first, what's the ordered checklist of fixes, and how do you decide when to bump the warehouse vs when to rewrite the query?"

Solution Using the top-down profile checklist with pruning, spill, and operator time %

-- Step 1 — find the query and its query_id
SELECT query_id, query_text, execution_time / 1000 AS seconds
FROM SNOWFLAKE.INFORMATION_SCHEMA.QUERY_HISTORY
WHERE start_time > DATEADD(hour, -1, CURRENT_TIMESTAMP)
ORDER BY execution_time DESC
LIMIT 20;

-- Step 2 — pull the per-operator stats
SELECT *
FROM TABLE(GET_QUERY_OPERATOR_STATS('abc123-def456'))
ORDER BY execution_time_breakdown_overall_percentage DESC;

-- Step 3 — check pruning on TableScans
SELECT
  operator_type,
  operator_attributes:table_name::string AS table_name,
  operator_statistics:pruning:partitions_scanned::int AS scanned,
  operator_statistics:pruning:partitions_total::int AS total_partitions,
  1 - (operator_statistics:pruning:partitions_scanned::float
       / operator_statistics:pruning:partitions_total::float) AS pruning_pct
FROM TABLE(GET_QUERY_OPERATOR_STATS('abc123-def456'))
WHERE operator_type = 'TableScan';

-- Step 4 — check spillage
SELECT
  bytes_spilled_to_local_storage,
  bytes_spilled_to_remote_storage,
  warehouse_size
FROM SNOWFLAKE.INFORMATION_SCHEMA.QUERY_HISTORY
WHERE query_id = 'abc123-def456';

Step-by-step trace.

Step	Signal	Action
1	Find query_id from QUERY_HISTORY	Isolate the slow query
2	List operators by execution time %	Identify the hot operator (usually TableScan or Sort/Join)
3	Check pruning % on all TableScans	If any < 50%, cluster or rewrite filter
4	Check remote spill in QUERY_HISTORY	If non-zero, bump warehouse OR rewrite
5	Check Broadcast joins on non-small tables	Force NO_BROADCAST hint or filter first
6	Verify `SELECT *` isn't the culprit	Project only needed columns
7	Re-run and diff	Confirm expected drop in wall clock

Output:

Signal	Root cause	One-line fix
Pruning % < 50% on filter column	Table not clustered	`ALTER TABLE t CLUSTER BY (col)`
Remote spill > 0	Warehouse too small OR bad plan	Bump 1 tier OR rewrite
Broadcast join with build > 1M rows	Stale stats or growing dim	`/+ NO_BROADCAST(t) /` hint
`SELECT *` on 100-column table	Excess columnar reads	Project only needed columns
Sort spilling on top-N	No LIMIT push-down	Add explicit LIMIT + pre-aggregate

Why this works — concept by concept:

Top-down operator time % — Snowflake profiles are naturally ordered by execution time %. The hot operator jumps to the top; you don't have to hunt for it. Read from there.
Pruning % is the pre-scan lever — every byte you don't scan is free. Clustering shifts the pruning % from 0 to 99% for the filter column. That single move often 10×'s dashboard queries.
Remote spill = memory pressure — remote is 10–100× slower than local spill. Zero remote spill is the sizing target. When bumping a warehouse takes remote spill to zero, you've found the right tier.
Rewrite > scale, when possible — rewriting (adding LIMIT, pre-aggregating, filtering dims first) is durable and doesn't inflate cost per query. Bumping warehouse is fast but pays 2× per tier.
Cost — profile diagnosis is $0 (a few metadata queries). Fix costs vary — clustering a 200 TB table takes credits to reclusters; bumping a warehouse for one query is fractional cost; rewriting the SQL is free but takes engineer-time. Net-cost usually favours rewrite > cluster > scale.

SQL
Topic — optimization
SQL optimization drills

Practice →

SQL Topic — aggregation Pre-aggregation and GROUP BY drills

Practice →

4. BigQuery execution graph

`bigquery execution plan` — the stage-based DAG, slot-milliseconds, shuffle bytes, `--dry_run` cost estimation, and `INFORMATION_SCHEMA.JOBS`

The mental model in one line: BigQuery's execution model is a directed acyclic graph of stages, where each stage is a set of parallel workers reading input, applying a piece of the query, and writing output that gets shuffled into the next stage; reading the graph is a matter of finding the hot stage (largest slot_ms), checking shuffle bytes (network cost), and reading the INFORMATION_SCHEMA.JOBS row for the query to correlate slot time against wait time. Once you can read the stage graph and the JOBS row, you can size reservations, spot skew, and translate cost per query into cost per week.

Slot 1 — the stage-based model.

Stages are the unit of parallel work. Each stage has an input (from disk or a previous stage's shuffle output), a compute step (filter, aggregate, join), and a shuffle output. Stages run in parallel wherever their inputs are ready.
Slots are the compute unit. Each stage runs on some number of slots (BigQuery's word for "worker threads"). On-demand pricing gives 2000 slots per project by default; flat-rate reservations pin exact slot counts.
Slot-milliseconds (slot_ms) — the fundamental unit of BigQuery compute cost. If a stage uses 100 slots for 4.2 seconds, that's 100 × 4200 = 420,000 slot_ms. The whole-query total_slot_ms is the sum across all stages.
Wall clock ≠ slot time. Wall clock is what the user waits. Slot time is total compute. A stage using 100 slots for 4.2 s has 420,000 slot_ms but only 4.2 s of wall clock. High parallelism means wall clock much less than slot time.

Slot 2 — reading the stage graph.

Console view — BigQuery UI → Query History → click query → Execution details tab. Shows the DAG with each stage's slot time, wait time, and shuffle bytes.
Stage columns. slot_ms, wait_ms, read_ms, compute_ms, write_ms, shuffle_output_bytes, records_read, records_written.
Skew signal. If a stage has slot_ms significantly larger than expected given records_read, the workers are unbalanced — some workers read 100× more than others. Skew is expensive.
Wait vs compute. wait_ms > compute_ms means the stage was queued waiting for slots (concurrency pressure). compute_ms > wait_ms means the query was compute-bound (worked flat out).
Hot stage. The stage with the largest slot_ms is the bottleneck. Optimising a stage with 200,000 slot_ms of a 220,000-total is worthwhile; optimising a 200-slot_ms stage isn't.

Slot 3 — --dry_run for cost estimation without execution.

bq query --dry_run --use_legacy_sql=false 'SELECT ...' — returns the estimated bytes processed without running the query.
--dry_run is free. No slots used, no bytes billed.
Estimate accuracy. Usually within 5% of actual on partition-pruned queries. Larger error on complex joins where the planner can't estimate output.
Use case — cost gates in CI. Wrap queries in a dry-run + assertion: fail if estimated bytes exceed a threshold. Catches accidental SELECT * on a 100 TB table before deployment.
Web console equivalent. The query editor shows "This query will process X TB when run" — same estimator.

Slot 4 — INFORMATION_SCHEMA.JOBS audit trail.

SELECT * FROM \region-us.INFORMATION_SCHEMA.JOBS WHERE job_id = 'abc'; — full metadata row per query.
Key columns. total_bytes_processed, total_slot_ms, total_bytes_billed, start_time, end_time, cache_hit, statement_type, query.
job_stages — nested array with per-stage details.
referenced_tables — array of tables the query touched. Useful for lineage.
INFORMATION_SCHEMA.JOBS_BY_ORGANIZATION — organisation-wide view (requires admin role).
Retention. ~180 days. Sufficient for weekly cost reviews.

Slot 5 — slot utilisation vs wait time (the CPU vs queue distinction).

Compute time = time slots actually worked. Green area in the console utilisation chart.
Wait time = time slots were assigned but blocked (usually on I/O or shuffle). Amber area.
Read time = time reading from disk / column storage. Grey area.
Write time = time writing shuffle output. Blue area.
Diagnosis reading. If wait time > compute time, the query is I/O-bound or shuffle-bound; a bigger reservation won't help. If compute time > wait time, more slots would speed it up (assuming you have them).

Slot 6 — on-demand vs flat-rate slot billing.

On-demand. $5 per TB scanned (US region). Bills by total_bytes_billed. Slots are elastic — up to 2000 per project. Best for spiky workloads.
Flat-rate reservations. Commit to X slots per month. Predictable bill. Queries wait in queue when they exceed the reservation. Best for consistent workloads.
BI Engine. In-memory cache for dashboarding queries. Reduces slot cost when the same query runs repeatedly.
Storage cost is separate. ~$20/TB/month for active storage; ~$10/TB/month for long-term (unchanged for 90 days).
Egress cost is separate. Charged per byte leaving GCP.

Slot 7 — reading the graph in five moves.

Move 1 — find the hot stage. Sort by slot_ms desc. The top stage is the bottleneck.
Move 2 — check hot stage's compute vs wait. Compute > wait = more slots would help. Wait > compute = I/O bound.
Move 3 — check shuffle bytes. Large shuffle between two stages means the intermediate result is big. Aggregate earlier if possible.
Move 4 — check records_read vs partition prune. If a stage read 1 T records but the table is 100 T records, pruning worked. If it read 100 T, pruning failed — check partition and cluster keys.
Move 5 — name the fix. Partition the table on the filter column, cluster on high-cardinality keys, rewrite SELECT *, use approximate aggregations for cardinality estimates.

Slot 8 — the eight most common BigQuery execution-graph pathologies.

Full-table scan on a partitioned table. Filter isn't partition-pruning-friendly. WHERE DATE(created_at) = '2026-07-10' on a table partitioned by DATE(created_at) prunes. WHERE EXTRACT(YEAR FROM created_at) = 2026 doesn't.
SELECT * on a wide table. Every column is read even if only three are used. Fix — project only what you need.
Skewed join. One key value has 100× more rows than others; one worker gets stuck. Fix — pre-aggregate, sample, or handle skew explicitly.
Cross join without predicate. The classic FROM a, b WHERE a.x = b.x typo. BigQuery aborts on huge cross joins.
Large shuffle between stages. The intermediate is bigger than either input. Sometimes fine (aggregation reduces later); usually a bad plan.
Cache miss on repeated dashboarding queries. BI Engine not configured. Fix — add a reservation with BI Engine.
On-demand querying a 100 TB table daily. $500/query. Fix — partition, cluster, materialised view, or move to flat-rate.
Slot wait time > compute time. Queue pressure. Fix — larger reservation, or run job in a less contended time window.

Slot 9 — interview probes on BigQuery plan reading.

"How do you read a BigQuery execution plan?" — Open Query History, click into query, open Execution details. Find hot stage by slot_ms. Check compute vs wait. Diagnose shuffle. Name fix.
"What's slot_ms?" — Slot × milliseconds. The fundamental compute unit. Total slot_ms = sum across all stages.
"When would you use --dry_run?" — For cost estimation before running expensive queries. Also in CI to gate PRs that would scan too much.
"How do you use INFORMATION_SCHEMA.JOBS?" — Query it for slow-query audits, cost attribution, or plan history. Correlates well with team dashboards.
"On-demand or flat-rate?" — On-demand for spiky ad-hoc analytics ($5/TB is fine when volume is modest). Flat-rate for consistent workloads where you can commit to steady slot usage.

Common beginner mistakes

Confusing slot_ms with wall clock — 100 slots × 4 s = 400 slot_s, but wall clock was 4 s.
Forgetting to filter on the partition column — full-table scan on a partitioned table costs the same as an unpartitioned table.
Using SELECT * in dashboards — every extra column is a byte scanned = a cent billed.
Ignoring shuffle bytes — the intermediate stages may be bigger than the input if the plan is bad.
Running expensive queries without --dry_run first.

Worked example — the full-table scan on a partitioned table

Detailed explanation. The most common BigQuery bill surprise: a table partitioned by DATE(created_at) but the query filter is WHERE EXTRACT(YEAR FROM created_at) = 2026 — the filter can't prune partitions because the planner can't inverse-map the function.

Question. Given the query below on a 100 TB events table partitioned by DATE(created_at), why does --dry_run estimate 100 TB? What's the fix?

Input.

SELECT user_id, COUNT(*) AS n
FROM `project.dataset.events`
WHERE EXTRACT(YEAR FROM created_at) = 2026
  AND EXTRACT(MONTH FROM created_at) = 7
GROUP BY user_id
LIMIT 100;

Code.

-- Diagnose
bq query --dry_run --use_legacy_sql=false '
SELECT user_id, COUNT(*) AS n
FROM `project.dataset.events`
WHERE EXTRACT(YEAR FROM created_at) = 2026
  AND EXTRACT(MONTH FROM created_at) = 7
GROUP BY user_id
LIMIT 100'
-- Output: This query will process 100 TB when run.

-- Fix: rewrite to a sargable partition filter
SELECT user_id, COUNT(*) AS n
FROM `project.dataset.events`
WHERE created_at >= '2026-07-01'
  AND created_at <  '2026-08-01'
GROUP BY user_id
LIMIT 100;
-- Dry run: This query will process 3 TB when run. (One month of 100 TB × 12.)

Step-by-step explanation.

Original query wraps created_at in EXTRACT(...). BigQuery's partition pruner can't invert the function to identify which partitions match — it falls back to full-table scan.
--dry_run reveals the damage: 100 TB estimated. At $5/TB on-demand, that's $500 per run.
Fix — rewrite the filter as a range on the raw column: WHERE created_at >= '2026-07-01' AND created_at < '2026-08-01'. The partition pruner can now identify exactly the July 2026 partitions.
Re-run --dry_run: 3 TB estimated. At $5/TB, that's $15 per run — 30× cheaper.
Rule of thumb — always filter partitioned tables on the raw partition column with >= / < / BETWEEN. Never wrap in a function.

Output.

Query	Dry-run bytes	On-demand cost
`WHERE EXTRACT(YEAR FROM ...)`	100 TB	$500
`WHERE created_at BETWEEN ...`	3 TB	$15
`WHERE created_at BETWEEN ... AND cluster_col = X`	300 GB	$1.50

Rule of thumb. Always --dry_run before running big-table queries. Always filter on the partition column with sargable predicates. Cluster the table on the next-most-common filter column for even more pruning.

Worked example — the skewed join that dominates one stage

Detailed explanation. A join where one key value has 100× more rows than others creates a skewed stage — one worker gets stuck processing the hot key while others finish quickly. Symptom: one stage's slot_ms is much larger than expected.

Question. Given the execution graph fragment below showing a skewed join stage, name the diagnosis and the fix.

Input.

Stage 3 — HashJoin (fact_events × dim_user)
  Records read:     4,200,000,000
  Records written:  4,200,000,000
  slot_ms:          82,000,000
  wait_ms:          400
  compute_ms:       410,000
  Shuffle output:   660 GB
  Skew warning:     True — user_id=SYSTEM has 800M rows

Code.

-- Fix A — filter the hot key out (if valid)
WITH filtered_events AS (
  SELECT *
  FROM fact_events
  WHERE user_id != 'SYSTEM'    -- exclude the system pseudo-user
    AND created_at BETWEEN '2026-07-01' AND '2026-07-31'
),
system_events AS (
  SELECT *
  FROM fact_events
  WHERE user_id = 'SYSTEM'
    AND created_at BETWEEN '2026-07-01' AND '2026-07-31'
)
SELECT ... FROM filtered_events JOIN dim_user USING (user_id)
UNION ALL
SELECT ... FROM system_events   -- handle SYSTEM separately

-- Fix B — hash-partition the join key with a salt
SELECT ...
FROM fact_events e
JOIN dim_user u ON u.user_id = e.user_id
  AND MOD(e.hash_salt, 10) = MOD(u.hash_salt, 10)

Step-by-step explanation.

Stage 3 shows 82 M slot_ms — the dominant stage. Records read is 4.2 B, but the skew warning notes one user_id (SYSTEM) accounts for 800 M rows.
In a hash join, all rows with the same key hash to the same worker. SYSTEM sends 800 M rows to one worker while 100+ others finish quickly.
Fix A — split the hot key. Query the SYSTEM subset separately, then UNION ALL with the non-SYSTEM subset. Each subset's join is balanced.
Fix B — salt the join. Add a hash-salt column that varies within the hot key, so the 800 M rows spread across 10 workers instead of 1. Requires schema change to the dim table.
Interview signal — recognising skew from the execution graph and naming a mitigation is a senior-level probe.

Output.

Fix	slot_ms	Wall clock
None	82,000,000	8 min
A: filter hot key	42,000,000	4 min
B: salted key	44,000,000	3 min
A + partition prune	8,000,000	45 s

Rule of thumb. Any stage with slot_ms >> average is a skew candidate. Fix by isolating the hot key or salting it.

Worked example — moving from on-demand to a reservation for cost savings

Detailed explanation. A team running 50 similar dashboards on-demand at $5/TB every hour discovers they're paying $30,000/month in BigQuery. A flat-rate reservation can cap the bill while providing the same throughput — often 40% cheaper.

Question. Given the monthly usage below, decide whether to stay on-demand or switch to flat-rate. What size reservation would break even?

Input.

Monthly total bytes billed: 6 PB
Monthly on-demand cost:      $30,000  ($5/TB * 6,000 TB)
Average slot_ms/day:         2.1 * 10^10  (typical 400 slots utilized on avg)
Peak slots concurrent:       ~1200 (short bursts)
Off-peak:                    ~200
Query cadence:               15,000 queries/day

Code.

-- Estimate flat-rate cost at various commitments
-- Assumes $2000/mo per 100 slots (approximate US pricing; verify current)
-- 500 slots  = $10,000/mo — throttled at peaks
-- 700 slots  = $14,000/mo — few queue stalls
-- 1000 slots = $20,000/mo — covers 95% peaks
-- 1200 slots = $24,000/mo — covers 99% peaks
-- Autoscale up to 500 flex slots = ~$1000/mo extra

Step-by-step explanation.

On-demand at $30 K/month; queries run at up to 1200 slots peak but average 400 slots.
A 700-slot flat-rate reservation ($14 K/month) covers the average with headroom. Queries can queue during peaks — usually OK for dashboards.
Add 500 flex-slot autoscaling ($1 K/month typical) to cover the top 5% of peak-slot-demand queries without queueing.
Total ≈ $15 K/month vs $30 K on-demand — 50% cheaper for the same throughput.
Trade-off — reservations require a monthly commitment; on-demand is elastic. If usage drops 50% next quarter, on-demand shrinks with it; flat-rate doesn't.

Output.

Model	Slots	Cost / month	Notes
On-demand	2000 (elastic)	$30,000	Elastic; pays per TB scanned
Flat 500 slots	500	$10,000	Queues at peaks
Flat 700 + 500 flex	700+500	$15,000	Sweet spot
Flat 1200 slots	1200	$24,000	Almost no queueing

Rule of thumb. Flat-rate wins when average slot utilisation is above ~30% of peak. On-demand wins for spiky ad-hoc analytics. INFORMATION_SCHEMA.JOBS gives you the raw material to model this.

`bigquery execution plan` interview question on cost + performance tuning

A senior interviewer often asks: "A dashboard on a 200 TB partitioned + clustered events table is slow and expensive — 4 minutes per load, $8 per query, running every hour. Walk me through your diagnosis using the execution graph and INFORMATION_SCHEMA.JOBS, and name the top three fixes ranked by ROI."

Solution Using stage-graph diagnosis + partition pruning + materialised view

-- Step 1 — pull job metadata
SELECT
  job_id,
  total_bytes_processed / POW(1024, 4) AS tb_scanned,
  total_slot_ms,
  end_time - start_time AS wall_clock
FROM `region-us`.INFORMATION_SCHEMA.JOBS_BY_PROJECT
WHERE job_id = 'abc123';

-- Step 2 — read the stage graph via query_plan (JSON)
SELECT
  step.name        AS stage_name,
  step.slot_ms     AS stage_slot_ms,
  step.records_read AS records_read,
  step.shuffle_output_bytes AS shuffle_bytes
FROM `region-us`.INFORMATION_SCHEMA.JOBS_BY_PROJECT,
  UNNEST(job_stages) AS step
WHERE job_id = 'abc123'
ORDER BY step.slot_ms DESC;

-- Fix A — narrow the partition filter
-- Was: WHERE created_at >= DATE_SUB(CURRENT_DATE(), INTERVAL 90 DAY)
-- Now: WHERE created_at >= DATE_SUB(CURRENT_DATE(), INTERVAL 7 DAY)
-- Effect: 200TB → 15TB scanned per query

-- Fix B — pre-aggregate into a materialised view
CREATE MATERIALIZED VIEW `project.dataset.mv_daily_events` AS
SELECT
  DATE(created_at) AS event_date,
  user_id,
  COUNT(*) AS event_count,
  SUM(revenue) AS revenue
FROM `project.dataset.events`
GROUP BY event_date, user_id;

-- Dashboard now queries the MV, scanning ~10 MB per day per user instead of raw events

-- Fix C — add BI Engine to the reservation
-- (Console → BigQuery → Reservations → Add BI Engine capacity — e.g. 20 GB)
-- Cached repeat queries served in milliseconds

Step-by-step trace.

Fix	Bytes scanned	Slot_ms	Wall clock	Cost / query
Baseline (90-day filter, raw events)	200 TB	15 * 10^10	4 min	$1,000
A — narrow to 7 days	15 TB	1.3 * 10^10	45 s	$75
A + B — MV pre-aggregation	300 MB	1 * 10^8	3 s	$0.02
A + B + C — BI Engine cache	300 MB (first) / 0 (cache)	~0	200 ms	$0 (cache hit)

Output:

Metric	Baseline	Optimised
Wall clock	4 min	200 ms (cache) / 3 s (miss)
Bytes scanned	200 TB	300 MB
Cost / query	$1,000	$0.02
Hourly cost	$1,000/hr	$0.02/hr
Weekly cost	$168,000/wk	$3.36/wk

Why this works — concept by concept:

Partition pruning by narrowing the filter window — most dashboards actually only need the last 7 or 30 days. Narrowing from 90 days to 7 days is a mechanical 13× reduction in bytes scanned. Verify with --dry_run.
Materialised view for pre-aggregation — dashboards summarise the raw data. Pre-summarise once, query many times. Automatic incremental refresh keeps the MV current. Bytes scanned per dashboard load drops from TB to MB.
BI Engine cache — an in-memory cache for repeat queries. Sub-second latency; zero bytes scanned on cache hit. Perfect for dashboards where the same query runs every hour.
INFORMATION_SCHEMA.JOBS for cost attribution — the metadata table lets you attribute per-team, per-project, and per-user costs. Weekly review of the top 20 queries by total_bytes_processed finds the biggest optimisation targets.
Cost — Diagnosis: $0 (metadata queries). Fixes: MV creation is a one-off compute cost (~$100 for 200 TB); BI Engine adds a fixed monthly cost (~$100 for 20 GB). Ongoing: 50,000× cheaper. ROI in hours.

SQL
Topic — optimization
SQL optimization drills

Practice →

SQL Topic — window functions Window function and aggregation drills

Practice →

5. Dialect matrix + 5-step reading strategy

`read execution plan` across four engines — Postgres, Snowflake, BigQuery, SQL Server — one query, four artefacts, one senior reading strategy

The mental model in one line: the same conceptual query (join, filter, aggregate, sort, limit) compiles to a nested plan tree on Postgres, an operator stack on Snowflake, a stage DAG on BigQuery, and an XML plan graph on SQL Server; the artefacts look different but the reading discipline is the same — find the hot node, check the estimate-vs-actual, check for spill or shuffle, check parallelism, name the fix. Learn the five-step discipline once; apply it everywhere.

Slot 1 — the dialect matrix.

Engine	EXPLAIN keyword	Profile artefact	Cost surface
Postgres	`EXPLAIN (ANALYZE, BUFFERS, VERBOSE, FORMAT TEXT/JSON)`	Nested plan tree with per-node cost / rows / actual / loops / buffers	pg_stat_statements, log_min_duration_statement
Snowflake	`EXPLAIN` (plan only) + Query Profile UI (actual)	Operator stack with pruning % + spill panel	QUERY_HISTORY, ACCOUNT_USAGE, GET_QUERY_OPERATOR_STATS
BigQuery	`bq query --dry_run` + Execution details tab	Stage DAG with slot_ms + shuffle_bytes	INFORMATION_SCHEMA.JOBS, dry_run cost estimator
SQL Server	`SET STATISTICS TIME/IO ON` + Actual Execution Plan (XML)	Graphical plan in SSMS + XML	Query Store, DMVs (sys.dm_exec_query_stats)

Slot 2 — SQL Server-specific plan reading.

SET STATISTICS TIME ON — reports CPU and elapsed time per statement.
SET STATISTICS IO ON — reports logical reads (from cache) and physical reads (from disk) per table.
Actual Execution Plan (Ctrl+M in SSMS) — the graphical plan tree. Nodes are labelled with cost %, actual row count, and estimated row count.
Estimated vs Actual Plan — Estimated is planning-only (no execution); Actual is post-execution. Prefer Actual for diagnosis.
Plan XML — the underlying representation. Machine-readable; can be diffed programmatically.
Query Store — SQL Server's built-in query performance history. Similar to pg_stat_statements but with plan diffs and forced-plan capability.
DMVs — sys.dm_exec_query_stats, sys.dm_exec_query_plan, sys.dm_os_wait_stats. Real-time perf data.
Missing Index DMVs — sys.dm_db_missing_index_details — SQL Server's suggestion engine. Use as a starting point, not a truth.
Common operators. Index Seek, Clustered Index Scan, Hash Match, Nested Loops, Sort, Stream Aggregate, Table Spool.
Cost estimate. Each operator has a "cost of subtree" %. The percentages sum to 100%; the operator with the highest % is the bottleneck.

Slot 3 — the same query across four engines.

-- Same conceptual query on all four engines
SELECT
  customer_id,
  SUM(total) AS lifetime_value
FROM orders
WHERE created_at >= '2026-06-10'
GROUP BY customer_id
ORDER BY lifetime_value DESC
LIMIT 100;

Each engine shows this differently:

Postgres — nested tree: Limit → Sort → HashAggregate → Bitmap Heap Scan on orders.
Snowflake — operator stack: Result → Sort → Aggregate → Filter → TableScan(orders).
BigQuery — 3-stage DAG: Stage 1 (read + filter) → Shuffle → Stage 2 (aggregate + sort) → Stage 3 (limit + output).
SQL Server — XML plan tree with Limit → Sort → Hash Match (Aggregate) → Index Seek on orders(created_at).

Slot 4 — the 5-step senior read strategy (applies to all four engines).

Step 1 — find the hot node. The one with the largest fraction of total wall clock (or slot_ms on BigQuery). Every profile UI sorts by this.
Step 2 — check estimate vs actual. Every plan has an estimate and an actual row count per node. If they diverge by 10× or more, the optimiser was wrong — refresh stats or add extended stats.
Step 3 — check spill or shuffle. Postgres Buffers: temp read. Snowflake bytes_spilled_to_remote_storage. BigQuery shuffle_bytes (large intermediate). SQL Server Sort Warning in the XML plan. Any of these is a specific fix.
Step 4 — check parallelism. Postgres Workers Launched. Snowflake warehouse size. BigQuery slot utilisation vs wait time. SQL Server MAXDOP setting. If parallelism is disabled or unused, decide whether it should be.
Step 5 — fix the plan. Add index, run ANALYZE, rewrite join, add hint, bump memory, cluster the table, pre-aggregate. The hot-node diagnosis names the one-line fix.

Slot 5 — hints and last-resort tools.

Postgres — pg_hint_plan. An extension for query hints. /*+ SeqScan(t) */ forces seq scan. Use rarely — hints lock the plan against future data shifts.
Snowflake — NO_BROADCAST, BROADCAST, MATERIALIZED_VIEW_HINT. Fewer hints than other engines; the optimiser is intentionally opaque.
BigQuery — no hints per se. Rewrite the SQL to force a plan shape. Use materialised views or authorised views for repeated shapes.
SQL Server — many hints. WITH (NOLOCK), OPTION (HASH JOIN), OPTION (MAXDOP 4). Powerful but easily abused.
Rule. Hints are a last resort. First — refresh stats. Second — rewrite. Third — index. Fourth — hint (and add a comment explaining the hint's justification for the next maintainer).

Slot 6 — cost per query (bill-line thinking).

Postgres self-hosted. Cost = servers × hours. Query cost = fraction of server-time. Diagnose to keep utilisation reasonable; scale up when needed.
Postgres cloud (RDS, Aurora). Cost = instance-hours + IO. Bad queries can inflate IO — check pg_stat_statements and diff top-N by total time.
Snowflake. Cost = warehouse-tier × query-wall-clock. Bad plans = long wall clock = big bill. Optimise for shorter wall clock at a given tier.
BigQuery on-demand. Cost = TB scanned × $5. Bad plans = full-table scans = big bill. Optimise for pruned partitions and smart clustering.
BigQuery flat-rate. Cost = slot commitment × month. Bad plans = queue waits or overrun. Optimise for slot efficiency (compute > wait).
SQL Server. Cost = server + license. Query cost = share of server; long queries block others via lock contention. Diagnose lock waits with DMVs.

Slot 7 — reading discipline vs tuning discipline.

Reading. Every plan artefact you can read is a chance to catch a regression before it ships. Add plan diffs to code review. Compare prod vs staging plans for the same query. Set alerts on pg_stat_statements for queries whose mean-time doubles.
Tuning. Once the read names the fix, apply the fix. Prefer stats refresh (cheap) → query rewrite (durable) → index (durable, mild write cost) → materialised view (durable, storage cost) → warehouse bump (fast, expensive). Hints are last.

Slot 8 — the SQL Server actual execution plan XML example.

-- SQL Server: turn on both flags
SET STATISTICS TIME ON;
SET STATISTICS IO ON;

-- Include Actual Execution Plan (Ctrl+M) then run:
SELECT customer_id, SUM(total) AS lifetime_value
FROM orders
WHERE created_at >= '2026-06-10'
GROUP BY customer_id
ORDER BY lifetime_value DESC
OFFSET 0 ROWS FETCH NEXT 100 ROWS ONLY;

-- Right-click the plan → Show Execution Plan XML → paste into a diff tool

-- Turn off after diagnosis
SET STATISTICS TIME OFF;
SET STATISTICS IO OFF;

Common beginner mistakes

Reading only one engine's plan without recognising the other three shapes.
Confusing Snowflake's operator stack (top-down) with Postgres's plan tree (bottom-up).
Forgetting that BigQuery slot_ms ≠ wall clock — a 100-slot query is fast relative to slot time.
Skipping the estimate-vs-actual check — the single most reliable signal of a bad plan.
Bumping resources first — always diagnose the plan first; scale is the last resort.

Worked example — reading the same query on all four engines

Detailed explanation. The clearest way to lock in the mental model: read the same query's plan on Postgres, Snowflake, BigQuery, and SQL Server, and identify the equivalent nodes.

Question. Given the query and the plan artefacts from each engine, map each engine's nodes to the conceptual operators (Filter, GroupBy, Sort, Limit).

Input.

-- The query
SELECT customer_id, SUM(total) AS lifetime_value
FROM orders
WHERE created_at >= '2026-06-10'
GROUP BY customer_id
ORDER BY lifetime_value DESC
LIMIT 100;

Code. (No code — a mapping exercise.)

Conceptual operator | Postgres node        | Snowflake operator | BigQuery stage | SQL Server node
Read                | Bitmap/Index Scan    | TableScan          | Stage 1 input  | Index Seek
Filter              | Recheck Cond / Filter | Filter / Scan     | Stage 1 filter | (subsumed)
GroupBy             | HashAggregate        | Aggregate          | Stage 2 agg    | Hash Match (Aggregate)
Sort                | Sort                 | Sort               | Stage 2 sort   | Sort
Limit               | Limit                | Result             | Stage 3 output | Top

Step-by-step explanation.

Read — every engine has a scan. Postgres shows the physical access method (Bitmap / Index / Seq); Snowflake shows partitions scanned; BigQuery shows records_read per stage; SQL Server shows the seek predicate.
Filter — subsumed into scan on most engines when the predicate is sargable (index-lookup key on Postgres / SQL Server; partition prune on BigQuery; pruning % on Snowflake).
GroupBy — always a hash aggregate for high-cardinality group keys. Postgres and SQL Server can also pick sort-based aggregation when input is already sorted.
Sort — the ORDER BY SUM(total) DESC requires sorting after aggregation. Postgres and SQL Server may pick top-N heapsort when combined with LIMIT.
Limit — the top node. Postgres Limit, Snowflake Result (subsumes limit), BigQuery output stage, SQL Server Top.

Output. The mapping in the table above. Learn one column deeply; the others come naturally once you can read one.

Rule of thumb. Read one engine deeply first (usually Postgres because the syntax is most explicit). Then translate to Snowflake, BigQuery, SQL Server as you encounter them. The five-step reading discipline is invariant across all four.

Worked example — the same query optimised differently on each engine

Detailed explanation. Optimising the same conceptual query on each engine involves engine-specific levers — Postgres wants indexes, Snowflake wants clustering, BigQuery wants partition + cluster keys, SQL Server wants an index seek plus a covering include column.

Question. For the query above, give the one-line optimisation for each engine.

Input.

-- The query
SELECT customer_id, SUM(total) AS lifetime_value
FROM orders
WHERE created_at >= '2026-06-10'
GROUP BY customer_id
ORDER BY lifetime_value DESC
LIMIT 100;

Code.

-- Postgres
CREATE INDEX CONCURRENTLY idx_orders_created_customer_total
  ON orders (created_at, customer_id) INCLUDE (total);

-- Snowflake
ALTER TABLE orders CLUSTER BY (created_at);

-- BigQuery
CREATE OR REPLACE TABLE `project.dataset.orders`
PARTITION BY DATE(created_at)
CLUSTER BY customer_id
AS SELECT * FROM `project.dataset.orders`;

-- SQL Server
CREATE NONCLUSTERED INDEX idx_orders_created_at
  ON orders (created_at)
  INCLUDE (customer_id, total);

Step-by-step explanation.

Postgres — composite index on (created_at, customer_id) with INCLUDE (total) covers the filter, the group key, and the aggregate — index-only scan possible.
Snowflake — cluster by created_at so the WHERE filter prunes micro-partitions. SELECTing only three columns already benefits from columnar storage.
BigQuery — partition by DATE(created_at) so the filter prunes date partitions. Cluster by customer_id so the group-by benefits from block-level locality.
SQL Server — nonclustered index on created_at with INCLUDE (customer_id, total) — covers the filter and projects the two needed columns.
All four fixes achieve the same shape: a small scan + hash aggregate + sort + limit. Wall clock drops from tens of seconds to sub-second.

Output.

Engine	Before	After	Speed-up
Postgres	Seq Scan / 1.2 s	Index Only Scan / 15 ms	80×
Snowflake	0% prune / 45 s	99% prune / 4 s	11×
BigQuery	Full-scan / 200 TB	Partition + cluster / 2 GB	100,000× cost
SQL Server	Clustered Index Scan / 2.5 s	Index Seek + Include / 30 ms	83×

Rule of thumb. Every engine has an equivalent "cover the query with the index" pattern. Learn the syntax for each; the concept is the same.

`execution plan analysis` interview question on cross-engine plan reading

A senior interviewer often asks: "Design a runbook a mid-level engineer can follow to read execution plans across Postgres, Snowflake, BigQuery, and SQL Server. What are the five steps, and what's the one-line signal on each engine that tells you 'this plan is broken'?"

Solution Using the invariant 5-step read strategy plus a per-engine cheat card

STEP 1 — find the hot node/stage.
  Postgres:  sort plan by actual_time; deepest actual_time × loops = hot node.
  Snowflake: sort by execution_time_breakdown_overall_percentage; top card = hot.
  BigQuery:  sort by slot_ms; largest stage = hot.
  SQL Server: sort by cost of subtree %; highest = hot.

STEP 2 — check estimate vs actual.
  Postgres:  rows= vs actual rows= per node. Gap ≥ 10× is bad.
  Snowflake: input rows vs output rows on operators. Compare to statistics.
  BigQuery:  records_read vs estimated records. INFO_SCHEMA.JOBS has both.
  SQL Server: Estimated Number of Rows vs Actual Number of Rows on each node.

STEP 3 — check spill or shuffle.
  Postgres:  BUFFERS: temp read non-zero; Sort Method: external merge.
  Snowflake: bytes_spilled_to_remote_storage non-zero.
  BigQuery:  large shuffle_output_bytes between stages.
  SQL Server: Sort Warning; Hash Warning; TempDB usage.

STEP 4 — check parallelism.
  Postgres:  Workers Launched. Should be > 0 on big scans.
  Snowflake: warehouse size and MULTI_STATEMENT_COUNT.
  BigQuery:  slot count and wait_ms.
  SQL Server: MAXDOP and Parallelism operator in the plan.

STEP 5 — fix the plan.
  Postgres:  ANALYZE; add index; rewrite join; SET work_mem session.
  Snowflake: CLUSTER BY; rewrite filter; bump warehouse tier.
  BigQuery:  partition + cluster; materialised view; rewrite SELECT.
  SQL Server: rebuild statistics; CREATE INDEX; Query Store forced plan.

BROKEN-PLAN one-line signals (per engine):
  Postgres:  "Rows Removed by Filter: 99000000" — missing index.
  Snowflake: "Partitions scanned = Partitions total" — no clustering / bad filter.
  BigQuery:  "Full scan on partitioned table" — filter not sargable.
  SQL Server: "Clustered Index Scan on 100M-row table with cost 98%" — missing index.

Step-by-step trace.

Step	Postgres	Snowflake	BigQuery	SQL Server
1 hot	actual_time × loops	exec time %	slot_ms per stage	cost of subtree %
2 est vs act	rows vs actual rows	input vs output rows	records_read	Est vs Actual rows
3 spill	temp read	remote spill bytes	large shuffle	Sort/Hash warning
4 parallel	Workers Launched	WH size	slot count vs wait	MAXDOP + parallel op
5 fix	ANALYZE + INDEX	CLUSTER + rewrite	PARTITION + MV	REBUILD + INDEX

The runbook is invariant across all four engines. The signals differ; the discipline is the same.

Output:

Engine	Fastest one-signal diagnosis
Postgres	Any `Seq Scan` with `Rows Removed by Filter` > 1M
Snowflake	Any TableScan with pruning < 50% on a filter column
BigQuery	Any query where `dry_run` bytes > 10 TB on a partitioned table
SQL Server	Any plan node with cost of subtree > 80% and no index seek

Why this works — concept by concept:

Invariant discipline — every plan diagnosis reduces to five steps: hot node, estimate vs actual, spill/shuffle, parallelism, fix. The signals differ per engine but the mental model is one.
One-signal diagnosis — for every engine, there's a single reading that catches the most common plan pathology. Teaching a mid-level engineer that signal gives them 80% of senior plan-reading skill for that engine.
Runbook portability — an engineer who learns the runbook on Postgres can port it to Snowflake or BigQuery in a day. The mental model is the durable asset; syntax is trivia.
Escalation path — cheap fixes first (refresh stats), durable fixes second (index, cluster, partition), expensive fixes last (scale up, hint, materialise). Applies universally.
Cost — the runbook itself is a one-page cheat sheet; teaching it takes an hour. Ongoing cost is engineer-time saved on diagnosis (typically 30–60 min per incident reduced to 5).

SQL
Topic — optimization
SQL optimization drills

Practice →

SQL
Topic — joins
SQL join optimisation drills

Practice →

Cheat sheet — plan-reading recipe list

Bottom-up reading rule. On Postgres and SQL Server, always read the plan tree bottom-up — leaf scans first, Limit last. On Snowflake and BigQuery, sort operators or stages by execution time % / slot_ms and read the top.
Cost vs actual time. cost= is planner units (Postgres, SQL Server), not milliseconds. actual time= is wall clock (ms). Never quote cost to non-DBA humans.
The four numbers per Postgres node. cost (startup..total), rows (estimate), actual time (first..last), loops. Wall-clock cost = actual time × loops.
Estimate vs actual gap. Any node with rows off by 10×+ from actual rows is a bad-stats or bad-plan signal. Fix — ANALYZE, add extended statistics, or force the plan.
Rows Removed by Filter. Any Postgres Seq Scan with Rows Removed by Filter in the millions is a missing-index diagnosis. Add the index.
BUFFERS: temp read. Non-zero on Postgres = spilling. Fix — rewrite (LIMIT, pre-aggregate) or SET work_mem per session.
Sort Method. Postgres — quicksort (in-mem), top-N heapsort (bounded), external merge (spilled). external merge = spill = fix.
Batches: N > 1. Postgres Hash node with N > 1 = spilling. Fix — bump work_mem or filter build side down.
Workers Launched: 0. Postgres parallelism disabled. Check max_parallel_workers_per_gather and min_parallel_table_scan_size.
Nested Loop on two large sides. Bad plan almost always. Fix — refresh stats or force Hash Join (last resort).
Snowflake pruning %. The single most important number. 99% = great. Under 50% = fix the filter or CLUSTER BY.
Snowflake remote spill. Non-zero bytes_spilled_to_remote_storage = warehouse too small OR bad plan. Bump one tier or rewrite.
Snowflake local spill. Non-zero bytes_spilled_to_local_storage but zero remote = borderline. Not urgent.
Snowflake SELECT *. Costs more than SELECT a, b, c — columnar storage. Project only needed columns.
Snowflake broadcast join. Fine for < 10 K rows on the build side. Over ~1 M, force NO_BROADCAST or filter first.
BigQuery slot_ms. Fundamental compute cost unit. Sort stages by slot_ms desc to find the hot stage.
BigQuery --dry_run. Free bytes-processed estimate. Always run before expensive queries. Gate PRs in CI on estimated bytes.
BigQuery partition pruning. Filter on the raw partition column with >= / BETWEEN. Never wrap in a function (EXTRACT, DATE).
BigQuery clustering. Cluster on high-cardinality filter columns after partition. Block-level pruning inside each partition.
BigQuery materialised view. Pre-aggregate for dashboards. Automatic incremental refresh. Bytes scanned drops 1000×+.
BigQuery on-demand vs flat-rate. On-demand ($5/TB) for spiky ad-hoc. Flat-rate (per-slot commitment) for consistent workloads.
SQL Server SET STATISTICS TIME/IO ON. Reports CPU time, elapsed time, logical reads, physical reads. Turn on for diagnosis, off after.
SQL Server actual execution plan. Ctrl+M in SSMS. Cost of subtree % on every node sums to 100%; hottest node is the fix target.
SQL Server Query Store. Historical query perf + plan diffs. Forced plans available.
The 5-step senior read strategy. Hot node → estimate vs actual → spill/shuffle → parallelism → fix. Invariant across all four engines.
Refresh stats first, always. ANALYZE on Postgres. Snowflake refreshes automatically. BigQuery via INFORMATION_SCHEMA. SQL Server UPDATE STATISTICS. Fresh stats fix ~30% of bad plans immediately.
Rewrite before scaling. Query rewrite is free forever. Scaling is a monthly cost. Prefer rewrite when feasible.
Hints are last resort. Every hint locks a plan against future data shifts. Prefer index / cluster / partition changes to hints when possible.
Cost per query. Postgres self-hosted = server share. Snowflake = warehouse × wall clock. BigQuery on-demand = TB × $5. Flat-rate = monthly commit. Know your cost model when reading plans.
Plan diffs in code review. Add EXPLAIN (ANALYZE, BUFFERS) outputs to PR descriptions for schema or query changes. Catches regressions before ship.
CI cost gates. Wrap expensive queries with --dry_run in CI. Fail PR if estimated bytes exceed a threshold.
Monitoring stack. Postgres pg_stat_statements. Snowflake QUERY_HISTORY. BigQuery INFORMATION_SCHEMA.JOBS. SQL Server Query Store. Weekly review of top-N by mean time or total bytes.
Rule of thumb. 80% of bad plans come from 20% of queries. Diagnose the top 20 by cost weekly; that's usually where the leverage is.

Frequently asked questions

What is SQL EXPLAIN ANALYZE and how do I read the output?

sql explain analyze is the primitive every engine ships for surfacing the physical plan the executor actually walked — on Postgres it's EXPLAIN (ANALYZE, BUFFERS, VERBOSE, FORMAT TEXT) and it returns a nested plan tree annotated with cost=startup..total (planner units), rows=N (estimate), actual time=first..last (wall clock ms), loops=N (executions), and optional BUFFERS: shared hit / shared read / temp read (memory vs disk vs spill). Read it bottom-up — the leaf scans execute first and feed intermediate nodes (joins, aggregates), which feed the top node (Limit, Sort); the leaf whose actual time × loops dominates total wall clock is the bottleneck. The reading discipline is five steps — find the hot node, compare estimate to actual (10×+ gap = stale stats or bad plan), check for spill (Buffers: temp read non-zero, Sort Method: external merge), check parallelism (Workers Launched > 0), and name the fix (add index, run ANALYZE, rewrite join, bump work_mem). On Snowflake, the equivalent is the Snowsight Query Profile with operator cards and pruning %; on BigQuery, it's the execution graph with per-stage slot_ms; on SQL Server, it's the actual execution plan XML with cost-of-subtree percentages. The read is invariant across engines; only the artefact differs.

What's the difference between Postgres cost and actual time?

Cost is a made-up planner unit used only to rank plan alternatives during optimisation — a Seq Scan might cost 0.00..25000.00, but that does not mean 25 seconds; it means "the CBO's internal price for this scan compared to alternatives." Actual time is wall clock in milliseconds — actual time=42.512..42.548 means the first row from this node was ready 42.512 ms after query start; the last was ready at 42.548 ms. The two are not comparable — cost is arbitrary and only meaningful within a single plan; actual time is real and directly interpretable. On top of that, if a node has loops=1000 it executed a thousand times, so the wall-clock cost of the node is actual time × loops, not just actual time. When you're diagnosing a slow query, ignore cost entirely and read actual time × loops per node; when you're explaining why the optimiser chose plan A over plan B, cost is the axis that matters. Never quote cost to a business stakeholder — "this query cost 250,000" means nothing to them; "this query took 42 seconds" means everything.

How do I read a Snowflake query profile?

Open Snowsight → Activity → Query History → click into the query → Profile tab — the graphical view shows a stack of operator cards (TableScan, Filter, Aggregate, Join, Sort, WindowFunction), each with Execution time %, Bytes scanned, Rows produced, and per-operator statistics. Read top-down, sorted by execution time %, and the hot operator surfaces first. On the hot card — check pruning % for TableScans (99% is great, under 50% is a clustering or filter problem), check local vs remote disk spill for aggregates and sorts (bytes_spilled_to_remote_storage non-zero is the strongest single signal that the warehouse is too small), and check join algorithm for joins (Broadcast for genuinely small dim tables, hash / shuffle for large tables). The underlying data lives in SNOWFLAKE.INFORMATION_SCHEMA.QUERY_HISTORY (near-real-time, 7-day retention) and SNOWFLAKE.ACCOUNT_USAGE.QUERY_HISTORY (delayed but 1-year retention), with per-operator stats via GET_QUERY_OPERATOR_STATS(query_id). The five-step reading discipline is identical to other engines — hot operator, estimate vs actual (input vs output rows), spill (remote > local > none), parallelism (warehouse size), name the fix (CLUSTER BY, rewrite filter, bump warehouse, force NO_BROADCAST hint).

How do I read a BigQuery execution plan?

Open BigQuery Console → Query History → click the query → Execution details tab — the graphical view shows a stage-based DAG where each stage is a set of parallel workers reading input, applying part of the query, and shuffling output to the next stage. The fundamental compute unit is slot_ms (slot-milliseconds); a stage using 100 slots for 4.2 s costs 420 000 slot_ms with only 4.2 s of wall clock. Read stages sorted by slot_ms desc — the top stage is the bottleneck — and inspect records_read (partition pruning worked if this is a small fraction of the table), shuffle_output_bytes (large shuffle = big intermediate = optimise upstream), wait_ms vs compute_ms (wait > compute = slots queued; compute > wait = flat out), and skew warnings (one hot key monopolising a worker). The underlying data is INFORMATION_SCHEMA.JOBS with total_slot_ms, total_bytes_processed, and job_stages (nested per-stage details). Use bq query --dry_run to estimate bytes_processed before running — free, matches actual within ~5%, and pairs with CI gates to reject expensive PRs. The senior fix set is partition on the filter column (PARTITION BY DATE(created_at)), cluster on the next-most-common filter (CLUSTER BY user_id), create materialised views for dashboards (1000× cost drop typical), and choose on-demand vs flat-rate based on utilisation patterns.

What does "rows removed by filter" mean in Postgres?

Rows Removed by Filter: N on a Postgres plan node means the executor read N rows and threw them away — the filter predicate matched some fraction of the input, and the discarded rows are the loss. On a Seq Scan with Rows Removed by Filter: 99000000, the executor read 100 M rows and kept only 1 M — a scan that should have been an index seek. On an Index Scan with non-zero Rows Removed by Index Recheck, the index covered part of the predicate (say, the leading column) but a remaining condition (say, a trailing column not in the index) had to be re-checked at the heap; extending the index to a covering INCLUDE index eliminates the recheck. On a Bitmap Heap Scan with Rows Removed by Filter: N, the index bitmap was over-inclusive — the index isn't selective enough on its own; a partial index or a composite index that covers more of the predicate reduces the recheck. The general fix pattern is: any node with Rows Removed by Filter in the millions is a missing index or weak index diagnosis — add or extend an index so the filter becomes an Index Cond (part of the seek) instead of a Filter (applied after the read).

How do I fix a plan where the estimate is 10× off the actual?

An estimate that's 10×+ off the actual usually means one of five things — stale statistics (last ANALYZE was days ago and data has shifted; run ANALYZE table to refresh), skewed columns without extended stats (a column with a highly-skewed distribution fools the histogram; create CREATE STATISTICS s ON col FROM t; and run ANALYZE), correlated columns treated as independent (say, city and country are correlated but the planner multiplies their selectivities as if independent; CREATE STATISTICS s (dependencies, ndistinct) ON city, country FROM t; fixes it), function-wrapped predicates (WHERE EXTRACT(YEAR FROM created_at) = 2026 is not sargable; rewrite as a range WHERE created_at BETWEEN '2026-01-01' AND '2027-01-01'), or plan cache mode mismatch (a prepared statement was planned with generic parameters that don't match this call's actual values; force plan_cache_mode = 'force_custom_plan' for this session). The fastest triage — always try ANALYZE first — it's free, fast, and fixes ~30% of estimate divergences on its own. If ANALYZE doesn't help, look at the specific columns and predicates; extended statistics and predicate rewrites cover the remaining ~50%. As a last resort, force the plan via pg_hint_plan or its equivalents on other engines — but hints lock the plan against future data shifts, so always add a comment explaining why and revisit periodically.

Practice on PipeCode

Drill the SQL optimization practice library → for EXPLAIN ANALYZE reading, plan-tree diffs, Rows Removed by Filter diagnostics, and one-line index or rewrite fixes.
Sharpen the composite-index reflex with SQL indexing drills → — covering indexes, INCLUDE columns, partial indexes, index-only scans, and cluster keys.
Layer the join-optimisation surface with SQL join drills → — hash vs merge vs nested loop, broadcast vs shuffle joins on warehouses, and skew mitigation.
Push the SQL join-language depth with join drills on SQL → — semi-joins, anti-joins, lateral joins, and self-joins that show up in plan-reading interview questions.
Warm up with aggregation practice → — hash aggregate vs group aggregate, pre-aggregation patterns, and materialised-view design.
Layer window function drills → — ROW_NUMBER() / RANK() / LAG() plans, partition-key alignment with clustering, and window-frame reading.
Sharpen the general SQL surface with the SQL practice library → — 450+ DE-focused questions covering plan reading, index design, EXPLAIN ANALYZE, and every adjacent pattern.
For the broader SQL interview surface, take the SQL for Data Engineering course →.

Pipecode.ai is Leetcode for Data Engineering — every `sql explain analyze` recipe above ships with hands-on practice rooms where you type `EXPLAIN (ANALYZE, BUFFERS, VERBOSE)` on a live Postgres, read the plan tree bottom-up, spot the missing index from `Rows Removed by Filter`, migrate the same query to a Snowflake profile with pruning % and remote spill diagnosis, translate it into a BigQuery stage graph with `slot_ms` and `--dry_run` cost analysis, and finally read the SQL Server actual execution plan with `SET STATISTICS TIME/IO ON` — the exact four-engine `read execution plan` fluency that senior DE interviews probe. PipeCode pairs every plan-reading concept with 450+ DE-focused problems and a real-time scoring engine, so you never have to wonder whether your plan-diagnosis answer holds up under a senior interviewer's depth probes.

Practice plan reading now →
Indexing drills →

SQL Pagination Done Right: OFFSET / FETCH / LIMIT / Keyset Cursors

Gowtham Potureddi — Sat, 11 Jul 2026 16:22:36 +0000

sql pagination is the single most-shipped, least-benchmarked primitive in every REST endpoint, GraphQL resolver, admin dashboard, and infinite-scroll feed a data engineer builds — and the single largest silent source of tail-latency incidents once the underlying table crosses 10 million rows. Every engineer types LIMIT 20 OFFSET 40 on their first week, ships it to production, and only discovers that sql limit offset performance collapses at page 100,000 when the on-call graph turns red at 3 a.m. This guide is the honest, dialect-aware tour of what actually happens inside the planner when you ask for "page 47" of a 100-million-row table, why the answer looks fine at page 1 and terrifying at page 1,000,000, and what to reach for instead.

The tour walks the four primitives you have to keep straight in 2026 — the classic sql offset fetch family (Postgres / MySQL LIMIT n OFFSET k, ANSI OFFSET k ROWS FETCH FIRST n ROWS ONLY, SQL Server 2012+ OFFSET k ROWS FETCH NEXT n ROWS ONLY, Oracle 12c+ FETCH FIRST n ROWS ONLY, BigQuery and Snowflake LIMIT n OFFSET k), the honest performance story of why OFFSET is O(k + n) per page and why that curve goes vertical past a few hundred thousand rows, the keyset pagination (a.k.a. seek method pagination) pattern that replaces "skip k rows" with "give me everything after this cursor tuple" and produces the O(log n + K) deep-page path that every modern feed on the internet actually uses, and full sql cursor pagination for sql infinite scroll feeds — opaque base64 tokens, HMAC signing, the GraphQL Relay connections spec, and the six-engine dialect matrix that tells you which keyword lands where. Every section ships a teaching block followed by a Solution-Tail interview answer — code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works — so you leave with the two-line skeleton and the reason it wins.

When you want hands-on reps immediately after reading, drill the SQL pagination practice library →, rehearse on order-by-limit problems →, and sharpen the perf axis with the SQL indexing drill room →.

On this page

Why pagination matters in 2026
OFFSET / FETCH / LIMIT — dialect matrix
Why OFFSET is slow at page 1M
Keyset (seek method) pagination
Cursor pagination + dialect matrix
Cheat sheet — pagination recipe list
Frequently asked questions
Practice on PipeCode

1. Why pagination matters in 2026

The three shapes of `sql pagination` — infinite scroll, page-numbered UIs, and CSV-style exports — and why the OFFSET default fails all three at scale

The one-sentence invariant: sql pagination is the primitive that lets a client request "the next N rows of this ordered result set without re-fetching the previous rows," and the choice between OFFSET-based and keyset-based pagination is the single decision that determines whether your endpoint tail latency stays flat as the underlying table grows from 10K rows to 10B rows. Every backend eventually ships pagination; only a subset of them survive contact with a 10M-row table.

Where pagination actually shows up in production.

Infinite scroll feeds. Twitter / X home timeline, Instagram feed, TikTok For You page, GitHub notifications, Slack conversation history. The client asks "give me 20 more" every time the user scrolls, forever. Random-access-to-page-N is not a requirement; strict ordering and stability under new inserts are.
Page-numbered admin dashboards. "Show orders 1–50, page 2 is 51–100, ..." with a « ‹ 1 2 3 ... 47 › » pager at the bottom. Random-access-to-page-47 is a hard requirement; total row count usually is too. This is the shape where OFFSET is genuinely hard to replace.
Bulk CSV / Parquet exports. "Give me every row that matches this filter, split into 10K-row batches." Effectively a stream, not a paginator; users don't skip pages, they read from page 1 to the end.
API list endpoints. REST GET /orders?limit=20&cursor=... or GraphQL orders(first: 20, after: $endCursor). Almost always keyset-based in 2026 — the Relay spec bakes cursors into the wire protocol.
Data-warehouse dashboards. Superset / Metabase / Looker "next page" buttons on aggregate tables. Aggregates are usually small enough that OFFSET works; row-level fact-table pagination in warehouses needs keyset with a clustering key.

The OFFSET trap in one sentence.

LIMIT 20 OFFSET 1000000 asks the planner to read the first 1,000,020 rows in order and then discard the first 1,000,000. There is no "skip ahead" — the engine walks the ordered scan row by row until it has counted past the offset, then emits the next 20. Cost is O(k + n) per page. Across a full pagination from page 1 to page P, total cost is O(P × (k + n)) = O(P² × n) on a fixed page size — quadratic in the number of pages fetched.

The "consistent read across pages" problem.

Between page 1 and page 2 of an OFFSET paginator, other transactions may have INSERTed rows above the current window or DELETEd rows within it. The offset counter is positional, not keyed, so the same row can be seen twice or skipped entirely.
Example — a feed sorted by created_at DESC. Page 1 returns rows with timestamps [10:05, 10:04, 10:03, ..., 09:46] (20 rows). Between the two calls a new row lands at 10:06. Page 2 with OFFSET 20 LIMIT 20 now starts at the new row 21, which is the old row 20 (09:46) — the user sees 09:46 twice.
The dual — a row on page 1 gets deleted. Page 2 now starts one row earlier, skipping what was previously row 21. The user never sees that row.
Both problems are latent in every OFFSET-based paginator against a mutating source. Keyset pagination cures them because it navigates by key (the last row's tuple), not by position.

Why interviewers love pagination as a probe.

It's a fluency check on plan reading. Type EXPLAIN SELECT * FROM events ORDER BY created_at DESC LIMIT 20 OFFSET 1000000, then describe the plan out loud. A senior candidate says "Seq Scan or Index Scan of the first 1,000,020 rows plus a Limit that emits the last 20" in one breath.
It's a fluency check on index design. "What index does this need?" — a candidate who says "(created_at DESC, id DESC) for both OFFSET and keyset, but keyset is the one that actually uses it as a seek" signals senior instincts.
It's a fluency check on system design. "Design the feed for a Twitter-scale product." Every good answer includes keyset cursors as the default and OFFSET as an anti-pattern.
It's a fluency check on API design. "Design a REST endpoint that returns paginated results." Every good answer includes an opaque cursor token, hasNext / hasPrev, and no page=N query parameter.
It's a fluency check on failure modes. "What happens if two users insert between your page 1 and page 2 calls?" A senior candidate walks through the double-read / skipped-row failure and shows how keyset avoids it.

Six-engine keyword matrix — one-line summary.

Postgres 8.0+ / MySQL 4.0+ / Snowflake / BigQuery. LIMIT n OFFSET k. The de-facto default. Postgres also accepts the ANSI spelling.
ANSI SQL:2008. OFFSET k ROWS FETCH FIRST n ROWS ONLY. Verbose, deterministic, standard. Postgres, Oracle 12c+, SQL Server 2012+, and Db2 all support it.
SQL Server 2012+. OFFSET k ROWS FETCH NEXT n ROWS ONLY. The NEXT is a synonym for FIRST. Requires an ORDER BY clause — the parser refuses.
Oracle 12c+. OFFSET k ROWS FETCH FIRST n ROWS ONLY. Legacy Oracle (pre-12c) used the ROWNUM pseudocolumn pattern.
All engines. No dialect will save you from the O(k + n) cost. OFFSET is a plan shape, not a keyword.

What senior interviewers actually probe.

Do you know the OFFSET cost model? O(k + n) per page — the planner scans and discards.
Do you know when OFFSET is fine? For k ≤ 1,000 on a modestly wide table with a covering index — the scan is fast enough. Past k ≥ 100,000, latency climbs into the "someone will file a ticket" range.
Do you know the keyset skeleton? WHERE (order_col, id) < (:cursor_col, :cursor_id) ORDER BY order_col DESC, id DESC LIMIT n. One-shot tuple compare, one composite index seek.
Do you know the tie-breaker rule? Every paginated ORDER BY needs a deterministic tie-breaker on a unique column (usually id), or two rows with identical created_at will alternate between pages.
Do you know the NULLs rule? ORDER BY created_at DESC sorts NULLs first on some engines, last on others. Always spell it out — ORDER BY created_at DESC NULLS LAST.
Do you know the API-shape rule? Cursors are opaque, base64-encoded, and HMAC-signed. Never expose the raw tuple to clients — it's a leaky abstraction and a tampering vector.

Worked example — the OFFSET query that ships to production on day one

Detailed explanation. The archetype: given events(id, user_id, created_at, payload), return the first page of 20 most recent events. This is the query every backend intern types in week one. It is fine — for page 1. It becomes a tail-latency incident somewhere past page 10,000.

Question. Write a Postgres query that returns the N-th page of 20 most recent events, given a page number P (0-indexed). Then note the cost model.

Input.

id	user_id	created_at	payload
1	42	2026-07-01 09:00	login
2	42	2026-07-01 09:05	click
3	17	2026-07-01 09:10	login
...	...	...	...
1000000	91	2026-07-10 08:59	logout

Code.

-- Page P of size 20, 0-indexed
SELECT id, user_id, created_at, payload
FROM events
ORDER BY created_at DESC, id DESC
LIMIT 20
OFFSET (:page * 20);

Step-by-step explanation.

ORDER BY created_at DESC, id DESC — the compound sort key. The id DESC tie-breaker ensures deterministic ordering when two events share a millisecond.
LIMIT 20 — the page size. Clients typically pass page_size as a query parameter capped at 100 or 500.
OFFSET (:page * 20) — skip this many rows. On page 0, the offset is 0 and the query is fast. On page 50,000, the offset is 1,000,000 and the planner walks a million rows before emitting anything.
The planner picks either a Seq Scan (if no covering index) or an Index Scan on (created_at DESC, id DESC). Either way, it walks the ordered stream one row at a time until it has counted past the offset, then emits the next 20.
Cost model — O(k + n) where k is the offset and n is the limit. Wall-clock latency grows linearly with k.

Output (page 0, most recent 20 first).

id	user_id	created_at	payload
1000000	91	2026-07-10 08:59	logout
999999	42	2026-07-10 08:58	click
...	...	...	...

Rule of thumb. OFFSET is fine up to k ≈ 1,000 on a table with a covering index. Past that, latency becomes user-visible; past k ≈ 100,000, latency crosses SLA. If your product needs a « ‹ 1 2 ... 47 › » pager over a 10M+ row table, either cap the offset (Facebook's "5,000 pages" cap) or switch to keyset for the deep pages.

Worked example — the OFFSET failure mode across concurrent writes

Detailed explanation. A common source of pagination bugs — between page 1 and page 2, other transactions mutate the underlying table and the OFFSET counter drifts. The interviewer wants to hear "OFFSET is positional; keyset is keyed."

Question. Given the same events table, describe what happens if a user reads page 1 (offset 0, limit 20), then between calls someone inserts a new event with a created_at newer than every existing row, then the user reads page 2 (offset 20, limit 20). Which rows does the user see?

Input.

Pre-insert — 100 events with created_at from 10:00:00 (id=1) to 10:01:39 (id=100). Sorted DESC, row 1 is 10:01:39.
Between calls — one new row lands with created_at = 10:02:00, id=101.

Code.

-- Page 1 (returned to the user first)
SELECT id, created_at
FROM events
ORDER BY created_at DESC, id DESC
LIMIT 20 OFFSET 0;
-- Returns id=100 down to id=81

-- Concurrent insert lands: id=101, created_at=10:02:00

-- Page 2 (returned to the user second)
SELECT id, created_at
FROM events
ORDER BY created_at DESC, id DESC
LIMIT 20 OFFSET 20;
-- Returns id=81 down to id=62 — id=81 IS SEEN TWICE

Step-by-step explanation.

Page 1 returns rows 100 → 81 (the twenty most recent). The user reads them.
Between calls, one new row lands (id=101). The ordered stream is now 101, 100, 99, ..., 1.
Page 2 asks for offset 20 limit 20 from the new ordered stream. Offset 20 is now id=81; offset 39 is id=62.
The user sees id=81 on both page 1 and page 2 — the duplicate — because the positional offset drifted by one when id=101 was inserted.
Symmetrically, if id=90 had been deleted between calls, page 2 would start at what was previously id=80 — the row at previous position 21 (id=80) would be skipped.

Output. Page 1 shows ids 100 → 81. Page 2 shows ids 81 → 62. id=81 appears in both.

Rule of thumb. OFFSET is positional and drifts under concurrent inserts / deletes. Keyset navigates by key (the last row's tuple), so a new row appearing above the cursor doesn't affect subsequent pages, and a deleted row inside a previous page doesn't cause a skip. Every social feed in the world uses keyset for this reason.

Worked example — engine dialect quickstart

Detailed explanation. A quick sanity check — the six most common warehouses in 2026 and the pagination keyword each supports. Knowing this cold is a senior signal in system-design interviews.

Question. Given a list of engines, mark each with the OFFSET / LIMIT / FETCH keyword combination it supports as of 2026.

Input. Postgres 16, MySQL 8, SQL Server 2022, Oracle 19c, Snowflake, BigQuery.

Code. (No code — a dialect matrix answer.)

Engine          | Preferred syntax                          | Notes
Postgres 9+     | LIMIT n OFFSET k (ANSI also supported)    | ORDER BY strongly recommended
MySQL 8         | LIMIT n OFFSET k or LIMIT k, n            | ORDER BY strongly recommended
SQL Server 2012+| OFFSET k ROWS FETCH NEXT n ROWS ONLY      | ORDER BY REQUIRED
Oracle 12c+     | OFFSET k ROWS FETCH FIRST n ROWS ONLY     | pre-12c uses ROWNUM
Snowflake       | LIMIT n OFFSET k                          | avoid OFFSET past 1M
BigQuery        | LIMIT n OFFSET k                          | OFFSET slow; consider partition prune

Step-by-step explanation.

Postgres has supported the LIMIT n OFFSET k spelling since 8.0 and the ANSI OFFSET k FETCH FIRST n ROWS ONLY since 8.4. Both compile to identical plans.
MySQL supports LIMIT n OFFSET k and the shorthand LIMIT k, n (note the reversed operand order). Both compile to the same plan.
SQL Server 2012 introduced OFFSET / FETCH and requires an ORDER BY clause — the parser refuses without one, unlike Postgres and MySQL which allow (but discourage) unordered pagination.
Oracle 12c (2013) added ANSI OFFSET / FETCH. Older Oracle used a ROWNUM filter — WHERE ROWNUM <= 100 MINUS WHERE ROWNUM <= 80 — messy but the only pre-12c option.
Snowflake and BigQuery both accept LIMIT n OFFSET k but neither warehouse is optimised for deep OFFSET — micro-partitions on Snowflake or partition-and-cluster pruning on BigQuery mean the scan-and-discard cost is amplified by columnar-read overhead.

Output. The matrix above.

Rule of thumb. Memorise the six-row list. In an interview, if asked "how do you paginate on X," the first sentence should be the keyword combination and the second should be "with ORDER BY plus a deterministic tie-breaker on id."

Senior interview question on when to use OFFSET vs keyset

A senior interviewer often opens with: "You're designing the paginated /api/v1/orders endpoint for a two-billion-row orders table. The product wants both an infinite-scroll consumer app and a page-numbered admin dashboard with « ‹ 1 2 ... 100000 › ». Which pagination strategy do you pick for each, and why?"

Solution Using dual-endpoint design — keyset for infinite-scroll, capped OFFSET for admin

# Consumer app — infinite scroll, keyset-only, opaque cursors
# GET /api/v1/orders?first=20&after=<cursor>
def list_orders_keyset(first: int, after_cursor: str | None):
    # Cap first at 100 to protect the DB
    first = min(first, 100)

    if after_cursor:
        ts, oid = decode_and_verify_hmac(after_cursor)
        rows = db.query(
            """
            SELECT id, created_at, user_id, total
            FROM orders
            WHERE (created_at, id) < (%s, %s)
            ORDER BY created_at DESC, id DESC
            LIMIT %s + 1
            """,
            (ts, oid, first),
        )
    else:
        rows = db.query(
            """
            SELECT id, created_at, user_id, total
            FROM orders
            ORDER BY created_at DESC, id DESC
            LIMIT %s + 1
            """,
            (first,),
        )

    has_next = len(rows) > first
    rows = rows[:first]
    end_cursor = (
        encode_hmac(rows[-1].created_at, rows[-1].id) if rows else None
    )
    return {
        "edges": [{"node": r, "cursor": encode_hmac(r.created_at, r.id)} for r in rows],
        "pageInfo": {"hasNext": has_next, "endCursor": end_cursor},
    }


# Admin dashboard — page-numbered, capped OFFSET, total_count
# GET /admin/orders?page=<int>&page_size=<int>
def list_orders_admin(page: int, page_size: int):
    # Cap page at 5000, page_size at 200
    page = min(max(page, 0), 5_000)
    page_size = min(page_size, 200)

    total_count = db.query_scalar("SELECT COUNT(*) FROM orders")
    rows = db.query(
        """
        SELECT id, created_at, user_id, total
        FROM orders
        ORDER BY created_at DESC, id DESC
        LIMIT %s OFFSET %s
        """,
        (page_size, page * page_size),
    )
    return {
        "rows": rows,
        "page": page,
        "page_size": page_size,
        "total_count": total_count,
        "total_pages": (total_count + page_size - 1) // page_size,
    }

Step-by-step trace.

Step	Consumer app (keyset)	Admin dashboard (OFFSET)
1	client calls with `after=null`, `first=20`	client calls with `page=0`, `page_size=50`
2	first=capped at 100; no cursor	page capped at 5000; page_size at 200
3	SELECT with LIMIT 21 (first+1)	SELECT with LIMIT 50 OFFSET 0
4	21 rows returned; hasNext=true	50 rows returned
5	rows[:20] emitted; endCursor from row 20	rows emitted with total_count and total_pages
6	next call passes endCursor as after=	next call passes page=1
7	O(log n + 20) per page, flat forever	O(k + 50), fine for page ≤ 5000

The consumer app uses keyset because the infinite-scroll UX has no page-number requirement and needs constant-time deep pagination. The admin dashboard uses OFFSET because "jump to page 47" is a real product requirement, but caps the maximum page at 5,000 (like Facebook's search-result cap) so no admin can accidentally trigger a 100-million-row scan.

Output:

Endpoint	Cost model	Deep-page behaviour	Stability under writes
Consumer (keyset)	O(log n + K)	flat 5–10 ms at any depth	perfect — no drift
Admin (capped OFFSET)	O(k + n), k ≤ 5000×200=1M	slow-but-bounded past 100K rows	positional — mild drift

Why this works — concept by concept:

Dual-endpoint design matches the two consumer shapes — the consumer feed never needs "page 47"; it needs "the next 20 after this cursor forever." Keyset gives that with O(log n + K) deep pages. The admin dashboard does need "page 47"; capped OFFSET gives that with a bounded worst case.
LIMIT first + 1 is the peek-ahead trick — the endpoint fetches 21 rows but returns 20; the 21st row's existence tells the client whether hasNext is true. Zero extra queries, exact answer.
HMAC-signed opaque cursors — the cursor after parameter is a base64-encoded (created_at, id) tuple with an HMAC signature. Clients can't tamper with the tuple to leak rows they shouldn't see; servers can validate the signature in constant time before decoding.
Capped OFFSET protects the DB — a page-number cap of 5,000 with a page-size cap of 200 means the absolute worst case is OFFSET 999,800 LIMIT 200 — a ~1M-row scan, painful but bounded. Without the cap, an admin could drop page=1000000 and hang the DB.
Cost — Consumer endpoint is O(log n + K) per page where n is orders.rowcount and K = first. Admin endpoint is O(k + n) per page where k is page × page_size and n = page_size. On a 2B-row table with a covering (created_at DESC, id DESC) index, the consumer endpoint runs at ~8 ms per page at any depth; the admin endpoint runs at ~5 ms at page 0, ~50 ms at page 100, ~500 ms at page 10K, and ~5 s at page 100K (which the cap prevents from exceeding).

SQL
Topic — pagination
SQL pagination problems

Practice →

SQL Topic — order by / limit ORDER BY + LIMIT drills

Practice →

2. OFFSET / FETCH / LIMIT — dialect matrix

`sql offset fetch` and `LIMIT` across Postgres, MySQL, SQL Server, Oracle, Snowflake, and BigQuery — same primitive, six spellings, one shared perf model

The mental model in one line: every engine implements pagination with two knobs — a skip count (OFFSET k or LIMIT k, n's first operand) and a take count (LIMIT n or FETCH FIRST n ROWS ONLY) — and the choice of keyword changes nothing about the underlying plan shape, which is always "scan the ordered stream, discard k, emit n". Once you memorise the six spellings, translating a pagination query across dialects is mechanical.

Slot 1 — Postgres and MySQL (the LIMIT n OFFSET k family).

Postgres — SELECT ... ORDER BY ... LIMIT n OFFSET k. Also accepts the ANSI OFFSET k ROWS FETCH FIRST n ROWS ONLY since 8.4.
MySQL 8 — SELECT ... ORDER BY ... LIMIT n OFFSET k or the shorthand LIMIT k, n (note the reversed operand order — offset then limit). Both compile identically.
Snowflake — LIMIT n OFFSET k. Micro-partition pruning helps at page 1 but not at deep offsets.
BigQuery — LIMIT n OFFSET k. Documented as slow on deep offsets; the Google docs explicitly recommend keyset pagination past a few thousand rows.
Redshift, ClickHouse, DuckDB, SQLite — all support LIMIT n OFFSET k with the same semantics.

Slot 2 — the ANSI SQL:2008 spelling.

OFFSET k ROWS FETCH FIRST n ROWS ONLY — verbose, standard, deterministic. Every "ANSI-compliant" dialect accepts it.
OFFSET k ROWS FETCH NEXT n ROWS ONLY — the SQL-Server-idiomatic spelling; NEXT is a synonym for FIRST.
FETCH FIRST n ROWS ONLY — without OFFSET, equivalent to LIMIT n. Common on Oracle where LIMIT is not a keyword.
FETCH FIRST n ROWS WITH TIES — a variant that returns extra rows past the limit if they tie on the ORDER BY key. Rarely used in pagination (breaks page-size invariants) but the interviewer might ask about it.

Slot 3 — SQL Server-specific rules.

OFFSET k ROWS FETCH NEXT n ROWS ONLY — the SQL Server 2012+ syntax. Requires an ORDER BY clause — the parser refuses without one, unlike Postgres and MySQL.
Pre-2012 SQL Server — used ROW_NUMBER() OVER (ORDER BY ...) AS rn in a CTE and WHERE rn BETWEEN k+1 AND k+n to paginate. Ugly, but the only option.
SQL Server also supports TOP n for the first-page-only case (SELECT TOP 20 ...), which is not pagination — it just limits the result set without offset support.

Slot 4 — Oracle-specific rules.

Oracle 12c+ — OFFSET k ROWS FETCH FIRST n ROWS ONLY. Fully ANSI-compliant, same semantics as SQL Server 2012+.
Pre-12c Oracle (11g and earlier) — used the ROWNUM pseudocolumn: SELECT * FROM (SELECT rownum AS rn, t.* FROM t ORDER BY ... ) WHERE rn BETWEEN k+1 AND k+n. Legacy but still shows up in enterprise codebases.
Oracle 21c added FETCH FIRST n PERCENT ROWS ONLY — for percent-based limits — but this isn't a pagination primitive per se.

Slot 5 — BigQuery and Snowflake specifics.

BigQuery — LIMIT n OFFSET k, but OFFSET is documented as an anti-pattern past a few thousand rows. The recommended replacement is keyset. BigQuery also has LIMIT n with an implicit OFFSET 0 — same as everywhere.
Snowflake — LIMIT n OFFSET k. Snowflake's micro-partition metadata helps prune the initial scan but still has to walk the ordered result set past the offset. Deep OFFSET on a 1B-row Snowflake table is measurably slow.
Databricks SQL — LIMIT n OFFSET k since 2022. Same behaviour as Snowflake.

Slot 6 — the ORDER BY requirement.

Without ORDER BY, pagination is nondeterministic. Two calls with the same LIMIT / OFFSET may return different rows because the underlying scan order is not guaranteed stable.
SQL Server requires ORDER BY for OFFSET / FETCH — the parser rejects the query.
Postgres, MySQL, Snowflake, BigQuery allow omitting ORDER BY but will return unstable results. Every senior review comment says "add ORDER BY plus a tie-breaker on id."
Composite ORDER BY — the sort key must be a superset of the columns you want to compare in a cursor. For a "most recent" feed, the sort is ORDER BY created_at DESC, id DESC — id as tie-breaker.

Slot 7 — ties and tie-breakers.

If two rows share the same created_at (millisecond precision or worse), plain ORDER BY created_at DESC LIMIT 20 OFFSET 20 may put one on page 1 and the other on page 2 — or vice versa, on the next call. The results are unstable.
Fix — always include a unique tie-breaker: ORDER BY created_at DESC, id DESC. Every pagination code review comment worth reading says this.
FETCH FIRST n ROWS WITH TIES — the ANSI variant that returns all rows that tie on the sort key at the limit boundary. Rarely appropriate for pagination because it violates the "exactly n rows per page" invariant.

Slot 8 — NULLs ordering.

ORDER BY created_at DESC — where do NULLs go? Postgres and Oracle put NULLs first; MySQL and SQL Server put NULLs last; BigQuery puts NULLs first.
Fix — spell it out: ORDER BY created_at DESC NULLS LAST. Supported on Postgres, Oracle, and BigQuery. MySQL 8 supports the syntax but the behaviour matches the engine default anyway.
SQL Server does not support NULLS FIRST / LAST syntax — the workaround is ORDER BY (CASE WHEN created_at IS NULL THEN 1 ELSE 0 END), created_at DESC.

Common newbie mistakes.

Forgetting ORDER BY — the query "works" on page 1 but returns different rows on subsequent pages because the plan changed.
Omitting a unique tie-breaker — two rows with the same created_at can appear on both page 1 and page 2 or on neither.
Using SELECT * — projecting more columns than needed forces a heap fetch that OFFSET amplifies. Always project the minimum column set.
Assuming OFFSET is O(1) — it's O(k + n). On page 1M, that's a million rows scanned before the first byte is returned.
Assuming pagination is stable under writes — OFFSET is positional and drifts. Every social feed uses keyset.

How OFFSET interacts with EXPLAIN.

EXPLAIN SELECT * FROM events ORDER BY created_at DESC LIMIT 20 OFFSET 1000000 on Postgres — the plan shows Limit → Index Scan Backward using idx_events_created_at. The Limit step has actual rows=20 but actual loops = 1,000,020 — the engine walks a million rows before hitting the limit.
The same query with keyset — WHERE (created_at, id) < (:ts, :id) ORDER BY created_at DESC, id DESC LIMIT 20 — shows Limit → Index Scan Backward with actual rows=20 actual loops=20. One seek, twenty reads, done.
The Buffers: shared read=100000 line in EXPLAIN (ANALYZE, BUFFERS) is the smoking gun — OFFSET buffers grow linearly with k; keyset buffers stay constant.

Interview probes on syntax fluency.

"Which dialect requires ORDER BY for OFFSET/FETCH?" — SQL Server 2012+. Postgres, MySQL, Oracle, Snowflake, BigQuery all allow (but discourage) omitting it.
"What is the shorthand LIMIT 40, 20 in MySQL?" — offset 40, limit 20. Note the reversed operand order (offset first, then limit) — the most confused MySQL trivia question in interviews.
"How do you paginate on Oracle 11g?" — the legacy ROWNUM filter inside a subquery. Not the ANSI FETCH FIRST because that came in 12c.
"How do you handle NULLs on SQL Server's ORDER BY?" — the CASE WHEN … IS NULL trick, because SQL Server doesn't support NULLS FIRST / LAST.

Worked example — Postgres, MySQL, and SQL Server pagination side by side

Detailed explanation. The clearest way to see the dialect mechanics: write the same "20 most recent orders, page 47" query in three dialects. Once you can round-trip the syntax, the mental model locks in.

Question. Given a table orders(id, user_id, created_at, total), write the page-47-of-20 query in Postgres, MySQL 8, and SQL Server 2022. Note any syntactic requirements each engine imposes.

Input.

id	user_id	created_at	total
1	100	2026-07-10 09:00	25.00
2	101	2026-07-10 09:01	40.00
...	...	...	...

Code.

-- Postgres
SELECT id, user_id, created_at, total
FROM orders
ORDER BY created_at DESC, id DESC
LIMIT 20 OFFSET 940;

-- MySQL 8 (identical to Postgres; also accepts LIMIT 940, 20 shorthand)
SELECT id, user_id, created_at, total
FROM orders
ORDER BY created_at DESC, id DESC
LIMIT 20 OFFSET 940;

-- SQL Server 2022 — ORDER BY is REQUIRED
SELECT id, user_id, created_at, total
FROM orders
ORDER BY created_at DESC, id DESC
OFFSET 940 ROWS
FETCH NEXT 20 ROWS ONLY;

Step-by-step explanation.

Postgres uses LIMIT 20 OFFSET 940. The order is "take 20, skip 940." Postgres also supports the ANSI spelling — OFFSET 940 ROWS FETCH FIRST 20 ROWS ONLY — and treats it identically.
MySQL 8 supports the identical LIMIT 20 OFFSET 940 plus the classic shorthand LIMIT 940, 20 (note the operand order — offset first). The shorthand is the one 90% of MySQL codebases still use; the explicit form is more readable.
SQL Server 2022 uses OFFSET 940 ROWS FETCH NEXT 20 ROWS ONLY. The NEXT is a synonym for FIRST. The ROWS keyword is required. ORDER BY is required — the parser refuses without one, giving a clear compile-time error.
All three plans compile to the same shape — walk the sorted stream past 940 rows, emit the next 20. Actual latency is nearly identical on all three engines for the same page depth.
Migrating pagination between the three is mechanical — swap LIMIT n OFFSET k for OFFSET k ROWS FETCH NEXT n ROWS ONLY (and vice versa).

Output (page 47 = rows 941–960).

id	user_id	created_at	total
999059	...	2026-07-10 08:15	30.00
...	...	...	...

Rule of thumb. Learn both spellings. Every migration from a MySQL app to a SQL Server backend or from Postgres to Oracle rewrites pagination. The mechanical swap is a five-second edit; forgetting it is a compile error.

Worked example — Oracle 11g's `ROWNUM` pattern vs Oracle 12c+'s `FETCH FIRST`

Detailed explanation. Legacy Oracle codebases still use the ROWNUM pseudocolumn for pagination. Interviewers love this — it's a fluency check on both "you know the old syntax" and "you know the new one."

Question. Write the "page 47 of 20 orders" query on Oracle 11g (legacy) and Oracle 19c (modern). Explain why the legacy pattern needs a subquery.

Input. (Same as above.)

Code.

-- Oracle 11g and earlier — ROWNUM inside a subquery
SELECT * FROM (
  SELECT rownum AS rn, o.*
  FROM (
    SELECT id, user_id, created_at, total
    FROM orders
    ORDER BY created_at DESC, id DESC
  ) o
  WHERE rownum <= 960
)
WHERE rn > 940;

-- Oracle 12c+ — ANSI FETCH FIRST
SELECT id, user_id, created_at, total
FROM orders
ORDER BY created_at DESC, id DESC
OFFSET 940 ROWS
FETCH FIRST 20 ROWS ONLY;

Step-by-step explanation.

Oracle's ROWNUM is assigned after the WHERE clause but before the ORDER BY. The naive SELECT ROWNUM, ... FROM t ORDER BY x returns numbered-then-sorted rows, not sorted-then-numbered — the numbering is on the wrong data.
The workaround — a nested subquery. The inner SELECT orders the rows; the middle SELECT numbers them with ROWNUM; the outer SELECT filters by the numbered range.
The WHERE rownum <= 960 on the middle query is a common perf trick — Oracle can short-circuit once it has assigned 960 numbers, avoiding a full sort of the underlying table.
Oracle 12c (2013) shipped ANSI OFFSET / FETCH, so modern code should always prefer the second form. It's more readable, ships with the same plan shape, and is portable across engines.
If you inherit legacy Oracle code, recognise the three-level nested pattern instantly. It is one of the top-three most-migrated SQL constructs in Oracle-to-Postgres modernisation projects.

Output. Both queries return the same page 47.

Rule of thumb. On Oracle 12c+, always use OFFSET k ROWS FETCH FIRST n ROWS ONLY. On Oracle 11g and earlier, use the three-level ROWNUM subquery. Migrating from 11g to 12c is a mechanical rewrite; interview candidates who can spot both patterns and translate between them signal senior instincts.

Worked example — BigQuery's OFFSET warning and the recommended keyset alternative

Detailed explanation. BigQuery documents OFFSET as an anti-pattern past a few thousand rows. The recommended replacement is keyset — and the interviewer might ask why BigQuery is more sensitive to deep OFFSET than a row-store database.

Question. Given a 10B-row BigQuery events table partitioned by DATE(created_at), why is LIMIT 20 OFFSET 1000000 an anti-pattern? What's the recommended alternative?

Input. (Same events table as before, at 10B rows.)

Code.

-- BigQuery — anti-pattern past ~10K OFFSET
SELECT id, user_id, created_at, payload
FROM `project.dataset.events`
WHERE DATE(created_at) BETWEEN '2026-07-01' AND '2026-07-10'
ORDER BY created_at DESC, id DESC
LIMIT 20 OFFSET 1000000;

-- BigQuery — recommended keyset
SELECT id, user_id, created_at, payload
FROM `project.dataset.events`
WHERE DATE(created_at) BETWEEN '2026-07-01' AND '2026-07-10'
  AND (created_at, id) < (@cursor_ts, @cursor_id)
ORDER BY created_at DESC, id DESC
LIMIT 20;

Step-by-step explanation.

BigQuery is a columnar warehouse — each column is stored separately and reads are page-oriented. Skipping a million rows means reading a million-row-worth of column data across every projected column, then discarding it.
Even with partition pruning on the DATE(created_at) filter, the OFFSET step happens after the scan — the pruning helps but does not eliminate the scan-and-discard cost.
The keyset alternative uses WHERE (created_at, id) < (@cursor_ts, @cursor_id) — a range predicate that BigQuery can push into partition and cluster pruning. If the table is clustered on created_at, the query touches only the blocks that contain the target page.
On a 10B-row events table with DATE partitioning and created_at clustering, OFFSET 1000000 runs in ~30 seconds; the keyset variant runs in ~200 ms — a 150× win.
Every BigQuery pagination guide since 2019 has recommended keyset for anything past a few thousand rows. The Snowflake guidance is identical (micro-partitions instead of BigQuery partitions, but the same shape).

Output.

Query	Rows scanned	Slot time	Wall-clock
OFFSET 1000000	1,000,020	30 s slot-seconds	~30 s
Keyset	20	0.02 s slot-seconds	~200 ms

Rule of thumb. In a columnar warehouse (BigQuery, Snowflake, Redshift), OFFSET's cost is amplified by the columnar-read overhead — projecting fewer columns helps a little, but the fundamental scan-and-discard cost stays. Keyset is the only stable long-form pagination on warehouses.

Senior interview question on cross-warehouse pagination portability

A senior interviewer might ask: "Design a paginated orders endpoint that runs identically on Postgres (production OLTP), Snowflake (analytics), and BigQuery (data-warehouse). Where can you use OFFSET, where must you use keyset, and how do you keep the API contract stable across all three?"

Solution Using dialect-dispatched dbt macro with a shared cursor contract

-- macros/paginate_orders.sql (dbt)
{% macro paginate_orders(cursor_ts=None, cursor_id=None, page_size=20) %}
  {%- if target.type == 'postgres' -%}
    -- OLTP: keyset with covering index (created_at DESC, id DESC)
    SELECT id, user_id, created_at, total
    FROM orders
    WHERE
      {% if cursor_ts is not none %}
        (created_at, id) < ({{ cursor_ts }}, {{ cursor_id }})
      {% else %}
        true
      {% endif %}
    ORDER BY created_at DESC, id DESC
    LIMIT {{ page_size }} + 1
  {%- elif target.type == 'snowflake' -%}
    -- Analytics: keyset with clustering key on created_at
    SELECT id, user_id, created_at, total
    FROM orders
    WHERE
      {% if cursor_ts is not none %}
        (created_at, id) < ({{ cursor_ts }}, {{ cursor_id }})
      {% else %}
        TRUE
      {% endif %}
    ORDER BY created_at DESC, id DESC
    LIMIT {{ page_size }} + 1
  {%- elif target.type == 'bigquery' -%}
    -- DW: keyset + partition pruning on DATE(created_at)
    SELECT id, user_id, created_at, total
    FROM `{{ project }}.orders`
    WHERE
      DATE(created_at) BETWEEN
        DATE_SUB(CURRENT_DATE(), INTERVAL 90 DAY) AND CURRENT_DATE()
      {% if cursor_ts is not none %}
        AND (created_at, id) < ({{ cursor_ts }}, {{ cursor_id }})
      {% endif %}
    ORDER BY created_at DESC, id DESC
    LIMIT {{ page_size }} + 1
  {%- endif -%}
{% endmacro %}

Step-by-step trace.

Dispatch branch	What runs	Why
Postgres	keyset with `(created_at, id)` covering btree	OLTP; index seek is O(log n)
Snowflake	keyset with `created_at` clustering key	analytics; micro-partition prune
BigQuery	keyset + partition prune on DATE(created_at)	warehouse; partition + cluster prune
All	LIMIT page_size + 1 for hasNext peek-ahead	uniform API contract
All	ORDER BY created_at DESC, id DESC	deterministic ordering

The macro dispatches on target.type. Postgres, Snowflake, and BigQuery each get a keyset variant optimised for that engine's storage model (btree, clustering, partitioning). The output schema and cursor contract are identical — every caller sees (id, user_id, created_at, total) plus a peek-ahead row to compute hasNext.

Output:

target	page cost	deep-page cost	cursor stability
Postgres	~5 ms	~5 ms at any depth	perfect
Snowflake	~200 ms	~200 ms at any depth	perfect
BigQuery	~150 ms	~150 ms at any depth	perfect

Why this works — concept by concept:

One cursor contract, three plan shapes — the API contract is stable ({cursor_ts, cursor_id, page_size}), the underlying plan adapts per engine. Callers never learn which warehouse is behind the endpoint.
Compile-time dispatch, zero runtime overhead — dbt evaluates the Jinja block at compile time. The emitted SQL is one of three variants; no runtime branching, no reflection, no penalty.
Postgres gets the covering btree — the composite index (created_at DESC, id DESC) is index-aligned with the ORDER BY, so the plan is index scan + LIMIT with no sort step.
Snowflake gets the clustering key — CLUSTER BY (created_at) on the orders table plus the tuple predicate prunes to a small set of micro-partitions per page.
BigQuery gets partition + cluster pruning — partition the table by DATE(created_at) and cluster by created_at. The 90-day window on the outer WHERE prunes at partition level; the tuple predicate then prunes within clusters.
Cost — Postgres is O(log n + K) per page, Snowflake and BigQuery are O(P + K) where P is the number of relevant partitions/clusters (typically 1–3 per page). All three are effectively flat across page depth — the whole point of keyset.

SQL
Topic — SQL
SQL problem library — 450+ DE-focused questions

Practice →

SQL Topic — order by / limit ORDER BY + LIMIT drills

Practice →

3. Why OFFSET is slow at page 1M

`sql limit offset performance` — the O(k + n) cost model, the wide-row memory story, and the "consistent read across pages" failure mode

The mental model in one line: sql limit offset performance is O(k + n) per page because the planner must walk the ordered scan past k rows and discard them before emitting the next n, so a page at depth k=1,000,000 costs 50,000× more than a page at depth k=20 — a curve that stays flat until it isn't and then climbs a cliff. Everything you say about OFFSET at scale is a corollary of this cost model.

Slot 1 — the planner has to walk the scan.

OFFSET is not a "skip pointer." No engine in 2026 (row-store or columnar) has an O(1) skip-to-position primitive on an ordered scan.
The plan for SELECT ... ORDER BY x LIMIT n OFFSET k on Postgres is Limit (n rows) → Index Scan Backward using idx_x. The Limit step reads k + n rows from the index scan; the first k are counted-then-discarded; the last n are emitted.
Cost is O(k + n) per page. For a fixed n = 20, cost is effectively O(k).
Total cost across pages 1 through P is Σ (k_i + n) = Σ ((i-1) × n + n) = n × P × (P+1) / 2 = O(P² × n). Quadratic in the number of pages fetched.

Slot 2 — wide rows amplify the pain.

Every scanned-and-discarded row still incurs I/O for the columns projected in the SELECT list. SELECT * FROM t OFFSET 1000000 on a 1KB-per-row table reads ~1GB before emitting the first byte.
Even with SELECT id, created_at (a covering-index-only query), the index leaf pages must be read — smaller than the heap but non-zero.
Wide rows also amplify sort memory. If the ORDER BY isn't index-aligned, the query needs an explicit sort — a sort of k + n rows fitting in sort_mem (Postgres work_mem) or spilling to disk. Deep OFFSET on a non-indexed ORDER BY is catastrophic.

Slot 3 — the benchmark curve.

On a 10M-row Postgres table with covering btree (created_at DESC, id DESC):
- Page 1 (OFFSET 0 LIMIT 20) — ~3 ms.
- Page 100 (OFFSET 1,980) — ~4 ms.
- Page 1,000 (OFFSET 19,980) — ~15 ms.
- Page 10,000 (OFFSET 199,980) — ~140 ms.
- Page 100,000 (OFFSET 1,999,980) — ~1.4 s.
- Page 500,000 (OFFSET 9,999,980) — ~7 s (hits shared-buffer pressure).
The curve is roughly linear in k until buffer pressure kicks in, then super-linear.
On columnar warehouses (BigQuery, Snowflake), the same curve is shifted upward by 5–10× because of per-column read overhead.

Slot 4 — the "consistent read across pages" problem.

OFFSET is positional — it counts rows by their ordinal in the ORDER BY. If a row is inserted before the current position, the ordinal of every subsequent row shifts by one.
Duplicate read — a new row lands above the last cursor position between pages. The old row 21 now becomes new row 22; OFFSET 20 LIMIT 20 returns the old row 20 as its first row on page 2. The user sees it twice.
Skipped read — a row is deleted between pages. The old row 21 disappears; new row 21 is what was old row 22. OFFSET 20 LIMIT 20 returns starting from new row 21 (= old row 22), skipping what the user should have seen.
Both failure modes are silent — no error is thrown, no log line is emitted. The client just sees inconsistent data.

Slot 5 — mitigations that don't fix the fundamental problem.

Serialisable transactions. Wrapping the paginator in BEGIN TRANSACTION ISOLATION LEVEL SERIALIZABLE gives a stable snapshot for all reads. Works, but holds locks and pins MVCC snapshots across HTTP calls — hostile to any web-scale system.
Snapshot cursors. Server-side declared cursors (Postgres DECLARE CURSOR ... FOR ..., SQL Server cursor handles). Same problem — stateful on the DB side, needs sticky sessions, doesn't survive connection pooling.
Row versioning / audit tables. Add deleted_at and updated_at columns; filter out rows deleted after the initial page. Half a fix — new inserts still shift positions.
Cap the max page. Facebook / LinkedIn "5,000 pages max" hard cap. Doesn't fix drift, just bounds the pain.
Keyset pagination. The real fix. Navigate by key, not by position. Immune to inserts, deletes, and updates by design.

Slot 6 — when OFFSET is actually fine.

Small tables (< 1M rows). The whole table fits in RAM; even deep offsets are cheap.
Aggregate tables in warehouses. Superset / Looker dashboards on 10K-row summary tables. Fine.
Admin dashboards with page caps. If your admin UI caps the pager at page 500, deep OFFSET stays bounded.
Batched exports. The exporter reads pages 1 → P sequentially; total cost is quadratic, but if P is small the constant hides it.
Prototyping. Ship OFFSET, get the UI working, replace with keyset before scaling. This is the standard trajectory.

Slot 7 — reading the EXPLAIN plan.

Postgres EXPLAIN (ANALYZE, BUFFERS) — look for Buffers: shared read=N on the Limit step. N grows linearly with k for OFFSET; stays flat for keyset.
MySQL 8 EXPLAIN FORMAT=TREE — look for rows examined on the scan. Same story.
SQL Server SET STATISTICS IO ON — the logical reads count grows with k.
Snowflake — the query profile shows Rows Scanned on the source table. For OFFSET 1M LIMIT 20, Rows Scanned is ~1M; for keyset, it's ~20 (post-prune).
BigQuery — the "Bytes billed" tab shows OFFSET reads the full pre-limit column data. Keyset reads dramatically less.

Common newbie mistakes.

Assuming OFFSET is O(1). It's O(k + n). Every deep-page complaint traces back to this misconception.
Not adding a tie-breaker. Rows with duplicate ORDER BY keys drift across pages.
Not adding NULLS LAST. NULLs appear or disappear across pages depending on engine defaults.
Trusting OFFSET under concurrent writes. It drifts — silently.
Assuming the covering index makes OFFSET fast. It makes it faster than a sort-and-discard, but still O(k). The keyset path is O(log n + K) — dramatically different.

Worked example — measuring the OFFSET latency curve with EXPLAIN ANALYZE

Detailed explanation. The clearest demonstration — run EXPLAIN (ANALYZE, BUFFERS) at pages 1, 100, 1K, 10K, 100K, and 1M on the same query, then read the numbers. Every senior engineer has done this at least once; the interview question is often "walk me through what changes."

Question. Given a Postgres events(id, user_id, created_at, payload) table with 10M rows and a covering btree on (created_at DESC, id DESC), run EXPLAIN (ANALYZE, BUFFERS) for LIMIT 20 OFFSET k at k = 0, 20,000, 200,000, and 2,000,000. Report the latency curve.

Input. (10M-row events table, indexed on (created_at DESC, id DESC).)

Code.

EXPLAIN (ANALYZE, BUFFERS)
SELECT id, user_id, created_at
FROM events
ORDER BY created_at DESC, id DESC
LIMIT 20 OFFSET 0;
-- Limit  (cost=0.43..1.28 rows=20)
--   Buffers: shared hit=6
--   -> Index Scan Backward using idx_events (cost=0.43..425000.00 rows=10000000)
-- Planning Time: 0.1 ms
-- Execution Time: 3 ms

EXPLAIN (ANALYZE, BUFFERS)
SELECT id, user_id, created_at
FROM events
ORDER BY created_at DESC, id DESC
LIMIT 20 OFFSET 20000;
-- Limit  (cost=849.30..850.15 rows=20)
--   Buffers: shared hit=1200 read=800
--   Execution Time: 45 ms

EXPLAIN (ANALYZE, BUFFERS)
SELECT id, user_id, created_at
FROM events
ORDER BY created_at DESC, id DESC
LIMIT 20 OFFSET 200000;
-- Limit  (cost=8493.00..8493.85 rows=20)
--   Buffers: shared hit=8000 read=12000
--   Execution Time: 380 ms

EXPLAIN (ANALYZE, BUFFERS)
SELECT id, user_id, created_at
FROM events
ORDER BY created_at DESC, id DESC
LIMIT 20 OFFSET 2000000;
-- Limit  (cost=84930.00..84930.85 rows=20)
--   Buffers: shared hit=50000 read=150000
--   Execution Time: 3800 ms

Step-by-step explanation.

Page 1 (OFFSET 0) — 6 buffer hits, 3 ms. The plan reads 20 index leaves and emits 20 rows. Best case.
Page 1,001 (OFFSET 20,000) — 2,000 buffer accesses, 45 ms. Postgres walks 20,000 index leaves before emitting. Latency grows roughly linearly with k.
Page 10,001 (OFFSET 200,000) — 20,000 buffer accesses, 380 ms. shared read (buffer misses) start climbing because the working set exceeds shared_buffers.
Page 100,001 (OFFSET 2,000,000) — 200,000 buffer accesses, 3.8 s. Almost all reads are shared read — buffer pool saturated. This is the "someone will page the on-call" latency.
The Buffers: shared read count is the smoking gun — for OFFSET, it grows linearly with k; for keyset, it stays flat at 6–10 per page regardless of depth.

Output — latency curve.

Page (OFFSET)	Buffers	Latency
1 (0)	6	3 ms
1,001 (20,000)	2,000	45 ms
10,001 (200,000)	20,000	380 ms
100,001 (2,000,000)	200,000	3,800 ms

Rule of thumb. Deep OFFSET's latency curve is linear-then-super-linear. Everything under k ≤ 10,000 on a covering-indexed table is fine; everything past k ≥ 100,000 is a support ticket waiting to happen. Add a hard cap or migrate to keyset — the choice is not "if" but "when."

Worked example — the concurrent-write drift bug in action

Detailed explanation. The most infuriating pagination bug: pages that "shift" as users read them. This example builds a minimal reproduction using two sessions and shows exactly which row is missed and which row is duplicated.

Question. Simulate two sessions — session A reads page 1 of an OFFSET paginator, session B inserts a new row, session A reads page 2. Show which row is duplicated. Then show the same scenario with a delete instead of an insert.

Input. Rows 1–40 in events, sorted DESC by created_at, all with a unique created_at.

Code.

-- Session A: page 1
SELECT id FROM events ORDER BY created_at DESC LIMIT 20 OFFSET 0;
-- Returns: 40, 39, 38, ..., 21

-- Session B: insert a new row with created_at newer than any existing
INSERT INTO events (id, created_at) VALUES (41, '2026-07-11 00:00');

-- Session A: page 2 (OFFSET drifted)
SELECT id FROM events ORDER BY created_at DESC LIMIT 20 OFFSET 20;
-- Returns: 21, 20, 19, ..., 2
-- id=21 IS SEEN TWICE (once at end of page 1, once at start of page 2)


-- Reset. Repeat with delete.
-- Session A: page 1
SELECT id FROM events ORDER BY created_at DESC LIMIT 20 OFFSET 0;
-- Returns: 40, 39, ..., 21

-- Session B: delete row 30 (which was on page 1)
DELETE FROM events WHERE id = 30;

-- Session A: page 2
SELECT id FROM events ORDER BY created_at DESC LIMIT 20 OFFSET 20;
-- Returns: 20, 19, ..., 1
-- id=21 is SKIPPED (it should have been on page 2 but the offset now starts at 20)

Step-by-step explanation.

Pre-insert state — 40 rows sorted DESC. Page 1 (offset 0, limit 20) returns rows 40 → 21.
Session B inserts id=41 with a newer timestamp. The ordered stream is now 41, 40, 39, ..., 1 — 41 rows total.
Session A calls page 2 (offset 20, limit 20). Offset 20 in the new stream is row 21 (41, 40, ..., 22 is the first 20). Page 2 returns 21, 20, ..., 2 — the user sees id=21 twice.
In the delete scenario, session B deletes id=30 from the middle of page 1. The stream becomes 40, 39, ..., 31, 29, 28, ..., 1 — 39 rows.
Session A calls page 2. Offset 20 in the shrunken stream is row 21 in the original stream, which is now... row 20 in the new stream (since one was deleted). The paginator emits 20, 19, ..., 1 — id=21 is skipped entirely.

Output. Both scenarios produce inconsistent data — one duplicates, one skips.

Rule of thumb. Any OFFSET paginator against a mutable source has this drift. Every social feed, every "list of my orders" endpoint, every "list of my repos" page uses keyset instead. Keyset navigates by key (the last row's tuple), so inserts and deletes elsewhere in the ordered stream have no effect on subsequent pages.

Worked example — bounded-OFFSET as a safety net

Detailed explanation. If you can't migrate to keyset today (legacy admin dashboard, page-numbered pager), the cheap fix is a hard cap. Facebook, LinkedIn, and Google search results all do this — they refuse to serve past page N.

Question. Design a bounded-OFFSET endpoint that caps at 5,000 pages of 100 rows each. What are the trade-offs?

Input. A 100M-row orders table.

Code.

def list_orders_capped(page: int, page_size: int):
    # Hard caps
    MAX_PAGE = 5_000
    MAX_PAGE_SIZE = 200

    if page > MAX_PAGE:
        raise HttpError(400, "Page number exceeds maximum. Use search or filters.")

    page_size = min(page_size, MAX_PAGE_SIZE)
    offset = page * page_size

    # Worst case: page=5000, page_size=200 => OFFSET 1,000,000 LIMIT 200
    # Bounded ~600ms on a covering-indexed table
    rows = db.query(
        """
        SELECT id, created_at, total
        FROM orders
        ORDER BY created_at DESC, id DESC
        LIMIT %s OFFSET %s
        """,
        (page_size, offset),
    )
    return rows

Step-by-step explanation.

The MAX_PAGE = 5000 cap means the largest OFFSET is 5000 × 200 = 1,000,000. Bounded worst case.
The MAX_PAGE_SIZE = 200 cap means no client can ask for a 10K-row page and force a LIMIT 10000 OFFSET 100000 monster query.
If the client asks for page 5,001, the endpoint returns a 400 with a hint to use search filters (or infinite scroll via a keyset endpoint).
Deep-page latency at the cap is bounded — ~600 ms on a covering-indexed 100M-row table. Acceptable for an admin UI; unacceptable for consumer feeds (that's why consumer uses keyset).
This is the Facebook-search pattern — "your search returned too many results, please refine your query" is not a UX failure, it is a deliberate cost boundary.

Output.

Client on page ≤ 5,000 — normal response, bounded latency.
Client on page > 5,000 — 400 error with guidance.

Rule of thumb. Bounded OFFSET is the "we can't migrate today" fix. It doesn't cure the drift problem (concurrent writes still cause duplicates / skips), but it caps the perf blast radius. Every legacy admin dashboard in production ships this pattern.

Senior interview question on OFFSET perf diagnosis

A senior interviewer might ask: "You get a Slack ping — the orders list page is 8 seconds at page 500. The engineer who wrote it says 'we have an index.' Diagnose the failure in under two minutes."

Solution Using EXPLAIN ANALYZE + covering-index verification + keyset migration path

-- Step 1: EXPLAIN the failing query
EXPLAIN (ANALYZE, BUFFERS)
SELECT id, user_id, created_at, total
FROM orders
ORDER BY created_at DESC, id DESC
LIMIT 20 OFFSET 10000;

-- Look for:
--   1. Is the scan using the index? (Index Scan Backward using idx_orders_created_at)
--   2. Is the sort happening in the executor? (Sort node → covering index missing / mis-aligned)
--   3. Buffers: shared read count (growing with OFFSET = OFFSET trap)
--   4. Any Filter step after the index scan (missing covering columns)

-- Step 2: Verify the index actually covers the ORDER BY
\d+ orders
-- If the existing index is (created_at) but the ORDER BY is (created_at DESC, id DESC),
-- Postgres 15+ can still walk backward but old versions do a sort.

-- Step 3: Add the covering composite if missing
CREATE INDEX CONCURRENTLY idx_orders_paginate
  ON orders (created_at DESC, id DESC);
-- Now the Limit → Index Scan Backward path is optimal for OFFSET too.
-- But OFFSET remains O(k) — indexing does not fix the trap.

-- Step 4: Migrate to keyset for the consumer endpoint
SELECT id, user_id, created_at, total
FROM orders
WHERE (created_at, id) < (:cursor_ts, :cursor_id)
ORDER BY created_at DESC, id DESC
LIMIT 20;
-- Now O(log n + K) at any depth.

Step-by-step trace.

Step	What you check	What it tells you
1	EXPLAIN plan node type	Sort node = missing / mis-aligned index; Index Scan Backward = good
2	Buffers: shared read	Growing with OFFSET = classic OFFSET trap
3	\d+ orders index list	Composite `(created_at DESC, id DESC)` = optimal; single-col = suboptimal
4	Actual rows vs actual loops	20 rows but 10,020 loops = classic OFFSET scan-and-discard
5	Fix path A (short-term)	CREATE INDEX CONCURRENTLY (covering)
6	Fix path B (long-term)	Migrate to keyset for consumer endpoints

The senior diagnosis is: identify the OFFSET trap in EXPLAIN, verify the covering index is in place (adding it if missing), and set the migration to keyset for any endpoint expected to serve deep pages. The short-term win is a 2–3× improvement from index alignment; the long-term win is O(log n) from keyset — a 100× improvement at page 100K.

Output:

Fix	Page 500 latency	Page 100K latency	Sustainable?
Baseline (missing composite index)	8 s	Timeout	No
Composite index added	300 ms	4 s	For admin UI with page cap
Migrated to keyset	8 ms	8 ms	Yes — flat forever

Why this works — concept by concept:

EXPLAIN ANALYZE reveals the plan shape — the actual rows vs actual loops line is diagnostic. A LIMIT of 20 that runs 10,000 loops is the classic OFFSET signature. Every senior engineer reads this line first.
Composite index alignment — the ORDER BY (created_at DESC, id DESC) matches a composite index (created_at DESC, id DESC); Postgres walks the index leaves backward without a Sort node. Reducing the sort step is the cheapest short-term win.
OFFSET is O(k) even with covering index — indexing does not fix the trap; it just makes the scan-and-discard cheaper per row. At k = 100,000 on a covering-indexed table, latency is still seconds.
Keyset is O(log n + K) — the tuple predicate uses the same composite index as a seek. The plan is Index Scan Backward with actual loops = 20, regardless of depth.
Cost — Baseline (no composite) is O(k × log n) with sort; composite-only is O(k); keyset is O(log n + K). On a 100M-row table with K=20 and k=100,000, the numbers are ~40 s / ~4 s / ~10 ms — a 400× spread across the three fixes.

SQL
Topic — indexing
SQL indexing drills

Practice →

SQL Topic — optimization SQL optimization problems

Practice →

4. Keyset (seek method) pagination

`keyset pagination` and `seek method pagination` — WHERE (created_at, id) < (:cursor) ORDER BY DESC LIMIT n, composite index alignment, and the O(log n) deep-page path

The mental model in one line: keyset pagination (a.k.a. seek method pagination) replaces "skip k rows" with "give me the rows strictly after this cursor tuple," where the cursor is the last row of the previous page and the comparison is a lexicographic tuple compare — turning the deep-page cost from O(k + n) (scan + discard) into O(log n + K) (one index seek + K sequential reads). Every modern feed, every GraphQL Relay endpoint, and every "next 20" API on the internet is built on this primitive.

Slot 1 — the keyset skeleton.

Two-slot template: Slot A — the WHERE tuple comparison: WHERE (order_col, id) < (:cursor_col, :cursor_id) for DESC pagination, > for ASC. Slot B — the ORDER BY with matching direction and tie-breaker: ORDER BY order_col DESC, id DESC LIMIT n.
Every keyset query fills the two slots. The cursor is (last_row.order_col, last_row.id) from the previous page — no OFFSET, no page number.
The tuple comparison (a, b) < (:a, :b) is lexicographic — first a < :a, or a = :a AND b < :b. This is not the same as a < :a AND b < :b — that comparison would drop rows where a = :a AND b < :b.

Slot 2 — tuple comparison is one-shot, not per-column.

Postgres, MySQL 8, Oracle 12c+, Snowflake, and BigQuery all support WHERE (a, b) < (:a, :b) as native row-value comparison. This is the ANSI SQL:1999 spelling.
SQL Server and older MySQL don't support row-value comparison. The equivalent is WHERE a < :a OR (a = :a AND b < :b) — verbose but semantically identical.
The one-shot form is not just a syntactic nicety — it lets the planner recognise the comparison as a range scan on the composite index (a, b) and use a single index seek. The expanded per-column form may or may not compile to the same plan depending on planner cleverness.

Slot 3 — composite index alignment.

The composite index (order_col, id) in the same direction as the ORDER BY (DESC or ASC) enables an index-only backward or forward scan.
Postgres 15+ can walk any btree in either direction, so CREATE INDEX ... ON t (created_at DESC, id DESC) and CREATE INDEX ... ON t (created_at ASC, id ASC) work equivalently for backward scans.
MySQL 8.0.13+ supports descending indexes (CREATE INDEX ... ON t (created_at DESC, id DESC)). Older MySQL always stored indexes ascending — the planner walked them backward for DESC scans, which was slightly less efficient.
SQL Server has supported descending index columns since forever. Oracle since 8i.
Snowflake has no btree — the equivalent is CLUSTER BY (created_at), which reorders micro-partitions to co-locate rows with similar created_at values.
BigQuery uses table clustering — CLUSTER BY created_at — which does the same physical co-location.

Slot 4 — the O(log n + K) deep-page path.

The index seek on (created_at, id) < (:ts, :id) positions the scan at exactly the right leaf page in O(log n) — the depth of the btree.
From that leaf, the engine reads K sequential rows (where K is the LIMIT). Each read is O(1). Total forward-scan cost is O(K).
Overall: O(log n + K) per page — flat in the depth of the pagination. Page 1 and page 1,000,000 both cost the same.
Compared to OFFSET's O(k + n) — at page 100K with n = 20, keyset is 5,000× faster.

Slot 5 — DESC pagination vs ASC pagination.

DESC (newest-first feed) — WHERE (created_at, id) < (:cursor_ts, :cursor_id) ORDER BY created_at DESC, id DESC LIMIT n. Cursor is the last (oldest) row emitted.
ASC (oldest-first log) — WHERE (created_at, id) > (:cursor_ts, :cursor_id) ORDER BY created_at ASC, id ASC LIMIT n. Cursor is the last (newest) row emitted.
The < / > operator flips with the direction; the ORDER BY direction flips too; the tie-breaker direction must match.
Mixing directions — ORDER BY created_at DESC, id ASC — is possible but requires the tuple comparison and the ORDER BY to match. The composite index must be (created_at DESC, id ASC) or Postgres can't do an index-only scan.

Slot 6 — bidirectional pagination.

Forward — WHERE (a, b) < (:a, :b) ORDER BY a DESC, b DESC LIMIT n. Returns the next N older rows.
Backward — WHERE (a, b) > (:a, :b) ORDER BY a ASC, b ASC LIMIT n, then reverse the result set in application code. Returns the previous N newer rows.
The Relay spec's first / after / last / before maps directly to this — first + after is forward, last + before is backward.
Some engines support the ORDER BY a ASC, b ASC LIMIT n variant with the result reversed — Postgres does this via a subquery: SELECT * FROM (... ORDER BY a ASC LIMIT n) t ORDER BY a DESC.

Slot 7 — trade-offs and honest limitations.

No random access. You can't ask for "page 47" — keyset only lets you paginate forward or backward from a known cursor. The trade-off is: no page numbers, no total-count-of-pages banner.
Total row count is expensive. Keyset gives you hasNext cheaply (via LIMIT n+1 peek-ahead) but no total. Total count on a mutable table is O(n) anyway — the OFFSET pager pays that cost only because the total-count query is a separate SELECT COUNT(*).
Skipping a page is impossible. Client-side "jump to page 100" doesn't map to keyset. The only workaround is a hybrid — keyset for infinite scroll, bounded OFFSET for admin dashboards.
Cursor must include all ORDER BY columns. If the ORDER BY is (created_at, id), the cursor is (created_at, id). Adding a third sort column adds a third cursor component.

Slot 8 — handling NULLs correctly.

If created_at can be NULL, the tuple comparison behaves unexpectedly. On most engines, NULL < anything and NULL > anything both return NULL — the row is not returned.
Fix — use ORDER BY created_at DESC NULLS LAST and, in the WHERE, use a COALESCE(created_at, '-infinity') cast to give NULLs a defined position in the order.
Better fix — never allow NULLs on the paginated sort column. Make it NOT NULL and populate on insert.
On BigQuery, the default for ORDER BY x DESC is NULLs first. Use NULLS LAST explicitly.

Slot 9 — handling duplicate ORDER BY keys (ties).

Two rows with the same created_at — the tie-breaker on id disambiguates. WHERE (created_at, id) < (:ts, :id) correctly excludes only the exact cursor row and continues from there.
Without the id tie-breaker, WHERE created_at < :ts would skip all rows with created_at = :ts — you might miss valid rows.
The tie-breaker column must be unique across the table (usually the primary key id) — otherwise the composite comparison can still tie and cause drift.

Slot 10 — cursor stability under mutations.

New insert with created_at > cursor — the cursor's tuple stays the same; the new row appears on future backward pages (relative to a newer cursor), never on already-paginated results. Perfect.
New insert with created_at < cursor — same story; the new row is in front of the cursor and thus in future forward pages.
Delete a row already paginated — its removal doesn't affect anything; the cursor points to a row that's still in the ordered sequence (or, if the cursor itself is deleted, the next call skips ahead to the next valid row).
Update the paginated sort column (e.g. UPDATE events SET created_at = ...) — the updated row can move in the sequence, but since keyset uses the cursor's remembered tuple, not the current row's tuple, subsequent pages remain consistent.

Common newbie mistakes.

Using per-column WHERE instead of tuple comparison — WHERE created_at < :ts AND id < :id drops rows where created_at < :ts AND id ≥ :id. Wrong semantically.
Forgetting the tie-breaker — two rows with the same created_at alternate between pages.
Wrong ORDER BY direction — ORDER BY created_at DESC, id ASC with a < comparison drops the wrong rows.
Missing composite index — the query works but plans as a Filter after a full sort. Latency stays O(n).
Encoding the cursor as page=N — this is not keyset. Cursors are opaque tuples, not page numbers.

Interview probes on syntax fluency.

"Why is tuple comparison better than per-column AND/OR?" — one-shot lexicographic comparison, single index-range scan, no chance of dropping edge rows.
"What index does keyset need?" — composite on the ORDER BY columns in the ORDER BY direction, matching direction to the DESC/ASC.
"What if two rows share a millisecond?" — the unique tie-breaker (id) disambiguates. Every keyset paginator ends the ORDER BY with , id or another unique column.
"What if the sort column is NULL for some rows?" — either the column is NOT NULL, or you use COALESCE(sort_col, sentinel) in both the WHERE and ORDER BY. The NULL problem is one of the two most common keyset bugs.
"How do you paginate backward?" — swap < for >, flip the ORDER BY direction, reverse the result in application code. Or use first/last + after/before if you're implementing the Relay spec.

Worked example — the "next 20 events" keyset query on Postgres

Detailed explanation. The canonical keyset query — DESC pagination on (created_at, id) with a composite index. Every consumer feed on the internet uses this shape.

Question. Given events(id BIGINT PRIMARY KEY, created_at TIMESTAMPTZ NOT NULL, user_id BIGINT, payload TEXT) with an index on (created_at DESC, id DESC), write the Postgres query that returns the next 20 events after a cursor (cursor_ts, cursor_id).

Input.

10M rows, sorted DESC by (created_at, id).
Cursor is (2026-07-05 12:00, 5000000) — the last row of the previous page.

Code.

SELECT id, user_id, created_at, payload
FROM events
WHERE (created_at, id) < (:cursor_ts, :cursor_id)
ORDER BY created_at DESC, id DESC
LIMIT 20;

Step-by-step explanation.

WHERE (created_at, id) < (:cursor_ts, :cursor_id) — tuple comparison. Lexicographic — first created_at < :cursor_ts, or created_at = :cursor_ts AND id < :cursor_id.
This maps to a range predicate on the composite index (created_at DESC, id DESC). Postgres does one index seek to position at the cursor, then scans backward.
ORDER BY created_at DESC, id DESC — matches the index order. No sort step.
LIMIT 20 — bounds the scan. The engine reads exactly 20 rows past the cursor position and stops.
Total plan cost — Index Scan Backward using idx_events_paginate (cost=0.43..1.28) with actual rows=20 actual loops=1. Flat regardless of pagination depth.

Output (20 rows just older than the cursor).

id	user_id	created_at	payload
4999999	42	2026-07-05 11:59:59	click
4999998	91	2026-07-05 11:59:58	login
...	...	...	...
4999980	17	2026-07-05 11:59:00	logout

Rule of thumb. The two-line skeleton — tuple compare + ORDER BY DESC — is the entire keyset primitive. Memorise both slots; every future keyset query is a variation on this template.

Worked example — expanded per-column form for SQL Server (no row-value comparison)

Detailed explanation. SQL Server (and older MySQL) doesn't support the ANSI row-value comparison WHERE (a, b) < (:a, :b). The expanded form is verbose but semantically identical.

Question. Rewrite the previous keyset query using per-column WHERE clauses for SQL Server compatibility. Verify the planner still uses a range scan.

Input. (Same events table, on SQL Server.)

Code.

-- SQL Server — expanded per-column form
SELECT TOP 20 id, user_id, created_at, payload
FROM events
WHERE
     created_at < @cursor_ts
  OR (created_at = @cursor_ts AND id < @cursor_id)
ORDER BY created_at DESC, id DESC;

Step-by-step explanation.

Row-value comparison (a, b) < (:a, :b) is not SQL Server-native. The expanded form is a < :a OR (a = :a AND b < :b).
The OR decomposition is critical — a < :a AND b < :b is wrong — it drops rows where a < :a AND b >= :b.
SQL Server's optimiser recognises the OR (equal AND less) shape and typically compiles it to a range seek on the composite index. Verify with the execution plan.
SELECT TOP 20 replaces LIMIT 20 — SQL Server's spelling. Semantically identical.
If the planner does not recognise the pattern (older SQL Server versions, or complex predicates), it may fall back to a scan + filter — always verify with the actual plan.

Output. Same 20 rows as the tuple-form query.

Rule of thumb. On SQL Server and older MySQL, use the expanded OR (equal AND less) form. It's more verbose but semantically identical. Every keyset paginator against SQL Server ships this pattern.

Worked example — backward pagination (previous page)

Detailed explanation. The Relay spec's last + before — "give me the 20 rows immediately newer than this cursor." Implementable with a flipped tuple compare and a reversed result set.

Question. Write a Postgres query that returns the 20 events immediately newer than a cursor (cursor_ts, cursor_id), in DESC order.

Input. (Same events table; cursor (2026-07-05 12:00, 5000000).)

Code.

-- Backward keyset — get the 20 rows immediately newer than the cursor
SELECT * FROM (
  SELECT id, user_id, created_at, payload
  FROM events
  WHERE (created_at, id) > (:cursor_ts, :cursor_id)
  ORDER BY created_at ASC, id ASC
  LIMIT 20
) t
ORDER BY created_at DESC, id DESC;

Step-by-step explanation.

Inner query — flip < to > (newer, not older) and flip ORDER BY to ASC (so the closest rows to the cursor are read first). LIMIT 20.
This produces 20 rows in ASC order — the "reading direction" for backward pagination but the wrong final order for the UI.
Outer query — re-sort DESC to match the standard feed ordering. Rows are the same; the presentation flips.
The two-step (inner-ASC, outer-DESC) is unavoidable — you need the closest-first read direction internally but the newest-first presentation externally.
On engines with FETCH FIRST 20 ROWS ONLY inside a subquery restriction (Oracle 12c+), the same pattern works with the ANSI spelling instead of LIMIT.

Output (20 rows just newer than the cursor, presented newest-first).

id	user_id	created_at	payload
5000020	...	2026-07-05 12:00:20	...
...	...	...	...
5000001	...	2026-07-05 12:00:01	...

Rule of thumb. Backward pagination flips the operator, flips the direction, and reverses the result. The composite index on (created_at DESC, id DESC) works for both directions on Postgres 15+.

Worked example — keyset over composite ORDER BY with mixed directions

Detailed explanation. A subtle case — the ORDER BY has one DESC column and one ASC tie-breaker (ORDER BY score DESC, id ASC for leaderboards). The tuple comparison needs to handle the mixed direction.

Question. Write a Postgres keyset query for a leaderboard sorted by score DESC, id ASC, given a cursor (cursor_score, cursor_id).

Input. leaderboard(id INT PRIMARY KEY, score INT NOT NULL). Composite index (score DESC, id ASC).

Code.

-- Leaderboard: score DESC, id ASC as tie-breaker
SELECT id, score
FROM leaderboard
WHERE
     score < :cursor_score
  OR (score = :cursor_score AND id > :cursor_id)
ORDER BY score DESC, id ASC
LIMIT 20;

Step-by-step explanation.

The ORDER BY has mixed directions — score DESC, id ASC. Row-value comparison doesn't fit — (score, id) < (:cursor_score, :cursor_id) would compare both columns in the same direction.
Use the expanded form — score < :cursor_score (strictly worse score) OR (score = :cursor_score AND id > :cursor_id) (same score, larger id — the tie-breaker direction).
The composite index must match — (score DESC, id ASC). Postgres supports mixed-direction indexes since 8.3.
The OR (score = :cursor_score AND id > :cursor_id) — note >, not < — because the tie-breaker is ASC.
Interviewers love this — a candidate who reflexively types (score, id) < (:cs, :ci) shows they haven't thought through the mixed-direction case.

Output (20 rows just past the cursor in the leaderboard).

id	score
101	4800
102	4800
55	4795
...	...

Rule of thumb. Mixed-direction ORDER BY needs the expanded OR (equal AND direction-specific) form. Row-value comparison only works when all columns share the same direction. Every leaderboard paginator ships this pattern.

Senior interview question on keyset design

A senior interviewer might ask: "Design the composite index and the SQL for a keyset-paginated chat_messages endpoint that returns the last 30 messages of a specific channel_id. Explain how the index alignment guarantees O(log n) at any depth."

Solution Using channel-scoped composite index + keyset tuple compare

-- Table
CREATE TABLE chat_messages (
  id           BIGINT PRIMARY KEY,
  channel_id   BIGINT NOT NULL,
  created_at   TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  user_id      BIGINT NOT NULL,
  body         TEXT
);

-- Channel-scoped composite index
CREATE INDEX idx_chat_channel_paginate
  ON chat_messages (channel_id, created_at DESC, id DESC);

-- Page 1: last 30 messages of channel 42
SELECT id, user_id, created_at, body
FROM chat_messages
WHERE channel_id = :channel_id
ORDER BY created_at DESC, id DESC
LIMIT 30;

-- Page 2: next 30 older messages after the cursor (last row of page 1)
SELECT id, user_id, created_at, body
FROM chat_messages
WHERE channel_id = :channel_id
  AND (created_at, id) < (:cursor_ts, :cursor_id)
ORDER BY created_at DESC, id DESC
LIMIT 30;

Step-by-step trace.

Step	What happens	Why
1	Composite index `(channel_id, created_at DESC, id DESC)`	Channel as prefix; sort keys after
2	Page 1 query hits index at `channel_id = 42`	O(log n) seek to channel prefix
3	Reads backward from first leaf	O(30) sequential reads
4	Page 2 adds tuple predicate	Positions at cursor within channel prefix
5	Reads next 30 rows backward	O(30) more sequential reads
6	Cursor is opaque `base64((created_at, id))`	Server issues; client passes back

The composite index has channel_id as its leading column so every message in a given channel lives in a contiguous range of the index. Within that range, rows are ordered (created_at DESC, id DESC), so both the initial page and every subsequent keyset page hit the same tight index range. Plan is Index Scan Backward with actual rows = 30 actual loops = 1 for both pages.

Output:

Metric	Page 1	Page 100	Page 10,000
Plan	Index Scan Backward	Index Scan Backward	Index Scan Backward
Rows scanned	30	30	30
Latency (10M msgs, 10K channels)	4 ms	4 ms	4 ms
Cursor size (base64)	~48 B	~48 B	~48 B

Why this works — concept by concept:

Composite index with channel_id as prefix — the leading column matches the WHERE filter (channel_id = :cid), so the index seek positions at the start of the channel's rows in O(log n). Everything after that is a sequential index-leaf scan.
ORDER BY matches the composite index — (created_at DESC, id DESC) inside the channel prefix is stored contiguously; Postgres reads backward with no sort node.
Tuple compare uses the same index range — AND (created_at, id) < (:cursor_ts, :cursor_id) is a range predicate on the composite; the planner tightens the initial seek by one more equality-then-less-than step.
Flat latency at any pagination depth — every page hits the exact same plan shape. Latency at page 1 = latency at page 10,000 = latency at page 100,000.
Cursor size is bounded — a (TIMESTAMPTZ, BIGINT) tuple base64-encodes to ~48 bytes. Fits in a URL query parameter without percent-encoding overhead. HMAC-signed it grows to ~80 bytes — still fine.
Cost — O(log n + K) per page where n is the total row count and K is the LIMIT. For n = 10M and K = 30, that's ~24 index-leaf reads (log₂(10M) ≈ 24) plus 30 sequential reads — well under 5 ms on any modern hardware.

SQL
Topic — indexing
Composite indexing drills

Practice →

SQL Topic — top-N per group Top-N per group problems

Practice →

5. Cursor pagination + dialect matrix

`sql cursor pagination` and `sql infinite scroll` — opaque base64 tokens, HMAC signing, the GraphQL Relay spec, and the six-engine dialect matrix that ties it all together

The mental model in one line: sql cursor pagination is the API-shape layer on top of keyset — the client sends an opaque after token, the server decodes it into (cursor_col, cursor_id), runs the keyset SQL, and returns the next page plus an endCursor for the client to send on the next call, and the whole family of sql infinite scroll endpoints (REST, GraphQL Relay, gRPC streaming) reduces to this one wire contract with slightly different naming. Once you nail the opaque-token discipline and the Relay edges/pageInfo shape, every consumer feed is a variation on the same theme.

Slot 1 — the opaque cursor token.

Cursors are opaque to clients — a base64-encoded string, not a page number, not a raw tuple. Clients treat cursors as black boxes; the server owns the encoding.
Payload — usually a JSON blob {"ts": "2026-07-05T12:00:00Z", "id": 5000000, "dir": "desc"} or a msgpack / protobuf equivalent. JSON is simplest; msgpack shrinks the on-the-wire size.
HMAC-signed. Every cursor is signed with a server-side secret so a client can't tamper with it to skip rows or leak data. cursor = base64(json_payload).hmac(secret). Verification is O(1) on the way in.
Encoding — base64url (no +/ padding characters) is URL-safe and drop-in for query parameters.
Rotation — the HMAC secret rotates every N months; cursors older than the rotation window return a 400 with a hint to re-fetch page 1.

Slot 2 — the Relay Connections spec.

GraphQL's Relay spec (2015, still the standard) defines cursor pagination as first / after / last / before with edges and pageInfo envelope.
Query — orders(first: 20, after: "eyJ0cyI6..."). edges { node { id, ...fields }, cursor } pageInfo { hasNextPage, endCursor }.
Every edge has its own cursor (usable as the after for a "page start here" request). The endCursor is a shorthand for the last edge's cursor.
hasNextPage — cheap via the LIMIT n+1 peek-ahead trick. Fetch 21 rows, return 20, hasNext = (fetched > 20).
hasPreviousPage — same trick backward, or hasPrev = (after cursor is set). Not always cheap to compute exactly.

Slot 3 — REST cursor endpoints.

The naming varies — Twitter uses cursor, GitHub uses page+per_page with Link headers, Stripe uses starting_after / ending_before. All are cursor-based under the hood.
Response envelope — {"data": [...], "next_cursor": "eyJ0cyI6...", "prev_cursor": null}. The client passes next_cursor on the next call.
Link header (GitHub-style) — RFC 5988 links: <https://api.example.com/orders?cursor=eyJ...>; rel="next", <...>; rel="prev". Elegant; clients can parse without knowing the JSON envelope.
Idempotency — the same cursor + limit returns the same rows (assuming no schema changes). Clients can retry safely.

Slot 4 — cursor lifecycle and invalidation.

Cursors survive inserts — they point to a specific row, so new rows in the ordered stream don't affect the cursor's target.
Cursors survive deletes — if the cursor's target row is deleted, the next call reads from just past the (now missing) target position. Effectively the cursor still works.
Cursors survive schema changes — as long as the sort columns are unchanged. If the sort column changes semantics (e.g. renaming created_at to posted_at), old cursors become invalid and the server should reject them with a clear error.
Cursors survive secret rotation — with graceful degradation. Rotate secrets; grandfather the previous secret for one rotation window; reject cursors older than that.
Cursors are short-lived in practice — clients drop them on tab close. But an idle browser tab holding a cursor for a week should still work if the schema hasn't changed.

Slot 5 — the six-engine dialect matrix for keyset.

Postgres — full support. (a, b) < (:a, :b) tuple compare + composite btree in matching direction. Best case.
MySQL 8 — full support since 8.0.16 for row-value comparison; also supports descending indexes since 8.0.13. Perfect for keyset.
SQL Server — no row-value comparison; use expanded OR (equal AND less) form. Composite index seek works via query planner recognition; verify with the plan.
Oracle 12c+ — supports row-value comparison and composite descending indexes. Same story as Postgres.
Snowflake — supports row-value comparison. No btree indexes; the equivalent is CLUSTER BY (sort_col) for micro-partition co-location. Deep-page latency depends on cluster health.
BigQuery — supports row-value comparison in WHERE. No btree; the equivalent is table CLUSTER BY sort_col plus partition pruning if the sort_col has a date component. Best practice: partition by DATE(sort_col), cluster by sort_col, then keyset.

Slot 6 — dialect gotchas.

NULL ordering. Postgres and Oracle default to NULLs first for DESC; MySQL and SQL Server default to NULLs last; BigQuery defaults to NULLs first. Always spell out NULLS LAST explicitly.
Millisecond precision. Different engines store TIMESTAMP at different resolutions — Postgres microseconds, MySQL up to microseconds, SQL Server 100-ns, BigQuery microseconds. Two "identical" timestamps in one engine may differ in another — always include the id tie-breaker.
Case sensitivity in text sort columns. created_at is numeric so no problem; a name-based sort has to consider collation. ORDER BY name COLLATE "C" or the equivalent, plus include the collation in the cursor to avoid inconsistency across mixed locale servers.
Composite index size. A composite (a, b, c) index is 30–50% larger than a single-column (a) index. Keyset requires the composite; the storage cost is the price of admission. Usually worth it.
Row-value comparison support. SQL Server, older MySQL, and some BigQuery legacy modes don't recognise (a, b) < (:a, :b) as a row-value comparison — use the expanded form.

Slot 7 — the peek-ahead trick for hasNextPage.

Client asks for first: 20. Server queries with LIMIT 21 (one extra). If the query returns 21 rows, hasNextPage = true and the server returns only the first 20; if it returns ≤ 20, hasNextPage = false and the server returns all of them.
Zero-cost — the 21st row is a single extra index leaf read. No separate SELECT COUNT(*) query.
The 21st row's cursor is the endCursor for the next page's after parameter — perfect handoff.
Every Relay implementation ships this trick. If your endpoint doesn't do it, hasNextPage requires a separate query that's expensive on large tables.

Slot 8 — HMAC signing and secret management.

Server has a secret S. Cursor payload is P = json({"ts": ..., "id": ...}). Signed cursor is base64url(P) + "." + hmac_sha256(S, base64url(P)).
On the way in — verify signature first. If invalid, return 400 with "invalid cursor." No SQL runs.
Secrets live in a KMS / secret manager. Rotate every 90 days. On rotation, keep the previous secret valid for a grace period equal to the maximum expected client idle time (usually 24–72 hours).
Alternative — encrypt the payload symmetrically with the secret. Adds tamper-proofing plus payload privacy. Not strictly needed for pagination (the payload is not confidential), but common in enterprise deployments.

Slot 9 — total count and its trade-offs.

Keyset gives you hasNextPage cheaply; it does not give you totalCount.
Exact total on a large mutable table is O(n) regardless — SELECT COUNT(*) scans the whole thing or the index.
Approximations — pg_class.reltuples on Postgres, information_schema.tables.table_rows on MySQL, SYSTEM_SAMPLING on Snowflake. Fast but stale.
Best practice — don't return total count on infinite-scroll feeds. Show a scroll bar that grows dynamically instead. On admin dashboards, cache the total count and refresh every few minutes.

Slot 10 — the interview litmus test.

"Design a paginated timeline endpoint." Every good answer starts with cursor pagination, keyset SQL, opaque tokens, and the Relay spec. If the candidate reaches for page=1&limit=20, they're a junior.
"How do you handle 'jump to page 47'?" — you don't, on infinite scroll. On admin dashboards, use capped OFFSET; on data exports, iterate keyset from page 1.
"Explain hasNextPage without an extra query." — the LIMIT n+1 peek-ahead. Instant senior signal.
"Explain why cursors are opaque." — tamper resistance, forward compatibility (server can change the payload schema without breaking clients), and abstraction (clients shouldn't reason about database internals).

Common newbie mistakes.

Exposing raw tuples in cursors. ?cursor=2026-07-05T12:00:00Z_5000000 is a leaky abstraction and a tampering vector.
Skipping HMAC. Anyone can craft a cursor to read rows they shouldn't see. Sign every cursor.
Assuming keyset gives you totalCount. It doesn't; that's a separate expensive query.
Not planning for schema changes. When the sort column changes semantics, old cursors must be rejected cleanly.
Using page numbers instead of cursors on a "list this user's posts" endpoint. If two viewers see slightly different data (privacy filters), page numbers don't align across viewers.

Worked example — encoding and decoding an opaque cursor with HMAC

Detailed explanation. The full round-trip — encode a (created_at, id) tuple into a base64 HMAC-signed cursor, decode it on the way in with signature verification. Every consumer feed does this exactly once.

Question. Write Python that encodes a cursor from (created_at, id) with HMAC-SHA256 signing, and decodes it back on the way in with signature verification. Show the wire format.

Input. created_at = datetime(2026, 7, 5, 12, 0), id = 5000000, secret = b"pagination-secret-2026".

Code.

import base64
import hashlib
import hmac
import json
from datetime import datetime, timezone


SECRET = b"pagination-secret-2026"


def encode_cursor(created_at: datetime, row_id: int) -> str:
    payload = {
        "ts": created_at.astimezone(timezone.utc).isoformat(),
        "id": row_id,
        "dir": "desc",
    }
    payload_bytes = json.dumps(payload, separators=(",", ":"), sort_keys=True).encode()
    payload_b64 = base64.urlsafe_b64encode(payload_bytes).rstrip(b"=").decode()
    signature = hmac.new(SECRET, payload_b64.encode(), hashlib.sha256).digest()
    sig_b64 = base64.urlsafe_b64encode(signature).rstrip(b"=").decode()
    return f"{payload_b64}.{sig_b64}"


def decode_cursor(cursor: str) -> tuple[datetime, int]:
    try:
        payload_b64, sig_b64 = cursor.split(".", 1)
    except ValueError:
        raise ValueError("malformed cursor")
    # Verify signature
    expected_sig = hmac.new(SECRET, payload_b64.encode(), hashlib.sha256).digest()
    expected_sig_b64 = base64.urlsafe_b64encode(expected_sig).rstrip(b"=").decode()
    if not hmac.compare_digest(sig_b64, expected_sig_b64):
        raise ValueError("cursor signature invalid")
    # Decode payload
    padding = b"=" * (-len(payload_b64) % 4)
    payload_bytes = base64.urlsafe_b64decode(payload_b64.encode() + padding)
    payload = json.loads(payload_bytes)
    return datetime.fromisoformat(payload["ts"]), payload["id"]


# Round-trip
ts = datetime(2026, 7, 5, 12, 0, tzinfo=timezone.utc)
cursor = encode_cursor(ts, 5_000_000)
print(cursor)
# eyJkaXIiOiJkZXNjIiwiaWQiOjUwMDAwMDAsInRzIjoiMjAyNi0wNy0wNVQxMjowMDowMCswMDowMCJ9.CzRK9pKu1EWvKQrKn2H8kMXvpQXwZR7Tu-y1O4iF1Ho

ts2, id2 = decode_cursor(cursor)
assert (ts2, id2) == (ts, 5_000_000)

Step-by-step explanation.

Payload — a JSON dict with ts, id, and dir. Sort keys for determinism (so re-encoding produces the same bytes).
base64.urlsafe_b64encode — URL-safe base64 (uses -_ instead of +/), strips = padding so the cursor is drop-in for URL query parameters.
Signature — HMAC-SHA256 over the base64 payload (not the raw JSON, so the signature covers the exact wire form). Base64-encode the signature too.
Wire format — payload_b64.signature_b64. Two base64url strings separated by a dot, JWT-like but without a header.
Decode — split on the dot, verify signature with hmac.compare_digest (constant-time compare to prevent timing attacks), then decode the payload.

Output.

Cursor eyJkaXIiOiJkZXNjIiwiaWQiOjUwMDAwMDAsInRzIjoiMjAyNi0wNy0wNVQxMjowMDowMCswMDowMCJ9.CzRK9pKu1EWvKQrKn2H8kMXvpQXwZR7Tu-y1O4iF1Ho (or similar; signature varies with the secret).
Decode returns (datetime(2026,7,5,12,0,tzinfo=UTC), 5_000_000).

Rule of thumb. JWT-like signed cursors are the industry standard. Every consumer API in 2026 uses this exact shape (with minor variations in payload schema and signing algorithm). Roll it yourself in 30 lines of code or use a library like itsdangerous.

Worked example — the Relay Connections envelope

Detailed explanation. The GraphQL Relay spec, in one worked query. This is what every graphql-ruby, graphene, strawberry, and apollo-server codebase writes verbatim.

Question. Given the events table and the keyset SQL from before, write the resolver for a Relay-style events(first: 20, after: "...") connection query.

Input. events(id, user_id, created_at, payload) table.

Code.

from typing import Optional

def resolve_events_connection(first: int, after: Optional[str]) -> dict:
    # Clamp first to prevent unbounded page sizes
    first = min(max(first, 1), 100)

    # Decode the after cursor (if any)
    if after:
        try:
            cursor_ts, cursor_id = decode_cursor(after)
        except ValueError:
            raise GraphQLError("invalid cursor")
        rows = db.query(
            """
            SELECT id, user_id, created_at, payload
            FROM events
            WHERE (created_at, id) < (%s, %s)
            ORDER BY created_at DESC, id DESC
            LIMIT %s
            """,
            (cursor_ts, cursor_id, first + 1),
        )
    else:
        rows = db.query(
            """
            SELECT id, user_id, created_at, payload
            FROM events
            ORDER BY created_at DESC, id DESC
            LIMIT %s
            """,
            (first + 1,),
        )

    has_next = len(rows) > first
    rows = rows[:first]
    edges = [
        {
            "node": {
                "id": r["id"],
                "user_id": r["user_id"],
                "created_at": r["created_at"],
                "payload": r["payload"],
            },
            "cursor": encode_cursor(r["created_at"], r["id"]),
        }
        for r in rows
    ]

    return {
        "edges": edges,
        "pageInfo": {
            "hasNextPage": has_next,
            "hasPreviousPage": after is not None,
            "startCursor": edges[0]["cursor"] if edges else None,
            "endCursor": edges[-1]["cursor"] if edges else None,
        },
    }

Step-by-step explanation.

Clamp first to a safe range (1–100). Prevents a client from asking for a million-row page.
If after is provided, decode it (with signature verification). If invalid, return a GraphQL error.
SQL — keyset if after is set, initial-page if not. Both queries fetch first + 1 rows for the peek-ahead.
has_next = (len(rows) > first) — the peek-ahead trick. Trim to first rows before returning.
Build edges with a per-row cursor (generated from (created_at, id)) plus pageInfo with hasNextPage / hasPreviousPage / startCursor / endCursor.

Output — Relay-shaped JSON envelope, ready for apollo-client or relay-runtime.

Rule of thumb. The first + 1 peek-ahead + edges + pageInfo envelope is the entire Relay spec, condensed. Every GraphQL feed on the internet ships this exact pattern.

Worked example — BigQuery keyset with partition pruning

Detailed explanation. The BigQuery specialisation — partition by DATE(sort_col) and cluster by sort_col. Keyset works, but you have to be careful with the outer WHERE to prune partitions.

Question. Design the BigQuery keyset paginator for a 10B-row events table, partitioned by DATE(created_at) and clustered by (created_at, id). Include the partition prune predicate.

Input. events table on BigQuery with PARTITION BY DATE(created_at) CLUSTER BY created_at, id.

Code.

-- BigQuery keyset with partition prune
SELECT id, user_id, created_at, payload
FROM `project.dataset.events`
WHERE
  -- Partition prune: only relevant days
  DATE(created_at) BETWEEN
    DATE_SUB(DATE(@cursor_ts), INTERVAL 7 DAY) AND DATE(@cursor_ts)
  -- Keyset tuple comparison
  AND (created_at, id) < (@cursor_ts, @cursor_id)
ORDER BY created_at DESC, id DESC
LIMIT 21;

Step-by-step explanation.

Partition prune — DATE(created_at) BETWEEN cursor_date - 7 AND cursor_date. Reads only the last 7 days of partitions relative to the cursor. Tighter windows read fewer partitions but risk missing older rows if the user paginates deep.
Alternative — DATE(created_at) <= DATE(@cursor_ts). Prunes all partitions newer than the cursor; reads all older ones. Broader but always correct.
Keyset tuple compare — same as Postgres/MySQL. BigQuery has supported row-value comparison since 2019.
CLUSTER BY (created_at, id) — within each partition, rows are physically co-located by (created_at, id). The tuple compare uses the clustering to prune blocks.
LIMIT 21 — peek-ahead for hasNextPage.

Output. 20 rows per page, flat latency at any depth, ~150 ms per page on a 10B-row table.

Rule of thumb. On BigQuery (or any partitioned columnar warehouse), always combine keyset with a partition-prune predicate. Skipping the prune predicate means a full-table scan even with keyset — the tuple compare alone won't prune partitions on its own.

Senior interview question on API-shape and cursor discipline

A senior interviewer might ask: "Design the wire format for an infinite-scroll messages endpoint. What goes in the cursor, how do you sign it, what does the response envelope look like, and how do you handle secret rotation?"

Solution Using signed opaque cursor + Relay envelope + graceful secret rotation

import base64
import hashlib
import hmac
import json
import os
from datetime import datetime, timezone
from typing import Optional

# Two secrets — active and previous — to allow graceful rotation
SECRETS = {
    "v2": os.environ["CURSOR_SECRET_V2"],  # current
    "v1": os.environ.get("CURSOR_SECRET_V1"),  # previous, may be None
}
ACTIVE_VERSION = "v2"


def encode_cursor(ts: datetime, mid: int) -> str:
    payload = {
        "ts": ts.astimezone(timezone.utc).isoformat(),
        "id": mid,
        "dir": "desc",
        "v": ACTIVE_VERSION,
    }
    b = base64.urlsafe_b64encode(
        json.dumps(payload, separators=(",", ":"), sort_keys=True).encode()
    ).rstrip(b"=").decode()
    sig = hmac.new(
        SECRETS[ACTIVE_VERSION].encode(), b.encode(), hashlib.sha256
    ).digest()
    return b + "." + base64.urlsafe_b64encode(sig).rstrip(b"=").decode()


def decode_cursor(cursor: str) -> tuple[datetime, int]:
    b, sig = cursor.split(".", 1)
    # Peek at the version in the payload
    padding = b"=" * (-len(b) % 4)
    payload = json.loads(base64.urlsafe_b64decode(b.encode() + padding))
    version = payload.get("v", "v1")
    secret = SECRETS.get(version)
    if secret is None:
        raise ValueError(f"cursor version {version} no longer supported")
    expected = base64.urlsafe_b64encode(
        hmac.new(secret.encode(), b.encode(), hashlib.sha256).digest()
    ).rstrip(b"=").decode()
    if not hmac.compare_digest(sig, expected):
        raise ValueError("cursor signature invalid")
    return datetime.fromisoformat(payload["ts"]), payload["id"]


def messages_connection(channel_id: int, first: int, after: Optional[str]) -> dict:
    first = min(max(first, 1), 100)
    if after:
        try:
            cts, cid = decode_cursor(after)
        except ValueError as e:
            raise ApiError(400, str(e))
        rows = db.query(KEYSET_SQL, (channel_id, cts, cid, first + 1))
    else:
        rows = db.query(FIRST_PAGE_SQL, (channel_id, first + 1))

    has_next = len(rows) > first
    rows = rows[:first]
    edges = [
        {"node": r, "cursor": encode_cursor(r["created_at"], r["id"])}
        for r in rows
    ]
    return {
        "edges": edges,
        "pageInfo": {
            "hasNextPage": has_next,
            "hasPreviousPage": after is not None,
            "endCursor": edges[-1]["cursor"] if edges else None,
        },
    }

Step-by-step trace.

Step	Component	Purpose
1	Payload includes `v: "v2"`	Version tag for secret rotation
2	Sign with `SECRETS[ACTIVE_VERSION]`	Current secret
3	On decode, peek `v` before verification	Pick which secret to verify against
4	If `v` not in SECRETS → reject	Cursor too old, client re-fetches page 1
5	Verify signature constant-time	Reject tampered cursors
6	SQL uses keyset SQL if `after`, initial otherwise	O(log n + K) per page
7	Response uses Relay `edges + pageInfo`	Standard GraphQL envelope
8	LIMIT first + 1 for peek-ahead	hasNextPage without extra query

The design is production-ready: opaque signed cursors with version tags, graceful secret rotation via a two-secret map, Relay-standard response envelope, and the peek-ahead trick for hasNextPage. When a secret rotates, the new secret becomes v2 and the old becomes v1; clients with old cursors continue to work for one rotation window. After the window, old cursors return 400 and the client re-fetches page 1.

Output:

Property	Value
Wire cursor size	~90 bytes (base64 payload + base64 signature)
Verification time	~50 μs (HMAC-SHA256 constant-time)
Rotation window	typically 24–72 hours (max client idle)
DB latency per page	~5 ms on covering-indexed 10B-row table

Why this works — concept by concept:

Opaque, version-tagged, HMAC-signed cursor — the payload includes v: "v2" so the server knows which secret to verify against. Tamper-proof and forward-compatible with rotation.
Constant-time signature compare — hmac.compare_digest prevents timing attacks. If you use == on the signature, an attacker can measure the compare time and forge a valid cursor byte-by-byte.
Graceful rotation with a two-secret map — on rotation, promote v2 → v3 and v1 → v2 (drop the oldest). Clients with cursors signed by the previous secret continue to work until the window expires.
Relay envelope with peek-ahead — LIMIT first + 1 fetches one extra row; has_next = (fetched > first); the extra row is discarded and its cursor becomes the next after. No extra query.
Cost — HMAC verify O(1). Cursor decode O(payload size). SQL keyset O(log n + K) per page. Total per-request cost is dominated by the SQL — which is flat across pagination depth. The Relay envelope adds negligible overhead.

SQL
Topic — pagination
Cursor pagination problems

Practice →

SQL
Topic — window functions
Window function drills

Practice →

Cheat sheet — pagination recipe list

Two-primitive rule. sql pagination reduces to two primitives — OFFSET-based (page-numbered, admin dashboards, small tables) and keyset-based (infinite scroll, APIs, large mutable tables). Every real system ships one, the other, or both.
OFFSET / LIMIT skeleton. SELECT ... ORDER BY sort_col, id LIMIT n OFFSET k. Cost O(k + n) — fine up to k ≈ 1,000, painful past k ≥ 100,000.
ANSI OFFSET FETCH skeleton. SELECT ... ORDER BY sort_col, id OFFSET k ROWS FETCH FIRST n ROWS ONLY. Postgres, Oracle 12c+, SQL Server 2012+, Db2. Verbose but standard.
SQL Server specifics. OFFSET k ROWS FETCH NEXT n ROWS ONLY — NEXT is a synonym for FIRST. ORDER BY is required — the parser refuses without one.
MySQL shorthand. LIMIT k, n — offset first, limit second (reversed operand order). Equivalent to LIMIT n OFFSET k.
Oracle 11g legacy. Nested ROWNUM subquery — sort inside, number in the middle, filter outside. Migrate to OFFSET ... FETCH FIRST on 12c+.
Deterministic ORDER BY rule. Every paginated query needs a unique tie-breaker (usually id) — ORDER BY sort_col DESC, id DESC. Ties without the tie-breaker drift across pages.
NULLS ordering rule. Spell out ORDER BY sort_col DESC NULLS LAST. Postgres/Oracle default to NULLs first; MySQL/SQL Server default to NULLs last; BigQuery to NULLs first. Never rely on the default.
Concurrent-write failure mode. OFFSET is positional and drifts under inserts / deletes — the classic "row seen twice" or "row skipped" bug. Keyset is keyed and immune.
Deep-OFFSET cost. O(k + n) — for k = 1M, n = 20, the engine walks 1M rows before emitting 20. Latency curve stays flat until it isn't, then climbs a cliff.
Keyset skeleton. WHERE (sort_col, id) < (:cursor_col, :cursor_id) ORDER BY sort_col DESC, id DESC LIMIT n. Deep-page cost O(log n + K) — flat forever.
Tuple comparison rule. (a, b) < (:a, :b) is lexicographic — first a < :a, or a = :a AND b < :b. Not the same as a < :a AND b < :b (which drops valid rows).
Composite index rule. Keyset requires a composite index on the ORDER BY columns in the ORDER BY direction — (sort_col DESC, id DESC). Enables index-only backward scan.
SQL Server row-value fallback. Expand tuple compare to sort_col < :cursor_ts OR (sort_col = :cursor_ts AND id < :cursor_id). Same semantics, verbose but portable to older engines.
Mixed-direction ORDER BY. For ORDER BY score DESC, id ASC, expand to score < :s OR (score = :s AND id > :i). Row-value compare doesn't fit mixed directions.
Backward pagination. Flip < to >, flip ORDER BY direction, reverse result in application code. Or use Relay's last / before pair.
Peek-ahead trick. LIMIT n + 1 — fetch one extra row, use its existence to compute hasNextPage, discard from the response. Zero extra queries.
Opaque cursor rule. Cursors are base64-encoded JSON, HMAC-signed with a server-side secret. Clients treat them as black boxes. Never expose raw tuples.
Cursor rotation rule. Version-tag the payload; keep two active secrets (current + previous) for a rotation window equal to max client idle time. Reject cursors older than the window with a 400 and re-fetch-page-1 hint.
Relay Connections envelope. edges[] (with per-edge cursor) + pageInfo (hasNextPage, hasPreviousPage, endCursor, startCursor). GraphQL standard since 2015.
REST cursor envelope. {"data": [...], "next_cursor": "eyJ...", "prev_cursor": null} — Stripe, GitHub, Twitter shapes. Or RFC-5988 Link headers for HATEOAS-style clients.
Warehouse specifics. Snowflake — cluster by sort_col; use tuple compare. BigQuery — partition by DATE(sort_col), cluster by sort_col, keyset with partition-prune predicate. Never use OFFSET past ~10K on either.
Bounded-OFFSET safety net. For legacy admin dashboards you can't migrate yet, cap page × page_size ≤ 1M. Bounded worst-case latency of ~600 ms on a covering-indexed 100M-row table.
SELECT COUNT(*) warning. Total count on a mutable table is O(n) regardless. On infinite scroll, don't return it. On admin dashboards, cache it.
Dialect matrix. LIMIT / OFFSET: Postgres, MySQL, Snowflake, BigQuery, SQLite, Redshift, DuckDB. OFFSET / FETCH FIRST: Postgres, Oracle 12c+, SQL Server 2012+, Db2. Row-value compare: Postgres, MySQL 8, Oracle 12c+, Snowflake, BigQuery. Expanded form fallback: SQL Server, older MySQL.
When OFFSET is fine. Small tables (< 1M rows), aggregate dashboards, page-capped admin UIs (page ≤ 5000), one-off batched exports. Everything else — keyset.
When keyset is fine. Infinite-scroll feeds, mobile pagination, GraphQL Relay endpoints, big-table list APIs, any endpoint expected to serve deep pages with stable results under concurrent writes.

Frequently asked questions

What is SQL pagination and how do you choose between OFFSET and keyset?

sql pagination is the primitive that lets a client fetch "the next N rows of an ordered result set" without re-fetching what came before. The two implementations you have to know are OFFSET-based — SELECT ... ORDER BY x LIMIT n OFFSET k — and keyset (seek method) based — WHERE (x, id) < (:cursor_x, :cursor_id) ORDER BY x DESC, id DESC LIMIT n. Choose OFFSET when the product requires random access to page N (admin dashboards, "jump to page 47" pagers), the table is small (< 1M rows) or aggregate-sized, and the page depth is capped (page × page_size ≤ 1M on a covering-indexed table). Choose keyset for every consumer feed, every GraphQL Relay endpoint, every "list my orders / posts / messages" endpoint, and any pagination that must be stable under concurrent inserts and deletes — because OFFSET is positional and drifts under mutations, whereas keyset navigates by key and is immune. The cost story cements the choice: OFFSET is O(k + n) per page (linear in depth), keyset is O(log n + K) per page (flat forever). On a 100M-row table with a covering index, OFFSET at page 100K takes about 1.4 s while keyset at any depth stays around 5 ms — a 300× gap that widens as the table grows.

Why is OFFSET slow on deep pages?

Because the planner has no O(1) skip primitive on an ordered scan. When you write LIMIT 20 OFFSET 1000000, the engine walks the ordered stream one row at a time, counting past a million rows, discarding each, before emitting the last twenty. Cost is O(k + n) per page — for k = 1M, that's a million row reads per page. On a covering-indexed table with SELECT id, created_at, Postgres's EXPLAIN (ANALYZE, BUFFERS) shows Index Scan Backward using idx_events with actual rows=20 actual loops=1000020 — the "20 rows but a million loops" line is the smoking gun. Buffer pressure amplifies this — past a few hundred thousand rows, the working set exceeds shared_buffers and reads spill from cache-hit to disk-read, latency jumps from milliseconds to seconds. Wide rows (with SELECT * on a 1KB row) make it worse — each discarded row still pays its projection cost. And on columnar warehouses like BigQuery and Snowflake, per-column read overhead multiplies the effect further — deep OFFSET reads gigabytes of column data before returning bytes. Total cost across a full pagination from page 1 to page P is quadratic — O(P² × n) — which is why every social feed on the internet moved off OFFSET a decade ago.

What is keyset (seek method) pagination?

keyset pagination (a.k.a. seek method pagination) replaces "skip k rows" with "give me the rows strictly after this cursor tuple." The two-line skeleton is WHERE (sort_col, id) < (:cursor_col, :cursor_id) ORDER BY sort_col DESC, id DESC LIMIT n — the WHERE tuple comparison positions the scan at the cursor via an index seek on the composite (sort_col DESC, id DESC) index, and the LIMIT bounds the forward scan to exactly n rows. Total cost is O(log n + K) per page — the index seek is O(log n) (depth of the btree), and K = LIMIT sequential reads finish it. Every page — page 1 or page 1,000,000 — hits the exact same plan shape, so latency stays flat across the entire pagination depth. The tuple comparison (a, b) < (:a, :b) is lexicographic — first a < :a, or a = :a AND b < :b — supported natively on Postgres, MySQL 8, Oracle 12c+, Snowflake, and BigQuery. On SQL Server and older MySQL, expand to a < :a OR (a = :a AND b < :b) — same semantics, verbose but portable. The trade-off is no random access — you can't ask for "page 47" — but every infinite-scroll consumer feed on the internet uses keyset because it beats OFFSET on both cost and consistency.

How do you build cursor pagination for a REST or GraphQL API?

Cursors are opaque base64-encoded, HMAC-signed tokens — clients treat them as black boxes and pass them back verbatim on the next call. The payload is a JSON blob {"ts": "2026-07-05T12:00:00Z", "id": 5000000, "dir": "desc", "v": "v2"} — base64url-encoded, then HMAC-SHA256-signed with a server-side secret, then joined as payload_b64.signature_b64 (JWT-like without a header). Servers verify the signature in constant time (hmac.compare_digest) before decoding, so tampered cursors are rejected before any SQL runs. For GraphQL, follow the Relay Connections spec — orders(first: 20, after: "eyJ...") returns edges[] (with a per-edge cursor) plus pageInfo (hasNextPage, endCursor, startCursor). For REST, use {"data": [...], "next_cursor": "eyJ...", "prev_cursor": null} (Stripe / GitHub style) or RFC-5988 Link headers (<https://.../orders?cursor=eyJ...>; rel="next"). The hasNextPage uses the peek-ahead trick — fetch LIMIT n + 1, use the existence of the (n+1)th row as hasNextPage, discard it before returning. Rotate the signing secret every 90 days with a two-secret map (current + previous) so cursors survive one rotation window; reject older versions with a 400 and a hint to re-fetch page 1.

How does pagination differ across Postgres, MySQL, SQL Server, Oracle, Snowflake, and BigQuery?

Postgres — LIMIT n OFFSET k (native) and OFFSET k ROWS FETCH FIRST n ROWS ONLY (ANSI); full row-value comparison (a, b) < (:a, :b) for keyset; composite descending indexes since forever. Best case. MySQL 8 — LIMIT n OFFSET k or shorthand LIMIT k, n (reversed operands); row-value compare since 8.0.16; descending indexes since 8.0.13. Full keyset support. SQL Server 2012+ — OFFSET k ROWS FETCH NEXT n ROWS ONLY with required ORDER BY; no row-value compare (use expanded a < :a OR (a = :a AND b < :b) form); no NULLS FIRST/LAST (use CASE WHEN … IS NULL workaround). Oracle 12c+ — OFFSET k ROWS FETCH FIRST n ROWS ONLY (ANSI); full row-value compare; pre-12c uses nested ROWNUM subquery. Snowflake — LIMIT n OFFSET k; row-value compare supported; no btree indexes — use CLUSTER BY (sort_col) for micro-partition co-location; avoid OFFSET past ~10K. BigQuery — LIMIT n OFFSET k; row-value compare since 2019; no btree — use PARTITION BY DATE(sort_col) CLUSTER BY sort_col and combine keyset with an outer partition-prune predicate; OFFSET is documented as an anti-pattern past a few thousand rows. Learn the six spellings; every migration between engines rewrites pagination as a mechanical translation.

How do you handle ties and NULLs in a paginated ORDER BY?

Ties — two rows sharing the ORDER BY key (same created_at to the millisecond, same score in a leaderboard) can alternate between pages unless you add a unique tie-breaker. Every paginated ORDER BY ends with a unique column, usually the primary key: ORDER BY created_at DESC, id DESC. The tuple comparison in keyset must include the tie-breaker too — (created_at, id) < (:cursor_ts, :cursor_id). Without the tie-breaker, WHERE created_at < :ts would skip all rows tied on :ts even if only one was the cursor row. NULLs — Postgres and Oracle default to NULLs first for DESC; MySQL and SQL Server default to NULLs last; BigQuery defaults to NULLs first. Never rely on the default — spell it out: ORDER BY created_at DESC NULLS LAST. Postgres, Oracle, MySQL 8, and BigQuery all support the NULLS FIRST / LAST syntax; SQL Server doesn't — use the CASE WHEN created_at IS NULL THEN 1 ELSE 0 END, created_at DESC workaround. For keyset with a nullable sort column, either mark the column NOT NULL (the safest fix) or use COALESCE(created_at, '-infinity') on both sides of the comparison to give NULLs a defined position. Every keyset paginator ships either the NOT NULL constraint or the coalescing predicate.

Practice on PipeCode

Drill the SQL pagination practice library → for LIMIT / OFFSET, FETCH FIRST n ROWS ONLY, keyset tuple comparison, and cursor endpoints across Postgres, MySQL 8, SQL Server, Oracle, Snowflake, and BigQuery dialects.
Rehearse on ORDER BY + LIMIT problems → — the ordering, tie-breaker, and NULL discipline that make every paginated query deterministic.
Sharpen the SQL indexing drill room → for composite (sort_col, id) indexes, index-only backward scans, and the difference between OFFSET's O(k) and keyset's O(log n + K) deep-page paths.
Push the difficulty ceiling with SQL optimization drills → — read EXPLAIN ANALYZE plans, spot the OFFSET trap, and migrate legacy paginators to keyset.
Warm up with top-N per group problems → — the sibling primitive to pagination, sharing the composite (partition_col, sort_col, id) index pattern.
Layer window function drills → — the ROW_NUMBER() / RANK() / DENSE_RANK() family that shows up when a legacy paginator can't be migrated to keyset yet.
Sharpen the general SQL surface with the SQL practice library → which contains 450+ DE-focused questions covering pagination, keyset cursors, index alignment, and every adjacent pattern.
For the broader SQL interview surface, take the SQL for Data Engineering course →.

Pipecode.ai is Leetcode for Data Engineering — every `sql pagination` recipe above ships with hands-on practice rooms where you type the OFFSET / LIMIT skeleton, migrate it to `sql offset fetch` (ANSI) and `OFFSET ... FETCH NEXT` (SQL Server) side by side, benchmark the `sql limit offset performance` cliff at page 1M with `EXPLAIN (ANALYZE, BUFFERS)`, rewrite the same paginator as `keyset pagination` with tuple comparison, align the composite index for `seek method pagination`'s `O(log n)` deep-page path, wire the whole thing behind an opaque HMAC-signed cursor for `sql cursor pagination`, and finally serve the Relay `edges / pageInfo` envelope that every `sql infinite scroll` feed on the internet ships. PipeCode pairs every reading with 450+ DE-focused problems and a real-time scoring engine, so you never have to wonder whether your pagination answer holds up under a senior interviewer's depth probes.

Practice pagination now →
Indexing drills →

SQL LATERAL Joins & CROSS APPLY: When They Beat Subqueries

Gowtham Potureddi — Sat, 11 Jul 2026 15:31:46 +0000

sql lateral join is the single most under-taught primitive in modern SQL — and the single largest tool a senior data engineer can pull out of the ANSI SQL/1999 standard when the plain correlated subquery hits a wall. Once you internalise "LATERAL means: for each outer row, run this inner query with access to the outer columns," a whole family of otherwise-painful queries collapses into a two-line skeleton — the classic sql top-n per group ("give me the last three orders per customer"), JSON and array unnest with correlation, table-valued function composition, and every other "run a mini-query per row" ask an interviewer might throw at a whiteboard.

This guide is the mid-to-senior tour you wished existed the first time a reviewer asked why your query was doing an N × subquery scan, when an interviewer probed the difference between lateral vs subquery semantics, when the team lead demanded a cross apply sql server port of your postgres lateral recipe, when the analyst asked whether bigquery lateral was a thing, or when the on-call engineer showed a outer apply plan running 10× faster than the LEFT JOIN + ROW_NUMBER() alternative. It walks through the LATERAL syntax and mental model (LEFT JOIN LATERAL vs CROSS JOIN LATERAL, correlation binding, set-returning-function unnest), the CROSS APPLY / OUTER APPLY family that SQL Server ships as its LATERAL equivalent (the translation table, APPLY + TVF composition, JSON_TABLE / OPENJSON unnest), the top-N-per-group performance story (LATERAL + LIMIT with an index seek versus ROW_NUMBER() OVER with a sort), and the six-engine performance matrix (Postgres, MySQL 8, Oracle 12c+, SQL Server, Snowflake, BigQuery) so you always know which keyword to reach for on which warehouse. Every section pairs a teaching block with a Solution-Tail interview answer — code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works.

When you want hands-on reps immediately after reading, drill the SQL joins practice library →, rehearse on top-N-per-group problems →, and sharpen the correlated-subquery axis with the subqueries drill room →.

On this page

Why LATERAL matters in 2026
LATERAL syntax & mental model
CROSS APPLY & OUTER APPLY
Top-N per group pattern
When LATERAL beats subquery — perf matrix
Cheat sheet — LATERAL / CROSS APPLY recipe list
Frequently asked questions
Practice on PipeCode

1. Why LATERAL matters in 2026

The "for each outer row, run this inner query" primitive — the missing piece between joins, subqueries, and window functions

The one-sentence invariant: a sql lateral join runs the right-hand subquery once per row of the left-hand relation, with the inner query allowed to reference the outer row's columns — turning "SELECT the top-3 orders per customer" from a window-function-plus-filter puzzle into a two-line correlated join. Once you internalise "for each outer row, run this query," the LATERAL interview surface collapses into a single mental model with a handful of dialect variants.

Where LATERAL fits in the join family.

Inner join / outer join. Rows are matched on an equality (or range) predicate. Both sides are evaluated independently, then the planner picks a join algorithm — hash, merge, or nested loop.
Correlated subquery in the SELECT list. For each outer row, the planner runs the scalar subquery to produce one column value. Cost is N × cost(subquery) in the worst case — many planners fail to unnest correlated scalars.
Correlated subquery in the WHERE / EXISTS clause. For each outer row, the planner runs the boolean subquery to filter. Modern planners try hard to rewrite these to a semi-join, but a rewrite is not guaranteed.
LATERAL / CROSS APPLY. For each outer row, the planner runs a table-producing subquery — and joins the outer row to each of the inner rows returned. The inner query can reference outer columns freely. Semantically, LATERAL is a correlated join that produces zero, one, or many inner rows per outer row.

Why the primitive is under-taught.

The SQL/1999 standard introduced the keyword, but implementations lagged for over a decade. Postgres added it in 9.3 (2013), MySQL waited until 8.0.14 (2019), Oracle shipped it in 12c (2013), and SQL Server never adopted the keyword — instead offering CROSS APPLY and OUTER APPLY since 2005 with identical semantics.
Every classic SQL textbook was written before LATERAL was widely available, so an entire generation of engineers learned to reach for WHERE (partition, ranked_col) IN (SELECT ...) or window-function-plus-filter patterns even when a two-line LATERAL would beat both.
LATERAL is also invisible to the naive query author — you have to know the keyword to type it. There is no CTE-style rewrite that emerges from re-reading your own draft.

Dialect coverage in 2026 — the six engines you actually ship to.

Postgres 9.3+. Full support. Reads exactly like the ANSI syntax — SELECT ... FROM outer, LATERAL (SELECT ... WHERE inner.k = outer.k) AS inner. Both CROSS JOIN LATERAL and LEFT JOIN LATERAL ... ON TRUE variants supported.
MySQL 8.0.14+. Full support. Same syntax as Postgres. Older MySQL versions (5.7, 8.0.0–8.0.13) require the ROW_NUMBER() workaround.
Oracle 12c+. Ships both LATERAL (ANSI) and CROSS APPLY / OUTER APPLY (SQL-Server-compatible). Interchangeable — pick the one your team reads more fluently.
SQL Server 2005+. No LATERAL keyword. Uses CROSS APPLY (inner) and OUTER APPLY (left) instead. Same semantics as ANSI LATERAL.
Snowflake. Full LATERAL support since 2020. Also supports the FLATTEN table function inside LATERAL for JSON / array unnest.
BigQuery. No explicit LATERAL keyword. Correlation is implicit — an unnested array reference in the FROM clause behaves like CROSS JOIN LATERAL UNNEST(...). Scalar correlation is expressed via subqueries, not LATERAL.

Three problem shapes LATERAL nails.

Top-N per group. "Latest 3 orders per customer" — the archetype. LATERAL + LIMIT 3 compiles to a nested-loop-with-limit plan that stops fetching inner rows the moment the third one arrives; ROW_NUMBER() OVER + WHERE rn ≤ 3 has to materialise every ranked row across the whole table before filtering.
JSON / array unnest with correlation. "Unroll every item in the JSON payload, keeping the parent row." LATERAL + jsonb_array_elements(payload) in Postgres, CROSS APPLY OPENJSON(payload) in SQL Server. The unnest function must see the outer row's payload column — that correlation is exactly what LATERAL exists for.
TVF composition. "Run this parametrised table-valued function per row of an input list." SELECT c.*, t.* FROM customers c CROSS APPLY dbo.top_orders(c.id) t on SQL Server; the same pattern with a Postgres SETOF-returning function inside LATERAL.

Why interviewers love LATERAL as a probe.

It's a compact fluency check. A senior candidate reaches for LATERAL on the top-3-orders-per-customer question in the first minute; a junior reaches for ROW_NUMBER() OVER and has to be nudged toward the LATERAL rewrite.
It forces mental-model precision. "What does the inner query see?" (the current outer row's columns) "What does the outer query see?" (each inner row emitted). "What if the inner returns zero rows?" (CROSS JOIN LATERAL drops the outer row; LEFT JOIN LATERAL keeps it with NULLs).
It reveals planner intuition. "When does LATERAL beat a plain correlated subquery?" (when the planner can push a limit or unique into the inner scan). "When does it lose?" (when N is huge and the inner query can't be indexed).
It exposes dialect awareness. "Would this run on SQL Server?" — a candidate who instantly answers "yes, but you write CROSS APPLY instead of CROSS JOIN LATERAL" signals cross-warehouse fluency.

Where LATERAL / APPLY lands in your pipelines.

Analytics on wide fact tables. "Attach the most recent status change to each order," "attach the top three sessions per user," "attach the last five sensor readings to each device row." All classic top-N-per-group patterns.
Feature engineering. "For each user row, compute the sum of the last 30 days of spend." LATERAL + windowed aggregate inside — the inner query filters by the outer's user_id and produces one summary row per outer.
ETL of nested data. "Explode each JSON payload into its item rows, keeping the parent metadata." LATERAL + jsonb_array_elements / OPENJSON / UNNEST — one row per (parent, item) pair.
Stored-procedure-to-SQL migrations. Legacy code with a WHILE loop that runs a query per row of a driver table maps directly to LATERAL / CROSS APPLY over that driver table.

What senior interviewers actually probe.

Do you know the syntax? CROSS JOIN LATERAL (…) (inner) vs LEFT JOIN LATERAL (…) ON TRUE (left). The ON TRUE is a common typo target — every candidate forgets it once.
Do you know the semantic difference? Inner drops outer rows with no inner match; left keeps them with NULLs. Same as regular join families.
Do you know the SQL Server keyword? CROSS APPLY = CROSS JOIN LATERAL; OUTER APPLY = LEFT JOIN LATERAL ... ON TRUE. Interchangeable on Oracle.
Do you know when LATERAL wins on perf? When the inner query is expensive per row and can be indexed with (partition_col, order_col) — the nested-loop-with-limit plan is dramatically cheaper than a sort-and-rank plan.
Do you know the BigQuery variant? No LATERAL keyword; UNNEST in the FROM clause is implicitly correlated. For scalar-correlation asks, BigQuery falls back to subqueries.

Worked example — top-3 orders per customer, the first LATERAL query most engineers write

Detailed explanation. The archetype: given customers(id, name) and orders(id, customer_id, order_ts, total), return the three most recent orders per customer. This is the fastest way to spot whether a candidate has ever reached for LATERAL — the two-line answer is a fingerprint on the whiteboard.

Question. Write a Postgres query that returns, for each customer, the three most recent orders (order_ts, total). Use LATERAL. Assume there is an index on orders(customer_id, order_ts DESC).

Input.

customer_id	name
1	Alice
2	Bob
3	Carol

id	customer_id	order_ts	total
101	1	2026-07-01	40
102	1	2026-07-05	25
103	1	2026-07-08	90
104	1	2026-07-09	60
105	2	2026-07-02	70
106	2	2026-07-07	30

Code.

SELECT c.id AS customer_id, c.name, o.order_ts, o.total
FROM customers c
CROSS JOIN LATERAL (
  SELECT o.order_ts, o.total
  FROM orders o
  WHERE o.customer_id = c.id
  ORDER BY o.order_ts DESC
  LIMIT 3
) o
ORDER BY c.id, o.order_ts DESC;

Step-by-step explanation.

FROM customers c — the outer relation. Every row here is visited once.
CROSS JOIN LATERAL (…) o — for each outer row c, run the parenthesised subquery. The subquery reads c.id freely — that is the correlation binding.
Inside the LATERAL — WHERE o.customer_id = c.id ORDER BY o.order_ts DESC LIMIT 3. With the index on (customer_id, order_ts DESC), this is a single index seek plus a bounded scan of at most three rows.
CROSS JOIN LATERAL drops any customer with no matching orders. To keep customers with no orders, swap to LEFT JOIN LATERAL (…) ON TRUE and the missing-inner rows become NULL.
Final ORDER BY c.id, o.order_ts DESC produces a stable output — customers grouped, each customer's orders newest first.

Output.

customer_id	name	order_ts	total
1	Alice	2026-07-09	60
1	Alice	2026-07-08	90
1	Alice	2026-07-05	25
2	Bob	2026-07-07	30
2	Bob	2026-07-02	70

Rule of thumb. LATERAL + LIMIT K with a covering index is the fastest top-N-per-group plan on Postgres, MySQL 8, and Oracle. The nested-loop-with-limit stops early per customer, so total cost is K × outer_rows × log(inner_rows) — dramatically better than the ROW_NUMBER() OVER + WHERE rn ≤ 3 sort-then-filter plan on skewed data.

Worked example — LATERAL vs plain correlated subquery — where the semantics diverge

Detailed explanation. A common source of confusion — the difference between "correlated subquery in the SELECT list" (returns exactly one value) and "LATERAL subquery in the FROM clause" (returns zero-to-many rows). The interviewer wants to hear you say "correlated scalar subqueries are one-column-one-row; LATERAL is one-outer-row-to-many-inner-rows."

Question. Given the same tables, write two versions of "for each customer, get the most recent order total": one using a correlated scalar subquery, one using LATERAL. Explain the semantic difference.

Input. (Same as above.)

Code.

-- Version A: correlated scalar subquery in SELECT list
SELECT
  c.id AS customer_id,
  c.name,
  (SELECT o.total
   FROM orders o
   WHERE o.customer_id = c.id
   ORDER BY o.order_ts DESC
   LIMIT 1) AS last_total
FROM customers c;

-- Version B: LATERAL join
SELECT
  c.id AS customer_id,
  c.name,
  o.total AS last_total,
  o.order_ts AS last_order_ts
FROM customers c
LEFT JOIN LATERAL (
  SELECT o.total, o.order_ts
  FROM orders o
  WHERE o.customer_id = c.id
  ORDER BY o.order_ts DESC
  LIMIT 1
) o ON TRUE;

Step-by-step explanation.

Version A returns exactly one column (last_total) per customer. If the inner query returned two columns, the whole query would fail — the SELECT-list subquery must be scalar (one column, one row).
Version B returns many columns from the inner query (total, order_ts, and any others you project). The LATERAL join yields a full inner row per outer, projected alongside the outer columns.
Version A silently returns NULL for customers with no orders — the subquery returns zero rows, which the SELECT-list treats as NULL.
Version B with LEFT JOIN LATERAL ... ON TRUE behaves the same way — outer row preserved with NULL inner columns. With CROSS JOIN LATERAL, customers with no orders would be dropped entirely.
Perf-wise, most planners rewrite Version A to Version B under the hood — but "most" is not "all." Older planners and specific dialects (BigQuery on certain shapes) leave the correlated scalar as an N × subquery plan.

Output (both versions).

customer_id	name	last_total	last_order_ts
1	Alice	60	2026-07-09
2	Bob	30	2026-07-07
3	Carol	NULL	NULL

Rule of thumb. Reach for LATERAL the moment you need more than one column from the inner query per outer row, or when you want explicit control over the plan. Correlated scalar subqueries are fine for one-column-one-row lookups, but only when you trust the planner to unnest them.

Worked example — engine dialect quickstart

Detailed explanation. A quick sanity check the interviewer might drop: "which of these engines can I run LATERAL / APPLY on today?" The answer is a short six-row list — knowing it cold is a senior signal.

Question. Given a list of engines, mark each with the LATERAL / APPLY keyword it supports as of 2026.

Input. Postgres 16, MySQL 8.0.14+, Oracle 12c+, SQL Server 2022, Snowflake, BigQuery.

Code. (No code — a dialect matrix answer.)

Engine          | LATERAL support (as of 2026)              | Notes
Postgres 9.3+   | LATERAL (ANSI), CROSS/LEFT JOIN LATERAL   | first mover in OSS
MySQL 8.0.14+   | LATERAL (ANSI), CROSS/LEFT JOIN LATERAL   | 5.7 lacks it
Oracle 12c+     | LATERAL + CROSS APPLY + OUTER APPLY        | ships both keywords
SQL Server 2005+| CROSS APPLY + OUTER APPLY (no LATERAL kw)  | same semantics
Snowflake       | LATERAL, LATERAL FLATTEN                    | FLATTEN for JSON
BigQuery        | UNNEST is implicit LATERAL; no keyword     | correlated array only

Step-by-step explanation.

Postgres was the OSS first-mover in 2013. If you're on Postgres 12+, LATERAL is table stakes.
MySQL waited until 8.0.14 in 2019. If your project supports MySQL 5.7, LATERAL is off-limits; use the ROW_NUMBER() workaround.
Oracle 12c (2013) ships both keywords — LATERAL (ANSI) and CROSS/OUTER APPLY (SQL-Server-compatible). Pick whichever your team reads better.
SQL Server never added LATERAL. Every LATERAL query maps mechanically to CROSS APPLY (inner) or OUTER APPLY (left).
Snowflake ships LATERAL and adds LATERAL FLATTEN for VARIANT / JSON columns — the standard unnest pattern for semi-structured data.
BigQuery has no LATERAL keyword. Array unnest via UNNEST(...) in the FROM clause is implicitly correlated; scalar correlations use subqueries.

Output. The matrix above.

Rule of thumb. Memorise the six-row list. When an interviewer asks "would you use LATERAL here?" the first question back is "what's the warehouse?" — say it out loud and you signal senior instincts.

Senior interview question on the LATERAL mental model

A senior interviewer often opens with: "I hand you two tables — customers and orders — and ask for the two most-recent orders per customer. Walk me through the LATERAL query, tell me what changes on SQL Server, and tell me when this beats ROW_NUMBER() OVER + WHERE rn ≤ 2."

Solution Using CROSS JOIN LATERAL with LIMIT 2

-- ANSI / Postgres / MySQL 8 / Oracle 12c+ / Snowflake
SELECT c.id AS customer_id, c.name, o.order_ts, o.total
FROM customers c
CROSS JOIN LATERAL (
  SELECT o.order_ts, o.total
  FROM orders o
  WHERE o.customer_id = c.id
  ORDER BY o.order_ts DESC
  LIMIT 2
) o
ORDER BY c.id, o.order_ts DESC;

-- SQL Server 2005+ — same semantics, CROSS APPLY keyword
SELECT c.id AS customer_id, c.name, o.order_ts, o.total
FROM customers c
CROSS APPLY (
  SELECT TOP 2 o.order_ts, o.total
  FROM orders o
  WHERE o.customer_id = c.id
  ORDER BY o.order_ts DESC
) o
ORDER BY c.id, o.order_ts DESC;

Step-by-step trace.

Step	Outer row (customer_id)	Inner query result	Emitted rows
1	c1 = Alice	last 2 = (07-09, 60), (07-08, 90)	(Alice, 07-09, 60), (Alice, 07-08, 90)
2	c2 = Bob	last 2 = (07-07, 30), (07-02, 70)	(Bob, 07-07, 30), (Bob, 07-02, 70)
3	c3 = Carol	last 2 = (empty)	(dropped — CROSS JOIN LATERAL)
4	Sort	ORDER BY c.id, o.order_ts DESC	Final output

The engine walks each customer, runs the inner LIMIT 2 query with the outer's c.id bound in, and emits at most two rows per customer. Carol has no orders — the CROSS JOIN LATERAL drops her; swap to LEFT JOIN LATERAL ... ON TRUE to preserve her with NULLs.

Output:

customer_id	name	order_ts	total
1	Alice	2026-07-09	60
1	Alice	2026-07-08	90
2	Bob	2026-07-07	30
2	Bob	2026-07-02	70

Why this works — concept by concept:

CROSS JOIN LATERAL binds the outer row into the inner query — inside the parentheses, c.id is a live value from the current outer row, not a table reference. The planner treats this as a correlated join, not a Cartesian product.
LIMIT 2 is a per-outer-row limit, not a global limit — the LATERAL subquery runs once per outer row and each run applies its own LIMIT. This is why you can't achieve the same result with a plain SELECT ... LIMIT 2 at the top level.
Index seek + bounded scan — with an index on (customer_id, order_ts DESC), each inner run is one seek (position on the customer_id) plus at most two forward reads. Total cost is outer_rows × (seek + 2 × row_fetch) — dramatically cheaper than sorting all orders.
CROSS APPLY on SQL Server is a syntactic rebrand — the semantics are identical to CROSS JOIN LATERAL. The TOP 2 keyword replaces LIMIT 2; everything else is the same query.
Cost — O(N_c × (log N_o + K)) where N_c is customer count, N_o is order count, and K = 2 is the LIMIT. Compare with ROW_NUMBER() OVER (PARTITION BY customer_id ORDER BY order_ts DESC), which is O(N_o log N_o) for the sort — bad when N_o is huge and N_c is small. LATERAL wins on skewed data with a good index; the window function wins when the (customer, order) distribution is uniform and no index is available.

SQL
Topic — top-N per group
Top-N per group problems

Practice →

SQL Topic — joins (SQL) SQL joins — full library

Practice →

2. LATERAL syntax & mental model

`postgres lateral` and `lateral vs subquery` — CROSS JOIN LATERAL, LEFT JOIN LATERAL, and the correlation binding that makes it all work

The mental model in one line: LATERAL turns the right-hand FROM-clause subquery into a per-outer-row invocation, letting the inner query freely reference outer columns and producing zero, one, or many rows per outer — behaving like an inner join (CROSS JOIN LATERAL) or a left join (LEFT JOIN LATERAL … ON TRUE) depending on the keyword. Once you say "for each outer row, run this inner query and either drop or preserve the outer row based on whether the inner returned anything," the LATERAL syntax interview surface reduces to a two-slot template.

Slot 1 — the join keyword.

CROSS JOIN LATERAL (…) alias — inner-join semantics. If the inner subquery returns zero rows, the outer row is dropped from the result. Equivalent to CROSS APPLY on SQL Server / Oracle.
LEFT JOIN LATERAL (…) alias ON TRUE — left-outer-join semantics. If the inner subquery returns zero rows, the outer row is preserved with NULLs in the inner columns. Equivalent to OUTER APPLY on SQL Server / Oracle.
INNER JOIN LATERAL (…) alias ON TRUE — spelled out; identical to CROSS JOIN LATERAL on most engines.
RIGHT JOIN LATERAL — not defined. LATERAL is asymmetric — the inner references the outer, not vice versa. There's no way to reverse the direction.

Slot 2 — the inner subquery.

Any SELECT that can reference the outer columns. Includes CTEs (WITH … SELECT …), set-returning function calls (SELECT * FROM generate_series(1, outer.n)), unnest expressions (SELECT * FROM jsonb_array_elements(outer.payload)), table-valued function calls (Oracle / SQL Server), and window-function subqueries.
Returns zero, one, or many rows. Multiple columns supported — the LATERAL is a table, not a scalar.
Can be aliased — LATERAL (…) AS o — with column aliases in the ANSI form LATERAL (…) AS o(col1, col2).

Correlation binding — the rule that makes LATERAL a "join" instead of a "cross product".

The inner query sees the outer row as if it were a set of parameters. Any column of the outer table can appear inside the LATERAL subquery.
Without the LATERAL keyword, an ordinary subquery in the FROM clause cannot reference outer columns — a query like FROM customers c, (SELECT * FROM orders WHERE customer_id = c.id) o is a syntax error on strict parsers.
The LATERAL keyword tells the parser "this subquery is allowed to see the outer scope." On Postgres, LATERAL is required whenever the subquery references outer columns from the same FROM clause.
On Postgres, LATERAL is implicit on set-returning functions in the FROM clause — SELECT * FROM t, unnest(t.arr) works without the keyword. Being explicit is the safer style.

Two syntactic forms on Postgres — comma vs JOIN.

Comma form — FROM outer, LATERAL (…) inner — equivalent to CROSS JOIN LATERAL. The comma is a cross join by default.
JOIN form — FROM outer CROSS JOIN LATERAL (…) inner or FROM outer LEFT JOIN LATERAL (…) inner ON TRUE. The JOIN form is more explicit and preferred for review clarity.
Both compile to the same plan. The JOIN form is what most modern style guides recommend.

Common newbie mistakes.

Forgetting ON TRUE on LEFT JOIN LATERAL. Every LEFT JOIN needs an ON clause; the LATERAL subquery has no join column to reference, so the correct spelling is LEFT JOIN LATERAL (…) x ON TRUE.
Omitting the LATERAL keyword. Without it, the outer columns are invisible inside the subquery and the query errors out.
Assuming the inner query is executed once. It runs once per outer row — the whole point of LATERAL.
Assuming the outer query can see the inner. It can't — the outer projects only its own columns plus whatever the LATERAL alias exposes.
Placing an aggregate outside the LATERAL block expecting per-outer aggregation. Aggregates outside operate on the joined result set; per-outer aggregation belongs inside the LATERAL.

How LATERAL interacts with GROUP BY and window functions.

GROUP BY outer_col after a LATERAL join still works — the group is over the joined result set.
ROW_NUMBER() OVER (PARTITION BY outer_col) after a LATERAL is redundant if the LATERAL already used LIMIT K internally.
Aggregates inside the LATERAL (SELECT SUM(x) FROM inner WHERE inner.k = outer.k) produce one summary row per outer — equivalent to a correlated subquery with better plan predictability.

Set-returning functions inside LATERAL — the JSON / array pattern.

Postgres: SELECT t.id, e.value FROM t, LATERAL jsonb_array_elements(t.payload) e — one row per (parent, JSON element) pair.
Postgres: SELECT t.id, u.n FROM t, LATERAL unnest(t.arr) u(n) — one row per (parent, array element) pair.
Postgres: SELECT o.id, r FROM orders o, LATERAL generate_series(1, o.qty) r — one row per (order, series element) — useful for expanding aggregate quantities into rows.
Snowflake: SELECT t.id, f.value FROM t, LATERAL FLATTEN(input => t.payload) f — the VARIANT / JSON unnest primitive.

Interview probes on syntax fluency.

"Why CROSS JOIN LATERAL and not just a plain subquery?" — the plain subquery in the FROM clause cannot reference outer columns without LATERAL; LATERAL is the keyword that authorises correlation.
"When would you switch to LEFT JOIN LATERAL?" — whenever you must preserve outer rows even if the inner returns nothing. Reporting "one row per customer, latest order or NULL" is the canonical case.
"What's the difference between LATERAL and a CTE?" — a CTE is evaluated once, produces one result set, and joins to the outer table by its own predicates. A LATERAL runs once per outer row and is naturally correlated. CTEs can't express per-row-limited inner queries the way LATERAL can.
"Why does Postgres require the keyword but BigQuery not?" — Postgres is strict about correlation scope; BigQuery's UNNEST on an array column in the FROM clause is implicitly LATERAL because the array reference is inherently correlated to its parent row.

Worked example — LATERAL vs a lateral-less rewrite

Detailed explanation. The clearest way to see what LATERAL adds: try the same query without it. On Postgres the lateral-less form errors out because the inner subquery can't see the outer columns. Writing both side-by-side is the fastest way to internalise the correlation-binding rule.

Question. Given customers and orders, get the most recent order per customer. Write two Postgres queries — one with LATERAL, one attempting the same without LATERAL — and explain why the lateral-less version fails.

Input. (Same customers and orders as the first section.)

Code.

-- With LATERAL — compiles
SELECT c.id, c.name, o.order_ts, o.total
FROM customers c
LEFT JOIN LATERAL (
  SELECT order_ts, total
  FROM orders
  WHERE customer_id = c.id
  ORDER BY order_ts DESC
  LIMIT 1
) o ON TRUE;

-- Without LATERAL — ERROR: invalid reference to FROM-clause entry
SELECT c.id, c.name, o.order_ts, o.total
FROM customers c
LEFT JOIN (
  SELECT order_ts, total
  FROM orders
  WHERE customer_id = c.id           -- ERROR: no c.id in scope
  ORDER BY order_ts DESC
  LIMIT 1
) o ON TRUE;

Step-by-step explanation.

The lateral-less version places the subquery in the FROM clause but without the LATERAL keyword. Postgres treats it as an independent subquery whose scope is only its own FROM clause — c.id is not visible.
The parser rejects the query with ERROR: invalid reference to FROM-clause entry for table "c". On MySQL 8 the same error appears as Unknown column 'c.id' in 'where clause'.
Adding LATERAL to the second query fixes it — the keyword authorises the correlation.
Without LATERAL, the equivalent trick is a correlated scalar subquery in the SELECT list or a ROW_NUMBER() + filter — both work, both compile, both have different perf characteristics.
The rule of thumb — if your subquery in the FROM clause needs to see the outer row's columns, you need LATERAL. Every time.

Output.

id	name	order_ts	total
1	Alice	2026-07-09	60
2	Bob	2026-07-07	30
3	Carol	NULL	NULL

Rule of thumb. LATERAL is the keyword that turns a "same-level subquery" into a "correlated join." Without it, subquery scopes are isolated; with it, the inner query sees the outer row like function parameters.

Worked example — `LEFT JOIN LATERAL ... ON TRUE` for preserving outer rows

Detailed explanation. The most common LATERAL variant in production analytics — "one row per customer, plus the latest order or NULLs if there isn't one." The ON TRUE looks weird until you realise every LEFT JOIN needs an ON clause and there's no natural join column between a LATERAL block and its outer.

Question. Write a Postgres query that returns every customer along with their most recent order (or NULLs for customers who never ordered). Sort by customer id.

Input. (Same customers and orders as above.)

Code.

SELECT
  c.id AS customer_id,
  c.name,
  o.id AS last_order_id,
  o.order_ts AS last_order_ts,
  o.total AS last_order_total
FROM customers c
LEFT JOIN LATERAL (
  SELECT o.id, o.order_ts, o.total
  FROM orders o
  WHERE o.customer_id = c.id
  ORDER BY o.order_ts DESC
  LIMIT 1
) o ON TRUE
ORDER BY c.id;

Step-by-step explanation.

LEFT JOIN LATERAL — LEFT-outer semantics. If the inner subquery returns zero rows, the outer row is preserved with NULLs in the inner columns.
ON TRUE — every LEFT JOIN needs an ON clause. The LATERAL has no natural join column (the correlation happens inside the parentheses), so ON TRUE says "match unconditionally on whatever the inner emits."
Inner subquery — one row per outer via LIMIT 1. Reads c.id from the outer scope.
Carol has no orders — the inner returns zero rows — the LEFT JOIN LATERAL preserves Carol with NULLs in last_order_id, last_order_ts, last_order_total.
Alice and Bob each have orders — the inner returns exactly one row — the LEFT JOIN LATERAL emits the pair.

Output.

customer_id	name	last_order_id	last_order_ts	last_order_total
1	Alice	104	2026-07-09	60
2	Bob	106	2026-07-07	30
3	Carol	NULL	NULL	NULL

Rule of thumb. LEFT JOIN LATERAL (…) x ON TRUE is the canonical "outer preserved, inner optional" pattern. Every reporting query with a "most recent event per entity" ask uses this shape.

Worked example — LATERAL over a set-returning function

Detailed explanation. The killer LATERAL pattern for semi-structured data — unnesting arrays or JSON per outer row. Postgres exposes unnest, jsonb_array_elements, generate_series; Snowflake exposes FLATTEN; MySQL uses JSON_TABLE. All read cleanly inside a LATERAL.

Question. Given orders(id, items jsonb) where items is a JSON array of {sku, qty, price} objects, unroll each item into its own row alongside the parent order id.

Input.

id	items
1	`[{"sku":"A","qty":2,"price":10},{"sku":"B","qty":1,"price":25}]`
2	`[{"sku":"C","qty":3,"price":5}]`

Code.

SELECT
  o.id AS order_id,
  item ->> 'sku' AS sku,
  (item ->> 'qty')::int AS qty,
  (item ->> 'price')::numeric AS price
FROM orders o
CROSS JOIN LATERAL jsonb_array_elements(o.items) AS item
ORDER BY o.id, sku;

Step-by-step explanation.

jsonb_array_elements(o.items) is a set-returning function that produces one row per element of the JSON array — perfect for unnesting.
Inside the LATERAL, o.items is bound from the outer row. Order 1 emits two elements; order 2 emits one.
CROSS JOIN LATERAL — inner-join semantics. Orders with an empty or NULL items array would be dropped; swap to LEFT JOIN LATERAL … ON TRUE to preserve them.
Outside the LATERAL, we destructure each element with ->> (returns text) and cast to int / numeric as needed.
Snowflake equivalent: LATERAL FLATTEN(input => o.items) f then f.value:sku::string, f.value:qty::int. SQL Server: CROSS APPLY OPENJSON(o.items) WITH (sku VARCHAR(50), qty INT, price DECIMAL(10,2)).

Output.

order_id	sku	qty	price
1	A	2	10
1	B	1	25
2	C	3	5

Rule of thumb. LATERAL + set-returning function is the way to unnest arrays and JSON in modern SQL. Every ETL pipeline that ingests semi-structured payloads uses this shape somewhere.

Senior interview question on LATERAL correlation

A senior interviewer might ask: "Write a Postgres query that, for each customer, returns the customer's id, the total number of orders they've placed, the total spend, and the details of their most recent order in one row. Use LATERAL and explain why this beats three separate correlated subqueries."

Solution Using LEFT JOIN LATERAL with per-outer aggregate + top-1

SELECT
  c.id                                 AS customer_id,
  c.name,
  agg.order_count,
  agg.total_spend,
  last_ord.id                          AS last_order_id,
  last_ord.order_ts                    AS last_order_ts,
  last_ord.total                       AS last_order_total
FROM customers c
LEFT JOIN LATERAL (
  SELECT COUNT(*) AS order_count, COALESCE(SUM(total), 0) AS total_spend
  FROM orders
  WHERE customer_id = c.id
) agg ON TRUE
LEFT JOIN LATERAL (
  SELECT id, order_ts, total
  FROM orders
  WHERE customer_id = c.id
  ORDER BY order_ts DESC
  LIMIT 1
) last_ord ON TRUE
ORDER BY c.id;

Step-by-step trace.

Outer	agg subquery result	last_ord subquery result	Emitted row
c1 = Alice	(4, 215)	(104, 07-09, 60)	(1, Alice, 4, 215, 104, 07-09, 60)
c2 = Bob	(2, 100)	(106, 07-07, 30)	(2, Bob, 2, 100, 106, 07-07, 30)
c3 = Carol	(0, 0)	(NULL, NULL, NULL)	(3, Carol, 0, 0, NULL, NULL, NULL)

Two LATERAL blocks per outer row — one aggregates, the other picks the top-1. Both are LEFT-outer, so customers with zero orders still appear (agg returns (0, 0) from the empty set with COALESCE; last_ord returns zero rows so the LEFT JOIN emits NULLs).

Output:

customer_id	name	order_count	total_spend	last_order_id	last_order_ts	last_order_total
1	Alice	4	215	104	2026-07-09	60
2	Bob	2	100	106	2026-07-07	30
3	Carol	0	0	NULL	NULL	NULL

Why this works — concept by concept:

One LATERAL block per per-outer computation — the aggregation lives in one LATERAL, the top-1 lookup lives in another. Each block runs once per outer row with the outer's c.id bound in. Adding a third computation adds a third LATERAL — the pattern composes cleanly.
COALESCE inside the aggregate LATERAL — SUM(total) on an empty result set is NULL. Wrapping it in COALESCE(..., 0) turns Carol's spend into a proper zero instead of a NULL. This is idiomatic for "count-plus-sum-with-zero-default" reports.
LEFT JOIN LATERAL for the top-1 lookup — Carol has no orders, so the top-1 subquery returns zero rows. LEFT JOIN LATERAL preserves her outer row with NULLs; CROSS JOIN LATERAL would drop her, which is wrong for a "one row per customer" report.
Three separate correlated scalars would be slower — one (SELECT COUNT(*) FROM orders WHERE …) in the SELECT list plus one (SELECT SUM(total) FROM …) plus one (SELECT id FROM … ORDER BY … LIMIT 1) is three inner passes per outer. Two LATERAL blocks is two inner passes — the aggregate combines COUNT and SUM in one pass — and the plan is more predictable.
Cost — O(N_c × (P_agg + P_top1)) where P_agg is the cost of the aggregate scan per customer (index on (customer_id) gives O(matching_rows)) and P_top1 is the cost of the top-1 seek (index on (customer_id, order_ts DESC) gives O(log N_o)). With both indexes, the whole query is O(N_c × (matching_rows + log N_o)) — much better than three uncorrelated table scans.

SQL
Topic — joins
Joins problem library

Practice →

SQL Topic — subqueries Subquery drills

Practice →

3. CROSS APPLY & OUTER APPLY

`cross apply sql server` and `outer apply` — SQL Server's LATERAL equivalent, the translation table, and TVF composition

The mental model in one line: CROSS APPLY is SQL Server's spelling of CROSS JOIN LATERAL, and OUTER APPLY is its spelling of LEFT JOIN LATERAL ... ON TRUE — identical semantics with a different keyword, plus a killer composition trick with table-valued functions (TVFs) that turns "run this parametrised query per row" into a first-class citizen. Once you know the two-line translation table, every ANSI LATERAL recipe maps cleanly to SQL Server.

The translation table — every ANSI LATERAL maps to an APPLY.

CROSS JOIN LATERAL (…) alias → CROSS APPLY (…) alias. Inner-join semantics; drops outer row when inner returns zero.
LEFT JOIN LATERAL (…) alias ON TRUE → OUTER APPLY (…) alias. Left-outer semantics; preserves outer row with NULLs when inner returns zero.
SELECT * FROM t, LATERAL (…) → SELECT * FROM t CROSS APPLY (…). Same semantics.
LIMIT K inside the inner → TOP K inside the inner (SQL Server uses TOP, not LIMIT).
jsonb_array_elements(t.payload) (Postgres) → OPENJSON(t.payload) (SQL Server 2016+).
Set-returning functions → CROSS APPLY of a TVF or of OPENJSON / STRING_SPLIT / OPENROWSET.

Why SQL Server has APPLY instead of LATERAL.

Microsoft shipped CROSS APPLY and OUTER APPLY in SQL Server 2005 — years before Postgres, MySQL, or Oracle added LATERAL. Once the keyword was baked into the SQL Server dialect, changing it would break every stored procedure ever written.
Semantically APPLY and LATERAL are equivalent. There is no operation you can do in ANSI LATERAL that you can't do with CROSS APPLY / OUTER APPLY.
Oracle 12c (2013) shipped both keywords for portability — ANSI LATERAL for standards-compliance and CROSS/OUTER APPLY for teams migrating from SQL Server.

APPLY + TVF — the composition superpower.

A table-valued function (TVF) is a user-defined function that returns a table. SQL Server, Oracle, and Postgres all support them (Postgres calls them SETOF-returning functions).
The killer pattern — SELECT c.*, t.* FROM customers c CROSS APPLY dbo.top_n_orders(c.id, 3) t — invokes the TVF once per outer row, passing the outer's c.id and a fixed 3 as arguments. The TVF returns a table; the outer row is joined to every row it emits.
This turns a piece of reusable business logic (top_n_orders) into a plug-and-play building block. Callers get the correlated behaviour for free.
Postgres equivalent — LATERAL (SELECT * FROM top_n_orders(c.id, 3)) or, if the function is SETOF orders, just LATERAL top_n_orders(c.id, 3) t.

JSON unnest via APPLY on SQL Server.

OPENJSON(json_column) is SQL Server's JSON parser. Called inside CROSS APPLY, it returns one row per JSON element / property, with columns you specify via WITH (…).
CROSS APPLY OPENJSON(o.items) WITH (sku VARCHAR(50), qty INT, price DECIMAL(10,2)) — reads exactly like Postgres's jsonb_array_elements with per-column extraction.
For older SQL Server (pre-2016), the equivalent is a custom split TVF or STUFF + FOR XML PATH string manipulation — much uglier.

OPENJSON / OPENROWSET / STRING_SPLIT with APPLY — the SQL Server unnest family.

OPENJSON(json) — unnest a JSON string or JSON column. Returns key/value pairs by default; with WITH (…) returns typed columns.
STRING_SPLIT(delimited_string, delimiter) — unnest a delimited string. Returns one row per token.
OPENROWSET(BULK 'file.csv', ...) — read an external file. Not a LATERAL primitive per se, but often composed with APPLY.
All three are commonly used with CROSS APPLY to unnest a column into rows per parent.

Oracle's LATERAL + APPLY duality.

Oracle 12c ships both keywords. LATERAL (…) and CROSS APPLY (…) are interchangeable; LEFT JOIN LATERAL (…) ON TRUE and OUTER APPLY (…) are also interchangeable.
Most Oracle style guides prefer LATERAL for ANSI-conformance and use APPLY only when porting SQL Server code.
Oracle also supports TABLE(function_name(args)) inside a FROM clause — a legacy syntax for calling PL/SQL table functions, superseded by the LATERAL / APPLY approach.

Common APPLY gotchas.

TOP N inside CROSS APPLY without ORDER BY — non-deterministic. Always pair TOP N with ORDER BY.
Forgetting to include the outer alias — CROSS APPLY (SELECT TOP 3 * FROM orders WHERE customer_id = c.id ORDER BY order_ts DESC) o — the reference to c.id must live inside the parentheses, not at the top level.
Using SELECT * inside APPLY that returns overlapping column names — SQL Server may error on ambiguous columns.
Using APPLY in an old compatibility level — APPLY is available from 2005 onwards, so any modern SQL Server supports it, but SQL Server 2000 databases upgraded in-place without changing compatibility level may still have issues.

Perf story — APPLY vs alternatives on SQL Server.

APPLY is often faster than WITH cte AS (… ROW_NUMBER() OVER (…)) SELECT WHERE rn <= K when a covering index on (partition_col, order_col) exists. The APPLY plan seeks per outer row and stops early; the CTE plan sorts every row.
APPLY beats correlated scalar subquery when you need multiple columns from the inner query.
APPLY loses to a plain hash join when the join is a simple equality on a bounded inner set — the planner picks hash by default anyway.

Which APPLY variant when.

CROSS APPLY — use when you want inner-join semantics. Drop outer rows with no matching inner rows.
OUTER APPLY — use for reporting queries that need "one row per outer, plus optional inner." Every "customer report with most-recent-order-or-NULL" query uses OUTER APPLY.
Never use CROSS APPLY when you need to preserve outer rows — you'll silently lose data.

Worked example — LATERAL → CROSS APPLY translation of the top-3-orders query

Detailed explanation. The most common port — take a working Postgres LATERAL query and rewrite it for SQL Server. The translation is mechanical.

Question. Rewrite the top-3-orders-per-customer LATERAL query as SQL Server CROSS APPLY.

Input. (Same customers and orders as before.)

Code.

-- Postgres: LATERAL
SELECT c.id, c.name, o.order_ts, o.total
FROM customers c
CROSS JOIN LATERAL (
  SELECT order_ts, total
  FROM orders
  WHERE customer_id = c.id
  ORDER BY order_ts DESC
  LIMIT 3
) o
ORDER BY c.id, o.order_ts DESC;

-- SQL Server: CROSS APPLY
SELECT c.id, c.name, o.order_ts, o.total
FROM customers c
CROSS APPLY (
  SELECT TOP 3 order_ts, total
  FROM orders
  WHERE customer_id = c.id
  ORDER BY order_ts DESC
) o
ORDER BY c.id, o.order_ts DESC;

Step-by-step explanation.

Keyword swap — CROSS JOIN LATERAL becomes CROSS APPLY. The parentheses stay the same.
Limit keyword — LIMIT 3 becomes TOP 3 at the beginning of the SELECT. SQL Server has no LIMIT.
ORDER BY inside the inner query stays. SQL Server requires it for deterministic TOP results.
Correlation binding — the reference to c.id inside the inner query works identically. APPLY authorises correlation just like LATERAL.
Outer ORDER BY at the end works the same way.

Output. (Same as the LATERAL version.)

Rule of thumb. Every LATERAL query maps to CROSS APPLY / OUTER APPLY with two keyword swaps. Keep the same mental model; change only the words.

Worked example — CROSS APPLY + inline TVF for reusable per-row logic

Detailed explanation. The composition win — wrap the per-row logic in a TVF, then compose it via CROSS APPLY. Every caller gets the correlated behaviour with one keyword.

Question. Create an inline TVF dbo.top_n_orders(@cust INT, @n INT) that returns the top-N most-recent orders for a customer, then use CROSS APPLY to fetch the top-3 for every customer.

Input. (Same as before.)

Code.

-- TVF definition (SQL Server 2008+)
CREATE FUNCTION dbo.top_n_orders (@cust INT, @n INT)
RETURNS TABLE
AS RETURN
  SELECT TOP (@n) id, order_ts, total
  FROM orders
  WHERE customer_id = @cust
  ORDER BY order_ts DESC;

-- Caller uses CROSS APPLY to invoke the TVF per outer row
SELECT c.id AS customer_id, c.name, t.id AS order_id, t.order_ts, t.total
FROM customers c
CROSS APPLY dbo.top_n_orders(c.id, 3) t
ORDER BY c.id, t.order_ts DESC;

Step-by-step explanation.

CREATE FUNCTION … RETURNS TABLE AS RETURN <SELECT> defines an inline TVF — the body is a single SELECT that returns a table.
Inline TVFs are optimised alongside the caller — the planner inlines the function body and rewrites it as if you'd typed the SELECT inline. No function-call overhead.
CROSS APPLY dbo.top_n_orders(c.id, 3) t — for each customer row, invoke the TVF with the outer's c.id and the constant 3. The TVF returns the top-3 orders; APPLY joins them to the outer row.
Downstream callers use one keyword — CROSS APPLY top_n_orders(...) — instead of duplicating the ORDER BY + TOP + WHERE. Business logic centralised.
Compare with multi-statement TVFs (RETURNS @t TABLE(...) AS BEGIN INSERT INTO @t SELECT ... END) — these are NOT inlined and have significantly worse perf. Prefer inline TVFs.

Output.

customer_id	name	order_id	order_ts	total
1	Alice	104	2026-07-09	60
1	Alice	103	2026-07-08	90
1	Alice	102	2026-07-05	25
2	Bob	106	2026-07-07	30
2	Bob	105	2026-07-02	70

Rule of thumb. Inline TVFs + CROSS APPLY is the SQL Server pattern for reusable per-row logic. One function definition, N callers, zero copy-paste.

Worked example — OPENJSON unnest with CROSS APPLY

Detailed explanation. The JSON unnest story on SQL Server. OPENJSON is a table-valued function that parses JSON and emits rows; CROSS APPLY runs it per parent row.

Question. Given orders(id, items NVARCHAR(MAX)) where items is a JSON array of {sku, qty, price} objects, unroll each item into its own row.

Input.

id	items
1	`[{"sku":"A","qty":2,"price":10},{"sku":"B","qty":1,"price":25}]`
2	`[{"sku":"C","qty":3,"price":5}]`

Code.

SELECT
  o.id       AS order_id,
  j.sku,
  j.qty,
  j.price
FROM orders o
CROSS APPLY OPENJSON(o.items)
  WITH (
    sku   VARCHAR(50)     '$.sku',
    qty   INT             '$.qty',
    price DECIMAL(10, 2)  '$.price'
  ) AS j
ORDER BY o.id, j.sku;

Step-by-step explanation.

OPENJSON(o.items) parses the JSON string in o.items and returns one row per element of the top-level array.
WITH (col type '$.path', ...) — the "explicit-schema" form of OPENJSON. Each column extracts a JSON path from the current element. Types are enforced.
CROSS APPLY runs OPENJSON once per outer order row, passing o.items in. Order 1 emits two element rows; order 2 emits one.
WITH is important — without it, OPENJSON returns a key/value/type triplet per element, which is much harder to use. Always use the explicit-schema form for typed unnest.
The equivalent on Postgres — jsonb_array_elements(o.items) in LATERAL, then ->> 'sku' etc. Snowflake — LATERAL FLATTEN(input => o.items) then f.value:sku::string.

Output.

order_id	sku	qty	price
1	A	2	10.00
1	B	1	25.00
2	C	3	5.00

Rule of thumb. CROSS APPLY OPENJSON(json_col) WITH (col type '$.path', ...) is the SQL Server pattern for JSON unnest. Every ETL that lands JSON payloads uses this shape.

Senior interview question on APPLY composition

A senior interviewer might ask: "Design a reporting query on SQL Server that, for each product, returns the top-3 orders by revenue and the total revenue across all orders. Use CROSS APPLY for the top-3, an inline TVF for reuse, and OUTER APPLY for the aggregate so products with no orders still appear."

Solution Using inline TVF + CROSS APPLY + OUTER APPLY composition

-- Inline TVF for top-N by revenue
CREATE FUNCTION dbo.top_n_orders_by_rev (@prod INT, @n INT)
RETURNS TABLE
AS RETURN
  SELECT TOP (@n) id, order_ts, qty * price AS revenue
  FROM order_lines
  WHERE product_id = @prod
  ORDER BY qty * price DESC;

-- Reporting query
SELECT
  p.id            AS product_id,
  p.name          AS product_name,
  t.id            AS top_order_id,
  t.order_ts      AS top_order_ts,
  t.revenue       AS top_order_revenue,
  agg.total_rev,
  agg.order_count
FROM products p
CROSS APPLY dbo.top_n_orders_by_rev(p.id, 3) t
OUTER APPLY (
  SELECT
    COUNT(*)                    AS order_count,
    COALESCE(SUM(qty * price), 0) AS total_rev
  FROM order_lines
  WHERE product_id = p.id
) agg
ORDER BY p.id, t.revenue DESC;

Step-by-step trace.

Step	Outer (product_id)	CROSS APPLY t	OUTER APPLY agg	Emitted rows
1	p1	top 3 = (o11, o12, o13)	(order_count=8, total_rev=520)	3 rows joined with the agg per row
2	p2	top 3 = (o21, o22) (only 2 orders)	(order_count=2, total_rev=90)	2 rows joined
3	p3	top 3 = (empty)	(order_count=0, total_rev=0)	0 rows (CROSS APPLY drops)

The CROSS APPLY t drops products with no orders — desirable for the "top order per product" report. The OUTER APPLY agg would preserve the outer row if it were the only join; since it's paired with CROSS APPLY t, the outer row is only emitted when t emits at least one inner row.

Output:

product_id	product_name	top_order_id	top_order_ts	top_order_revenue	total_rev	order_count
1	Widget A	o11	2026-07-10	250	520	8
1	Widget A	o12	2026-07-08	150	520	8
1	Widget A	o13	2026-07-05	100	520	8
2	Widget B	o21	2026-07-09	50	90	2
2	Widget B	o22	2026-07-06	40	90	2

Why this works — concept by concept:

Inline TVF centralises the "top N by revenue" logic — one function definition, called from every reporting query that needs the pattern. The planner inlines the body, so there's zero function-call overhead.
CROSS APPLY for the top-N, OUTER APPLY for the aggregate — CROSS APPLY drops products with zero orders (they shouldn't appear in a "top orders per product" report); OUTER APPLY preserves the outer row if only the aggregate matters. Choosing between the two is a semantic call, not a syntactic one.
COALESCE(SUM(...), 0) inside the aggregate APPLY — SUM over an empty set is NULL. Wrapping in COALESCE yields a proper zero, so downstream reports don't need to handle NULL revenues.
Two APPLY blocks compose cleanly — each block runs once per outer row with the outer's p.id bound in. Adding a third computation (say, "average order size per product") adds a third block with no side effects on the first two.
Cost — O(N_p × (P_top3 + P_agg)) where P_top3 is the cost of the top-3 seek (with an index on (product_id, revenue DESC) it's O(log N_o + 3)) and P_agg is the cost of the aggregate scan (with an index on (product_id) it's O(matching_rows)). Both indexes present, this is fast — dramatically better than doing two separate correlated scans in the SELECT list, and beats a ROW_NUMBER() OVER + WHERE + GROUP BY alternative on skewed data.

SQL
Topic — joins (SQL) — hard
Hard SQL joins

Practice →

SQL Topic — subquery Subquery mastery

Practice →

4. Top-N per group pattern

`sql top-n per group` — the LATERAL + LIMIT recipe versus ROW_NUMBER() OVER, and the index shape that makes it fly

The mental model in one line: top-N-per-group is the archetype LATERAL query, and it beats the classic ROW_NUMBER() OVER (PARTITION BY group_col ORDER BY order_col DESC) WHERE rn ≤ N plan whenever the (group_col, order_col DESC) index exists and N is small — because the LATERAL plan is a nested loop with a per-group LIMIT, and the window function is a full sort. Once you know the two shapes and their index requirements, "give me the latest K events per entity" becomes a mechanical choice, not an art.

Three canonical approaches to top-N-per-group.

LATERAL + LIMIT — FROM outer CROSS JOIN LATERAL (SELECT ... FROM inner WHERE inner.k = outer.k ORDER BY inner.ts DESC LIMIT N) x. Plan is nested loop with a bounded inner. Best when (outer.k, inner.ts DESC) is indexed and N is small.
ROW_NUMBER() + filter — WITH r AS (SELECT ..., ROW_NUMBER() OVER (PARTITION BY k ORDER BY ts DESC) AS rn FROM inner) SELECT * FROM r WHERE rn <= N. Plan sorts every row, ranks, then filters. Best when index is missing or N is close to inner_rows / outer_rows.
Correlated scalar subquery per column — SELECT (SELECT MAX(x) FROM inner WHERE inner.k = outer.k) FROM outer. Only for single-column-single-row lookups. Cannot express "top 3."

Why LATERAL wins with the right index.

With an index on (customer_id, order_ts DESC), the inner query is a B-tree seek to the customer_id + a bounded scan of the next N rows. Cost per outer row is O(log N_inner + N).
The window function has no way to stop early — it must sort all rows in the inner table (or all rows within each partition, which is asymptotically the same) before applying the filter. Cost is O(N_inner log N_inner).
On a table with 10M orders and 100K customers, LATERAL + LIMIT 3 is roughly 100K × (log 10M + 3) ≈ 100K × 27 ≈ 2.7M ops. The window function is 10M × log 10M ≈ 10M × 23 ≈ 230M ops. Two orders of magnitude difference — real, reproducible, unmistakable.

When ROW_NUMBER() wins.

No index available. If you can't add (group_col, order_col DESC), LATERAL loses — every inner query becomes a full sequential scan.
N is close to inner_rows / outer_rows. If you're asking for "top 100 out of an average 105," you may as well sort everything anyway.
Cross-DB portability with legacy engines. MySQL 5.7 lacks LATERAL; ROW_NUMBER() is the portable answer.
Multi-window computations in the same pass. If you also need RANK() OVER and PERCENT_RANK() OVER in the same query, ROW_NUMBER() piggybacks on the existing sort.

The DISTINCT ON shortcut (Postgres only).

Postgres has SELECT DISTINCT ON (customer_id) * FROM orders ORDER BY customer_id, order_ts DESC — returns exactly one row per customer_id, the first one in the sort order.
Great for top-1 per group. Doesn't extend to top-N (you'd need N passes).
LATERAL is more general and reads more clearly across the top-1 → top-N spectrum.

The QUALIFY shortcut (Snowflake / BigQuery / Databricks).

SELECT * FROM orders QUALIFY ROW_NUMBER() OVER (PARTITION BY customer_id ORDER BY order_ts DESC) <= 3.
Same plan as WITH ... WHERE rn <= 3, but reads in one clause.
Portable across Snowflake / BigQuery / Databricks / Teradata; not available on Postgres / MySQL / SQL Server / Oracle.

Skewed-distribution matters — the "hot key" problem.

If 90% of orders belong to 10% of customers, ROW_NUMBER() OVER still has to sort all 100% of orders. LATERAL runs its bounded scan only against each customer's slice — the 10% of hot customers get slightly more expensive scans, but the 90% of cold customers get near-zero cost.
On heavy-tailed data (which is 99% of real production workloads), LATERAL's per-group cost adapts naturally.

BigQuery — no LATERAL keyword, but the pattern exists.

BigQuery uses ARRAY_AGG(... ORDER BY ... LIMIT N) inside a subquery — a materialisation-heavy alternative.
Or the QUALIFY shortcut with ROW_NUMBER().
Neither is as clean as ANSI LATERAL, but the semantics are the same.

Ties — a subtle correctness issue.

ORDER BY order_ts DESC LIMIT 3 returns exactly 3 rows even when there are ties on order_ts. The choice among tied rows is non-deterministic.
ROW_NUMBER() OVER (…) <= 3 behaves identically — non-deterministic tie-breaking.
For deterministic top-N with ties, use RANK() <= 3 (returns all tied rows) or add a tie-breaker column to the ORDER BY.

Benchmark shape — the shape every senior candidate should know.

Setup: 10M orders, 100K customers, index on (customer_id, order_ts DESC), K = 3.
LATERAL + LIMIT 3: ≈ 1.2s. Plan: nested loop join + index seek + LIMIT.
ROW_NUMBER() + WHERE rn ≤ 3: ≈ 4.8s. Plan: index scan + sort + filter.
Correlated scalar per column: ≈ 9.5s (three passes). Plan: N × subquery.
Ratio: LATERAL is 4× faster than window function, 8× faster than correlated scalar.

Worked example — latest 3 orders per customer with LATERAL

Detailed explanation. The archetype from an interview lens. Given the standard schema and the standard index, write the LATERAL query and explain the plan.

Question. Given orders(id, customer_id, order_ts, total) with an index on (customer_id, order_ts DESC), write a Postgres query returning the three most recent orders per customer.

Input. (10M orders across 100K customers.)

Code.

SELECT c.id, c.name, o.id AS order_id, o.order_ts, o.total
FROM customers c
CROSS JOIN LATERAL (
  SELECT id, order_ts, total
  FROM orders
  WHERE customer_id = c.id
  ORDER BY order_ts DESC
  LIMIT 3
) o
ORDER BY c.id, o.order_ts DESC;

Step-by-step explanation.

Outer scan of customers — one row per customer. Cost is O(N_c).
Per customer, an index seek into orders using the (customer_id, order_ts DESC) index. Cost O(log N_o).
Fetch the next 3 rows sequentially from the index — the ORDER BY is satisfied by the index, so no sort is needed. Cost O(3).
LATERAL emits three rows per customer (or fewer, for customers with fewer than three orders). Total inner cost O(N_c × (log N_o + 3)).
Final outer ORDER BY is a small sort over ~3 × N_c rows — cheap.

Output. (First few rows.)

id	name	order_id	order_ts	total
1	Alice	104	2026-07-09	60
1	Alice	103	2026-07-08	90
1	Alice	102	2026-07-05	25
2	Bob	106	2026-07-07	30
2	Bob	105	2026-07-02	70

Rule of thumb. LATERAL + LIMIT 3 + covering index is the canonical top-N-per-group plan on Postgres, MySQL 8, and Oracle. When (partition_col, order_col DESC) is indexed, this is dramatically faster than any window-function alternative.

Worked example — latest 3 orders per customer with ROW_NUMBER()

Detailed explanation. The portable alternative. Works everywhere but has to sort every row.

Question. Rewrite the top-3 query using ROW_NUMBER() OVER without LATERAL.

Input. (Same as above.)

Code.

WITH ranked AS (
  SELECT
    o.*,
    ROW_NUMBER() OVER (PARTITION BY customer_id ORDER BY order_ts DESC) AS rn
  FROM orders o
)
SELECT c.id, c.name, r.id AS order_id, r.order_ts, r.total
FROM customers c
JOIN ranked r ON r.customer_id = c.id AND r.rn <= 3
ORDER BY c.id, r.order_ts DESC;

Step-by-step explanation.

The CTE ranked computes ROW_NUMBER() over the whole orders table, partitioned by customer_id and ordered by order_ts DESC.
The window function requires a sort on the (customer_id, order_ts DESC) composite key. If the index covers both columns, the planner may use an index scan without an explicit sort — but not all planners do.
WHERE r.rn <= 3 filters to the top 3 per customer. This happens after the window function has ranked every row.
The outer JOIN to customers re-attaches the customer name.
Cost O(N_o log N_o) for the sort, plus O(N_o) for the ranking pass. On 10M orders, this is roughly 10M × 23 ≈ 230M ops — 2 orders of magnitude worse than the LATERAL plan.

Output. (Same rows as the LATERAL version.)

Rule of thumb. ROW_NUMBER() OVER (… PARTITION BY g ORDER BY o DESC) <= N is portable across every major engine (except MySQL 5.7 which lacks window functions). Use it when LATERAL is unavailable or when N is close to inner_rows / outer_rows.

Worked example — QUALIFY variant on Snowflake / BigQuery

Detailed explanation. The one-clause shortcut on modern cloud warehouses. Reads better than the CTE + WHERE combo.

Question. Rewrite the top-3-per-customer query using QUALIFY.

Input. (Same as above.)

Code.

-- Snowflake / BigQuery / Databricks / Teradata
SELECT c.id, c.name, o.id AS order_id, o.order_ts, o.total
FROM customers c
JOIN orders o ON o.customer_id = c.id
QUALIFY ROW_NUMBER() OVER (PARTITION BY o.customer_id ORDER BY o.order_ts DESC) <= 3
ORDER BY c.id, o.order_ts DESC;

Step-by-step explanation.

QUALIFY filters on window-function results — a shorthand for the CTE + WHERE combo.
Semantically identical to the ROW_NUMBER() + WHERE plan — same sort, same filter, same output.
Snowflake, BigQuery, Databricks, and Teradata support QUALIFY; Postgres, MySQL, SQL Server, and Oracle do not.
On engines that support both, QUALIFY reads better; on engines that support only one, LATERAL wins on perf.

Output. (Same rows.)

Rule of thumb. QUALIFY is syntactic sugar for the CTE + WHERE pattern. It doesn't change the plan, but it does reduce the query from three CTEs to one — a real readability win on complex reports.

Senior interview question on top-N-per-group perf

A senior interviewer might ask: "Design the query to fetch the latest 3 orders per customer on Postgres. Explain when you'd reach for LATERAL, when for ROW_NUMBER() OVER, and what index you'd add to make the LATERAL plan sing."

Solution Using CROSS JOIN LATERAL with the covering index

-- Index (usually already exists on production DBs)
CREATE INDEX IF NOT EXISTS idx_orders_customer_ts_desc
  ON orders (customer_id, order_ts DESC);

-- Query
SELECT
  c.id                                     AS customer_id,
  c.name,
  o.id                                     AS order_id,
  o.order_ts,
  o.total,
  ROW_NUMBER() OVER
    (PARTITION BY c.id ORDER BY o.order_ts DESC) AS rn_per_customer
FROM customers c
CROSS JOIN LATERAL (
  SELECT id, order_ts, total
  FROM orders
  WHERE customer_id = c.id
  ORDER BY order_ts DESC
  LIMIT 3
) o
ORDER BY c.id, o.order_ts DESC;

Step-by-step trace.

Outer (customer_id)	Inner scan	Rows emitted	rn_per_customer
c1 = Alice	seek + 3 rows	(o104, 07-09, 60), (o103, 07-08, 90), (o102, 07-05, 25)	1, 2, 3
c2 = Bob	seek + 2 rows	(o106, 07-07, 30), (o105, 07-02, 70)	1, 2
c3 = Carol	seek + 0 rows	(dropped — CROSS JOIN LATERAL)	—

The outer scans customers once. Per customer, the inner does one index seek + one bounded scan of up to 3 rows. The optional ROW_NUMBER() OVER (PARTITION BY c.id …) adds a within-per-customer ordering label — useful for downstream reports that want "1st most recent," "2nd most recent," "3rd most recent."

Output:

customer_id	name	order_id	order_ts	total	rn_per_customer
1	Alice	104	2026-07-09	60	1
1	Alice	103	2026-07-08	90	2
1	Alice	102	2026-07-05	25	3
2	Bob	106	2026-07-07	30	1
2	Bob	105	2026-07-02	70	2

Why this works — concept by concept:

Covering index on (customer_id, order_ts DESC) — the LATERAL plan can seek the index to the exact customer, then walk forward for up to three rows in the correct order. No sort, no filter, no rescanning the table. This is the single most important perf lever for LATERAL top-N.
LIMIT 3 inside the LATERAL is a per-outer-row bound — the inner scan stops after emitting three rows. Compared to WHERE rn <= 3, which needs the window function to rank every row before filtering, the LATERAL plan pays exactly 3 inner-row reads per customer.
CROSS JOIN LATERAL for the top-N semantics — customers with zero orders should not appear in the report (they have no "latest 3"). CROSS drops them; swap to LEFT JOIN LATERAL ... ON TRUE if the report demands "customers with no orders shown as zeros."
ROW_NUMBER() OVER runs over the LATERAL output, not the base table — because the LATERAL already limited to 3 rows per customer, the window function operates on at most 3 × N_customers rows. Free labels, no extra sort.
Cost — with the covering index, O(N_c × (log N_o + K)) where K = 3. On 10M orders × 100K customers, this is ≈ 2.7M ops — sub-second on any modern database. Without the index, the plan degrades to O(N_c × N_o) = O(10^12) — unrun-able. The index is the whole game.

SQL
Topic — top-N per group
Top-N per group drills

Practice →

SQL Topic — window functions Window function library

Practice →

5. When LATERAL beats subquery — perf matrix

`bigquery lateral`, planner behaviour, and LATERAL as an optimizer hint — the six-dialect perf matrix and the rules of thumb

The mental model in one line: LATERAL is worth reaching for whenever the planner would otherwise leave a correlated subquery un-optimised, whenever an index on (group_col, order_col) makes a per-outer LIMIT the winning plan, or whenever set-returning-function unnest requires correlation — and its dialect surface across Postgres, MySQL 8, Oracle 12c+, SQL Server, Snowflake, and BigQuery follows one primitive with six keywords. Once you internalise the four decision axes — plan quality, index availability, unnest requirement, dialect — the LATERAL-vs-subquery choice becomes a checklist.

Four decision axes.

Plan quality. Does the planner unnest the correlated subquery, or does it leave it as a per-row scan? On modern Postgres 14+, most correlated scalars get unnested to a hash join; on older versions and on some cloud warehouses, unnesting is not guaranteed.
Index availability. Does an index on (group_col, order_col DESC) exist? If yes, LATERAL + LIMIT is optimal. If no, LATERAL loses to ROW_NUMBER() over a table scan.
Unnest requirement. Does the inner query need to expand a set-returning function or JSON array? LATERAL is the only clean way to correlate an unnest with an outer row.
Dialect. Which engine, which keyword? LATERAL / CROSS APPLY / OUTER APPLY / QUALIFY / UNNEST — pick the right one.

Planner behaviour — correlated-scalar unnest failures.

Postgres 14+ unnests most (SELECT ... FROM inner WHERE inner.k = outer.k) scalar subqueries into a hash join or a merge join. Excellent plan quality.
Postgres 11–13 unnest correlated scalars less aggressively. Rewrite as LATERAL to force a predictable plan.
MySQL 8 tries harder than 5.7 but still misses some correlated-scalar cases. LATERAL is the safe rewrite.
Oracle 12c+ is aggressive about unnesting. LATERAL still helps when you want explicit multi-column output.
SQL Server 2016+ is competitive. CROSS APPLY is the idiomatic form; the planner recognises the pattern.
Snowflake and BigQuery are aggressive about scalar unnesting on the query engine side, but LATERAL / UNNEST remain the readable choice.

When LATERAL wins on perf.

Top-N per group with covering index. The definitive win. LATERAL + LIMIT + index seek is O(N_outer × (log N_inner + K)); window-function alternatives are O(N_inner log N_inner).
Multi-column top-1 with covering index. Same story — the LATERAL gives you multiple columns from the inner row for one seek.
JSON / array unnest. No competition — LATERAL is the only clean primitive.
TVF composition. LATERAL / CROSS APPLY calls the TVF per outer row; the alternative is a SELECT ... FROM outer WHERE outer.k IN (SELECT ...) which flattens the TVF away.
Correlated aggregates in reporting queries. LATERAL + SUM(...) / COUNT(...) per outer beats correlated scalar subqueries in the SELECT list when the planner isn't unnesting them.

When LATERAL loses (or ties) on perf.

No index on (group_col, order_col). LATERAL degrades to N × full scan of inner. Window function wins.
Very small inner tables. With inner_rows < 1000, the planner picks a hash join anyway; LATERAL adds no benefit.
Cross-warehouse code with legacy engines. MySQL 5.7 lacks LATERAL; QUALIFY / ROW_NUMBER() is the only portable path.
Very large N. With N close to inner_rows / outer_rows, the per-group LIMIT stops saving reads. Window function ties.
Skew of 0 (uniform distribution). With inner_rows / outer_rows ≈ constant per group, LATERAL and window function converge on cost.

LATERAL as an optimizer hint.

Even when a plain correlated subquery would produce the same plan, wrapping it in LATERAL forces the planner into the "nested-loop-with-limit" shape. This is useful when:
- The default plan is unstable across parameters.
- You want deterministic worst-case behaviour.
- The team wants a review-visible signal that per-row semantics are intentional.
LATERAL is not a hint in the formal sense (like /*+ INDEX(...) */) but functions as one in practice.

Dialect performance matrix — six engines, four patterns.

Top-N per group. LATERAL / CROSS APPLY / OUTER APPLY / QUALIFY / UNNEST — LATERAL wins on Postgres, MySQL 8, Oracle, SQL Server (as CROSS APPLY), Snowflake (both LATERAL and QUALIFY work); BigQuery uses QUALIFY.
JSON / array unnest. LATERAL + jsonb_array_elements on Postgres; CROSS APPLY + OPENJSON on SQL Server; LATERAL FLATTEN on Snowflake; JSON_TABLE on Oracle; UNNEST on BigQuery. All engines have a clean primitive.
TVF compose. CROSS APPLY on SQL Server / Oracle; LATERAL on Postgres / Snowflake / MySQL 8 (with SETOF functions). BigQuery has no user-defined TVF; scalar UDFs only.
Correlated scalar. All engines support the pattern; unnest quality varies. Postgres 14+ / Snowflake / BigQuery are aggressive; Postgres 11-13 / MySQL 8 less so; LATERAL wraps the pattern for predictability everywhere.

BigQuery specifics — LATERAL by another name.

BigQuery has no LATERAL keyword.
UNNEST(array_col) in the FROM clause is implicitly correlated to its parent row. SELECT t.id, x FROM t, UNNEST(t.arr) AS x behaves like CROSS JOIN LATERAL UNNEST(t.arr).
Scalar-correlation asks are expressed via subqueries — SELECT (SELECT MAX(x) FROM inner WHERE inner.k = outer.k) FROM outer. BigQuery's optimizer is aggressive about unnesting these.
Top-N per group is QUALIFY + ROW_NUMBER(), not LATERAL + LIMIT.

Snowflake specifics — LATERAL and LATERAL FLATTEN.

Snowflake ships LATERAL for correlated joins and LATERAL FLATTEN for VARIANT / JSON unnest.
QUALIFY is available and idiomatic for top-N per group.
Both LATERAL + LIMIT and QUALIFY + ROW_NUMBER() have similar performance on Snowflake because the columnar engine parallelises the ranking pass.

Rules of thumb, memorised.

Top-N per group with a covering index? LATERAL + LIMIT.
Top-N per group with no index? ROW_NUMBER() OVER + WHERE rn ≤ N, or QUALIFY on supporting engines.
JSON / array unnest? LATERAL (or CROSS APPLY on SQL Server). Every engine has a primitive.
Reusable per-row logic? CROSS APPLY of an inline TVF on SQL Server; LATERAL over a SETOF function on Postgres.
Cross-warehouse code? QUALIFY where available; ROW_NUMBER() + WHERE where not.
Correlated scalar with predictable plan? Wrap in LATERAL for explicit shape.

Worked example — LATERAL + LIMIT vs ROW_NUMBER() OVER — plan shape

Detailed explanation. The interview classic — show the two plans side-by-side and explain the difference.

Question. For orders(id, customer_id, order_ts, total) with an index on (customer_id, order_ts DESC), write both the LATERAL + LIMIT top-3 query and the ROW_NUMBER() + WHERE variant, and describe the difference in plan shape.

Input. (10M orders across 100K customers.)

Code.

-- Plan A: LATERAL + LIMIT
SELECT c.id, o.id, o.order_ts, o.total
FROM customers c
CROSS JOIN LATERAL (
  SELECT id, order_ts, total
  FROM orders
  WHERE customer_id = c.id
  ORDER BY order_ts DESC
  LIMIT 3
) o;

-- Plan B: ROW_NUMBER() + WHERE
SELECT c.id, o.id, o.order_ts, o.total
FROM customers c
JOIN (
  SELECT
    id, customer_id, order_ts, total,
    ROW_NUMBER() OVER (PARTITION BY customer_id ORDER BY order_ts DESC) AS rn
  FROM orders
) o ON o.customer_id = c.id AND o.rn <= 3;

Step-by-step explanation.

Plan A — Nested Loop → Index Seek on (customer_id, order_ts DESC) → Limit 3. One seek per customer_id + 3 reads. Cost O(N_c × (log N_o + 3)).
Plan B — Sort → Windowed ROW_NUMBER → Filter rn <= 3 → Hash Join to customers. Full sort over all 10M orders + ranking pass. Cost O(N_o log N_o).
On the given dataset (10M orders, 100K customers, index present): Plan A ≈ 1.2s wall clock; Plan B ≈ 4.8s wall clock. LATERAL wins by ≈ 4× on this shape.
Without the index, Plan A degrades to O(N_c × N_o) — dramatically worse — while Plan B stays at O(N_o log N_o). Adding the index is a hard requirement for LATERAL to beat.
When N is large (say, top-1000 per customer): the per-group LIMIT stops saving reads because each customer has fewer than 1000 orders on average, and Plan A degrades to a per-customer full scan; Plan B stays constant. Choose based on N vs inner_rows / outer_rows.

Output. Same rows for both plans; different perf.

Rule of thumb. For small N (top 1–100) with a covering index, LATERAL + LIMIT is the definitive winner. For large N or missing index, window function wins.

Worked example — correlated scalar subquery plan quality

Detailed explanation. A tricky performance case — the same logic as a scalar subquery in the SELECT list. On modern engines, the planner may unnest it; on older engines, it won't. LATERAL forces the good plan.

Question. For each customer, return their name plus the total spend across all their orders. Write both the correlated-scalar version and the LATERAL version, and explain when they diverge on perf.

Input. (Same as above.)

Code.

-- Correlated scalar in SELECT list — plan depends on unnesting
SELECT
  c.id,
  c.name,
  (SELECT COALESCE(SUM(total), 0)
   FROM orders
   WHERE customer_id = c.id) AS total_spend
FROM customers c;

-- LATERAL — deterministic hash-join plan
SELECT
  c.id,
  c.name,
  agg.total_spend
FROM customers c
LEFT JOIN LATERAL (
  SELECT COALESCE(SUM(total), 0) AS total_spend
  FROM orders
  WHERE customer_id = c.id
) agg ON TRUE;

Step-by-step explanation.

On Postgres 14+, the correlated scalar version is unnested into a hash-aggregate plan — semantically equivalent to a GROUP BY customer_id join. Fast.
On Postgres 11–13, MySQL 8, and some cloud engines, the unnest may not happen. The scalar runs as N × subquery, cost O(N_c × avg_orders_per_customer) — bad on skewed data.
The LATERAL version compiles to a nested-loop-with-aggregate or a hash-aggregate join — either way, deterministic. No planner-dependent behaviour.
On the same 10M-order dataset with 100K customers: correlated-scalar-unnested ≈ 3s; LATERAL ≈ 3s (same plan); correlated-scalar-un-unnested ≈ 30s.
Rule: if you don't trust the planner to unnest, or if you want review-visible determinism, use LATERAL.

Output. (Same rows for both.)

Rule of thumb. LATERAL is a plan-stabiliser for correlated aggregates. When the planner is smart, both versions perform the same; when it's not, LATERAL is the safety net.

Worked example — BigQuery UNNEST as implicit LATERAL

Detailed explanation. BigQuery has no LATERAL keyword but supports the pattern via UNNEST on array columns. The correlation is implicit.

Question. Given orders(id, items ARRAY<STRUCT<sku STRING, qty INT64, price NUMERIC>>) on BigQuery, unroll each item into its own row alongside the parent order id.

Input.

id	items
1	`[{sku:"A", qty:2, price:10}, {sku:"B", qty:1, price:25}]`
2	`[{sku:"C", qty:3, price:5}]`

Code.

-- BigQuery: UNNEST is implicit LATERAL
SELECT
  o.id     AS order_id,
  item.sku,
  item.qty,
  item.price
FROM orders o,
     UNNEST(o.items) AS item
ORDER BY o.id, item.sku;

Step-by-step explanation.

FROM orders o, UNNEST(o.items) AS item — the comma is a CROSS JOIN; UNNEST(o.items) reads the outer row's items array. This is implicit LATERAL.
BigQuery's parser recognises that UNNEST on a column reference must be per-row. No LATERAL keyword needed.
To get LEFT-outer semantics — preserve orders with an empty items array — use LEFT JOIN UNNEST(o.items) AS item ON TRUE.
UNNEST supports arrays of structs, arrays of primitives, and even repeated fields inside nested structs. The syntax stays the same.
Equivalent to SELECT o.id, item.* FROM orders o CROSS JOIN LATERAL UNNEST(o.items) item in ANSI form.

Output.

order_id	sku	qty	price
1	A	2	10
1	B	1	25
2	C	3	5

Rule of thumb. On BigQuery, UNNEST(array_col) in the FROM clause is LATERAL by another name. No keyword, same semantics.

Senior interview question on cross-warehouse portability

A senior interviewer might ask: "Your company runs analytics on Snowflake (batch) and BigQuery (batch). Design the top-3-orders-per-customer query that produces the same output on both warehouses. Where would you use LATERAL, where QUALIFY, and why?"

Solution Using dbt macro with dialect dispatch

-- macros/top_n_orders_per_customer.sql
{% macro top_n_orders_per_customer(customers_rel, orders_rel, n=3) %}
  {%- if target.type == 'snowflake' -%}
    -- Snowflake: LATERAL + LIMIT for index-seek path
    SELECT c.id AS customer_id, c.name, o.id AS order_id, o.order_ts, o.total
    FROM {{ customers_rel }} c
    CROSS JOIN LATERAL (
      SELECT id, order_ts, total
      FROM {{ orders_rel }}
      WHERE customer_id = c.id
      ORDER BY order_ts DESC
      LIMIT {{ n }}
    ) o
  {%- elif target.type == 'bigquery' -%}
    -- BigQuery: QUALIFY + ROW_NUMBER (no LATERAL keyword)
    SELECT c.id AS customer_id, c.name, o.id AS order_id, o.order_ts, o.total
    FROM {{ customers_rel }} c
    JOIN {{ orders_rel }} o ON o.customer_id = c.id
    QUALIFY ROW_NUMBER() OVER (PARTITION BY o.customer_id ORDER BY o.order_ts DESC) <= {{ n }}
  {%- else -%}
    -- Portable fallback: CTE + WHERE rn <= n
    WITH ranked AS (
      SELECT o.*,
        ROW_NUMBER() OVER (PARTITION BY customer_id ORDER BY order_ts DESC) AS rn
      FROM {{ orders_rel }} o
    )
    SELECT c.id AS customer_id, c.name, r.id AS order_id, r.order_ts, r.total
    FROM {{ customers_rel }} c
    JOIN ranked r ON r.customer_id = c.id AND r.rn <= {{ n }}
  {%- endif -%}
{% endmacro %}

Step-by-step trace.

Dispatch branch	What runs	Why
Snowflake	LATERAL + LIMIT n	Uses per-outer bound for the tightest plan
BigQuery	QUALIFY ROW_NUMBER() OVER ... <= n	No LATERAL keyword on BigQuery
Postgres / MySQL / SQL Server (fallback)	CTE + ROW_NUMBER + WHERE rn <= n	Portable across the remaining engines
All	Same output columns (customer_id, name, order_id, order_ts, total)	Downstream stays warehouse-agnostic

The macro dispatches on target.type at compile time. Snowflake gets the LATERAL variant; BigQuery gets QUALIFY; every other target gets the portable CTE version. All three emit the same output schema.

Output:

customer_id	name	order_id	order_ts	total
1	Alice	104	2026-07-09	60
1	Alice	103	2026-07-08	90
1	Alice	102	2026-07-05	25
2	Bob	106	2026-07-07	30
2	Bob	105	2026-07-02	70

Why this works — concept by concept:

Compile-time dispatch, zero runtime overhead — dbt evaluates the Jinja block at compile time. The emitted SQL is one of three variants; no runtime branching, no reflection, no penalty.
Snowflake gets LATERAL for the tighter plan — Snowflake supports LATERAL and pushes the LIMIT into the inner scan. With a supporting clustering key on (customer_id, order_ts), the plan is the tightest available.
BigQuery gets QUALIFY because there is no LATERAL keyword — QUALIFY + ROW_NUMBER() is idiomatic on BigQuery, reads well, and produces the same output shape as the LATERAL variant.
Portable fallback uses only CTE + WHERE — every warehouse supports window functions (with the exception of MySQL 5.7, which the fallback would need a further branch for). One SQL that runs on Postgres, MySQL 8, SQL Server, Oracle, and Databricks with no changes.
Cost — Snowflake LATERAL is O(N_c × (log N_o + K)) with a clustering key; BigQuery QUALIFY is O(N_o log N_o) (columnar sort is cheap); portable CTE is O(N_o log N_o). All three run in single-digit seconds on the 10M × 100K dataset on medium warehouse sizes. The macro adds zero runtime cost; the maintainability win is that downstream consumers see identical output columns regardless of which warehouse produced them.

SQL
Topic — joins (SQL) — medium
Medium SQL joins

Practice →

SQL
Topic — SQL
SQL problem library — 450+ DE-focused questions

Practice →

Cheat sheet — LATERAL / CROSS APPLY recipe list

Two-slot skeleton. Slot 1: the join keyword (CROSS JOIN LATERAL / LEFT JOIN LATERAL ... ON TRUE / CROSS APPLY / OUTER APPLY). Slot 2: the inner subquery that references outer columns. Memorise both; every LATERAL answer fills the two slots.
CROSS JOIN LATERAL — inner-join semantics; drops outer rows with no inner match. Same as CROSS APPLY on SQL Server.
LEFT JOIN LATERAL ... ON TRUE — left-outer semantics; preserves outer rows with NULLs when the inner returns zero. Same as OUTER APPLY on SQL Server. The ON TRUE is not optional — every LEFT JOIN needs an ON clause.
Correlation binding rule. Inside a LATERAL block, the outer row's columns are in scope like function parameters. Without LATERAL, the outer scope is invisible inside the subquery.
Postgres syntax variants. FROM outer, LATERAL (…) x (comma) or FROM outer CROSS JOIN LATERAL (…) x (explicit JOIN). Same plan; explicit JOIN form is preferred for review clarity.
SQL Server translation. CROSS APPLY (…) = CROSS JOIN LATERAL (…); OUTER APPLY (…) = LEFT JOIN LATERAL (…) ON TRUE; TOP N inside APPLY = LIMIT N inside LATERAL.
Top-N per group primitive. CROSS JOIN LATERAL (SELECT … WHERE inner.k = outer.k ORDER BY inner.ts DESC LIMIT N) x. With an index on (k, ts DESC), the plan is nested loop + index seek + LIMIT — dramatically faster than ROW_NUMBER() OVER + WHERE rn ≤ N on skewed data.
JSON unnest primitive. Postgres: LATERAL jsonb_array_elements(t.payload). SQL Server: CROSS APPLY OPENJSON(t.payload) WITH (…). Snowflake: LATERAL FLATTEN(input => t.payload). BigQuery: UNNEST(t.array_col) in the FROM clause (implicit LATERAL).
Array unnest primitive. Postgres: LATERAL unnest(t.arr) x(v). Oracle: TABLE(t.arr) x or CROSS APPLY UNNEST(t.arr). BigQuery: UNNEST(t.arr). All produce one row per (parent, element) pair.
TVF composition primitive. SQL Server: CROSS APPLY dbo.top_n_orders(c.id, 3) t. Postgres: LATERAL top_n_orders(c.id, 3) t for SETOF-returning functions. Reusable per-row business logic centralised in one function.
Multi-column top-1 primitive. LEFT JOIN LATERAL (SELECT id, order_ts, total FROM orders WHERE customer_id = c.id ORDER BY order_ts DESC LIMIT 1) o ON TRUE. Beats three separate correlated scalars because it does one inner scan for all columns.
Per-outer aggregate primitive. LEFT JOIN LATERAL (SELECT COUNT(*) AS cnt, COALESCE(SUM(total), 0) AS spend FROM orders WHERE customer_id = c.id) agg ON TRUE. Beats correlated scalar per aggregate; predictable plan.
Composable LATERAL blocks. Multiple LATERAL blocks in one FROM clause each run once per outer row. Add a third computation — add a third block; no interaction between them.
DISTINCT ON shortcut (Postgres only). SELECT DISTINCT ON (customer_id) * FROM orders ORDER BY customer_id, order_ts DESC — top-1 per group in one keyword. Doesn't extend to top-N.
QUALIFY shortcut (Snowflake / BigQuery / Databricks / Teradata). SELECT * FROM orders QUALIFY ROW_NUMBER() OVER (…) <= N. Reads like WHERE-on-window-function. Same plan as CTE + WHERE.
When LATERAL beats ROW_NUMBER() OVER. Index on (group_col, order_col DESC) present + small N (top 1–100) + skewed distribution + engine supports LATERAL. All four → LATERAL wins by 2–10×.
When ROW_NUMBER() OVER beats LATERAL. No supporting index, or large N, or uniform distribution, or MySQL 5.7 (no LATERAL). All four → window function wins or ties.
Dialect matrix. LATERAL: Postgres 9.3+, MySQL 8.0.14+, Oracle 12c+, Snowflake. CROSS APPLY / OUTER APPLY: SQL Server 2005+, Oracle 12c+. UNNEST (implicit LATERAL): BigQuery. QUALIFY: Snowflake, BigQuery, Databricks, Teradata.
LATERAL as an optimizer hint. Even when a plain correlated subquery gets unnested, wrapping in LATERAL forces the "nested-loop-with-limit" plan — deterministic worst case, review-visible per-row intent.
Cost model. LATERAL + LIMIT with covering index: O(N_outer × (log N_inner + K)). ROW_NUMBER() OVER: O(N_inner log N_inner). Correlated scalar: O(N_outer × per-outer-cost) with unnesting; O(N_outer × N_inner) without.

Frequently asked questions

What is a SQL LATERAL join and when do you use it?

A sql lateral join is a FROM-clause subquery that runs once per row of the outer relation, with the inner query allowed to reference the outer row's columns like function parameters. Reach for it whenever you need "for each outer row, run this correlated query" — top-N per group with LIMIT (the classic latest-3-orders-per-customer), JSON / array unnest with jsonb_array_elements or OPENJSON or UNNEST correlated to the parent, table-valued-function composition where the TVF takes outer-row arguments, or multi-column top-1 lookups where a scalar subquery would need three separate correlated queries. LATERAL is a keyword introduced in SQL/1999 and shipped in Postgres 9.3 (2013), MySQL 8.0.14 (2019), Oracle 12c (2013), and Snowflake; SQL Server never adopted the keyword but offers CROSS APPLY (inner semantics) and OUTER APPLY (left-outer semantics) with identical behaviour since SQL Server 2005. BigQuery uses implicit correlation via UNNEST on array columns in the FROM clause. Every LATERAL query has a two-slot shape — the join keyword (CROSS JOIN LATERAL vs LEFT JOIN LATERAL … ON TRUE) plus the inner subquery — and once you internalise "for each outer row, run this inner query," the whole family reads like a single primitive across every dialect.

What is the difference between LATERAL and a subquery?

An ordinary subquery in the FROM clause is evaluated independently of the outer query — its scope is only the tables it names, and it cannot see outer columns. A LATERAL subquery is evaluated per outer row with the outer's columns bound in scope, so the inner query can reference outer.k, outer.ts, or any outer column freely. This is the correlation binding rule — LATERAL is the keyword that authorises the outer-column reference. A correlated scalar subquery in the SELECT list (SELECT (SELECT MAX(x) FROM inner WHERE k = outer.k) FROM outer) has a similar per-row behaviour but is restricted to one column, one row — LATERAL generalises that to a whole inner table with multiple columns and multiple rows. Perf-wise, modern planners often unnest correlated scalars into hash joins, but the unnesting is not guaranteed; LATERAL forces a predictable "nested-loop-with-limit" plan that is often faster on skewed data with a supporting index. In short — subqueries are for isolated computations, LATERAL is for correlated ones that need outer-column visibility and typically emit more than one column or more than one row per outer.

How does CROSS APPLY compare to LATERAL?

cross apply sql server is SQL Server's spelling of ANSI LATERAL — same semantics, different keyword. CROSS APPLY (…) maps directly to CROSS JOIN LATERAL (…); outer apply maps to LEFT JOIN LATERAL (…) ON TRUE. The only syntactic differences are the keyword itself and SQL Server's use of TOP N instead of LIMIT N inside the inner query. Every ANSI LATERAL recipe — top-N per group, JSON unnest via OPENJSON, TVF composition, multi-column top-1 — ports to CROSS APPLY with two keyword swaps. Oracle 12c+ ships both keywords (LATERAL and CROSS APPLY / OUTER APPLY), so Oracle teams can pick whichever their style guide prefers. SQL Server has offered CROSS APPLY / OUTER APPLY since 2005, years before Postgres or MySQL added LATERAL, so the APPLY family is deeply embedded in SQL Server codebases and shows up in every "top-N per group" or "TVF-per-row" pattern. When porting Postgres code to SQL Server (or vice versa), the mechanical rules are: CROSS JOIN LATERAL ↔ CROSS APPLY, LEFT JOIN LATERAL … ON TRUE ↔ OUTER APPLY, LIMIT N ↔ TOP N, jsonb_array_elements ↔ OPENJSON. Same primitive; same performance story; different spelling.

How do you get top-N per group with LATERAL?

The archetype: SELECT c.*, o.* FROM customers c CROSS JOIN LATERAL (SELECT * FROM orders WHERE customer_id = c.id ORDER BY order_ts DESC LIMIT N) o. The outer scans customers once; per outer row, the inner runs an index seek on the (customer_id, order_ts DESC) composite index and reads at most N rows. On SQL Server the same query is CROSS APPLY (SELECT TOP N * FROM orders WHERE customer_id = c.id ORDER BY order_ts DESC) o. To preserve customers with no orders, swap CROSS JOIN LATERAL for LEFT JOIN LATERAL … ON TRUE (or CROSS APPLY for OUTER APPLY on SQL Server). This plan is O(N_customers × (log N_orders + K)) — dramatically faster than ROW_NUMBER() OVER (PARTITION BY customer_id ORDER BY order_ts DESC) WHERE rn ≤ N, which is O(N_orders log N_orders) for the sort. On a 10M-order, 100K-customer table with the covering index, LATERAL + LIMIT 3 runs in about 1.2 seconds; the window-function variant runs in about 4.8 seconds — a 4× win. On engines without LATERAL (MySQL 5.7, older SQL Server), fall back to ROW_NUMBER() OVER + WHERE. On Snowflake, BigQuery, Databricks, and Teradata, QUALIFY ROW_NUMBER() OVER (…) <= N is a one-clause shortcut with the same plan as CTE + WHERE.

Which SQL engines support LATERAL / CROSS APPLY in 2026?

As of 2026: Postgres 9.3+ ships full ANSI LATERAL and both CROSS JOIN LATERAL and LEFT JOIN LATERAL ... ON TRUE. MySQL 8.0.14+ ships the same (MySQL 5.7 lacks LATERAL — use ROW_NUMBER() fallback). Oracle 12c+ ships both LATERAL (ANSI) and CROSS APPLY / OUTER APPLY (SQL-Server-compatible). SQL Server 2005+ ships CROSS APPLY and OUTER APPLY with identical semantics to LATERAL. Snowflake ships LATERAL and LATERAL FLATTEN (for VARIANT / JSON unnest). BigQuery has no LATERAL keyword but supports the pattern via implicit correlation on UNNEST(array_col) in the FROM clause; scalar correlation uses subqueries with aggressive planner unnesting. Databricks SQL supports LATERAL VIEW (a Hive-style syntax) plus QUALIFY. Redshift has no LATERAL; use ROW_NUMBER() + WHERE. Streaming SQL engines like Flink SQL and RisingWave inherit LATERAL semantics from ANSI. Memorise the six-row engine × keyword matrix — when an interviewer asks "would you use LATERAL here?" the first question back is always "what's the warehouse?"

Does BigQuery support LATERAL joins?

bigquery lateral is a common question — the short answer is "not by that name, but the semantics are there." BigQuery does not ship the LATERAL keyword. Instead, UNNEST(array_col) in the FROM clause is implicitly correlated to its parent row — SELECT o.id, item.sku FROM orders o, UNNEST(o.items) AS item behaves exactly like CROSS JOIN LATERAL UNNEST(o.items) in ANSI SQL. To preserve rows with an empty array, use LEFT JOIN UNNEST(o.items) AS item ON TRUE — this is BigQuery's LEFT JOIN LATERAL … ON TRUE equivalent. For scalar correlations ("for each row, compute a value from a related table"), BigQuery uses subqueries in the SELECT list; the planner is aggressive about unnesting correlated scalars into hash joins, so perf is competitive with ANSI LATERAL. For top-N per group, BigQuery uses QUALIFY ROW_NUMBER() OVER (PARTITION BY … ORDER BY … DESC) <= N — a one-clause shorthand for the CTE + WHERE pattern. There is no direct BigQuery equivalent to CROSS JOIN LATERAL (SELECT … LIMIT N) with an inner LIMIT — the top-N pattern goes through QUALIFY. When porting a Postgres LATERAL query to BigQuery, the mechanical rules are: LATERAL jsonb_array_elements(…) → UNNEST(…); LATERAL (SELECT … LIMIT N) for top-N → JOIN … QUALIFY ROW_NUMBER() OVER (…) <= N; scalar correlations stay as SELECT-list subqueries and trust the planner.

Practice on PipeCode

Drill the SQL joins practice library → for CROSS JOIN LATERAL, LEFT JOIN LATERAL ... ON TRUE, CROSS APPLY, and OUTER APPLY variants across Postgres, MySQL 8, Oracle, and SQL Server dialects.
Rehearse on top-N per group problems → — the archetype LATERAL win where a covering index on (group_col, order_col DESC) beats every ROW_NUMBER() OVER + WHERE rn ≤ N alternative.
Sharpen the subqueries drill room → to feel the difference between correlated scalar subqueries and LATERAL joins in the FROM clause.
Push the difficulty ceiling with hard SQL joins → for the composite APPLY + TVF patterns senior interviewers love — multi-column top-1, per-outer aggregate, JSON unnest with correlation.
Warm up with medium SQL joins → — the mid-difficulty rung where LATERAL vs ROW_NUMBER() OVER trade-offs show up.
Layer window function drills → — the ROW_NUMBER() + QUALIFY axis that pairs with LATERAL for the top-N-per-group family.
Practise self-join drills → for the "row-to-row within a table" pattern that LATERAL often replaces with a cleaner form.
Layer subquery drills → to build the mental map of correlated vs uncorrelated subqueries and where LATERAL fits between them.
Sharpen the general SQL surface with the SQL practice library → which contains 450+ DE-focused questions covering LATERAL, APPLY, top-N per group, JSON unnest, and every adjacent pattern.
For the broader SQL interview surface, take the SQL for Data Engineering course →.

Pipecode.ai is Leetcode for Data Engineering — every `sql lateral join` recipe above ships with hands-on practice rooms where you write the two-slot skeleton, wire `CROSS JOIN LATERAL` for top-3-per-customer with a covering index, translate to `cross apply sql server` and `outer apply` for the Microsoft stack, port the same pattern to `postgres lateral`, chase the `lateral vs subquery` performance question with real ANALYZE plans, rehearse the `sql top-n per group` interview classic, compose inline TVFs on SQL Server for reusable per-row logic, unnest JSON with `OPENJSON` and `jsonb_array_elements`, and cross-warehouse-dispatch the whole family to `bigquery lateral` via `UNNEST` and QUALIFY. PipeCode pairs every reading with 450+ DE-focused problems and a real-time scoring engine, so you never have to wonder whether your LATERAL answer holds up under a senior interviewer's depth probes.

Practice SQL joins now →
Top-N per group drills →

SQL MATCH_RECOGNIZE: Pattern Recognition for Sessions, Funnels & Anomalies

Gowtham Potureddi — Sat, 11 Jul 2026 15:25:53 +0000

sql match_recognize is the single most under-taught pattern in warehouse SQL — and the single largest edge a senior data engineer can pull out of the ANSI SQL/2016 standard on a whiteboard. The same "detect an ordered sequence of rows that satisfies a regex-like pattern" ask shows up in five different costumes: sessionization ("group events into 30-minute sessions"), funnel conversion ("who did signup → view → cart → checkout in that order?"), anomaly detection ("find the V-shape reversal in this price series"), fraud bursts ("five spend events inside sixty seconds"), and streaming CEP ("emit an alert as soon as the pattern completes"). Every answer starts from the same seven-slot skeleton: PARTITION BY entity ORDER BY time MEASURES ... PATTERN (A B+ C) DEFINE A AS ..., B AS ..., C AS .... The interviewer wants to hear you say "row pattern matching is regex over ordered rows" in the first sentence.

This guide is the mid-to-senior tour you wished existed the first time an interviewer asked you to write a row pattern matching sql query on the whiteboard, sketch an oracle match_recognize V-shape anomaly for a financial series, port a query to snowflake match_recognize, wire a flink sql pattern for streaming CEP, or design a funnel analysis sql pipeline that reports drop-off with sql pattern detection primitives. It walks through the ANSI clause anatomy (PARTITION BY / ORDER BY / MEASURES / ONE ROW PER MATCH vs ALL ROWS PER MATCH / AFTER MATCH SKIP / PATTERN with regex quantifiers / DEFINE), the sessionization pattern with PATTERN ((A B*)+) and per-user partitioning, the funnel pattern with anchored ordering and PERMUTE variants, the V-shape and W-shape stock anomaly the Oracle documentation made famous, and the dialect matrix — Oracle 12c+, Snowflake, Flink SQL, Trino/Presto, RisingWave versus Postgres and BigQuery with the portable LAG + SUM(new_group) OVER fallback. Every section pairs a teaching block with a Solution-Tail interview answer — code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works.

When you want hands-on reps immediately after reading, drill the pattern-matching practice library →, rehearse on gaps-and-islands problems → for the portable fallback, and sharpen the streaming axis with streaming drills →.

On this page

Why row pattern matching matters in 2026
MATCH_RECOGNIZE anatomy
Sessionization with MR
Funnel & conversion patterns
Anomaly detection & dialect matrix
Cheat sheet — MATCH_RECOGNIZE recipe list
Frequently asked questions
Practice on PipeCode

1. Why row pattern matching matters in 2026

The "regex over ordered rows" family — sessionization, funnels, anomalies, fraud bursts, and every other row-sequence question a senior interviewer might ask

The one-sentence invariant: a sql match_recognize question asks you to detect an ordered sequence of rows that satisfies a regex-like pattern inside a partition, and the canonical answer is a seven-clause skeleton where PATTERN is a regex over labelled row classes and DEFINE assigns predicates to those labels. Once you internalise that "PATTERN + DEFINE is the whole engine," the row-pattern interview surface collapses into a template you can write from muscle memory.

Five faces of the same problem.

Sessionization. Given an event stream, group consecutive events into sessions separated by a gap of inactivity — a pattern (A B*)+ where B continues while the inactivity gap is inside the threshold. Every product analytics platform under the sun — Google Analytics, Segment, Rudderstack, Snowplow, PostHog, Mixpanel — implements exactly this logic. The interviewer wants to see the MR version and the portable LAG + SUM version, and to hear you say which one you would ship on which warehouse.
Funnels. Given a table of user events, find users who went SIGNUP → VIEW_PRODUCT+ → ADD_TO_CART → CHECKOUT in that order. Growth teams live and die on funnel conversion — every product analytics dashboard does this, and every senior data engineer needs to write the query cold. MR reads like a regex and expresses the strict ordering trivially; the portable fallback needs a self-join per step.
Anomaly detection. Given a price series, find every V-shape reversal (N down candles followed by N up candles) or every W-shape double bottom. The classic Oracle MATCH_RECOGNIZE documentation example is exactly this shape — it's the query the ANSI SQL/2016 committee had in mind when they wrote the clause. Fraud detection uses PATTERN (SPEND{5,}) for spend bursts inside a time window.
CEP / streaming pattern detection. flink sql pattern and the Flink CEP library expose MATCH_RECOGNIZE as a streaming operator — the same pattern-detection semantics but with watermarks, event time, and low-latency emission. RisingWave ships the same clause with streaming semantics. Trino and Presto ship the batch variant.
Time-series segmentation. Given a sensor stream, find every run of contiguous HIGH readings followed by a LOW — a two-part MR pattern that reads more clearly than the equivalent LAG + CASE chain.

Two mental primitives.

PATTERN is a regex over row labels. The alphabet is user-defined row labels (A, B, C, or named labels like SIGNUP, VIEW, CART, CHECKOUT). Quantifiers are the standard regex set: * (zero or more), + (one or more), ? (zero or one), {n} (exactly n), {n,m} (n to m). Alternation is |. Grouping is parentheses.
DEFINE is a predicate per label. For each label used in PATTERN, DEFINE assigns a boolean predicate that the row must satisfy to be classified with that label. Predicates can reference PREV(col), NEXT(col), FIRST(col), LAST(col), and any other label's rows already matched — the state machine is aware of what came before.

Why senior interviewers love this family.

It tests window-function fluency plus regex thinking. Everyone can write ROW_NUMBER for a leaderboard; fewer can write PATTERN (STRT DOWN+ UP+) for a V-shape reversal. The interviewer probes whether you understand that pattern matching over rows is a distinct primitive from windowing over rows.
It has a portable answer and a native answer. The interviewer wants to see both: the LAG + SUM(new_group) OVER template from gaps-and-islands and the MATCH_RECOGNIZE version. Senior candidates know when to reach for which — MR on Oracle, Snowflake, Flink, Trino, RisingWave; portable everywhere else.
It's compositional. Once you can sessionize, funnels are the same skeleton with a longer PATTERN. Once you can do funnels, anomalies are a PATTERN with DEFINE predicates that reference PREV(col). Once you can do anomalies, streaming CEP is the same query with a watermark. A senior candidate walks through this progression without prompting.
It exposes partitioning and ordering discipline. Every answer must PARTITION BY user_id (or symbol, or device_id) and ORDER BY event_time. Candidates who forget the partition get wrong answers on multi-entity data — an instant red flag.

Where row-pattern queries land in your pipelines.

Product analytics. Every funnel report is a MR query underneath (or a chain of self-joins that could be one MR query). Every session is a PATTERN ((A B*)+) match. Every retention cohort is an anchored MR match against the first activity date.
Financial time series. V-shape reversals, W-shape double bottoms, breakout patterns, three-white-soldiers candlestick patterns — every classical technical-analysis pattern is a MR query. Fraud detection uses MR for burst detection, velocity checks, and sequence-of-events triggers.
Reliability / SRE. "N consecutive failed health-checks then an outage marker" is a MR pattern. Post-mortem timeline reconstruction is easier as MR than as a chain of window functions.
Streaming CEP. Flink SQL exposes MR as a streaming operator so an alert emits the moment the pattern completes on a live event stream. RisingWave ships the same primitive with streaming semantics. Real-time fraud, uptime monitoring, and order-flow anomaly detection all use this shape.
Data quality checks. "Assert that every START event is followed by a matching END within 5 minutes" is a MR query with a WITH UNMATCHED ROWS clause that exposes the violating rows.

What senior interviewers actually probe.

Anatomy first. Do you know the seven clauses? Can you name each one and say what it controls? PARTITION BY, ORDER BY, MEASURES, ONE ROW PER MATCH vs ALL ROWS PER MATCH, AFTER MATCH SKIP, PATTERN, DEFINE — say them in order without prompting and you're already in the top 20%.
PATTERN quantifiers. Do you know * vs + vs ? vs {n} vs {n,m} vs | vs grouping? Do you know the greedy vs reluctant distinction (+ vs +?)? Do you know PERMUTE(A, B, C) for unordered variants?
AFTER MATCH SKIP. Do you know SKIP TO NEXT ROW vs SKIP PAST LAST ROW vs SKIP TO FIRST <label> vs SKIP TO LAST <label>? The default is SKIP PAST LAST ROW on most engines; SKIP TO NEXT ROW lets matches overlap. Getting this wrong doubles or halves your output row count.
ONE ROW PER MATCH vs ALL ROWS PER MATCH. Do you know when to collapse a match into one summary row vs when to preserve the raw rows with a match id? Session summaries want ONE ROW; per-event enrichment wants ALL ROWS.
Dialect matrix. Can you name the engines that ship MR (Oracle, Snowflake, Flink SQL, Trino, Presto, RisingWave) and the ones that don't (Postgres, BigQuery, SQL Server, MySQL, Redshift as of 2026)? Do you default to the portable LAG + SUM pattern for a multi-warehouse codebase?

Worked example — the "consecutive down candles" anomaly starter

Detailed explanation. The canonical warm-up: given a prices(t, price) series, find every run of two or more consecutive price drops. This is the fastest way to check that a candidate recognises the MR family — the two-line PATTERN + DEFINE answer is a fingerprint on the whiteboard.

Question. Given prices(t, price) where t is a dense ascending integer and price is a positive integer, write an Oracle / Snowflake MR query that returns the start and end index of every run of two or more consecutive price drops.

Input.

t	price
1	100
2	98
3	95
4	97
5	90
6	88

Code.

SELECT *
FROM prices
MATCH_RECOGNIZE (
  ORDER BY t
  MEASURES
    FIRST(t)  AS run_start,
    LAST(t)   AS run_end,
    COUNT(*)  AS run_length
  ONE ROW PER MATCH
  AFTER MATCH SKIP PAST LAST ROW
  PATTERN (DOWN{2,})
  DEFINE
    DOWN AS price < PREV(price)
);

Step-by-step explanation.

ORDER BY t establishes the row order the pattern is matched against. Without it, the engine has no notion of "previous" or "next."
PATTERN (DOWN{2,}) — the regex says "match two or more consecutive rows classified as DOWN." The quantifier {2,} is "two or more" (regex convention).
DEFINE DOWN AS price < PREV(price) — a row is classified as DOWN only if its price is less than the previous row's price. PREV(col) reads the value from the previous row inside the current pattern match.
MEASURES FIRST(t) AS run_start, LAST(t) AS run_end, COUNT(*) AS run_length — for each match, emit the first row's t, the last row's t, and the count of matched rows.
AFTER MATCH SKIP PAST LAST ROW — after a match ends, resume matching at the row after the last matched row. This prevents overlapping matches — the default and the right choice for anomaly runs.

Output.

run_start	run_end	run_length
2	3	2
5	6	2

Rule of thumb. The consecutive-down-candles pattern is the MR "hello world" — recognise the shape in 5 seconds, write PATTERN + DEFINE in 30, and move on to the harder V-shape / W-shape patterns the interviewer really wants.

Worked example — the LAG + SUM portable equivalent

Detailed explanation. The senior follow-up: "write the same query without MATCH_RECOGNIZE." This is the portable answer that ships on Postgres, BigQuery, SQL Server, MySQL, Redshift, and every warehouse that doesn't ship MR. The equivalence check is a real interview probe — the interviewer wants to see that you can translate between the two.

Question. Rewrite the consecutive-down-candles query using LAG and cumulative SUM only.

Input. (Same as above.)

Code.

WITH flagged AS (
  SELECT
    t,
    price,
    CASE
      WHEN price < LAG(price) OVER (ORDER BY t) THEN 1
      ELSE 0
    END AS is_down,
    CASE
      WHEN (price < LAG(price) OVER (ORDER BY t))
        <> COALESCE(LAG(price) OVER (ORDER BY t) < LAG(price, 2) OVER (ORDER BY t), FALSE)
        THEN 1
      ELSE 0
    END AS is_new_run
  FROM prices
),
grouped AS (
  SELECT
    t,
    price,
    is_down,
    SUM(is_new_run) OVER (ORDER BY t) AS run_id
  FROM flagged
  WHERE is_down = 1
)
SELECT
  MIN(t)   AS run_start,
  MAX(t)   AS run_end,
  COUNT(*) AS run_length
FROM grouped
GROUP BY run_id
HAVING COUNT(*) >= 2
ORDER BY run_start;

Step-by-step explanation.

flagged CTE — mark every row where the price is less than the previous row's price as is_down = 1. The is_new_run flag fires on a DOWN row whose predecessor was not DOWN, so the cumulative sum starts a new run.
grouped CTE — keep only DOWN rows and cumulative-sum is_new_run into run_id. Rows inside the same run share the same id.
Final SELECT — group by run_id and aggregate. Filter to HAVING COUNT(*) >= 2 to enforce the "two or more" clause the PATTERN quantifier {2,} gave us for free.
This portable version is 15+ lines vs 12 lines for the MR version. On complex patterns the gap widens dramatically — a "buy then three drops then a stop-loss" pattern is 40+ lines portable vs 20 lines MR.
Rule of thumb: for portable code, the LAG + SUM template works. For readability on Oracle / Snowflake / Flink / Trino / RisingWave, MR wins as soon as the pattern is more than one row class.

Output. (Same as above — same rows, different plan.)

Rule of thumb. Every MR pattern has a LAG + SUM portable equivalent — the translation is mechanical for simple patterns and painful for regex-heavy ones. Portability first; MR when the pattern justifies it.

Worked example — engine dialect quickstart

Detailed explanation. A quick sanity check the interviewer might drop: "Which of these engines can I run MATCH_RECOGNIZE on today?" The answer is a short list — knowing it cold is a senior signal.

Question. Given a list of warehouses / engines, mark each as MR-capable or portable-fallback-only in 2026.

Input. Oracle 12c+, Snowflake, Flink SQL, Trino, Presto, RisingWave, Postgres 16, BigQuery, SQL Server 2022, MySQL 8, Redshift, Databricks SQL.

Code. (No code — a dialect matrix answer.)

Engine              | MATCH_RECOGNIZE support (as of 2026)
Oracle 12c+         | YES — full ANSI, shipped 2013 (first in the market)
Snowflake           | YES — full ANSI, GA since 2021
Flink SQL           | YES — streaming CEP semantics
Trino / Presto      | YES — batch, since Trino 361
RisingWave          | YES — streaming, since 1.7
Postgres 16         | NO — no planned support; use LAG + SUM
BigQuery            | NO — no planned support; use LAG + SUM
SQL Server 2022     | NO — no planned support; use LAG + SUM
MySQL 8             | NO — no planned support; use LAG + SUM
Redshift            | NO — no planned support; use LAG + SUM
Databricks SQL      | NO (as of 2026) — use LAG + SUM

Step-by-step explanation.

Oracle was first — the ANSI SQL/2016 committee based the standard on Oracle's original implementation from 12c.
Snowflake followed in 2021 GA — the batch semantics match Oracle's exactly.
Flink SQL and RisingWave ship the streaming variant — watermark-driven pattern completion with low-latency emission.
Trino / Presto ship the batch variant — most cloud data lakehouses (Athena, Dremio, Ahana) that build on Trino inherit MR support.
Postgres, BigQuery, SQL Server, MySQL, Redshift, and Databricks SQL do not ship MR in 2026; portable LAG + SUM is the fallback. If your codebase is multi-warehouse, default to portable.

Output. The matrix above.

Rule of thumb. Memorise the five-yes / six-no list. When an interviewer asks "would you use MATCH_RECOGNIZE here?" the first question back is "what's the warehouse?" — say it out loud and you signal senior instincts.

Senior interview question on the row-pattern mental model

A senior interviewer often opens with: "I hand you an event stream and ask for a metric — pick sessionization, funnel conversion, or anomaly detection — that requires you to detect an ordered row pattern. Walk me through the seven-clause skeleton, and tell me what changes across the three asks."

Solution Using the seven-clause MATCH_RECOGNIZE skeleton

-- Universal MATCH_RECOGNIZE skeleton — seven clauses, one regex over rows
SELECT <select-list>
FROM <input-table>
MATCH_RECOGNIZE (
  PARTITION BY <entity-col>                          -- Slot 1: per-entity scope
  ORDER BY <ordering-col>                            -- Slot 2: row order
  MEASURES                                           -- Slot 3: per-match columns
    FIRST(<label>.<col>) AS <alias>,
    LAST (<label>.<col>) AS <alias>,
    COUNT(*)             AS <alias>,
    CLASSIFIER()         AS matched_label
  ONE ROW PER MATCH                                  -- Slot 4: output shape
  AFTER MATCH SKIP PAST LAST ROW                     -- Slot 5: overlap policy
  PATTERN (<A> <B>+ <C>?)                            -- Slot 6: regex over labels
  DEFINE                                             -- Slot 7: label predicates
    <A> AS <predicate on A>,
    <B> AS <predicate on B, may reference PREV/NEXT>,
    <C> AS <predicate on C>
);

Step-by-step trace.

Ask	PARTITION BY	PATTERN	Key DEFINE predicate	Output shape
Sessionization	user_id	`((A B*)+)`	`B AS ts - LAG(B.ts) <= INTERVAL '30 minutes'`	ONE ROW PER MATCH → session summary
Funnel	user_id	`(SIGNUP VIEW+ CART CHECKOUT)`	`VIEW AS event_type = 'view_product'`	ONE ROW PER MATCH → converted user + timings
V-shape anomaly	symbol	`(STRT DOWN+ UP+)`	`DOWN AS price < PREV(price)`, `UP AS price > PREV(price)`	ONE ROW PER MATCH → reversal boundaries
W-shape anomaly	symbol	`(DOWN+ UP+ DOWN+ UP+)`	Same as V-shape	ONE ROW PER MATCH → double-bottom boundaries
Fraud burst	user_id	`(SPEND{5,})`	`SPEND AS ts - FIRST(SPEND.ts) <= INTERVAL '60' SECOND`	ONE ROW PER MATCH → burst boundaries
Per-event enrichment	user_id	Any	Same as above	ALL ROWS PER MATCH → per-row classifier

The universal skeleton turns "which query do I write?" from a taste question into a mechanical checklist. Pick the partition, pick the ordering, pick the PATTERN, write the DEFINE predicates — the whole query falls out.

Output:

Ask	Approx query length (MR)	Approx query length (portable LAG + SUM)
Sessionization	~15 lines	~20 lines
Funnel	~15 lines	~40 lines (4 self-joins)
V-shape anomaly	~15 lines	~25 lines
W-shape anomaly	~15 lines	~40+ lines
Fraud burst	~15 lines	~30 lines

Why this works — concept by concept:

PATTERN is a regex over row labels — every MR answer starts with a regex over labels drawn from an alphabet you define. The moment you can express the ask as a regex, PATTERN writes itself. A B+ C is the archetype; every variant is a quantifier tweak.
DEFINE is a predicate per label — for each label used in PATTERN, DEFINE gives the boolean predicate that classifies a row into that label. Predicates can reference PREV, NEXT, FIRST, LAST of the current or previous labels — the state machine is aware of history.
PARTITION BY entity is non-negotiable — multi-entity data always requires per-entity partitioning. Skipping it produces a global pattern that mixes users / symbols / devices, which is a wrong-answer buffer that the interviewer catches immediately.
ORDER BY inside the MR block matters — the ordering determines what "previous" and "next" mean. Sessionization orders by event_time; anomaly detection orders by ts; funnels order by event_time. Get the ordering wrong and PREV / NEXT lie.
Cost — MR compiles to a state-machine scan over each partition, cost O(N) per partition after the sort. The dominant cost is the ORDER BY sort — O(N log N) per partition. On modern engines the state-machine step is streamed row-by-row inside the sort output, so wall clock is essentially the sort cost. Portable LAG + SUM is the same cost class; MR just reads better for regex-shaped patterns.

SQL
Topic — pattern matching
Pattern matching problems

Practice →

SQL Topic — gaps and islands Gaps and islands drills

Practice →

2. MATCH_RECOGNIZE anatomy

The seven-clause skeleton — PARTITION BY, ORDER BY, MEASURES, ONE ROW / ALL ROWS PER MATCH, AFTER MATCH SKIP, PATTERN, DEFINE

The mental model in one line: MATCH_RECOGNIZE is a regex engine that runs over ordered rows inside a partition, where PATTERN is the regex, DEFINE is the alphabet's semantics, and MEASURES + ONE ROW / ALL ROWS PER MATCH shape the output. Once you say "PATTERN is regex, DEFINE is semantics, MEASURES is payload" out loud, the anatomy interview surface collapses to a slot-filling exercise.

Slot 1 — PARTITION BY <cols>.

Scopes the pattern engine per entity. Every partition is matched independently — one match per user, per symbol, per device, per session.
Optional but almost always present. Omitting it runs the pattern over the whole table as one partition — right only for single-entity data (a single sensor stream, a single stock price series).
Multiple columns supported — PARTITION BY user_id, device_id scopes per (user, device) pair.

Slot 2 — ORDER BY <cols>.

Required. Establishes the row order the pattern is matched against. PREV and NEXT are defined relative to this ordering.
Almost always event_time or a monotonically-increasing sequence. Ties are broken by additional columns — ORDER BY event_time, event_id is the defensive form.
Skipping ORDER BY inside MR raises a compilation error on every engine.

Slot 3 — MEASURES.

Declares the per-match output columns. Each measure is an expression evaluated at match time, using FIRST(label.col), LAST(label.col), COUNT(*), aggregations, and any expression that references the pattern's labels.
Standard measure functions:
- FIRST(A.col) — value of col from the first A-labelled row in the match.
- LAST(A.col) — value of col from the last A-labelled row.
- COUNT(A.*) — count of A-labelled rows.
- SUM(A.col), AVG(A.col), MIN(A.col), MAX(A.col) — aggregate over A-labelled rows.
- CLASSIFIER() — returns the label a specific row was classified into (used with ALL ROWS PER MATCH).
- MATCH_NUMBER() — a monotonically-increasing per-partition match id.

Slot 4 — ONE ROW PER MATCH vs ALL ROWS PER MATCH.

ONE ROW PER MATCH — one output row per matched pattern. The output row's values come from the MEASURES expressions. Right for session summaries, funnel outcomes, anomaly boundaries.
ALL ROWS PER MATCH — one output row per row inside the match. Each row carries the match id (MATCH_NUMBER()), its classifier (CLASSIFIER()), and any MEASURES you declared. Right for per-event enrichment inside a session, funnel step attribution.
Variants — ALL ROWS PER MATCH SHOW EMPTY MATCHES (include zero-length matches), ALL ROWS PER MATCH OMIT EMPTY MATCHES (skip them), ALL ROWS PER MATCH WITH UNMATCHED ROWS (also emit rows that never matched any pattern — key for drop-off attribution).

Slot 5 — AFTER MATCH SKIP ….

Controls where the engine resumes matching after a successful match. Four canonical options:
- AFTER MATCH SKIP PAST LAST ROW — resume at the row after the last matched row. No overlap. Default on Oracle / Snowflake / Trino / Flink.
- AFTER MATCH SKIP TO NEXT ROW — resume at the row after the first matched row. Allows overlap.
- AFTER MATCH SKIP TO FIRST <label> — resume at the first row labelled <label> in the match.
- AFTER MATCH SKIP TO LAST <label> — resume at the last row labelled <label> in the match.
Getting this wrong doubles or halves your output row count. For anomaly runs and funnel completions, SKIP PAST LAST ROW is right. For "count every possible occurrence" analytics, SKIP TO NEXT ROW is right.

Slot 6 — PATTERN (...).

The regex over row labels. Syntax:
- Concatenation — A B C matches an A row followed by a B row followed by a C row.
- * — zero or more.
- + — one or more.
- ? — zero or one.
- {n} — exactly n.
- {n,m} — n to m (inclusive).
- | — alternation (A | B matches either).
- Parentheses — grouping ((A B)+ matches one or more A B pairs).
- Reluctant quantifiers — +?, *?, ?? — match as few rows as possible (greedy is default).
- Anchors — ^ (start of partition), $ (end of partition) on engines that support them.
- PERMUTE(A, B, C) — match A, B, C in any order (Oracle / Trino / Flink).
Reads like a regex over the row alphabet — once you can write regex, PATTERN writes itself.

Slot 7 — DEFINE <label> AS <predicate>.

Assigns a boolean predicate to each PATTERN label. A row is classified into a label only if it satisfies the label's predicate at that step of the state machine.
If a label is used in PATTERN but not defined, it defaults to TRUE (matches any row). PATTERN (A B*) DEFINE B AS ... — A matches anything.
Predicates can reference:
- The current row's columns — DEFINE UP AS price > 100.
- The previous row inside the match — DEFINE UP AS price > PREV(price).
- The first row of the match — DEFINE B AS ts - FIRST(A.ts) <= INTERVAL '30 minutes'.
- The last row of a specific label — DEFINE C AS price > LAST(B.price).
- Aggregates over previously matched rows — DEFINE C AS SUM(A.qty) + SUM(B.qty) > 100.

Reading order for a MR query.

What's the partition? → PARTITION BY.
What's the row order? → ORDER BY.
What's the pattern I'm looking for? → PATTERN.
What predicates define each label? → DEFINE.
What do I emit per match? → MEASURES.
One row per match or all rows? → ONE ROW / ALL ROWS PER MATCH.
Where do I resume after a match? → AFTER MATCH SKIP.

Every senior candidate walks through the seven slots in this order without prompting.

Worked example — consecutive-price-drop with ONE ROW PER MATCH

Detailed explanation. The classic anatomy warm-up: use ONE ROW PER MATCH to emit one summary row per run of consecutive price drops. This is where every candidate should start when learning MR.

Question. Given prices(t, price) ordered by t, use MATCH_RECOGNIZE to emit one row per run of two or more consecutive price drops, with the run's start ts, end ts, count, and first / last prices.

Input.

t	price
1	100
2	98
3	95
4	93
5	97
6	90

Code.

SELECT *
FROM prices
MATCH_RECOGNIZE (
  ORDER BY t
  MEASURES
    FIRST(t)     AS run_start,
    LAST(t)      AS run_end,
    FIRST(price) AS start_price,
    LAST(price)  AS end_price,
    COUNT(*)     AS run_length
  ONE ROW PER MATCH
  AFTER MATCH SKIP PAST LAST ROW
  PATTERN (DOWN{2,})
  DEFINE
    DOWN AS price < PREV(price)
);

Step-by-step explanation.

ORDER BY t — the state machine walks rows in t order. PREV(price) refers to the immediately preceding row inside the match.
PATTERN (DOWN{2,}) — regex says "two or more DOWN rows in a row." Matches greedily by default — as many DOWN rows as possible.
DEFINE DOWN AS price < PREV(price) — a row is DOWN only if its price is less than the previous row's. PREV(price) on the first row of a partition is NULL — the predicate evaluates to NULL, treated as FALSE, so no match starts on row 1.
MEASURES — FIRST(t) and LAST(t) are the boundary timestamps; FIRST(price) and LAST(price) are the boundary prices; COUNT(*) is the run length.
AFTER MATCH SKIP PAST LAST ROW — after emitting a match ending at row k, resume at row k+1. No overlapping runs.

Output.

run_start	run_end	start_price	end_price	run_length
2	4	98	93	3
6	6	90	90	0

Wait — the second row is wrong because row 6 alone can't satisfy DOWN{2,}. Correct output:

run_start	run_end	start_price	end_price	run_length
2	4	98	93	3

Rule of thumb. ONE ROW PER MATCH collapses each pattern match into one summary row. Perfect for "how many runs did I have?" reporting. When you need per-row enrichment, switch to ALL ROWS PER MATCH.

Worked example — same pattern with ALL ROWS PER MATCH + CLASSIFIER

Detailed explanation. The follow-up variant: preserve every row inside the match and label each with its classifier. Useful for per-event enrichment — you keep every input row and enrich it with match id and label.

Question. Rewrite the consecutive-price-drop query to preserve every row and label each row with its match id and classifier.

Input. (Same as above.)

Code.

SELECT *
FROM prices
MATCH_RECOGNIZE (
  ORDER BY t
  MEASURES
    MATCH_NUMBER() AS match_id,
    CLASSIFIER()   AS matched_as
  ALL ROWS PER MATCH
  AFTER MATCH SKIP PAST LAST ROW
  PATTERN (DOWN{2,})
  DEFINE
    DOWN AS price < PREV(price)
);

Step-by-step explanation.

ALL ROWS PER MATCH — instead of one summary row per match, emit every input row that was part of a match.
MATCH_NUMBER() — a monotonically-increasing per-partition id. Rows from match 1 have match_id = 1; rows from match 2 have match_id = 2.
CLASSIFIER() — returns the label the row was classified into. In this query it's always DOWN (the only label).
Rows that didn't match any pattern are not emitted with ALL ROWS PER MATCH. To emit them too, use ALL ROWS PER MATCH WITH UNMATCHED ROWS.
Use this variant when downstream code needs the original rows enriched with match metadata — e.g., a Sankey diagram of the funnel or a per-event timeline replay.

Output.

t	price	match_id	matched_as
2	98	1	DOWN
3	95	1	DOWN
4	93	1	DOWN

Rule of thumb. ONE ROW PER MATCH for summary reporting; ALL ROWS PER MATCH for per-event enrichment. Match id + classifier are the standard columns you'd project.

Worked example — AFTER MATCH SKIP TO NEXT ROW for overlapping matches

Detailed explanation. The nuanced follow-up: sometimes you want every possible pattern occurrence, including overlapping ones. Switching AFTER MATCH SKIP PAST LAST ROW to AFTER MATCH SKIP TO NEXT ROW changes the output row count dramatically.

Question. Given prices(t, price), count every pair of consecutive drops — even when one pair's second drop is the next pair's first drop. Use AFTER MATCH SKIP TO NEXT ROW and PATTERN (DOWN DOWN).

Input.

t	price
1	100
2	98
3	95
4	90

Code.

SELECT *
FROM prices
MATCH_RECOGNIZE (
  ORDER BY t
  MEASURES
    FIRST(t) AS pair_start,
    LAST(t)  AS pair_end
  ONE ROW PER MATCH
  AFTER MATCH SKIP TO NEXT ROW
  PATTERN (DOWN DOWN)
  DEFINE
    DOWN AS price < PREV(price)
);

Step-by-step explanation.

PATTERN (DOWN DOWN) — match exactly two consecutive DOWN rows.
AFTER MATCH SKIP TO NEXT ROW — after a match starting at row k and ending at k+1, resume matching at row k+1 (the middle row), not k+2. This lets the next match start at what was the previous match's second row.
Row 2 → 3 forms one match (DOWN at 2, DOWN at 3). Engine resumes at row 3.
Row 3 → 4 forms another match (DOWN at 3, DOWN at 4). Engine resumes at row 4.
Result: two overlapping matches instead of one. With SKIP PAST LAST ROW, we'd only get the first match — the engine would resume at row 4 after emitting the 2 → 3 match.

Output.

pair_start	pair_end
2	3
3	4

Rule of thumb. SKIP PAST LAST ROW for non-overlapping runs (sessions, anomalies, funnels). SKIP TO NEXT ROW for "every possible occurrence" analytics (candlestick pair counting, sliding-window pattern probes). Choose deliberately.

Senior interview question on anatomy fluency

A senior interviewer might ask: "Walk me through a MATCH_RECOGNIZE query that finds every user's first session of length ≥ 3 events, ordered by event time. I want to see the seven clauses filled in with the right choices — including ONE ROW vs ALL ROWS, AFTER MATCH SKIP semantics, and the PATTERN quantifier."

Solution Using the seven-clause skeleton with first-match selection

SELECT *
FROM events
MATCH_RECOGNIZE (
  PARTITION BY user_id
  ORDER BY event_time
  MEASURES
    MATCH_NUMBER()      AS match_id,
    FIRST(A.event_time) AS session_start,
    LAST(B.event_time)  AS session_end,
    COUNT(*)            AS event_count
  ONE ROW PER MATCH
  AFTER MATCH SKIP PAST LAST ROW
  PATTERN (A B{2,})
  DEFINE
    A AS TRUE,
    B AS event_time - LAST(A.event_time) <= INTERVAL '30' MINUTE
       AND event_time - LAST(B.event_time, 1) <= INTERVAL '30' MINUTE
)
QUALIFY ROW_NUMBER() OVER (PARTITION BY user_id ORDER BY session_start) = 1;

Step-by-step trace.

Slot	Choice	Why
PARTITION BY	user_id	Per-user pattern scope
ORDER BY	event_time	Pattern runs in time order
MEASURES	match_id, session_start, session_end, event_count	Session summary payload
Output shape	ONE ROW PER MATCH	Session summary, not per-event enrichment
AFTER MATCH SKIP	SKIP PAST LAST ROW	No overlapping sessions
PATTERN	`(A B{2,})`	Anchor A + 2 or more continuation rows
DEFINE A	TRUE	Any row can start a session
DEFINE B	Within 30 min of prior row	Sessionization inactivity threshold
Post-filter	QUALIFY ROW_NUMBER() = 1	First session per user

The MR block finds every session of length ≥ 3 per user; the QUALIFY picks the earliest one per user. This is the sessionization + first-match composition pattern.

Output:

user_id	match_id	session_start	session_end	event_count
u1	1	2026-07-01 10:00	2026-07-01 10:20	3
u2	1	2026-07-02 09:00	2026-07-02 09:10	4

Why this works — concept by concept:

PATTERN (A B{2,}) encodes "anchor plus 2+ continuation" — the anchor A matches any row (DEFINE A AS TRUE); the B{2,} quantifier says "two or more continuation rows." Together, the pattern matches sessions of length ≥ 3 (one A + two or more B).
DEFINE B references LAST(A.event_time) and LAST(B.event_time, 1) — the predicate says "this B row is within 30 minutes of the previous B row (or the A row if this is the first B)." LAST(B.event_time, 1) is the "one B ago" — the immediately preceding B row inside the match.
QUALIFY for first-per-user selection — QUALIFY ROW_NUMBER() OVER (PARTITION BY user_id ORDER BY session_start) = 1 picks the earliest session per user. QUALIFY is the Snowflake / BigQuery shorthand; on Oracle / Trino wrap it in a subquery with WHERE rn = 1.
MATCH_NUMBER() in MEASURES — the per-partition match id is useful downstream for debugging and for joining back to the raw rows. Even when you don't display it, projecting it costs nothing.
Cost — one MR pass with a state-machine scan + one QUALIFY window pass. On a 1B-row event table with 10M users, the MR pass is O(N log N) for the sort + O(N) for the state machine; the QUALIFY is O(sessions). Fast on modern warehouses — single-digit minutes on a medium Snowflake warehouse.

SQL
Topic — pattern matching
Pattern matching drills

Practice →

SQL Topic — regular expressions Regular expression problems

Practice →

3. Sessionization with MR

`sql pattern detection` for event streams — the `PATTERN ((A B*)+)` sessionization primitive with per-user partitioning and inactivity DEFINE

The mental model in one line: sessionization with MATCH_RECOGNIZE reads "anchor A, then zero-or-more continuation rows B where B is within 30 minutes of the previous B or the anchor A" — expressed as PATTERN ((A B*)+) with a DEFINE predicate on B for the inactivity threshold. Once you say "A is any row, B stays inside the threshold, and (A B*)+ produces one match per session," the MR sessionization interview surface reduces to a two-line PATTERN + DEFINE recipe.

The canonical MR sessionization contract.

Definition. A session is a maximal run of events from one user where consecutive events are separated by no more than the inactivity threshold. PATTERN ((A B*)+) expresses this as "anchor + zero or more continuations, repeated one or more times." Each MR match is one session.
Anchor semantics. A matches any row; it's the session anchor. DEFINE A AS TRUE is the standard.
Continuation semantics. B matches a row within the inactivity threshold of the previous B row (or the anchor if this is the first B). DEFINE B AS event_time - LAST(A.event_time) <= INTERVAL '30' MINUTE AND event_time - LAST(B.event_time, 1) <= INTERVAL '30' MINUTE. On simple engines the second clause reduces to event_time - PREV(event_time) <= INTERVAL '30' MINUTE.
Match maximality. MR is greedy by default — it matches as many rows as possible. Sessions naturally end when the next row exceeds the threshold, and the match terminates at the last row that stayed inside.

Two equivalent PATTERNs.

PATTERN ((A B*)+) — nested pattern: one or more (A + zero-or-more B) blocks. Rarely used in this form; the outer + is redundant because AFTER MATCH SKIP handles the "next session" transition.
PATTERN (A B*) — the canonical form. One anchor A + zero or more continuations B. Each match is one session; AFTER MATCH SKIP PAST LAST ROW starts the next match at the next row.

Choosing the threshold.

30 minutes. The Google Analytics / Segment / Rudderstack / Snowplow / PostHog / Mixpanel default. Right for consumer web / mobile.
60 minutes. Right for long-attention apps — video streaming, IDEs, coding editors.
10 minutes. Right for high-frequency contexts — IoT dashboards, trading terminals.
Custom. Plot inter-event gap histograms; pick the elbow at the 95th percentile.

Per-user partitioning is non-negotiable.

PARTITION BY user_id inside the MR block is mandatory for multi-user data. Without it, the pattern engine matches across user boundaries — a session from u1 can absorb u2's next event if their timestamps are close.
Multiple partition columns supported — PARTITION BY user_id, device_id for per-(user, device) sessions.

Watermark ceiling for streaming.

On Flink SQL / RisingWave, sessionization runs in streaming mode with a watermark. The MR block can only close a session when the watermark passes the last event's timestamp + the inactivity threshold — otherwise a late event could still extend the session.
Batch MR doesn't need a watermark — the whole partition is available.

Session-level aggregations via MEASURES.

FIRST(event_time) — session start.
LAST(event_time) — session end.
LAST(event_time) - FIRST(event_time) — session duration.
COUNT(*) — event count.
COUNT(A.*) — anchor count (should always be 1 for PATTERN (A B*)).
COUNT(B.*) — continuation count.
FIRST(page_url) — landing page.
LAST(page_url) — exit page.

Common MR sessionization interview probes.

"Why PATTERN (A B*) and not (A+)?" — A+ uses one label with the same DEFINE for both anchor and continuation; the DEFINE would need to handle the "first row" case. A B* cleanly separates anchor semantics from continuation semantics.
"How do you handle the first row?" — A matches any row (DEFINE A AS TRUE), so the first row of the partition is always the anchor of the first match.
"What if a user has no events?" — no rows in the partition, no matches, no output. Fine.
"How do you re-sessionize when late data arrives?" — recompute the MR query over a rolling window. The pattern engine is deterministic given the input rows.
"MR sessionization vs LAG + SUM — which do you ship?" — MR on Oracle / Snowflake / Flink / Trino / RisingWave for readability; portable LAG + SUM for multi-warehouse dbt projects.

Worked example — 30-minute sessionization on Snowflake

Detailed explanation. The canonical Snowflake sessionization ask: given events(user_id, event_time, event_name, page_url), produce session summaries with MR. This is the query every senior candidate should be able to write from memory on Snowflake.

Question. Given events(user_id, event_time, event_name, page_url) with a 30-minute inactivity threshold, use MATCH_RECOGNIZE to produce (user_id, session_id, session_start, session_end, event_count, landing_page, exit_page).

Input.

user_id	event_time	event_name	page_url
u1	2026-07-01 10:00	page_view	/home
u1	2026-07-01 10:05	click	/home
u1	2026-07-01 10:20	page_view	/pricing
u1	2026-07-01 11:15	page_view	/home
u1	2026-07-01 11:20	click	/home

Code.

SELECT *
FROM events
MATCH_RECOGNIZE (
  PARTITION BY user_id
  ORDER BY event_time
  MEASURES
    MATCH_NUMBER()     AS session_id,
    FIRST(event_time)  AS session_start,
    LAST(event_time)   AS session_end,
    COUNT(*)           AS event_count,
    FIRST(page_url)    AS landing_page,
    LAST(page_url)     AS exit_page
  ONE ROW PER MATCH
  AFTER MATCH SKIP PAST LAST ROW
  PATTERN (A B*)
  DEFINE
    B AS DATEDIFF('minute', LAG(event_time), event_time) <= 30
);

Step-by-step explanation.

PARTITION BY user_id — per-user sessionization. Each user is independent.
ORDER BY event_time — rows walked in time order.
PATTERN (A B*) — one anchor A + zero or more continuations B. Each match is one session.
DEFINE B AS DATEDIFF('minute', LAG(event_time), event_time) <= 30 — a continuation row is one within 30 minutes of the previous row (which may be A on the first B, or the previous B). Snowflake's LAG inside DEFINE reads the previous row's value in the current match.
MEASURES — MATCH_NUMBER for session_id, FIRST/LAST for boundaries, COUNT for event count, FIRST/LAST of page_url for landing / exit page. AFTER MATCH SKIP PAST LAST ROW starts the next match right after the last matched row.

Output.

user_id	session_id	session_start	session_end	event_count	landing_page	exit_page
u1	1	2026-07-01 10:00	2026-07-01 10:20	3	/home	/pricing
u1	2	2026-07-01 11:15	2026-07-01 11:20	2	/home	/home

Rule of thumb. MR sessionization on Snowflake is one query — PATTERN (A B*), DEFINE B on inactivity, MEASURES for the boundaries. Reads more clearly than the LAG + CASE + SUM equivalent; ships faster if your warehouse supports it.

Worked example — the LAG + SUM portable equivalent

Detailed explanation. For a multi-warehouse project, ship the portable LAG + SUM version. This is the "same query, no MR" answer for Postgres, BigQuery, SQL Server, MySQL, Redshift, Databricks.

Question. Rewrite the sessionization query without MATCH_RECOGNIZE.

Input. (Same as above.)

Code.

WITH flagged AS (
  SELECT
    user_id,
    event_time,
    event_name,
    page_url,
    CASE
      WHEN event_time - LAG(event_time) OVER (PARTITION BY user_id ORDER BY event_time)
             > INTERVAL '30 minutes'
        OR LAG(event_time) OVER (PARTITION BY user_id ORDER BY event_time) IS NULL
        THEN 1
      ELSE 0
    END AS new_session
  FROM events
),
sessionized AS (
  SELECT
    user_id,
    event_time,
    event_name,
    page_url,
    SUM(new_session) OVER (PARTITION BY user_id ORDER BY event_time) AS session_id
  FROM flagged
)
SELECT
  user_id,
  session_id,
  MIN(event_time) AS session_start,
  MAX(event_time) AS session_end,
  COUNT(*)        AS event_count,
  MIN(page_url) FILTER (WHERE event_time = MIN(event_time) OVER (PARTITION BY user_id, session_id)) AS landing_page,
  MAX(page_url) FILTER (WHERE event_time = MAX(event_time) OVER (PARTITION BY user_id, session_id)) AS exit_page
FROM sessionized
GROUP BY user_id, session_id
ORDER BY user_id, session_id;

Step-by-step explanation.

flagged CTE — flag new_session = 1 when the inter-event gap exceeds 30 minutes, or when there's no previous event.
sessionized CTE — cumulative-sum the flag per user to produce a per-user session id.
Final SELECT — group by (user_id, session_id) and aggregate to session summary. The FILTER clauses for landing / exit page pick the URL of the first / last event in the session.
Landing / exit page trick — using a window function inside FILTER makes the aggregate pick the URL from the boundary row. Simpler alternative on Postgres 16+: (ARRAY_AGG(page_url ORDER BY event_time))[1] for the first, (ARRAY_AGG(page_url ORDER BY event_time DESC))[1] for the last.
Runtime — three window passes + one aggregate. Same cost class as the MR version; the query is longer but portable to every warehouse.

Output. (Same as above.)

Rule of thumb. MR is shorter; portable is longer but universally supported. For dbt projects that run on multiple warehouses, ship portable. For Snowflake-only projects, ship MR.

Worked example — MR sessionization with explicit sign-out

Detailed explanation. The nuanced follow-up: some products close sessions on explicit user actions (sign-out, tab-close). Extend the PATTERN with a terminator label C that closes the session when a specific event fires.

Question. Given events(user_id, event_time, event_name), treat event_name = 'sign_out' as an explicit session terminator. Use MR to produce session summaries.

Input.

user_id	event_time	event_name
u1	10:00	page_view
u1	10:05	sign_out
u1	10:07	page_view
u1	10:10	click

Code.

SELECT *
FROM events
MATCH_RECOGNIZE (
  PARTITION BY user_id
  ORDER BY event_time
  MEASURES
    MATCH_NUMBER()    AS session_id,
    FIRST(event_time) AS session_start,
    LAST(event_time)  AS session_end,
    COUNT(*)          AS event_count
  ONE ROW PER MATCH
  AFTER MATCH SKIP PAST LAST ROW
  PATTERN (A B* C?)
  DEFINE
    B AS DATEDIFF('minute', LAG(event_time), event_time) <= 30
       AND event_name != 'sign_out',
    C AS event_name = 'sign_out'
);

Step-by-step explanation.

PATTERN (A B* C?) — anchor A + zero-or-more continuations B + optional terminator C. The terminator is a sign-out event.
DEFINE B AS ... AND event_name != 'sign_out' — B continuations exclude sign-out events. The moment we hit a sign-out, B stops matching.
DEFINE C AS event_name = 'sign_out' — C matches only the sign-out event. C? makes it optional — sessions without a sign-out still match.
After a session with a sign-out C, AFTER MATCH SKIP PAST LAST ROW moves past the sign-out to the next event. That next event becomes the anchor of the following session.
Compared to the "LAG(event_name) = 'sign_out'" trick in the portable version, MR expresses the terminator cleanly with a labelled row class. Much more readable.

Output.

user_id	session_id	session_start	session_end	event_count
u1	1	10:00	10:05	2
u1	2	10:07	10:10	2

Rule of thumb. For any product with explicit session-close semantics, add a terminator label to PATTERN with ? quantifier and a DEFINE predicate matching the closing event. The rest of the pipeline stays the same.

Senior interview question on MR sessionization at scale

A senior interviewer might ask: "You run a product with 500M daily events and 50M users on Snowflake. Design the nightly sessionization pipeline using MATCH_RECOGNIZE, compute per-session aggregates, and forward-fill the last non-null utm_campaign across sessions per user for last-touch attribution. Walk me through the query and the plan cost."

Solution Using MR sessionization + per-session aggregate + forward-fill

-- Stage 1 — MR sessionize
WITH sessionized AS (
  SELECT *
  FROM events
  MATCH_RECOGNIZE (
    PARTITION BY user_id
    ORDER BY event_time
    MEASURES
      MATCH_NUMBER() AS session_id
    ALL ROWS PER MATCH
    AFTER MATCH SKIP PAST LAST ROW
    PATTERN (A B*)
    DEFINE
      B AS DATEDIFF('minute', LAG(event_time), event_time) <= 30
  )
),
-- Stage 2 — per-session aggregate + last non-null campaign
per_session AS (
  SELECT
    user_id,
    session_id,
    MIN(event_time) AS session_start,
    MAX(event_time) AS session_end,
    COUNT(*)        AS event_count,
    (ARRAY_AGG(utm_campaign ORDER BY event_time DESC)
       FILTER (WHERE utm_campaign IS NOT NULL))[1] AS last_touch_campaign
  FROM sessionized
  GROUP BY user_id, session_id
),
-- Stage 3 — forward-fill across sessions
per_user_last_touch AS (
  SELECT
    user_id,
    session_id,
    session_start,
    session_end,
    event_count,
    last_touch_campaign,
    LAST_VALUE(last_touch_campaign IGNORE NULLS) OVER (
      PARTITION BY user_id
      ORDER BY session_start
      ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW
    ) AS user_last_touch_campaign
  FROM per_session
)
SELECT * FROM per_user_last_touch
ORDER BY user_id, session_start;

Step-by-step trace.

Stage	What runs	Approx rows out on 500M input
1	MR sessionization (ALL ROWS PER MATCH)	500M rows enriched with session_id
2	Per-session aggregate	~100M sessions (assuming 5 events/session avg)
3	Forward-fill last_touch	100M rows with per-user attribution

The three-stage MR + aggregate + forward-fill pipeline sessionizes the stream, aggregates each session to a summary row, then propagates the most recent non-null campaign across the user's session history.

Output:

user_id	session_id	session_start	last_touch_campaign	user_last_touch_campaign
u1	1	2026-07-01	google_ads	google_ads
u1	2	2026-07-05	NULL	google_ads
u1	3	2026-07-10	facebook_ads	facebook_ads

Why this works — concept by concept:

ALL ROWS PER MATCH keeps raw rows for later aggregation — the MR block emits the raw event rows enriched with session_id via MATCH_NUMBER(). This is the cleanest way to sessionize when the downstream stage still needs per-event columns.
Per-session aggregate collapses via GROUP BY session_id — standard SQL from here — GROUP BY (user_id, session_id) with MIN / MAX / COUNT. The ARRAY_AGG(... ORDER BY event_time DESC) FILTER (WHERE ... IS NOT NULL)[1] idiom picks the most recent non-null value inside the session.
LAST_VALUE IGNORE NULLS for cross-session forward-fill — the ANSI-standard "carry the last non-null forward" primitive. The ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW frame is required — without it, LAST_VALUE reads the whole partition.
Modular pipeline — Stage 1 is MR-heavy (fine on Snowflake); Stages 2 and 3 are portable SQL. If you later move to a portable-only warehouse, only Stage 1 needs a rewrite to LAG + SUM.
Cost — Stage 1 is O(N log N) per partition — the MR sort dominates. Stages 2 and 3 are O(sessions) — cheap. On 500M events with 50M users on a medium Snowflake warehouse, the pipeline runs in 15-30 minutes. Well inside a nightly window.

SQL
Topic — streaming
Streaming analytics drills

Practice →

SQL Topic — streaming (medium) Streaming problems (medium)

Practice →

4. Funnel & conversion patterns

`funnel analysis sql` reads like a regex — `PATTERN (SIGNUP VIEW+ CART CHECKOUT)` collapses a four-step funnel into one MR block

The mental model in one line: funnel analysis with MATCH_RECOGNIZE encodes the step ordering directly in the PATTERN — PATTERN (SIGNUP VIEW+ CART CHECKOUT) reads left-to-right as "signup then one-or-more product views then add-to-cart then checkout," and DEFINE gives each label its event-type predicate. Once you say "PATTERN is the funnel shape, DEFINE is each step's event predicate, MEASURES emits the conversion metrics," the funnel interview surface reduces to a regex over your event stream.

The canonical funnel PATTERN.

PATTERN (SIGNUP VIEW+ CART CHECKOUT) — strict left-to-right ordering. User must fire SIGNUP, then one or more VIEW events, then CART, then CHECKOUT. Any deviation breaks the match.
Label naming — use event-type-suggestive labels (SIGNUP, VIEW, CART, CHECKOUT) rather than A, B, C, D. Reads better; downstream engineers get the semantics for free.
Quantifiers matter — VIEW+ (one or more) requires at least one product view; VIEW* (zero or more) allows the user to skip views; VIEW{2,} requires at least two views.

The DEFINE predicates.

DEFINE SIGNUP AS event_type = 'signup' — matches only signup events.
DEFINE VIEW AS event_type = 'view_product' — matches only product view events.
DEFINE CART AS event_type = 'add_to_cart' — matches only add-to-cart events.
DEFINE CHECKOUT AS event_type = 'checkout' — matches only checkout events.

Time constraints inside DEFINE.

Sometimes the funnel has an SLA — "checkout must occur within 24 hours of signup." Add a time predicate to CHECKOUT: DEFINE CHECKOUT AS event_type = 'checkout' AND event_time - FIRST(SIGNUP.event_time) <= INTERVAL '24' HOUR.
Cross-step SLAs — "the last view must be within 1 hour of add-to-cart": DEFINE CART AS event_type = 'add_to_cart' AND event_time - LAST(VIEW.event_time) <= INTERVAL '1' HOUR.

MEASURES — the funnel metrics payload.

FIRST(SIGNUP.event_time) AS signup_ts — funnel entry ts.
FIRST(CHECKOUT.event_time) AS checkout_ts — funnel exit ts.
LAST(CHECKOUT.event_time) - FIRST(SIGNUP.event_time) AS time_to_convert — end-to-end conversion time.
COUNT(VIEW.*) AS view_count — how many product views before add-to-cart.
COUNT(*) AS total_events — total events in the funnel path.
FIRST(VIEW.product_id) AS first_product_viewed — which product started the exploration.
LAST(VIEW.product_id) AS last_product_viewed — which product ended the exploration (often the one added to cart).

AFTER MATCH SKIP for funnels.

AFTER MATCH SKIP PAST LAST ROW — one funnel completion per user, ending at the checkout. The user's post-checkout events don't count for the current funnel. Right for "how many users completed the funnel?" reports.
AFTER MATCH SKIP TO NEXT ROW — allow overlapping funnels (a user starts a new SIGNUP after a checkout, before the next SIGNUP). Rare; only when the pattern legitimately overlaps.
AFTER MATCH SKIP TO LAST CHECKOUT — same as SKIP PAST LAST ROW since CHECKOUT is the terminal label.

PERMUTE — unordered variants.

PATTERN (PERMUTE(A, B, C, D)) — matches A, B, C, D in any order. Oracle / Trino / Flink support PERMUTE; Snowflake does not (as of 2026).
Use for onboarding funnels where the order between steps 2 and 3 doesn't matter — "verify_email, add_phone" is the same as "add_phone, verify_email."
Rare but powerful — PERMUTE explodes to N! alternations under the hood, so keep the arity ≤ 4.

Drop-off attribution — WITH UNMATCHED ROWS.

Standard funnel analysis reports both converters and non-converters. ALL ROWS PER MATCH WITH UNMATCHED ROWS emits every input row, either as part of a match or as an unmatched row.
Unmatched rows have MATCH_NUMBER() = NULL and CLASSIFIER() = NULL. Filter to those to identify drop-off events.
Powerful for "at which step did most users drop off?" reports — group unmatched rows by their event_type.

Common funnel analysis sql interview probes.

"Why is MR better than four self-joins?" — the self-join version is O(N^4) if you're unlucky; MR is O(N log N) per partition. Correctness is also easier — no accidental duplicate rows.
"How do you handle users who repeat steps?" — quantifiers. VIEW+ handles multiple views; CART? handles optional cart.
"How do you enforce a time window between steps?" — DEFINE predicates with event_time - LAST(...) inside them.
"How do you compute drop-off per step?" — WITH UNMATCHED ROWS + group unmatched by event_type.
"MR funnel vs LEAD chain — which do you ship?" — MR on capable engines; LEAD chain (four LEAD calls) on portable stacks.

Worked example — signup → view → cart → checkout funnel

Detailed explanation. The canonical funnel: given a user event stream, find every user who completed a four-step funnel. Emit one row per completing user with conversion time and view count.

Question. Given events(user_id, event_time, event_type, product_id), use MR to find every user who completed the signup → view+ → cart → checkout funnel. Emit (user_id, signup_ts, checkout_ts, time_to_convert_min, view_count, first_product, cart_product).

Input.

user_id	event_time	event_type	product_id
u1	09:00	signup	NULL
u1	09:02	view_product	101
u1	09:05	view_product	102
u1	09:08	add_to_cart	102
u1	09:12	checkout	102
u2	10:00	signup	NULL
u2	10:03	view_product	201
u2	10:05	pageview	NULL

Code.

SELECT *
FROM events
MATCH_RECOGNIZE (
  PARTITION BY user_id
  ORDER BY event_time
  MEASURES
    FIRST(SIGNUP.event_time)              AS signup_ts,
    LAST(CHECKOUT.event_time)             AS checkout_ts,
    DATEDIFF('minute', FIRST(SIGNUP.event_time), LAST(CHECKOUT.event_time)) AS time_to_convert_min,
    COUNT(VIEW.*)                          AS view_count,
    FIRST(VIEW.product_id)                 AS first_product,
    LAST(CART.product_id)                  AS cart_product
  ONE ROW PER MATCH
  AFTER MATCH SKIP PAST LAST ROW
  PATTERN (SIGNUP VIEW+ CART CHECKOUT)
  DEFINE
    SIGNUP   AS event_type = 'signup',
    VIEW     AS event_type = 'view_product',
    CART     AS event_type = 'add_to_cart',
    CHECKOUT AS event_type = 'checkout'
);

Step-by-step explanation.

PARTITION BY user_id ORDER BY event_time — per-user pattern scope, time-ordered.
PATTERN (SIGNUP VIEW+ CART CHECKOUT) — strict left-to-right: signup, one or more views, cart, checkout.
DEFINE — each label matches a specific event_type. Any deviation breaks the match — a user who sees a page_view between VIEWs doesn't match the strict pattern.
MEASURES — signup ts (FIRST of SIGNUP), checkout ts (LAST of CHECKOUT), time_to_convert (delta), view_count (COUNT of VIEW rows), first product viewed, product in cart.
u1 matches; u2 fails (never got to add_to_cart or checkout). Only u1 appears in the output.

Output.

user_id	signup_ts	checkout_ts	time_to_convert_min	view_count	first_product	cart_product
u1	09:00	09:12	12	2	101	102

Rule of thumb. MR funnel = one PATTERN with the strict step order + one DEFINE per step + MEASURES for the conversion metrics. Reads like the funnel diagram; ships as one query.

Worked example — funnel with drop-off attribution

Detailed explanation. The standard funnel report needs both converters and the drop-off point for non-converters. ALL ROWS PER MATCH WITH UNMATCHED ROWS exposes every input row along with match metadata — the query below emits converters as matched rows and drop-off events as unmatched rows.

Question. Extend the funnel to expose drop-off events. For every user, emit either their conversion path (matched) or their last event before drop-off (unmatched with CLASSIFIER = NULL).

Input. (Same as above.)

Code.

SELECT *
FROM events
MATCH_RECOGNIZE (
  PARTITION BY user_id
  ORDER BY event_time
  MEASURES
    MATCH_NUMBER() AS match_id,
    CLASSIFIER()   AS matched_step
  ALL ROWS PER MATCH WITH UNMATCHED ROWS
  AFTER MATCH SKIP PAST LAST ROW
  PATTERN (SIGNUP VIEW+ CART CHECKOUT)
  DEFINE
    SIGNUP   AS event_type = 'signup',
    VIEW     AS event_type = 'view_product',
    CART     AS event_type = 'add_to_cart',
    CHECKOUT AS event_type = 'checkout'
);

Step-by-step explanation.

ALL ROWS PER MATCH WITH UNMATCHED ROWS — emit every input row. Matched rows carry match_id and matched_step. Unmatched rows have NULL for both.
Downstream aggregation groups unmatched rows by event_type to see where drop-off happened: SELECT event_type, COUNT(*) FROM ... WHERE match_id IS NULL GROUP BY event_type ORDER BY COUNT(*) DESC.
For u2, the signup and view_product rows are unmatched — the pattern couldn't complete because there's no add_to_cart or checkout. Both are emitted with matched_step = NULL.
For u1, every event is emitted with the step it matched (SIGNUP, VIEW, VIEW, CART, CHECKOUT).
This is exactly the shape a Sankey funnel diagram wants — one row per event, enriched with match id and step label.

Output.

user_id	event_time	event_type	match_id	matched_step
u1	09:00	signup	1	SIGNUP
u1	09:02	view_product	1	VIEW
u1	09:05	view_product	1	VIEW
u1	09:08	add_to_cart	1	CART
u1	09:12	checkout	1	CHECKOUT
u2	10:00	signup	NULL	NULL
u2	10:03	view_product	NULL	NULL
u2	10:05	pageview	NULL	NULL

Rule of thumb. For funnel drop-off attribution, use ALL ROWS PER MATCH WITH UNMATCHED ROWS. Unmatched rows are your drop-off population — filter and aggregate by event_type to identify the leaky step.

Worked example — PERMUTE for onboarding step order

Detailed explanation. Onboarding funnels sometimes let users complete steps in any order — "verify_email, add_phone, complete_profile" can happen 1-2-3 or 2-1-3 or any permutation. PERMUTE inside PATTERN matches any ordering.

Question. Given events(user_id, event_time, event_type), find every user who completed the three-step onboarding (verify_email, add_phone, complete_profile) in any order. Emit (user_id, first_step_ts, last_step_ts, total_time_min).

Input.

user_id	event_time	event_type
u1	09:00	verify_email
u1	09:05	add_phone
u1	09:08	complete_profile
u2	10:00	add_phone
u2	10:03	complete_profile
u2	10:06	verify_email

Code.

SELECT *
FROM events
MATCH_RECOGNIZE (
  PARTITION BY user_id
  ORDER BY event_time
  MEASURES
    FIRST(event_time)                                         AS first_step_ts,
    LAST(event_time)                                          AS last_step_ts,
    DATEDIFF('minute', FIRST(event_time), LAST(event_time))   AS total_time_min
  ONE ROW PER MATCH
  AFTER MATCH SKIP PAST LAST ROW
  PATTERN (PERMUTE(EMAIL, PHONE, PROFILE))
  DEFINE
    EMAIL   AS event_type = 'verify_email',
    PHONE   AS event_type = 'add_phone',
    PROFILE AS event_type = 'complete_profile'
);

Step-by-step explanation.

PATTERN (PERMUTE(EMAIL, PHONE, PROFILE)) — match EMAIL, PHONE, PROFILE in any order. Under the hood the engine expands to all 3! = 6 alternations: (EMAIL PHONE PROFILE | EMAIL PROFILE PHONE | PHONE EMAIL PROFILE | PHONE PROFILE EMAIL | PROFILE EMAIL PHONE | PROFILE PHONE EMAIL).
Both u1 (1-2-3 order) and u2 (2-3-1 order) match.
FIRST(event_time) — the earliest event in the match (regardless of which label it was).
LAST(event_time) — the latest event in the match.
PERMUTE is available on Oracle, Trino, Flink. Snowflake requires explicit alternation as of 2026 — write the 6 branches manually or use a portable fallback.

Output.

user_id	first_step_ts	last_step_ts	total_time_min
u1	09:00	09:08	8
u2	10:00	10:06	6

Rule of thumb. PERMUTE handles unordered funnel steps declaratively — write PERMUTE(A, B, C) instead of a 6-way OR. Keep arity ≤ 4 (24 alternations); above that, the state machine gets expensive.

Senior interview question on multi-step funnel attribution

A senior interviewer might ask: "You run an e-commerce site and want a nightly report: for each user who completed the signup → view+ → cart → checkout funnel, emit (user_id, signup_ts, checkout_ts, time_to_convert, view_count, drop_off_step) — where drop_off_step is 'CONVERTED' for completers and the last step reached for non-completers. Walk me through the query."

Solution Using MR + WITH UNMATCHED ROWS + last-step aggregate

WITH walk AS (
  SELECT *
  FROM events
  MATCH_RECOGNIZE (
    PARTITION BY user_id
    ORDER BY event_time
    MEASURES
      MATCH_NUMBER() AS match_id,
      CLASSIFIER()   AS matched_step
    ALL ROWS PER MATCH WITH UNMATCHED ROWS
    AFTER MATCH SKIP PAST LAST ROW
    PATTERN (SIGNUP VIEW+ CART CHECKOUT)
    DEFINE
      SIGNUP   AS event_type = 'signup',
      VIEW     AS event_type = 'view_product',
      CART     AS event_type = 'add_to_cart',
      CHECKOUT AS event_type = 'checkout'
  )
),
per_user AS (
  SELECT
    user_id,
    MAX(CASE WHEN matched_step = 'CHECKOUT' THEN event_time END) AS checkout_ts,
    MIN(CASE WHEN matched_step = 'SIGNUP'   THEN event_time END) AS signup_ts,
    COUNT(CASE WHEN matched_step = 'VIEW'   THEN 1 END)          AS view_count,
    -- The last step reached in any run (matched or unmatched signup / view / cart)
    CASE
      WHEN MAX(CASE WHEN matched_step = 'CHECKOUT' THEN 1 END) = 1 THEN 'CONVERTED'
      WHEN MAX(CASE WHEN matched_step = 'CART'     THEN 1 END) = 1 THEN 'CART'
      WHEN MAX(CASE WHEN matched_step = 'VIEW'     THEN 1 END) = 1 THEN 'VIEW'
      WHEN MAX(CASE WHEN matched_step = 'SIGNUP'   THEN 1 END) = 1 THEN 'SIGNUP'
      WHEN MAX(CASE WHEN event_type   = 'signup'   THEN 1 END) = 1 THEN 'SIGNUP_UNMATCHED'
      ELSE 'NO_SIGNUP'
    END AS drop_off_step
  FROM walk
  GROUP BY user_id
)
SELECT
  user_id,
  signup_ts,
  checkout_ts,
  DATEDIFF('minute', signup_ts, checkout_ts) AS time_to_convert_min,
  view_count,
  drop_off_step
FROM per_user
ORDER BY user_id;

Step-by-step trace.

Step	What runs	Purpose
1	`walk` — MR with WITH UNMATCHED ROWS	Emit every event with match_id + matched_step (or NULLs)
2	`per_user` — aggregate	Per-user MIN(signup_ts), MAX(checkout_ts), COUNT(views), and the last step reached
3	Final SELECT	Compute time_to_convert; emit converters and non-converters uniformly

The pipeline emits one row per user, tagged with either 'CONVERTED' (checkout reached) or the last step they reached before dropping off. Downstream Sankey / attribution reports read this shape directly.

Output:

user_id	signup_ts	checkout_ts	time_to_convert_min	view_count	drop_off_step
u1	09:00	09:12	12	2	CONVERTED
u2	10:00	NULL	NULL	1	VIEW
u3	11:00	NULL	NULL	0	SIGNUP_UNMATCHED

Why this works — concept by concept:

ALL ROWS PER MATCH WITH UNMATCHED ROWS drives the whole report — one MR pass gives us every event enriched with match_id + matched_step. Matched rows drive conversion metrics; unmatched rows drive drop-off attribution.
CLASSIFIER() as a step-level label — projecting CLASSIFIER() into a matched_step column means downstream CASE-WHEN aggregates can pick out specific step timestamps and counts without joining back to the raw event stream.
CASE ladder to derive drop_off_step — the falling-through CASE ordered from CHECKOUT down to SIGNUP finds the deepest step reached. Rows that never signed up bubble to 'NO_SIGNUP'; rows that signed up but never entered the pattern (say, no view_product) show 'SIGNUP_UNMATCHED'.
Unified converters + non-converters — the same rowset carries both populations. Downstream funnel dashboards filter or aggregate as needed without another query.
Cost — one MR pass (O(N log N) per partition for the sort + O(N) state-machine scan) + one GROUP BY per user (O(N)) + one final SELECT (O(users)). On a 1B-event / 20M-user warehouse, the pipeline runs in single-digit minutes on a well-sized Snowflake medium warehouse. Nightly report cadence is easy.

SQL
Topic — user funnel analysis
User funnel analysis problems

Practice →

SQL Topic — window functions Window function drills

Practice →

5. Anomaly detection & dialect matrix

`oracle match_recognize` and `snowflake match_recognize` shine on anomalies — V-shape / W-shape reversals, fraud bursts, and the streaming-vs-batch dialect matrix

The mental model in one line: anomaly detection with MATCH_RECOGNIZE reads "match a shape defined by consecutive DOWN then UP transitions (V-shape), or DOWN then UP then DOWN then UP (W-shape / double bottom), or a burst of N spend events inside a time window (fraud)" — expressed as PATTERN (STRT DOWN+ UP+) or PATTERN (SPEND{5,}) with DEFINE using PREV / FIRST / LAST predicates. Once you say "PATTERN + PREV / FIRST — that's the anomaly engine," the interview surface reduces to a shape-to-PATTERN translation.

The classic V-shape reversal — Oracle's canonical example.

-- V-shape: N down candles followed by N up candles
PATTERN (STRT DOWN+ UP+)
DEFINE
  STRT AS TRUE,
  DOWN AS price < PREV(price),
  UP   AS price > PREV(price)

STRT is the anchor — matches any row (the row before the first DOWN).
DOWN+ — one or more consecutive rows where the price fell below the previous row's price.
UP+ — one or more consecutive rows where the price rose above the previous row's price.
Match = anchor + downtrend + uptrend = a V-shape reversal.

W-shape / double bottom.

PATTERN (STRT DOWN+ UP+ DOWN+ UP+)
DEFINE
  STRT AS TRUE,
  DOWN AS price < PREV(price),
  UP   AS price > PREV(price)

Twice the V-shape — two consecutive V-shapes make a W.
Common in technical analysis for "double bottom" reversal setup.

Fraud burst — PATTERN (SPEND{5,}) inside 60 seconds.

PATTERN (SPEND{5,})
DEFINE
  SPEND AS event_type = 'spend'
       AND event_time - FIRST(SPEND.event_time) <= INTERVAL '60' SECOND

SPEND{5,} — five or more SPEND events in a row.
DEFINE constrains that all SPEND events must occur within 60 seconds of the first — a "5+ transactions in 1 minute" burst.
Emit the boundaries + total amount + count. Downstream sends alerts.

Streaming CEP — Flink SQL / RisingWave.

Flink SQL exposes MATCH_RECOGNIZE with streaming semantics via its CEP (Complex Event Processing) library. Rows arrive continuously; the state machine tracks partial matches and emits the moment a match completes.
Watermark handling — Flink's watermark tells the pattern engine when it's safe to close a match. A pattern like PATTERN (A B{3,}) only emits after the watermark has advanced past the last matched row.
RisingWave ships the same clause with streaming semantics — MR runs continuously, matches emit as they complete.
The batch semantics on Oracle / Snowflake / Trino are a snapshot of the same engine — same pattern, all data at once.

Dialect matrix (as of 2026).

Engine	MATCH_RECOGNIZE	PERMUTE	AFTER MATCH SKIP variants	Streaming	Notes
Oracle 12c+	YES	YES	ALL	Batch	First to ship, 2013; the reference implementation
Snowflake	YES	NO (as of 2026)	Most	Batch	GA since 2021; the most-used commercial MR
Flink SQL	YES	YES	Most	Streaming (CEP)	Watermark-driven; emits on match completion
Trino / Presto	YES	YES	ALL	Batch	Since Trino 361; used by Athena / Dremio
RisingWave	YES	YES	Most	Streaming	Since 1.7; PostgreSQL-compatible streaming DB
Postgres 16	NO	—	—	—	No planned support; use LAG + SUM(new_group) OVER
BigQuery	NO	—	—	—	No planned support; use LAG + SUM
SQL Server 2022	NO	—	—	—	No planned support; use LAG + SUM
MySQL 8	NO	—	—	—	No planned support; use LAG + SUM
Redshift	NO	—	—	—	No planned support; use LAG + SUM
Databricks SQL	NO	—	—	—	No planned support (as of 2026); use LAG + SUM

Portable fallback — LAG + SUM(new_group) OVER.

The universal fallback for MR — LAG for previous-row lookups, CASE to flag change points, SUM OVER to cumulative-sum into a run id, then GROUP BY run id to aggregate.
Same computational complexity — O(N log N) per partition for the sort + O(N) for the CASE + SUM pass.
Longer code — 20-40 lines vs 10-15 for MR. The gap grows with pattern complexity.

Trino / Presto specifics.

Trino ships full ANSI MR since 361. AWS Athena inherits it from Trino. Dremio, Ahana, and Starburst all use the Trino MR implementation.
Watch for the SEEK variants — Trino's PATTERN parser is stricter than Oracle's; unquoted keywords may collide.

RisingWave / Flink specifics.

Streaming MR requires an ORDER BY on an event-time attribute with a defined watermark. Without a watermark, the pattern engine can't decide when to close a match.
Late data — rows arriving after the watermark are dropped (Flink) or reported to a side output (RisingWave configurable).
Match emission — Flink emits on match completion; RisingWave has an option to emit progressive updates.

Common sql pattern detection interview probes.

"Which engines ship MR?" — Oracle 12c+, Snowflake, Flink SQL, Trino / Presto, RisingWave. Not Postgres, BigQuery, SQL Server, MySQL, Redshift, Databricks SQL.
"Which engine has streaming MR?" — Flink SQL and RisingWave.
"Explain the V-shape PATTERN." — anchor + DOWN+ + UP+; DEFINE DOWN AS price < PREV(price), UP AS price > PREV(price).
"How do you detect a fraud burst?" — PATTERN (SPEND{5,}) + DEFINE using event_time - FIRST(SPEND.event_time) <= INTERVAL '60' SECOND.
"What's the portable fallback?" — LAG + CASE + SUM(new_group) OVER + GROUP BY run_id.

Worked example — V-shape reversal on Oracle / Snowflake

Detailed explanation. The classic Oracle documentation example: find every V-shape reversal in a stock price series. Emit the run's boundary timestamps and prices.

Question. Given ticker(symbol, ts, price), use MR to find every V-shape reversal per symbol (at least one down candle, at least one up candle after the trough).

Input.

symbol	ts	price
AAPL	1	100
AAPL	2	98
AAPL	3	95
AAPL	4	93
AAPL	5	95
AAPL	6	100
AAPL	7	98
AAPL	8	102

Code.

SELECT *
FROM ticker
MATCH_RECOGNIZE (
  PARTITION BY symbol
  ORDER BY ts
  MEASURES
    STRT.ts     AS start_ts,
    LAST(UP.ts) AS end_ts,
    MIN(DOWN.price) AS trough_price,
    LAST(UP.price)  AS rebound_price
  ONE ROW PER MATCH
  AFTER MATCH SKIP PAST LAST ROW
  PATTERN (STRT DOWN+ UP+)
  DEFINE
    DOWN AS price < PREV(price),
    UP   AS price > PREV(price)
);

Step-by-step explanation.

PARTITION BY symbol — each stock's series is matched independently.
PATTERN (STRT DOWN+ UP+) — anchor + one-or-more downs + one-or-more ups. Matches a V-shape.
DEFINE DOWN AS price < PREV(price) — a row is DOWN if its price is less than the previous row's. PREV(price) on the first row is NULL, so DOWN can't match the first row.
DEFINE UP AS price > PREV(price) — a row is UP if its price is greater than the previous row's.
MEASURES — STRT.ts is the anchor's timestamp; LAST(UP.ts) is the reversal's end; MIN(DOWN.price) is the trough; LAST(UP.price) is the rebound peak.
Matches: row 1 (STRT) + rows 2-4 (DOWN+) + rows 5-6 (UP+) = one V-shape. Then row 7 → row 8 forms another V-shape starting from row 6 (STRT at 6, DOWN at 7, UP at 8).

Output.

symbol	start_ts	end_ts	trough_price	rebound_price
AAPL	1	6	93	100
AAPL	6	8	98	102

Rule of thumb. V-shape reversal = PATTERN (STRT DOWN+ UP+) + DEFINE DOWN / UP using PREV(price). The classic Oracle example — the query every financial-data interview loves.

Worked example — fraud burst detection on Flink SQL

Detailed explanation. Flink SQL exposes MR as a streaming CEP operator. The fraud burst pattern — five or more spend events inside 60 seconds — matches as soon as the fifth spend arrives.

Question. On Flink SQL, given a streaming spend_events(user_id, event_time, amount) table with a defined watermark, use MR to emit an alert every time a user has 5+ spend events within 60 seconds.

Input. (Streaming — no snapshot table. Assume events arrive at 09:00:00, 09:00:05, 09:00:10, 09:00:20, 09:00:30, 09:00:45 for u1.)

Code.

SELECT *
FROM spend_events
MATCH_RECOGNIZE (
  PARTITION BY user_id
  ORDER BY event_time
  MEASURES
    FIRST(SPEND.event_time) AS burst_start,
    LAST(SPEND.event_time)  AS burst_end,
    COUNT(*)                AS spend_count,
    SUM(SPEND.amount)       AS total_amount
  ONE ROW PER MATCH
  AFTER MATCH SKIP PAST LAST ROW
  PATTERN (SPEND{5,})
  DEFINE
    SPEND AS event_time - FIRST(SPEND.event_time) <= INTERVAL '60' SECOND
);

Step-by-step explanation.

PATTERN (SPEND{5,}) — five or more consecutive SPEND rows.
DEFINE SPEND AS event_time - FIRST(SPEND.event_time) <= INTERVAL '60' SECOND — every SPEND row must be within 60 seconds of the first SPEND in the current match. The moment a row exceeds the 60-second window, the match terminates.
Because Flink uses streaming semantics, the state machine holds partial matches while waiting for more rows. It emits the moment the fifth row arrives and satisfies the pattern.
Watermark handling — Flink advances the watermark based on the source event-time attribute. A match is only finalised once the watermark has moved past the last matched row (guaranteeing no late data can extend the match).
AFTER MATCH SKIP PAST LAST ROW — after emitting a burst, resume matching at the next row. A user who fires 10 spends inside 60 seconds emits one burst (the first 5+), then may emit another one starting from the 6th event.

Output. (One row per completed burst.)

user_id	burst_start	burst_end	spend_count	total_amount
u1	09:00:00	09:00:30	5	250

Rule of thumb. Streaming fraud bursts on Flink SQL = PATTERN (SPEND{n,}) + DEFINE using a FIRST(SPEND.event_time) time-window constraint. The engine emits the moment the pattern completes; the watermark guards against late data.

Worked example — portable LAG + SUM fallback on BigQuery

Detailed explanation. BigQuery doesn't ship MR. For the same V-shape reversal, write the portable LAG + SUM(new_group) OVER version. Longer, but universally supported.

Question. Rewrite the V-shape reversal query for BigQuery (no MATCH_RECOGNIZE).

Input. (Same as V-shape example.)

Code.

WITH flagged AS (
  SELECT
    symbol,
    ts,
    price,
    CASE
      WHEN price < LAG(price) OVER (PARTITION BY symbol ORDER BY ts) THEN 'DOWN'
      WHEN price > LAG(price) OVER (PARTITION BY symbol ORDER BY ts) THEN 'UP'
      ELSE 'FLAT'
    END AS direction
  FROM ticker
),
runs AS (
  SELECT
    symbol,
    ts,
    price,
    direction,
    -- Start a new run when direction changes
    SUM(CASE
          WHEN direction <> LAG(direction) OVER (PARTITION BY symbol ORDER BY ts)
            OR LAG(direction) OVER (PARTITION BY symbol ORDER BY ts) IS NULL
          THEN 1 ELSE 0
        END) OVER (PARTITION BY symbol ORDER BY ts) AS run_id
  FROM flagged
),
run_summary AS (
  SELECT
    symbol,
    run_id,
    direction,
    MIN(ts)    AS run_start,
    MAX(ts)    AS run_end,
    MIN(price) AS run_min_price,
    MAX(price) AS run_max_price,
    COUNT(*)   AS run_length
  FROM runs
  GROUP BY symbol, run_id, direction
),
paired AS (
  SELECT
    symbol,
    run_start   AS down_start,
    run_end     AS down_end,
    run_min_price AS trough_price,
    LEAD(direction) OVER (PARTITION BY symbol ORDER BY run_start) AS next_dir,
    LEAD(run_start) OVER (PARTITION BY symbol ORDER BY run_start) AS up_start,
    LEAD(run_end)   OVER (PARTITION BY symbol ORDER BY run_start) AS up_end,
    LEAD(run_max_price) OVER (PARTITION BY symbol ORDER BY run_start) AS rebound_price
  FROM run_summary
  WHERE direction = 'DOWN'
)
SELECT
  symbol,
  down_start AS start_ts,
  up_end     AS end_ts,
  trough_price,
  rebound_price
FROM paired
WHERE next_dir = 'UP'
ORDER BY symbol, start_ts;

Step-by-step explanation.

flagged CTE — label each row with direction = DOWN / UP / FLAT relative to the previous row.
runs CTE — cumulative-sum a "direction changed" flag to produce run_id. Rows inside the same run share the id.
run_summary CTE — collapse each run into a summary row with direction, boundaries, and min / max prices.
paired CTE — pair consecutive runs by using LEAD to fetch the next run's direction and boundaries. A V-shape is a DOWN run immediately followed by an UP run.
Final SELECT — filter to direction = 'DOWN' AND next_dir = 'UP' and project the boundary columns.

Output. (Same as the Oracle version.)

Rule of thumb. The portable fallback for V-shape reversal is four CTEs + one LEAD chain. Cost is O(N log N) per partition — same as MR. The code is longer; the query is universally supported.

Senior interview question on dialect-portable anomaly detection

A senior interviewer might ask: "Your company runs analytics on Snowflake (batch, MR-capable) and BigQuery (batch, no MR). Design the anomaly-detection pipeline — V-shape reversals on a 100M-row daily price series — that produces the same output on both warehouses. Talk through the architecture and why you'd not use MR even on Snowflake."

Solution Using dbt macro with dialect dispatch + portable core

-- macros/detect_v_shape.sql — dispatches on target dialect
{% macro detect_v_shape(prices_relation, symbol_col='symbol', ts_col='ts', price_col='price') %}
  {%- if target.type == 'snowflake' -%}
    -- Snowflake: use MATCH_RECOGNIZE for readability + native plan
    SELECT
      {{ symbol_col }},
      start_ts,
      end_ts,
      trough_price,
      rebound_price
    FROM {{ prices_relation }}
    MATCH_RECOGNIZE (
      PARTITION BY {{ symbol_col }}
      ORDER BY {{ ts_col }}
      MEASURES
        STRT.{{ ts_col }}          AS start_ts,
        LAST(UP.{{ ts_col }})      AS end_ts,
        MIN(DOWN.{{ price_col }})  AS trough_price,
        LAST(UP.{{ price_col }})   AS rebound_price
      ONE ROW PER MATCH
      AFTER MATCH SKIP PAST LAST ROW
      PATTERN (STRT DOWN+ UP+)
      DEFINE
        DOWN AS {{ price_col }} < PREV({{ price_col }}),
        UP   AS {{ price_col }} > PREV({{ price_col }})
    )
  {%- else -%}
    -- Portable: LAG + SUM(new_group) OVER for BigQuery / Postgres / SQL Server / MySQL / Redshift / Databricks
    WITH flagged AS (
      SELECT
        {{ symbol_col }},
        {{ ts_col }},
        {{ price_col }},
        CASE
          WHEN {{ price_col }} < LAG({{ price_col }}) OVER (PARTITION BY {{ symbol_col }} ORDER BY {{ ts_col }}) THEN 'DOWN'
          WHEN {{ price_col }} > LAG({{ price_col }}) OVER (PARTITION BY {{ symbol_col }} ORDER BY {{ ts_col }}) THEN 'UP'
          ELSE 'FLAT'
        END AS direction
      FROM {{ prices_relation }}
    ),
    runs AS (
      SELECT
        *,
        SUM(CASE
              WHEN direction <> LAG(direction) OVER (PARTITION BY {{ symbol_col }} ORDER BY {{ ts_col }})
                OR LAG(direction) OVER (PARTITION BY {{ symbol_col }} ORDER BY {{ ts_col }}) IS NULL
              THEN 1 ELSE 0
            END) OVER (PARTITION BY {{ symbol_col }} ORDER BY {{ ts_col }}) AS run_id
      FROM flagged
    ),
    run_summary AS (
      SELECT
        {{ symbol_col }},
        run_id,
        direction,
        MIN({{ ts_col }})    AS run_start,
        MAX({{ ts_col }})    AS run_end,
        MIN({{ price_col }}) AS run_min_price,
        MAX({{ price_col }}) AS run_max_price
      FROM runs
      GROUP BY {{ symbol_col }}, run_id, direction
    ),
    paired AS (
      SELECT
        {{ symbol_col }},
        run_start   AS start_ts,
        run_min_price AS trough_price,
        LEAD(direction) OVER (PARTITION BY {{ symbol_col }} ORDER BY run_start) AS next_dir,
        LEAD(run_end)   OVER (PARTITION BY {{ symbol_col }} ORDER BY run_start) AS end_ts,
        LEAD(run_max_price) OVER (PARTITION BY {{ symbol_col }} ORDER BY run_start) AS rebound_price
      FROM run_summary
      WHERE direction = 'DOWN'
    )
    SELECT {{ symbol_col }}, start_ts, end_ts, trough_price, rebound_price
    FROM paired
    WHERE next_dir = 'UP'
  {%- endif -%}
{% endmacro %}

Step-by-step trace.

Dispatch branch	What runs	Why
Snowflake	MATCH_RECOGNIZE PATTERN (STRT DOWN+ UP+)	Native MR shorter, uses Snowflake's MR operator
BigQuery / Postgres / SQL Server / MySQL / Redshift / Databricks	LAG + SUM(new_group) OVER + LEAD pair	Portable fallback; universal coverage
Both	Same output columns (symbol, start_ts, end_ts, trough_price, rebound_price)	Downstream stays warehouse-agnostic

The macro dispatches on target.type at compile time. Snowflake models get the native MR query; every other target gets the portable version. Both branches produce the same output schema, so downstream consumers are unchanged.

Output:

symbol	start_ts	end_ts	trough_price	rebound_price
AAPL	2026-07-01	2026-07-05	178	195
AAPL	2026-07-08	2026-07-12	190	205
MSFT	2026-07-02	2026-07-06	415	430

Why this works — concept by concept:

Compile-time dispatch, not runtime dispatch — dbt evaluates the Jinja block at compile time, so the emitted SQL is either a MR query or a LAG-based query — never a runtime branch. Zero overhead at query time.
Same output columns across branches — both branches emit (symbol, start_ts, end_ts, trough_price, rebound_price). Downstream dashboards, reports, and joins don't need to know which warehouse produced the data.
MR is not always the right choice even where supported — some teams stick with the portable version even on Snowflake so the code review, the debug story, and the on-call ramp are the same everywhere. This macro gives the choice: opt into MR by adding a Snowflake branch, or ship portable everywhere.
Testability with the same fixture — write dbt tests against fixture data and run them on both warehouses. Same test file, same expected output — the macro guarantees output equivalence.
Cost — Snowflake's MR pass is O(N log N) per partition; the portable LAG + SUM pass is O(N log N) per partition. Same complexity class. On a 100M-row daily price table with 5000 symbols on Snowflake medium, both run in single-digit seconds; on BigQuery with the portable branch, similar (both use columnar storage + parallel window function operators). The dispatch macro adds zero runtime cost — the win is in maintainability, not performance.

SQL
Topic — window functions (SQL)
Window functions in SQL — full library

Practice →

SQL
Topic — SQL
SQL problem library — 450+ DE-focused questions

Practice →

Cheat sheet — MATCH_RECOGNIZE recipe list

Seven-clause skeleton. PARTITION BY (scope), ORDER BY (row order), MEASURES (per-match payload), ONE ROW PER MATCH vs ALL ROWS PER MATCH (output shape), AFTER MATCH SKIP … (overlap policy), PATTERN (...) (regex over row labels), DEFINE <label> AS <predicate> (label semantics). Memorise the order; every MR answer fills these seven slots.
PATTERN is a regex over row labels. Quantifiers: * (zero or more), + (one or more), ? (zero or one), {n} (exactly n), {n,m} (n to m). Alternation |. Grouping (...). Reluctant quantifiers +?, *?, ?? for shortest match.
DEFINE assigns predicates to labels. Predicates can reference PREV(col), NEXT(col), FIRST(label.col), LAST(label.col), aggregates over matched labels, and the current row's columns. Any label used in PATTERN but not defined defaults to TRUE.
PARTITION BY is per-entity scoping. Sessionization partitions by user_id; funnels partition by user_id; anomaly detection partitions by symbol. Omitting the partition matches across entity boundaries — a common wrong-answer bug.
ORDER BY is mandatory. Establishes what "previous" and "next" mean. Almost always the event time or a monotonic sequence. Missing ORDER BY is a compilation error.
ONE ROW PER MATCH for summary; ALL ROWS PER MATCH for enrichment. Session summaries use ONE ROW; per-event enrichment inside a match uses ALL ROWS. ALL ROWS PER MATCH WITH UNMATCHED ROWS also emits unmatched rows — critical for funnel drop-off attribution.
AFTER MATCH SKIP variants. SKIP PAST LAST ROW (default; no overlap; right for sessions, funnels, anomalies), SKIP TO NEXT ROW (allow overlap; right for "every possible occurrence" analytics), SKIP TO FIRST <label>, SKIP TO LAST <label> (skip to a labelled row inside the match).
Sessionization primitive. PATTERN (A B*) + DEFINE B AS DATEDIFF('minute', LAG(event_time), event_time) <= 30. One anchor + zero-or-more continuations inside the inactivity threshold. One match per session.
Funnel primitive. PATTERN (SIGNUP VIEW+ CART CHECKOUT) + one DEFINE per step. Strict left-to-right ordering; quantifiers on repeated steps. Encodes the funnel as regex; MR emits one row per completing user.
V-shape / W-shape anomaly primitive. PATTERN (STRT DOWN+ UP+) for V; PATTERN (STRT DOWN+ UP+ DOWN+ UP+) for W. DEFINE DOWN AS price < PREV(price), UP AS price > PREV(price). The classic Oracle documentation example — every financial-data interview loves this shape.
Fraud burst primitive. PATTERN (SPEND{5,}) + DEFINE SPEND AS event_time - FIRST(SPEND.event_time) <= INTERVAL '60' SECOND. Five or more spend events inside a 60-second window from the first.
CLASSIFIER() and MATCH_NUMBER(). CLASSIFIER() returns the label the current row was classified into; MATCH_NUMBER() is a monotonically-increasing per-partition match id. Project both into MEASURES when using ALL ROWS PER MATCH for downstream analytics.
PERMUTE for unordered patterns. PATTERN (PERMUTE(A, B, C)) matches A, B, C in any order — the engine expands to N! alternations under the hood. Oracle / Trino / Flink support PERMUTE; Snowflake requires manual alternation as of 2026.
WITH UNMATCHED ROWS for drop-off attribution. ALL ROWS PER MATCH WITH UNMATCHED ROWS emits both matched and unmatched rows. Unmatched rows have MATCH_NUMBER() = NULL and CLASSIFIER() = NULL — filter to those for drop-off events.
MR dialect matrix. YES: Oracle 12c+, Snowflake, Flink SQL (streaming CEP), Trino / Presto, RisingWave. NO: Postgres, BigQuery, SQL Server, MySQL, Redshift, Databricks SQL (as of 2026). Streaming semantics only on Flink SQL and RisingWave.
Portable fallback — LAG + SUM(new_group) OVER. For non-MR warehouses: LAG(col) for previous value, CASE WHEN col <> LAG(col) THEN 1 ELSE 0 END to flag change points, SUM(is_new) OVER (PARTITION BY entity ORDER BY ts) for run id, then GROUP BY run_id to aggregate. Same computational complexity, longer code.
Dbt macro dispatch pattern. Wrap MR in a Jinja macro with {% if target.type == 'snowflake' %} to emit the MR variant on Snowflake and the portable fallback everywhere else. Downstream consumers see identical output columns.
Flink streaming CEP. Same MR syntax, watermark-driven. Match emits when the watermark advances past the last matched row. Late data is dropped or routed to a side output. Great for real-time fraud detection and low-latency alerting.
When to use MR vs portable. MR when the pattern is genuinely multi-row and regex-like (funnels, V/W anomalies, PERMUTE variants) and the warehouse supports it. Portable for simple RLE, sessionization on multi-warehouse projects, and any codebase where dialect drift is a maintenance burden.
Cost model. MR compiles to sort + state-machine scan — O(N log N) per partition for the sort + O(N) for the state machine. Portable LAG + SUM is sort + two window passes + aggregate — same O(N log N) per partition. On billion-row inputs, both run in single-digit minutes on modern warehouses.

Frequently asked questions

What is SQL MATCH_RECOGNIZE and when do you use it?

sql match_recognize is the ANSI SQL/2016 row pattern matching clause — a mini regex engine that runs over ordered rows inside a partition. The seven-clause skeleton is PARTITION BY entity ORDER BY time MEASURES ... PATTERN (regex-over-labels) DEFINE label AS predicate ONE ROW PER MATCH AFTER MATCH SKIP …. Reach for it whenever you need to detect an ordered row sequence — sessionization (PATTERN (A B*) with an inactivity DEFINE), funnel conversion (PATTERN (SIGNUP VIEW+ CART CHECKOUT)), V-shape / W-shape stock anomaly (PATTERN (STRT DOWN+ UP+)), fraud bursts (PATTERN (SPEND{5,}) inside a 60-second window), or any streaming CEP task where a Flink SQL / RisingWave watermark closes matches as they complete. MR ships on Oracle 12c+, Snowflake, Flink SQL, Trino / Presto, and RisingWave; Postgres, BigQuery, SQL Server, MySQL, Redshift, and Databricks SQL don't ship it yet, so multi-warehouse code defaults to the portable LAG + SUM(new_group) OVER fallback.

How does MATCH_RECOGNIZE compare to gaps-and-islands with LAG and SUM?

Both express the same class of problems — sessionization, streaks, RLE, funnels, anomaly detection — but MATCH_RECOGNIZE reads as a regex over labelled row classes while gaps-and-islands cascades LAG, CASE, and SUM(is_new_group) OVER window functions. For simple RLE and sessionization, the two produce identical physical plans on Snowflake and roughly equivalent plans elsewhere; MR just reads more clearly once your team knows the anatomy. For complex multi-row patterns (V-shape / W-shape reversals, PATTERN (SIGNUP VIEW+ CART CHECKOUT) funnels, PERMUTE variants, fraud bursts inside a time window) MR is dramatically shorter — 10-15 lines vs 30-40 lines portable, with the gap growing as the pattern gets more regex-like. The strategic answer for a multi-warehouse codebase is a dbt macro that dispatches on target.type: emit MR on Snowflake / Oracle / Trino / Flink / RisingWave, emit the portable LAG + SUM version everywhere else, present the same output columns to downstream consumers.

What are the seven clauses of MATCH_RECOGNIZE?

The seven slots are, in canonical reading order: (1) PARTITION BY <cols> — scope the pattern per entity (user, symbol, device); (2) ORDER BY <cols> — establish row order (event_time or a monotonic sequence); (3) MEASURES <exprs> — per-match output columns using FIRST(label.col), LAST(label.col), COUNT(*), MATCH_NUMBER(), and CLASSIFIER(); (4) ONE ROW PER MATCH for summary output vs ALL ROWS PER MATCH (or ALL ROWS PER MATCH WITH UNMATCHED ROWS) for per-event enrichment; (5) AFTER MATCH SKIP … — the overlap policy (PAST LAST ROW for no overlap, TO NEXT ROW to allow overlap, TO FIRST/LAST <label> for label-anchored skips); (6) PATTERN (...) — the regex over row labels with quantifiers * + ? {n} {n,m}, alternation |, grouping (...), and PERMUTE(...); (7) DEFINE <label> AS <predicate> — the boolean predicate for each label, allowed to reference PREV, NEXT, FIRST, LAST, and aggregates over already-matched labels. Say them in order without prompting and you're already in the top 20% of MR fluency.

How do you write a funnel-conversion query with MATCH_RECOGNIZE?

funnel analysis sql with MATCH_RECOGNIZE reads like the funnel diagram itself: PATTERN (SIGNUP VIEW+ CART CHECKOUT) encodes the strict left-to-right step ordering, DEFINE SIGNUP AS event_type = 'signup', DEFINE VIEW AS event_type = 'view_product', DEFINE CART AS event_type = 'add_to_cart', DEFINE CHECKOUT AS event_type = 'checkout' assigns each label its event-type predicate, and MEASURES emits FIRST(SIGNUP.event_time) AS signup_ts, LAST(CHECKOUT.event_time) AS checkout_ts, DATEDIFF('minute', ...) AS time_to_convert_min, and COUNT(VIEW.*) AS view_count. Quantifiers control repetition (VIEW+ for one-or-more product views, CART? for optional add-to-cart, VIEW{2,} for at least two views). PERMUTE(A, B, C) handles unordered onboarding funnels on Oracle / Trino / Flink. ALL ROWS PER MATCH WITH UNMATCHED ROWS exposes drop-off events with MATCH_NUMBER() = NULL for the standard Sankey funnel report — one MR pass, both converters and drop-off attribution.

How do you detect V-shape or W-shape anomalies in SQL?

The classic oracle match_recognize documentation example: a V-shape stock reversal is PATTERN (STRT DOWN+ UP+) with DEFINE DOWN AS price < PREV(price) and DEFINE UP AS price > PREV(price) — one anchor row + one-or-more consecutive down candles + one-or-more consecutive up candles. STRT.ts is the reversal's start; LAST(UP.ts) is its end; MIN(DOWN.price) is the trough; LAST(UP.price) is the rebound. W-shape / double-bottom extends to PATTERN (STRT DOWN+ UP+ DOWN+ UP+) — two consecutive V-shapes in a row. The same pattern engine runs the fraud-burst detection — PATTERN (SPEND{5,}) with DEFINE SPEND AS event_time - FIRST(SPEND.event_time) <= INTERVAL '60' SECOND for a "5+ spends inside 60 seconds" alert. On flink sql pattern and RisingWave, the same syntax runs continuously against a live event stream with watermark-driven match completion — the same anomaly recipe, streaming semantics. Portable fallback for warehouses without MR: LAG + CASE (DOWN / UP / FLAT direction label) + SUM(is_new_group) OVER for run id + LEAD to pair consecutive runs.

Which SQL engines support MATCH_RECOGNIZE in 2026?

As of 2026, MATCH_RECOGNIZE ships natively on Oracle 12c+ (first to market in 2013, the reference implementation), Snowflake (batch, GA since 2021), Flink SQL (streaming CEP semantics with watermark-driven match completion), Trino / Presto (batch, since Trino 361 — inherited by AWS Athena, Dremio, Starburst, Ahana), and RisingWave (streaming, since 1.7, PostgreSQL-compatible). No MR support on Postgres 16, BigQuery, SQL Server 2022, MySQL 8, Redshift, or Databricks SQL — for those engines, use the portable LAG + SUM(new_group) OVER gaps-and-islands fallback. Streaming semantics only on Flink SQL and RisingWave; the other three (Oracle, Snowflake, Trino) are batch. PERMUTE is supported on Oracle, Trino, and Flink but not on Snowflake as of 2026. For a multi-warehouse codebase, wrap MR in a dbt macro with {% if target.type == 'snowflake' or target.type == 'trino' %} dispatch and fall back to the portable pattern everywhere else — same output columns, same tests, one macro.

Practice on PipeCode

Drill the pattern-matching practice library → for MATCH_RECOGNIZE anatomy, PATTERN quantifiers, DEFINE predicates, and AFTER MATCH SKIP semantics across Oracle, Snowflake, Trino, and Flink dialects.
Rehearse on gaps-and-islands problems → — the portable LAG + SUM(new_group) OVER fallback that every non-MR warehouse (Postgres, BigQuery, SQL Server, MySQL, Redshift, Databricks) requires.
Sharpen streaming drills → for the Flink SQL and RisingWave CEP variants — watermark-driven pattern completion, low-latency emission, side outputs for late data.
Push the difficulty ceiling with medium streaming problems → for real-time fraud detection, sessionization with watermarks, and multi-step funnel tracking.
Practise window-function drills → — the LAG / LEAD / FIRST_VALUE / LAST_VALUE primitives that both MR predicates and portable fallbacks lean on.
Sharpen the dialect-portable half with SQL window function problems → for Postgres, Snowflake, BigQuery, SQL Server, MySQL where portable is the default.
Layer user-funnel-analysis drills → for PATTERN (SIGNUP VIEW+ CART CHECKOUT), drop-off attribution via WITH UNMATCHED ROWS, and PERMUTE variants for unordered onboarding steps.
Practise regular-expression drills → — MATCH_RECOGNIZE's PATTERN clause is regex over rows; fluency with regex quantifiers transfers directly.
For general SQL sharpening, work through the SQL problem library → which contains 450+ DE-focused questions.
For the broader SQL interview surface, take the SQL for Data Engineering course →.

Pipecode.ai is Leetcode for Data Engineering — every `sql match_recognize` recipe above ships with hands-on practice rooms where you write the seven-clause anatomy, wire `PATTERN (A B*)` sessionization, chase the `PATTERN (SIGNUP VIEW+ CART CHECKOUT)` funnel with WITH UNMATCHED ROWS drop-off attribution, detect V-shape and W-shape reversals, rehearse the Flink SQL streaming CEP variant, and translate to the portable LAG + SUM(new_group) OVER fallback against real graded inputs. PipeCode pairs every reading with 450+ DE-focused problems and a real-time scoring engine, so you never have to wonder whether your `row pattern matching sql` answer holds up under a senior interviewer's depth probes.

Practice pattern matching now →
User funnel analysis drills →

SQL Gaps & Islands: Sessionization, Streaks & Run-Length Patterns

Gowtham Potureddi — Fri, 10 Jul 2026 17:35:17 +0000

sql gaps and islands is the single most-asked "consecutive rows" family in senior data-engineering interviews — and the single most under-taught one in undergraduate SQL curricula. The same "identify runs of consecutive events" ask shows up in five different colours: sessionization ("group events into 30-minute sessions"), login streaks ("longest consecutive-day streak per user"), uptime windows ("longest window with status = 'up'"), price plateaus ("compress the price series into runs of unchanged values"), and CDC state transitions ("emit one row per contiguous state"). Every answer starts from the same two-line trick: subtract a running counter from an ordered attribute, group by the difference, and each group is one island. The interviewer wants to hear you say "row_number minus date equals island id" in the first sentence.

This guide is the mid-to-senior tour you wished existed the first time an interviewer asked you to write a sessionization sql query on the whiteboard, sketch sql streaks for a growth analytics dashboard, chase a consecutive rows sql bug in production, derive a sql session id for a Segment-style event stream, or design an sql run length encoding compression for a status-transition table. It walks through the classic date-island Tabibitosan trick with ROW_NUMBER and date arithmetic, the sessionization pattern with LAG(event_time) plus SUM(new_session) OVER and per-user partitioning, the LeetCode-flavoured consecutive-day streak problem with longest-vs-current framing and holiday tolerance, and the run-length encoding pattern with SUM(CASE WHEN prev <> curr THEN 1 ELSE 0 END) OVER plus the MATCH_RECOGNIZE alternative and the Postgres / Snowflake / BigQuery / SQL Server dialect matrix. Every section pairs a teaching block with a Solution-Tail interview answer — code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works.

When you want hands-on reps immediately after reading, drill the gaps-and-islands practice library →, rehearse on window-function problems →, and sharpen the timing axis with time-series drills →.

On this page

Why gaps & islands matters in 2026
Classic date-island pattern
Sessionization with 30-min inactivity
Consecutive-day streaks
Run-length encoding + dialect matrix
Cheat sheet — Gaps & Islands recipe list
Frequently asked questions
Practice on PipeCode

1. Why gaps & islands matters in 2026

The "consecutive rows problem" family — sessionization, streaks, uptime, price plateaus, and every other run-of-events question a senior interviewer might ask

The one-sentence invariant: a sql gaps and islands question asks you to identify contiguous runs of rows that share some ordering property, and the canonical answer is always some form of "assign an island id, group by island id, aggregate per island". Once you internalise that "assign island id + group + aggregate" is the whole family, the consecutive-rows interview surface collapses into a two-step template you can write from muscle memory.

Five faces of the same problem.

Sessionization. Given an event stream, group consecutive events into sessions separated by a gap of inactivity (usually 30 minutes). Every product analytics tool — Google Analytics, Segment, Rudderstack, Snowplow, PostHog, Mixpanel — implements exactly this logic. The interviewer wants to see you write it in warehouse SQL, not paste an out-of-the-box definition.
Streaks. Given a table of user activity dates, find the longest consecutive-day streak per user (LeetCode's "Consecutive Numbers" and "Trips and Users" flavours). Growth teams live and die on daily-active retention, weekly streaks, workout streaks, learning-app streaks — all one query pattern.
Uptime / downtime windows. Given a table of status pings, find the longest window where status = 'up' (or the last outage of duration > 5 minutes). SRE dashboards and pager duty policies both consume this shape.
Price plateaus. Given a price time series with many unchanged values in a row, compress into runs of (start_ts, end_ts, price). Storage cost drops by 10-100× for slowly-changing values.
CDC state transitions. Given a slowly changing dimension type-2 stream, emit one row per contiguous state. Debezium, Airbyte, and Fivetran output this shape; downstream models consume it.

The two mental primitives.

Island id assignment. The mechanical trick that turns a sequence into groups. Two flavours: (a) subtract a running counter from the ordered attribute (login_date - ROW_NUMBER()); (b) cumulative-sum a "new group" flag (SUM(CASE WHEN gap > threshold THEN 1 ELSE 0 END) OVER (...)). Every gaps-and-islands answer uses one of these two.
Group + aggregate. Once you have an island id, GROUP BY user_id, island_id and aggregate — MIN(ts) as session start, MAX(ts) as session end, COUNT(*) as event count, etc. This half is boring; the whole trick is in the island-id derivation.

Why senior interviewers love this family.

It tests window-function fluency without being about ranking. Everyone can write ROW_NUMBER() for a leaderboard. Fewer candidates recognise that ROW_NUMBER() also works as a gap-detection primitive when subtracted from an ordered attribute.
It has a portable answer and a MATCH_RECOGNIZE answer. The interviewer probes whether you know both — the portable window-function recipe and the Snowflake/Oracle native syntax — and when to reach for each.
It's compositional. Once you can sessionize, streaks are trivial. Once you can do streaks, RLE is a variation. Once you can do RLE, uptime windows are the same query with a filter. A senior candidate walks through this progression without prompting.
It exposes multi-user partitioning discipline. Every answer must PARTITION BY user_id inside the window clauses. Candidates who forget the partition get wrong answers on the first test case — an instant red flag.

Where consecutive-rows queries land in your pipelines.

Product analytics. Every event stream is sessionized before it can be counted, deduped, or attributed. session_id is the atomic unit of product analytics.
Growth analytics. DAU/WAU/MAU retention, streak-based engagement metrics, funnel step deduplication — every metric that touches "consecutive days of activity" is a streaks query underneath.
Reliability / SRE. Uptime SLOs, outage duration histograms, incident post-mortems — every timeline reconstruction is an islands query with a status attribute.
Financial time series. Price plateaus, volatility clusters, order book state transitions, market-hours vs after-hours splits — every "runs of contiguous values" question is RLE.
CDC / SCD pipelines. Type-2 dimensions collapse consecutive unchanged rows into a single validity window using RLE. The valid_from / valid_to columns are computed exactly this way.

What senior interviewers actually probe.

Island-id derivation. Do you reach for ROW_NUMBER - date or SUM(new_group) OVER? Do you know when to use each? Can you say "date arithmetic works when the sequence is dense; cumulative sum works when the gap is a computed threshold" without prompting?
Sessionization threshold. Do you hard-code INTERVAL '30 minutes' or take it as a parameter? Do you know Segment / GA use 30 minutes by default? Do you handle timezone edge cases when the ETL runs across the DST boundary?
Streak-with-gap tolerance. Can you allow a 1-day miss without breaking the streak? Do you know how to pre-join a business-calendar table to exclude holidays and weekends from streak checks?
RLE compression math. Can you estimate the compression ratio for a slowly-changing series? Do you know when RLE beats DISTINCT ... LAG vs when a DISTINCT view is enough?
MATCH_RECOGNIZE vs portable. Do you know which warehouses ship MATCH_RECOGNIZE (Oracle, Snowflake) and which don't (Postgres, BigQuery, SQL Server)? Do you default to the portable LAG + SUM pattern for a multi-warehouse codebase?

Worked example — the "consecutive numbers" starter problem

Detailed explanation. The canonical whiteboard warm-up: given a logs(id, num) table, find every number that appears at least three times consecutively. This is LeetCode's "Consecutive Numbers" problem, and it's the fastest way to check that a candidate recognises the gaps-and-islands family. The two-line answer uses LAG twice or a ROW_NUMBER - RANK trick, and the interviewer wants to see whichever variant you like walked through cleanly.

Question. Given logs(id, num) where id is a dense ascending integer and num is any integer, write a query that returns every num that appears three or more times in a row.

Input.

id	num
1	1
2	1
3	1
4	2
5	1
6	2
7	2

Code.

-- Portable — LAG twice, check equality
SELECT DISTINCT l.num AS ConsecutiveNums
FROM (
  SELECT
    num,
    LAG(num, 1) OVER (ORDER BY id) AS prev1,
    LAG(num, 2) OVER (ORDER BY id) AS prev2
  FROM logs
) l
WHERE l.num = l.prev1 AND l.num = l.prev2;

Step-by-step explanation.

LAG(num, 1) OVER (ORDER BY id) produces the previous row's num, ordered by id. LAG(num, 2) ... produces two rows back. Together, each row now knows its own value and the two before it.
The WHERE l.num = l.prev1 AND l.num = l.prev2 predicate keeps only rows where the current value equals both the previous value and the one before that — a three-in-a-row match on the current row's timestamp.
SELECT DISTINCT l.num deduplicates the matches. If the same value appears in four consecutive rows, the predicate matches on rows 3 and 4; DISTINCT collapses the duplicate output.
The alternative form uses num - ROW_NUMBER() OVER (PARTITION BY num ORDER BY id) — the island-id trick — then groups by (num, island_id) and filters for COUNT(*) >= 3. Same answer, different mechanics.
Rule of thumb: for a fixed-length "N in a row" check, LAG chains are more readable. For a variable-length "runs of X" query, the island-id trick generalises better.

Output.

ConsecutiveNums
1

Rule of thumb. LeetCode "Consecutive Numbers" is the gaps-and-islands warm-up — recognise it in 5 seconds, write the LAG chain in 30, then move on to the harder island-id derivation the interviewer really wants.

Worked example — sessionize an event stream (preview)

Detailed explanation. The sessionization interview is a gaps-and-islands question wearing a product-analytics costume. The pattern is: compute the time gap to the previous event with LAG(event_time), flag rows where the gap exceeds a threshold, cumulative-sum the flag to produce a session id. This preview walks the mechanics; Section 3 goes deep on late-data and per-user partitioning.

Question. Given an events(user_id, event_time) stream and a 30-minute inactivity threshold, write a query that assigns a session_id (integer, 0-indexed within each user) to every row.

Input.

user_id	event_time
u1	2026-07-01 10:00
u1	2026-07-01 10:05
u1	2026-07-01 11:15
u1	2026-07-01 11:20

Code.

SELECT
  user_id,
  event_time,
  SUM(new_session) OVER (
    PARTITION BY user_id
    ORDER BY event_time
    ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW
  ) AS session_id
FROM (
  SELECT
    user_id,
    event_time,
    CASE
      WHEN EXTRACT(EPOCH FROM (event_time - LAG(event_time) OVER (PARTITION BY user_id ORDER BY event_time))) > 30 * 60
        THEN 1
      ELSE 0
    END AS new_session
  FROM events
) t;

Step-by-step explanation.

Inner query — LAG(event_time) OVER (PARTITION BY user_id ORDER BY event_time) yields the previous event's timestamp within the user. The first event of each user has LAG = NULL, and the epoch difference is NULL — treated as 0 by the CASE, so the first event stays in session 0.
EXTRACT(EPOCH FROM (event_time - LAG(...))) converts the interval to seconds. > 30 * 60 is the 30-minute threshold — any gap larger flags a new session.
CASE WHEN ... THEN 1 ELSE 0 END produces a 0/1 flag column called new_session. This is the "island-boundary" indicator.
Outer query — SUM(new_session) OVER (PARTITION BY user_id ORDER BY event_time) cumulatively sums the flags per user. The result increments by 1 at every gap larger than 30 minutes, producing a per-user 0-indexed session id.
ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW is the explicit window frame; some dialects default to it, but stating it explicitly avoids the "range vs rows" trap that trips senior candidates.

Output.

user_id	event_time	session_id
u1	2026-07-01 10:00	0
u1	2026-07-01 10:05	0
u1	2026-07-01 11:15	1
u1	2026-07-01 11:20	1

Rule of thumb. Sessionization is LAG + CASE + SUM OVER — three window primitives combined. Recognise the pattern in seconds; write the query in a minute.

Worked example — collapse a price series with RLE (preview)

Detailed explanation. Run-length encoding on a price series compresses consecutive unchanged values into one row per run. This preview shows the SUM-of-change-points trick; Section 5 goes deep on dialect variations and MATCH_RECOGNIZE alternatives.

Question. Given prices(ts, symbol, price) where price changes rarely, write a query that emits one row per (symbol, run) with the run's start_ts, end_ts, and price.

Input.

ts	symbol	price
1	AAPL	100
2	AAPL	100
3	AAPL	105
4	AAPL	105
5	AAPL	105
6	AAPL	100

Code.

WITH flagged AS (
  SELECT
    ts,
    symbol,
    price,
    CASE
      WHEN price <> LAG(price) OVER (PARTITION BY symbol ORDER BY ts)
        OR LAG(price) OVER (PARTITION BY symbol ORDER BY ts) IS NULL
        THEN 1
      ELSE 0
    END AS is_new_run
  FROM prices
),
grouped AS (
  SELECT
    symbol,
    price,
    ts,
    SUM(is_new_run) OVER (PARTITION BY symbol ORDER BY ts) AS run_id
  FROM flagged
)
SELECT
  symbol,
  price,
  MIN(ts) AS start_ts,
  MAX(ts) AS end_ts,
  COUNT(*) AS run_length
FROM grouped
GROUP BY symbol, price, run_id
ORDER BY start_ts;

Step-by-step explanation.

flagged CTE — for each row, compare the current price to the previous row's price (via LAG). If it differs (or if it's the first row), mark is_new_run = 1; otherwise 0. The special-case for NULL handles the very first row of the partition.
grouped CTE — cumulative-sum the is_new_run flag per symbol. Every 1 increments the counter; every 0 preserves it. The result is a run_id that's constant within one run and increments at each change point.
Outer SELECT — group by (symbol, price, run_id) to collapse each run into one row. MIN(ts) is the run start, MAX(ts) is the run end, COUNT(*) is the run length.
Including price in the GROUP BY is mostly cosmetic — the run_id already implies the price — but keeps the plan simple and the intent explicit.
Compression ratio: a series where 90% of rows are duplicates compresses ~10×. For slowly-changing prices or feature-flag rollouts, RLE cuts storage and query cost dramatically.

Output.

symbol	price	start_ts	end_ts	run_length
AAPL	100	1	2	2
AAPL	105	3	5	3
AAPL	100	6	6	1

Rule of thumb. RLE = flag change points with LAG + CASE, cumulative-sum to run_id, then GROUP BY run_id. Same three-step template as sessionization, different threshold.

Senior interview question on the gaps-and-islands mental model

A senior interviewer often opens with: "I hand you a table of user events over the last year and ask for a metric — pick one — that requires you to identify runs of consecutive activity. Walk me through your two-step template, and tell me what changes if the ask is sessionization vs streaks vs uptime windows."

Solution Using the "flag + cumulative sum" universal template

Universal gaps-and-islands template — three steps

Step 1 — Order the rows within each partition.
  PARTITION BY <entity> ORDER BY <time-or-sequence>

Step 2 — Assign an island id (two flavours).
  Flavour A — dense sequence subtraction (streaks, dense-date islands):
    island_id = ordered_attribute - ROW_NUMBER() OVER (PARTITION BY entity ORDER BY ordered_attribute)

  Flavour B — cumulative sum of change flag (sessions, RLE, uptime windows):
    is_new = CASE WHEN <boundary condition> THEN 1 ELSE 0 END
    island_id = SUM(is_new) OVER (PARTITION BY entity ORDER BY ordered_attribute)

Step 3 — Group by island id and aggregate.
  GROUP BY entity, island_id
  aggregates → MIN(ts) AS start, MAX(ts) AS end, COUNT(*) AS length, ...

Step-by-step trace.

Ask	Flavour	Boundary condition	Aggregate
Login streaks	A	consecutive dates (dense)	MIN(date), MAX(date), COUNT(*) as streak length
Sessionization	B	gap > 30 minutes	MIN(ts) as session_start, COUNT(*) as event count
Uptime windows	B	status changes from `up` to `down` or vice versa	MIN(ts), MAX(ts), duration
Price plateaus (RLE)	B	price <> LAG(price)	MIN(ts), MAX(ts), price, COUNT(*) as run length
Consecutive-N check	B	value = LAG(value)	filter to COUNT(*) >= N

The universal template turns "which query do I write?" from a taste question into a mechanical checklist. Pick the partition, pick the ordering, pick the flavour, apply the aggregate — the whole query falls out.

Output:

Ask	Query length	Time to whiteboard
Streak per user	8-12 lines	~2 min
Sessionization	12-18 lines	~3 min
Uptime windows	15-25 lines	~4 min
Price RLE	18-25 lines	~4 min
Consecutive-N	6-8 lines	~1 min

Why this works — concept by concept:

Two flavours cover the space — every gaps-and-islands question is either "dense sequence" (streaks, calendars) or "computed boundary" (sessions, RLE, uptime). Flavour A uses ROW_NUMBER subtraction; Flavour B uses cumulative sum of a flag. There is no third pattern.
PARTITION BY entity is non-negotiable — multi-user data always requires per-user partitioning. Skipping it produces a global island id that mixes users, which is a wrong-answer buffer that the interviewer catches immediately.
ORDER BY inside the window matters — the ordering determines what "consecutive" means. Sessionization orders by event_time; RLE orders by ts; streaks order by date. Get the ordering wrong and the island id is meaningless.
The aggregate is boring by design — once the island id is right, the aggregation is standard GROUP BY + MIN + MAX + COUNT. Interviewers spend 90% of the follow-up time on island-id derivation and 10% on the aggregate — mirror that ratio in your explanation.
Cost — the template is O(N log N) per partition for the ORDER BY inside the window, plus O(N) for the aggregate. On a 100M-row table with 1M users, that's ~1B operations — 10-30 seconds on modern warehouses. If it's too slow, pre-aggregate at ingestion or push partitions into a clustering key.

SQL
Topic — gaps-and-islands
Gaps and islands problems

Practice →

SQL Topic — window functions Window function drills

Practice →

2. Classic date-island pattern

`consecutive rows sql` in its purest form — the `date - ROW_NUMBER` Tabibitosan trick that turns a dense sequence into island groups

The mental model in one line: when a dense-date sequence is punctuated by gaps, subtracting a per-partition ROW_NUMBER() from the ordered attribute produces a constant per island — group by that constant and you have your runs. Once you say "date minus row_number equals island id" out loud, the classic date-island interview surface collapses to a five-line SQL query.

Why the trick works.

Dense sequences advance in lockstep. If a user logs in on 2026-07-01, 2026-07-02, 2026-07-03, and the ordered ROW_NUMBER() is 1, 2, 3, then date - row_number is 2026-06-30 for all three rows. The difference is constant inside an island.
A gap breaks the lockstep. If the next login is on 2026-07-06 and its row_number is 4, then date - row_number = 2026-07-02 — a different island id. The gap moved the row_number ahead of the date.
The island id has no meaning on its own — it's just a group key. You never expose it in the output; you only use it in GROUP BY. Aggregations recover the human-readable boundaries.

The Tabibitosan name. The trick is credited to a Japanese SQL blog post by Aketi Jyuuzou circa 2008. "Tabibitosan" translates to "arithmetic problem for travellers," and refers to the classic maths puzzle "two travellers walking at different speeds meet in the same place — where and when?" — which is the same "lockstep + gap" mental model.

Dialect variations on the date arithmetic.

Postgres. login_date - (ROW_NUMBER() OVER (...) || ' days')::interval — cast the row number to an interval. Or login_date - INTERVAL '1 day' * ROW_NUMBER() OVER (...).
Snowflake. DATEADD(day, -ROW_NUMBER() OVER (...), login_date).
BigQuery. DATE_SUB(login_date, INTERVAL ROW_NUMBER() OVER (...) DAY).
SQL Server. DATEADD(day, -ROW_NUMBER() OVER (...), login_date).
MySQL 8. login_date - INTERVAL ROW_NUMBER() OVER (...) DAY.

Alternative: use integer difference for weekly or monthly islands.

Weekly islands: WEEK(login_date) - ROW_NUMBER() OVER (...) — same trick, different granularity.
Monthly islands: MONTH(login_date) + YEAR(login_date) * 12 - ROW_NUMBER() OVER (...) — 12-month wrap.
Custom period islands: any monotonically-increasing integer works. EXTRACT(EPOCH FROM date) for second-granularity; EXTRACT(EPOCH FROM date) / 3600 for hour-granularity.

Handling non-daily grids.

Weekends off. Pre-filter to only Mon-Fri dates before applying the trick. Or pre-join a business-calendar table and use business_day_number instead of login_date.
Holidays off. Same as weekends — pre-join a holiday calendar and use the working_day_number as the ordered attribute.
Custom activity days. Any dense-integer attribute works. If a user must log in on the 1st, 15th, or 30th of each month to count, encode those dates into a dense sequence first.

Multi-user partitioning.

Always PARTITION BY user_id inside the ROW_NUMBER() window. Without the partition, all users share one row_number and the island ids mix across users.
The GROUP BY after the subtraction must include user_id too — same reason.
The final SELECT always projects user_id alongside the island columns.

Handling duplicates in the source data.

Distinct pre-step. If a user can have multiple rows on the same date, SELECT DISTINCT user_id, login_date FROM logins first. Otherwise, ROW_NUMBER() sees duplicates as separate rows and breaks the lockstep.
DENSE_RANK. An alternative — login_date - DENSE_RANK() OVER (PARTITION BY user_id ORDER BY login_date) — tolerates duplicates because DENSE_RANK assigns the same rank to ties. Same island id for a duplicated date. Read DENSE_RANK when the source may have duplicates.

Common sql gaps and islands date-pattern gotchas.

Wrong ORDER BY. ORDER BY login_date inside ROW_NUMBER produces the right island ids only if the dates are dense. ORDER BY user_id, login_date in a global window is a wrong-answer bug that mixes users.
Missing PARTITION BY. Global row_number across users mixes island ids. Every senior interviewer catches this on the second test case.
Off-by-one on the aggregation. The island start is MIN(login_date); the end is MAX(login_date); the length is MAX - MIN + 1 (inclusive of both endpoints). Reporting MAX - MIN under-counts by one — a classic streak-length bug.

Worked example — dense-date island detection

Detailed explanation. The teaching problem: given a logins(user_id, login_date) table where a user may log in on many consecutive days, find every user's login islands with start date, end date, and length. This is the canonical dense-date interview ask.

Question. Given logins(user_id, login_date) with distinct rows per (user_id, login_date), write a query that returns (user_id, island_start, island_end, island_length) for every login island of every user.

Input.

user_id	login_date
u1	2026-07-01
u1	2026-07-02
u1	2026-07-03
u1	2026-07-06
u1	2026-07-07
u2	2026-07-05
u2	2026-07-06

Code.

WITH numbered AS (
  SELECT
    user_id,
    login_date,
    ROW_NUMBER() OVER (PARTITION BY user_id ORDER BY login_date) AS rn
  FROM logins
),
islanded AS (
  SELECT
    user_id,
    login_date,
    login_date - (rn || ' days')::interval AS island_id
  FROM numbered
)
SELECT
  user_id,
  MIN(login_date)::date AS island_start,
  MAX(login_date)::date AS island_end,
  COUNT(*) AS island_length
FROM islanded
GROUP BY user_id, island_id
ORDER BY user_id, island_start;

Step-by-step explanation.

numbered CTE — per user, order the login dates ascending and assign rn = 1, 2, 3, .... The partition ensures each user gets its own row_number sequence.
islanded CTE — compute island_id = login_date - rn days. For a consecutive run 2026-07-01, 2026-07-02, 2026-07-03 with rn = 1, 2, 3, the island_id is 2026-06-30 for all three rows. The next login on 2026-07-06 with rn = 4 gives island_id = 2026-07-02 — a different island.
Outer SELECT — GROUP BY user_id, island_id collapses each island into one row. MIN(login_date) recovers the first date; MAX(login_date) the last; COUNT(*) the run length.
The ORDER BY user_id, island_start sorts the output for readability. Not strictly required — the caller can sort — but always nice on a whiteboard.
The ::date cast defends against timezone drift when login_date is a timestamp; on a pure date column, it's a no-op.

Output.

user_id	island_start	island_end	island_length
u1	2026-07-01	2026-07-03	3
u1	2026-07-06	2026-07-07	2
u2	2026-07-05	2026-07-06	2

Rule of thumb. Dense-date islands are three CTEs and one GROUP BY. Write the ROW_NUMBER, subtract from the date, group, aggregate — the whole pattern fits on a whiteboard in five minutes.

Worked example — handling weekend gaps (business-day islands)

Detailed explanation. A common follow-up: "your streak query treats weekends as breaks — how do you make weekends not count?" The answer is to compress the dense-date grid down to only business days first, then apply the standard island-id trick against the compressed grid. This is where a pre-joined business-calendar table earns its keep.

Question. Given logins(user_id, login_date) and a business_days(date, business_day_num) calendar table where business_day_num is a dense integer over Mon-Fri only, write a query that returns login islands treating weekends as "not a break."

Input (logins).

user_id	login_date
u1	2026-07-06 (Mon)
u1	2026-07-07 (Tue)
u1	2026-07-10 (Fri)
u1	2026-07-13 (Mon)
u1	2026-07-14 (Tue)

Input (business_days).

date	business_day_num
2026-07-06	1
2026-07-07	2
2026-07-08	3
2026-07-09	4
2026-07-10	5
2026-07-13	6
2026-07-14	7

Code.

WITH joined AS (
  SELECT
    l.user_id,
    l.login_date,
    b.business_day_num
  FROM logins l
  JOIN business_days b ON b.date = l.login_date
),
numbered AS (
  SELECT
    user_id,
    login_date,
    business_day_num,
    ROW_NUMBER() OVER (PARTITION BY user_id ORDER BY login_date) AS rn
  FROM joined
),
islanded AS (
  SELECT
    user_id,
    login_date,
    business_day_num - rn AS island_id
  FROM numbered
)
SELECT
  user_id,
  MIN(login_date)::date AS island_start,
  MAX(login_date)::date AS island_end,
  COUNT(*) AS active_days
FROM islanded
GROUP BY user_id, island_id
ORDER BY user_id, island_start;

Step-by-step explanation.

joined CTE — enrich each login with its business_day_num from the calendar. Weekends are absent from the calendar, so any weekend login (if it exists) is dropped by the join. If you want weekend logins to still count as activity but not break the streak, adjust the join to be a left join and coalesce a synthetic business-day number.
numbered CTE — same as before: per user, order by login_date and produce rn.
islanded CTE — business_day_num - rn is now the island id. Because business_day_num skips weekends, two logins on consecutive business days (Fri and Mon) have business_day_num differing by 1 — the streak continues.
Outer SELECT — group by island_id, take MIN/MAX/COUNT. island_length here is renamed to active_days to emphasise that it counts activity, not calendar days.
Trade-off: the query now depends on a business_days table. Every warehouse should have one — Snowflake's SYSTEM$GENERATE_CALENDAR, dbt's dbt_date package, or a hand-maintained table. If yours doesn't, generate one from generate_series (Postgres) or a recursive CTE (SQL Server).

Output.

user_id	island_start	island_end	active_days
u1	2026-07-06	2026-07-10	3
u1	2026-07-13	2026-07-14	2

Rule of thumb. For any streak or island query where weekends or holidays should not break the run, pre-join a business-day calendar and use the calendar's dense integer as the ordered attribute. The trick is the same; the grid changes.

Worked example — first and last event per island

Detailed explanation. A common variation: instead of aggregating an island to its length, ship the raw first and last events per island for downstream joins. This is what CDC pipelines do to compute type-2 slowly-changing dimension valid_from / valid_to. The trick is the same, plus a FIRST_VALUE / LAST_VALUE window pair to expose the endpoints on every row.

Question. Given logins(user_id, login_date) and a derived island id, write a query that returns every login row enriched with island_start and island_end as extra columns, without collapsing.

Input.

user_id	login_date
u1	2026-07-01
u1	2026-07-02
u1	2026-07-03
u1	2026-07-06

Code.

WITH numbered AS (
  SELECT
    user_id,
    login_date,
    ROW_NUMBER() OVER (PARTITION BY user_id ORDER BY login_date) AS rn
  FROM logins
),
islanded AS (
  SELECT
    user_id,
    login_date,
    login_date - (rn || ' days')::interval AS island_id
  FROM numbered
)
SELECT
  user_id,
  login_date,
  MIN(login_date) OVER (PARTITION BY user_id, island_id)::date AS island_start,
  MAX(login_date) OVER (PARTITION BY user_id, island_id)::date AS island_end
FROM islanded
ORDER BY user_id, login_date;

Step-by-step explanation.

The first two CTEs are identical to the earlier example — produce island_id per row using the row_number-subtracted-from-date trick.
Outer SELECT — replace the GROUP BY + MIN/MAX aggregation with MIN() OVER (PARTITION BY user_id, island_id) and MAX() OVER (...) — window aggregates that don't collapse the row.
Every row now has three interesting columns: its own login_date, the island's start, and the island's end. This shape is exactly what a downstream join wants — you can JOIN ... USING (user_id, island_id) or WHERE login_date = island_end to grab the "last event per island."
The PARTITION BY user_id, island_id inside the window is the critical piece. Without both, either the users mix or the islands mix.
Cost: two OVER clauses over the same partition are typically fused into one window pass by modern planners. On a 100M row table, expect the same runtime as a single-window query.

Output.

user_id	login_date	island_start	island_end
u1	2026-07-01	2026-07-01	2026-07-03
u1	2026-07-02	2026-07-01	2026-07-03
u1	2026-07-03	2026-07-01	2026-07-03
u1	2026-07-06	2026-07-06	2026-07-06

Rule of thumb. When you need the raw rows enriched with island boundaries — CDC, retention join, cohort assignment — swap the GROUP BY + MIN/MAX for MIN() OVER (PARTITION BY ..., island_id) window aggregates. Same island-id logic; different aggregation shape.

Senior interview question on multi-user island detection

A senior interviewer might ask: "Given a logins(user_id, login_date) table with 500M rows and 10M users, write the query that returns every user's longest login streak, and estimate the plan's cost. Show me both the standard Tabibitosan approach and how you'd optimise if the table is that large."

Solution Using per-user island id + aggregate + top-1 join

-- Step 1 — assign island ids per user
WITH numbered AS (
  SELECT
    user_id,
    login_date,
    ROW_NUMBER() OVER (PARTITION BY user_id ORDER BY login_date) AS rn
  FROM logins
),
islanded AS (
  SELECT
    user_id,
    login_date,
    login_date - (rn || ' days')::interval AS island_id
  FROM numbered
),
-- Step 2 — aggregate to island length per (user, island)
per_island AS (
  SELECT
    user_id,
    island_id,
    MIN(login_date)::date AS island_start,
    MAX(login_date)::date AS island_end,
    COUNT(*) AS streak_length
  FROM islanded
  GROUP BY user_id, island_id
),
-- Step 3 — rank per user, keep only the longest island
ranked AS (
  SELECT
    user_id,
    island_start,
    island_end,
    streak_length,
    ROW_NUMBER() OVER (PARTITION BY user_id ORDER BY streak_length DESC, island_end DESC) AS rn
  FROM per_island
)
SELECT
  user_id,
  island_start AS longest_streak_start,
  island_end AS longest_streak_end,
  streak_length AS longest_streak_days
FROM ranked
WHERE rn = 1;

Step-by-step trace.

Step	What runs	Rows out (approx)
1	ROW_NUMBER per user	500M
2	GROUP BY user_id, island_id	~50M islands (assuming ~5 islands/user avg)
3	ROW_NUMBER per user by streak length	50M with rank
4	Filter rn = 1	10M (one per user)

The three-step pattern reduces 500M raw rows down to 10M output rows (one per user) with the longest streak per user. Every step is a standard window-function pass — no self-joins, no cross-joins, no correlated subqueries.

Output:

user_id	longest_streak_start	longest_streak_end	longest_streak_days
u1	2026-07-01	2026-07-03	3
u2	2026-05-15	2026-05-19	5
u3	2026-06-10	2026-06-10	1

Why this works — concept by concept:

PARTITION BY user_id everywhere — every window function in every CTE partitions by user_id. Skipping it mixes users. The row_number, the min/max, and the rank all need the partition.
Aggregation before ranking — the per_island CTE collapses each island to one row before the final ranking. Ranking 500M row-level records is dramatically more expensive than ranking 50M island-level records.
Tiebreaker on island_end — if a user has two islands of the same maximum length, ORDER BY streak_length DESC, island_end DESC keeps the more recent one. Interviewers appreciate seeing a tie-break argument even if the test data doesn't hit it.
ROW_NUMBER = 1 vs LIMIT — for the "top-1 per group" pattern, ROW_NUMBER() OVER (PARTITION BY ...) = 1 is the portable idiom. DISTINCT ON (user_id) is a shorter Postgres-only equivalent; QUALIFY ROW_NUMBER() = 1 is Snowflake/BigQuery shorthand.
Cost — the plan is three window scans + one aggregate. On a 500M-row table with 10M users, the dominant cost is the first ROW_NUMBER (sort-based, O(N log N) per partition). Clustering the table by user_id reduces the sort cost per partition; on Snowflake, CLUSTER BY (user_id, login_date) accelerates automatic clustering to keep this partition-friendly.

SQL
Topic — gaps-and-islands
Gaps and islands drills

Practice →

SQL Topic — window functions (medium) Window function problems (SQL, medium)

Practice →

3. Sessionization with 30-min inactivity

`sessionization sql` is the flagship gaps-and-islands application — group event streams into sessions using LAG + SUM(new_session) OVER

The mental model in one line: sessionization turns an event stream into per-user sessions by flagging rows where the gap to the previous event exceeds an inactivity threshold, then cumulative-summing the flag to produce a per-user session id. Once you say "LAG for the gap, CASE for the flag, SUM OVER for the id," the sessionization interview surface reduces to a three-line window recipe.

The canonical sessionization contract.

Definition. A session is a maximal run of events from one user where consecutive events are separated by no more than the inactivity threshold. The first event of the day starts a new session; a gap greater than the threshold starts another new session.
Standard threshold. 30 minutes. Google Analytics, Segment, Rudderstack, and Snowplow all default to 30. Some product tools default to 60 minutes for lower-frequency apps.
Session id. Historically opaque — a UUID or a per-user monotonically-increasing integer. The integer form is what warehouse SQL usually produces; a UUID form is what real-time systems assign.
Session boundaries. session_start_ts (first event ts), session_end_ts (last event ts), event_count, session_duration_seconds, first_page, last_page, etc.

The three-line pattern.

-- Step 1 — gap to previous event
LAG(event_time) OVER (PARTITION BY user_id ORDER BY event_time) AS prev_ts

-- Step 2 — new-session flag when gap > threshold
CASE
  WHEN event_time - prev_ts > INTERVAL '30 minutes' OR prev_ts IS NULL
    THEN 1 ELSE 0
END AS new_session

-- Step 3 — cumulative-sum the flag to session id
SUM(new_session) OVER (PARTITION BY user_id ORDER BY event_time) AS session_id

Chained together in a CTE or subquery, this is the whole primitive.

Choosing the threshold.

30 min. Google Analytics standard. Good for consumer web and mobile apps where a user goes to lunch or answers a Slack ping.
1 hour. Loose. Good for long-attention apps like video streaming, coding editors, IDEs.
10 minutes. Tight. Good for high-frequency apps like IoT device dashboards or trading terminals.
Custom. Some teams use a data-driven threshold — histogram all inter-event gaps and pick the elbow at the 95th percentile. If your sessions look wrong at 30 min, plot the histogram first.

Timezone and DST handling.

Store event_time as UTC. Sessionization operates on event_time — the timezone of the user rarely matters for session boundaries.
If the ETL runs across a DST fall-back (2 AM twice), sessions can wrap the double hour. Because we compare against UTC, this is invisible — always store UTC and problem solved.
If a downstream report needs "session started before 6 PM local time" — convert session_start_ts to the user's timezone at the report level, not in the sessionization step.

Late-arriving events.

Real-world event streams have late events. A mobile app buffers events offline and flushes when the network returns; a server-side event goes through a queue with variable latency.
Watermark. Warehouse pipelines usually sessionize a rolling window (last 7 days) and re-sessionize on each run. Late events that arrive within the window are captured; older events are dropped.
Idempotent re-sessionization. The window function is deterministic given the input rows — re-running the query produces the same session ids for the same rows. Downstream consumers see stable ids as long as the input is stable.
UUID sessions. If you need stable ids across ETL runs, hash user_id and session_start_ts to a UUID: MD5(user_id || CAST(session_start_ts AS TEXT)). Deterministic, stable, but changes if the session start moves due to late data.

Per-user partitioning is non-negotiable.

Global session id. Wrong. Without PARTITION BY user_id in LAG and SUM(...) OVER, events from different users can "close" each other's sessions.
Per-user session id. Right. PARTITION BY user_id in every window function ensures session ids are computed independently per user.
Cross-user attribution. Some teams want a "global session ordinal" too — the Nth session across the whole system. That's a separate window pass; keep the per-user primitive clean first.

Session-level aggregations.

Once you have session_id, aggregations are straightforward:

SELECT
  user_id,
  session_id,
  MIN(event_time) AS session_start,
  MAX(event_time) AS session_end,
  MAX(event_time) - MIN(event_time) AS session_duration,
  COUNT(*) AS event_count,
  MIN(page_url) FILTER (WHERE event_type = 'pageview') AS landing_page
FROM sessionized
GROUP BY user_id, session_id;

Common sessionization sql interview probes.

"Why LAG and not LEAD?" — LAG references the previous event; the boundary flag fires when the current event is far from the previous one. LEAD would work symmetrically (fire on the last event of a session), but LAG is the canonical direction.
"What if the first event has no previous?" — LAG returns NULL. Handle explicitly in the CASE (OR prev_ts IS NULL) so the first event starts session 0.
"How do you handle multi-user tables?" — PARTITION BY user_id everywhere. The interviewer wants to hear it in the first sentence.
"How do you re-sessionize when late data arrives?" — recompute over a rolling window (7 days or 30 days). Assume late events within the window; drop older ones.
"What's the difference between session id and session UUID?" — the id is a per-user integer, the UUID is a global-unique identifier. Warehouse SQL usually produces the integer; downstream consumers convert to UUID via a deterministic hash.

Worked example — Segment-style event stream sessionization

Detailed explanation. The most-asked sessionization ask in a product-analytics interview: given a Segment-style events(user_id, event_time, event_name, page_url) stream, compute a session_id per user and per-session aggregations. This is the core pipeline any Segment / Rudderstack / Snowplow user has written.

Question. Given events(user_id, event_time, event_name, page_url) and a 30-minute inactivity threshold, produce a sessions table with (user_id, session_id, session_start, session_end, event_count, landing_page). Show both the sessionization CTE and the aggregation.

Input.

user_id	event_time	event_name	page_url
u1	2026-07-01 10:00	page_view	/home
u1	2026-07-01 10:05	click	/home
u1	2026-07-01 10:20	page_view	/pricing
u1	2026-07-01 11:15	page_view	/home
u1	2026-07-01 11:20	click	/home

Code.

WITH flagged AS (
  SELECT
    user_id,
    event_time,
    event_name,
    page_url,
    CASE
      WHEN event_time - LAG(event_time) OVER (PARTITION BY user_id ORDER BY event_time) > INTERVAL '30 minutes'
        OR LAG(event_time) OVER (PARTITION BY user_id ORDER BY event_time) IS NULL
        THEN 1
      ELSE 0
    END AS new_session
  FROM events
),
sessionized AS (
  SELECT
    user_id,
    event_time,
    event_name,
    page_url,
    SUM(new_session) OVER (PARTITION BY user_id ORDER BY event_time) AS session_id
  FROM flagged
)
SELECT
  user_id,
  session_id,
  MIN(event_time) AS session_start,
  MAX(event_time) AS session_end,
  MAX(event_time) - MIN(event_time) AS session_duration,
  COUNT(*) AS event_count,
  MIN(page_url) FILTER (WHERE event_name = 'page_view') AS landing_page
FROM sessionized
GROUP BY user_id, session_id
ORDER BY user_id, session_id;

Step-by-step explanation.

flagged CTE — compute new_session = 1 when the gap to the previous event exceeds 30 minutes, or when it's the first event of the user. The IS NULL branch handles the first event.
sessionized CTE — cumulative-sum the flag per user. Session 0 covers the events before the first gap; session 1 covers the next block; etc.
Final SELECT — group by (user_id, session_id) and aggregate. MIN(event_time) is the session start; MAX(event_time) the end. MIN(page_url) FILTER (WHERE event_name = 'page_view') picks the first pageview URL as the landing page.
The FILTER clause is ANSI standard (Postgres, Snowflake, BigQuery); SQL Server uses MIN(CASE WHEN event_name = 'page_view' THEN page_url END) as the equivalent. Same effect.
Extend with more per-session aggregates as needed — total revenue, exit page, campaign source, etc. The pattern generalises without re-sessionizing.

Output.

user_id	session_id	session_start	session_end	session_duration	event_count	landing_page
u1	0	2026-07-01 10:00	2026-07-01 10:20	00:20:00	3	/home
u1	1	2026-07-01 11:15	2026-07-01 11:20	00:05:00	2	/home

Rule of thumb. Sessionization is one CTE for the flag, one CTE for the id, one SELECT for the aggregation. Three tiny queries; one whole pipeline.

Worked example — configurable-threshold sessionization macro

Detailed explanation. In a real analytics stack, the threshold is a parameter — 30 min for consumer apps, 60 min for long-attention apps, 10 min for high-frequency apps. A dbt macro takes the threshold as an argument and generates the correct SQL.

Question. Write a dbt macro sessionize(events_relation, user_col, ts_col, threshold_minutes=30) that emits sessionization SQL parameterised on the threshold. Show the macro and one invocation.

Input. (Same as above — user_id, event_time, event_name, page_url.)

Code.

-- macros/sessionize.sql
{% macro sessionize(events_relation, user_col='user_id', ts_col='event_time', threshold_minutes=30) %}
  WITH flagged AS (
    SELECT
      *,
      CASE
        WHEN {{ ts_col }} - LAG({{ ts_col }}) OVER (PARTITION BY {{ user_col }} ORDER BY {{ ts_col }})
             > INTERVAL '{{ threshold_minutes }} minutes'
          OR LAG({{ ts_col }}) OVER (PARTITION BY {{ user_col }} ORDER BY {{ ts_col }}) IS NULL
          THEN 1
        ELSE 0
      END AS new_session
    FROM {{ events_relation }}
  )
  SELECT
    *,
    SUM(new_session) OVER (PARTITION BY {{ user_col }} ORDER BY {{ ts_col }}) AS session_id
  FROM flagged
{% endmacro %}

-- usage in a model
WITH sessionized AS (
  {{ sessionize(ref('events'), 'user_id', 'event_time', 30) }}
)
SELECT
  user_id,
  session_id,
  MIN(event_time) AS session_start,
  MAX(event_time) AS session_end,
  COUNT(*) AS event_count
FROM sessionized
GROUP BY user_id, session_id;

Step-by-step explanation.

The macro accepts the events relation, the user column name, the timestamp column name, and the threshold in minutes. Defaults match the Segment convention.
Inside the macro, the sessionization primitive is the same LAG + CASE + SUM OVER pattern. The threshold is templated into the INTERVAL '{{ threshold_minutes }} minutes' clause.
SELECT * inside the CTE forwards every downstream column — event_name, page_url, revenue, etc. — through the sessionization pass. Callers don't have to enumerate columns.
Usage — call {{ sessionize(ref('events'), ...) }} inside a model. The compiled SQL is a plain CTE that any warehouse can execute.
Adapting for BigQuery — replace INTERVAL '{{ threshold_minutes }} minutes' with INTERVAL {{ threshold_minutes }} MINUTE. Add a dispatch on target.type to pick the right syntax for every warehouse.

Output. (Same as the raw sessionization query — one row per (user, session) pair.)

Rule of thumb. Parameterise the threshold from day one. Hard-coding 30 minutes leads to code duplication when a new app onboards with a different threshold. A macro pays for itself the first time you need a second value.

Worked example — session close on explicit sign-out

Detailed explanation. Some products close sessions on explicit user actions — sign-out, tab-close, "end conversation." The pattern extends the boundary flag to include the explicit close: any row where event_name = 'sign_out' starts a new session on the next event.

Question. Given events(user_id, event_time, event_name) where event_name = 'sign_out' explicitly ends a session, write sessionization SQL that respects both the 30-min inactivity rule and the sign-out event.

Input.

user_id	event_time	event_name
u1	10:00	page_view
u1	10:05	sign_out
u1	10:07	page_view
u1	10:10	click

Code.

WITH flagged AS (
  SELECT
    user_id,
    event_time,
    event_name,
    CASE
      WHEN event_time - LAG(event_time) OVER (PARTITION BY user_id ORDER BY event_time) > INTERVAL '30 minutes'
        OR LAG(event_time) OVER (PARTITION BY user_id ORDER BY event_time) IS NULL
        OR LAG(event_name) OVER (PARTITION BY user_id ORDER BY event_time) = 'sign_out'
        THEN 1
      ELSE 0
    END AS new_session
  FROM events
)
SELECT
  user_id,
  event_time,
  event_name,
  SUM(new_session) OVER (PARTITION BY user_id ORDER BY event_time) AS session_id
FROM flagged
ORDER BY user_id, event_time;

Step-by-step explanation.

The CASE now has three branches — inactivity gap, first event, and "previous event was sign_out." Any of them flags a new session.
LAG(event_name) looks at the previous event's name. If it was sign_out, the current event starts a new session — even if the gap to the previous is 2 minutes.
Note that the sign_out event itself belongs to the ending session, not the new one. It's the next event after sign_out that opens a new session.
If the product wants "sign_out ends the session and the next event opens a new one immediately," this is correct. If it wants "sign_out itself is the last event of the ending session but the next event stays in the same session," drop the LAG branch.
Generalise: any explicit boundary event (page_close, conversation_end, "start new task") can be modelled by adding a LAG check on the boundary event name.

Output.

user_id	event_time	event_name	session_id
u1	10:00	page_view	0
u1	10:05	sign_out	0
u1	10:07	page_view	1
u1	10:10	click	1

Rule of thumb. For any product with explicit session-close semantics, add a LAG(event_name) = 'boundary_event' check to the new-session flag. The rest of the pipeline stays the same.

Senior interview question on session-level attribution

A senior interviewer might ask: "Given a Segment event stream, sessionize with 30-min inactivity, and then attribute the last-touch campaign per session and per user. Walk me through the query and explain how you'd handle a user whose last-touch campaign came from a session 30 days ago."

Solution Using sessionize + last-touch join + campaign attribution

WITH flagged AS (
  SELECT
    user_id,
    event_time,
    event_name,
    utm_campaign,
    CASE
      WHEN event_time - LAG(event_time) OVER (PARTITION BY user_id ORDER BY event_time) > INTERVAL '30 minutes'
        OR LAG(event_time) OVER (PARTITION BY user_id ORDER BY event_time) IS NULL
        THEN 1
      ELSE 0
    END AS new_session
  FROM events
),
sessionized AS (
  SELECT
    user_id,
    event_time,
    event_name,
    utm_campaign,
    SUM(new_session) OVER (PARTITION BY user_id ORDER BY event_time) AS session_id
  FROM flagged
),
per_session AS (
  SELECT
    user_id,
    session_id,
    MIN(event_time) AS session_start,
    MAX(event_time) AS session_end,
    COUNT(*) AS event_count,
    -- Last touch = the last non-null utm_campaign in the session
    (ARRAY_AGG(utm_campaign ORDER BY event_time DESC)
      FILTER (WHERE utm_campaign IS NOT NULL))[1] AS last_touch_campaign
  FROM sessionized
  GROUP BY user_id, session_id
),
per_user_last_touch AS (
  SELECT
    user_id,
    session_id,
    last_touch_campaign,
    -- Fill the last-touch forward across sessions
    LAST_VALUE(last_touch_campaign IGNORE NULLS) OVER (
      PARTITION BY user_id
      ORDER BY session_start
      ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW
    ) AS user_last_touch_campaign,
    session_start,
    session_end,
    event_count
  FROM per_session
)
SELECT * FROM per_user_last_touch
ORDER BY user_id, session_start;

Step-by-step trace.

Step	What runs	Output shape
1	Sessionize the event stream	Same rows plus session_id
2	Per-session aggregate	One row per (user, session) with last_touch_campaign
3	Forward-fill last-touch across sessions	Every session sees its user's most recent non-null campaign
4	Final SELECT	Session-level output with both session and user attribution

The three-step attribution pipeline sessionizes, aggregates to session-level, then forward-fills the last non-null campaign across a user's session history. The forward-fill is a LAST_VALUE IGNORE NULLS — the same trick that GA's last-non-direct-click attribution uses.

Output:

user_id	session_id	last_touch_campaign	user_last_touch_campaign	session_start
u1	0	google_ads	google_ads	2026-07-01
u1	1	NULL	google_ads	2026-07-05
u1	2	facebook_ads	facebook_ads	2026-07-10

Why this works — concept by concept:

Sessionize once, aggregate once, forward-fill once — the three-step pipeline mirrors the three-step gaps-and-islands universal template. Every extension (attribution, funnel, retention) reads the sessionized shape and adds one aggregation step.
FILTER + ARRAY_AGG for last non-null — the ARRAY_AGG(... ORDER BY ... DESC) FILTER (WHERE ... IS NOT NULL) trick picks the most recent non-null value inside a session. Portable to Postgres, Snowflake, BigQuery. SQL Server uses MAX(...) OVER (ORDER BY ...) with a case statement equivalent.
LAST_VALUE IGNORE NULLS for forward fill — the ANSI standard for "carry the last non-null value forward" — Postgres 16+, Snowflake, BigQuery, SQL Server 2022+ all ship it. The ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW frame is required — without it, LAST_VALUE reads the whole partition and returns the last row's value instead of the running last.
Cross-session state without joins — the LAST_VALUE OVER primitive replaces a self-join that would otherwise be O(N²). Window functions are the analytics engineer's go-to when a self-join is tempting.
Cost — three window passes plus one aggregate. On a 1B-row event table with 10M users, the sessionization pass is dominant (O(N log N)); the aggregation is O(N); the forward-fill is O(N). Total plan: a few window scans, no joins, very cache-friendly. Modern warehouses run this in single-digit minutes on billion-row inputs.

SQL
Topic — time-series
Time-series analytics drills

Practice →

SQL Topic — interval processing Interval processing problems

Practice →

4. Consecutive-day streaks

`sql streaks` — the LeetCode-flavoured "longest consecutive-day streak per user" problem, plus current-streak framing and gap tolerance

The mental model in one line: a consecutive-day streak is a dense-date island where the ordered attribute is the login date and the aggregate is COUNT(*); the "longest streak" is the max per user and the "current streak" is the island whose end date is today. Once you say "streaks = date islands + per-user max," the streak interview surface reduces to the standard Tabibitosan template plus one extra ranking step.

The two streak questions.

Longest streak per user. Given all-time logins, what's the longest consecutive-day run each user has ever had? Reported on user profiles ("your longest streak: 47 days").
Current streak per user. Given today's date, is the user still on a streak, and if so, how long is it? Powered by the daily-open notification that says "keep your 47-day streak alive."

The three-step recipe.

-- Step 1 — dense-date island id
island_id = login_date - ROW_NUMBER() OVER (PARTITION BY user_id ORDER BY login_date) * INTERVAL '1 day'

-- Step 2 — per-island aggregates
GROUP BY user_id, island_id
  → island_start = MIN(login_date), island_end = MAX(login_date), length = COUNT(*)

-- Step 3 — pick the longest or the current
longest → ROW_NUMBER() OVER (PARTITION BY user_id ORDER BY length DESC) = 1
current → WHERE island_end = CURRENT_DATE

Gap tolerance — the "one-day miss" allowed variant.

Sometimes a product allows a 1-day miss without breaking the streak — "you can miss one day per week." The recipe extends:

-- Flag "gap > tolerance" instead of "gap > 1"
CASE
  WHEN login_date - LAG(login_date) OVER (PARTITION BY user_id ORDER BY login_date) > INTERVAL '2 days'
    OR LAG(login_date) IS NULL
    THEN 1
  ELSE 0
END AS new_streak

Then cumulative-sum the flag to get an "extended island id," and aggregate as before. The threshold is the max allowed gap; 2 days tolerates a 1-day miss.

Holiday tolerance — the calendar-join variant.

Pre-join a business-calendar table with is_holiday and is_active_day flags. Filter login candidates to only active days before applying the island trick. Or use the calendar's active_day_num as the ordered attribute (as in Section 2's business-day example).
For "streak counts if the user is active on every non-holiday" — join the calendar into logins and check that every active day is present.

Current streak — the "streak is still alive today" check.

After computing per-island aggregates, filter to island_end IN (CURRENT_DATE, CURRENT_DATE - INTERVAL '1 day').
Including yesterday accounts for timezone edge cases — some users' "today" is still yesterday in UTC.
The user has an active streak if the filtered row exists; the streak length is the island's length.

Streak-end timestamp for retention analysis.

Growth teams want to know when users' streaks ended, not just their length. island_end is exactly this.
For churn cohort analysis, group users by island_end week: "how many users ended a streak in week N?" is a leading indicator of churn.
For "days since last streak," subtract island_end from CURRENT_DATE. Common in dashboards.

Common sql streaks interview probes.

"What's the difference between streak and session?" — streaks are typically per-day granularity; sessions are per-minute. Same island-id trick, different ordered attribute.
"How do you tolerate a 1-day miss?" — extend the boundary threshold to 2 days (or use the flag-and-sum flavour with gap > tolerance).
"How do you handle holidays?" — pre-join a business calendar; use working_day_num as the ordered attribute instead of raw dates.
"Current streak vs longest streak — same query?" — same island-id primitive; different filter/rank at the end. Current filters to island_end >= CURRENT_DATE - 1; longest picks the max per user.
"Streaks across multiple activity types?" — union the activities into a single "was_active_today" fact table, then apply the streak recipe.

Worked example — longest streak per user

Detailed explanation. The canonical LeetCode-style ask: given a logins(user_id, login_date) table with distinct rows per (user_id, login_date), return each user's longest consecutive-day streak with start, end, and length.

Question. Given logins(user_id, login_date) distinct per (user, date), write a query that returns (user_id, longest_streak_start, longest_streak_end, longest_streak_length) per user.

Input.

user_id	login_date
u1	2026-07-01
u1	2026-07-02
u1	2026-07-03
u1	2026-07-06
u1	2026-07-07
u2	2026-07-05
u2	2026-07-06
u2	2026-07-07
u2	2026-07-08
u2	2026-07-09

Code.

WITH numbered AS (
  SELECT
    user_id,
    login_date,
    ROW_NUMBER() OVER (PARTITION BY user_id ORDER BY login_date) AS rn
  FROM logins
),
islanded AS (
  SELECT
    user_id,
    login_date,
    login_date - (rn || ' days')::interval AS island_id
  FROM numbered
),
per_island AS (
  SELECT
    user_id,
    island_id,
    MIN(login_date)::date AS streak_start,
    MAX(login_date)::date AS streak_end,
    COUNT(*) AS streak_length
  FROM islanded
  GROUP BY user_id, island_id
),
ranked AS (
  SELECT
    user_id,
    streak_start,
    streak_end,
    streak_length,
    ROW_NUMBER() OVER (
      PARTITION BY user_id
      ORDER BY streak_length DESC, streak_end DESC
    ) AS rn
  FROM per_island
)
SELECT
  user_id,
  streak_start AS longest_streak_start,
  streak_end AS longest_streak_end,
  streak_length AS longest_streak_length
FROM ranked
WHERE rn = 1
ORDER BY user_id;

Step-by-step explanation.

numbered CTE — per user, order by login_date and assign rn. Same primitive as Section 2.
islanded CTE — compute island_id = login_date - rn days. Consecutive logins share the same island_id.
per_island CTE — group by (user_id, island_id) and aggregate to streak start/end/length.
ranked CTE — per user, rank the islands by length descending, tie-break on end date. ROW_NUMBER() = 1 picks the longest streak.
Final SELECT — filter to the winners and rename for clarity. Output is one row per user.

Output.

user_id	longest_streak_start	longest_streak_end	longest_streak_length
u1	2026-07-01	2026-07-03	3
u2	2026-07-05	2026-07-09	5

Rule of thumb. Longest streak = island-id trick + ROW_NUMBER() = 1 per user. Four CTEs, one output row per user. This is the answer any LeetCode "consecutive numbers" streak problem wants.

Worked example — current streak per user

Detailed explanation. The "streak is still alive today" variant: filter the per-island aggregates to those whose island_end is today or yesterday (accounting for timezone). Users without such an island have no current streak.

Question. Given the same logins table and today's date, return every user's current streak (start, end, length). Users with no current streak should not appear in the output.

Input.

user_id	login_date
u1	2026-07-08
u1	2026-07-09
u1	2026-07-10
u2	2026-07-05
u2	2026-07-06
u3	2026-07-10

Assume today is 2026-07-10.

Code.

WITH numbered AS (
  SELECT
    user_id,
    login_date,
    ROW_NUMBER() OVER (PARTITION BY user_id ORDER BY login_date) AS rn
  FROM logins
),
islanded AS (
  SELECT
    user_id,
    login_date,
    login_date - (rn || ' days')::interval AS island_id
  FROM numbered
),
per_island AS (
  SELECT
    user_id,
    island_id,
    MIN(login_date)::date AS streak_start,
    MAX(login_date)::date AS streak_end,
    COUNT(*) AS streak_length
  FROM islanded
  GROUP BY user_id, island_id
)
SELECT
  user_id,
  streak_start AS current_streak_start,
  streak_end AS current_streak_end,
  streak_length AS current_streak_length
FROM per_island
WHERE streak_end >= CURRENT_DATE - INTERVAL '1 day'
ORDER BY user_id;

Step-by-step explanation.

First three CTEs are identical to the "longest streak" query — compute per-island aggregates.
The filter WHERE streak_end >= CURRENT_DATE - INTERVAL '1 day' picks islands that ended today or yesterday. Yesterday accounts for timezone edge cases where a user's "today" may still be yesterday in UTC.
Users who haven't logged in recently have no matching island — they drop out of the output.
If a user has multiple islands ending in the window (unusual, but possible if the data is weird), the query returns both. In practice, a user has at most one current streak; add a ROW_NUMBER = 1 ranking step if you want to be defensive.
For "current streak = 1 day" (user logged in only today), the same query returns a length-1 island. That's usually the right answer for streak-based retention.

Output.

user_id	current_streak_start	current_streak_end	current_streak_length
u1	2026-07-08	2026-07-10	3
u3	2026-07-10	2026-07-10	1

Rule of thumb. Current streak = same island primitive + filter to island_end >= CURRENT_DATE - 1. No extra ranking needed — the filter alone selects the user's active streak.

Worked example — streak with 1-day-miss tolerance

Detailed explanation. A common product ask: "let users miss one day without breaking the streak." The recipe swaps the dense-date subtraction for the flag-and-sum flavour with a 2-day boundary.

Question. Given logins(user_id, login_date), allow up to a 1-day miss between consecutive logins and return the longest tolerant-streak per user.

Input.

user_id	login_date
u1	2026-07-01
u1	2026-07-02
u1	2026-07-04
u1	2026-07-05
u1	2026-07-09

Code.

WITH flagged AS (
  SELECT
    user_id,
    login_date,
    CASE
      WHEN login_date - LAG(login_date) OVER (PARTITION BY user_id ORDER BY login_date) > INTERVAL '2 days'
        OR LAG(login_date) OVER (PARTITION BY user_id ORDER BY login_date) IS NULL
        THEN 1
      ELSE 0
    END AS new_streak
  FROM logins
),
streaked AS (
  SELECT
    user_id,
    login_date,
    SUM(new_streak) OVER (PARTITION BY user_id ORDER BY login_date) AS streak_id
  FROM flagged
),
per_streak AS (
  SELECT
    user_id,
    streak_id,
    MIN(login_date)::date AS streak_start,
    MAX(login_date)::date AS streak_end,
    COUNT(*) AS login_count,
    (MAX(login_date) - MIN(login_date))::int + 1 AS calendar_span
  FROM streaked
  GROUP BY user_id, streak_id
),
ranked AS (
  SELECT
    user_id,
    streak_start,
    streak_end,
    login_count,
    calendar_span,
    ROW_NUMBER() OVER (
      PARTITION BY user_id
      ORDER BY calendar_span DESC
    ) AS rn
  FROM per_streak
)
SELECT
  user_id,
  streak_start,
  streak_end,
  login_count,
  calendar_span
FROM ranked
WHERE rn = 1
ORDER BY user_id;

Step-by-step explanation.

Because we're allowing 1-day misses, the dense-date subtraction trick no longer works (a 1-day miss changes the date - rn value). Switch to the flag-and-sum flavour.
flagged CTE — flag new_streak = 1 when the gap to the previous login exceeds 2 days. A 1-day gap (miss one day) stays inside the streak.
streaked CTE — cumulative-sum the flag per user to produce the tolerant streak_id.
per_streak CTE — aggregate to login_count (number of actual logins in the streak) and calendar_span (span from first to last day, inclusive). The two can differ — a tolerant streak of 10 calendar days may only have 8 logins.
ranked CTE — pick the streak with the longest calendar_span per user. If you'd rather rank by login_count, swap the ORDER BY.

Output.

user_id	streak_start	streak_end	login_count	calendar_span
u1	2026-07-01	2026-07-05	4	5

Rule of thumb. Tolerant streaks use the flag-and-sum flavour with a gap threshold. Reporting both login_count and calendar_span gives product a choice of streak definition without re-running the pipeline.

Senior interview question on streak analytics at scale

A senior interviewer might ask: "You run a language-learning app with 50M users and 10B daily-activity rows. Design a nightly pipeline that publishes each user's (longest_streak, current_streak, days_since_last_streak_end) so it's ready for the daily push notification. Where do the joins happen and what does the plan cost?"

Solution Using a two-stage nightly pipeline with materialised streaks

-- STAGE 1 (nightly) — materialise per-user streaks table
CREATE TABLE user_streaks AS
WITH numbered AS (
  SELECT
    user_id,
    activity_date,
    ROW_NUMBER() OVER (PARTITION BY user_id ORDER BY activity_date) AS rn
  FROM daily_activity
),
islanded AS (
  SELECT
    user_id,
    activity_date,
    activity_date - (rn || ' days')::interval AS island_id
  FROM numbered
),
per_island AS (
  SELECT
    user_id,
    island_id,
    MIN(activity_date)::date AS streak_start,
    MAX(activity_date)::date AS streak_end,
    COUNT(*) AS streak_length
  FROM islanded
  GROUP BY user_id, island_id
)
SELECT * FROM per_island;

-- STAGE 2 (each morning) — build the push-notification-ready table
CREATE TABLE user_streak_summary AS
WITH by_user AS (
  SELECT
    user_id,
    MAX(streak_length) AS longest_streak,
    MAX(CASE WHEN streak_end >= CURRENT_DATE - INTERVAL '1 day' THEN streak_length END) AS current_streak,
    MAX(streak_end) AS last_streak_end
  FROM user_streaks
  GROUP BY user_id
)
SELECT
  user_id,
  longest_streak,
  COALESCE(current_streak, 0) AS current_streak,
  (CURRENT_DATE - last_streak_end)::int AS days_since_last_streak_end
FROM by_user;

Step-by-step trace.

Stage	What runs	Approx cost on 10B rows
1	Compute per-user islands (window scan + GROUP BY)	~90% of pipeline cost
2	Aggregate to per-user summary	~5% of pipeline cost
2b	Join summary to push service via user_id	~5% for the export step

Stage 1 is where the heavy lifting happens; it's a nightly job and gets 30 minutes to run on a warehouse cluster. Stage 2 is cheap and can run in seconds after Stage 1 lands. The push service reads Stage 2 by user_id at delivery time.

Output:

user_id	longest_streak	current_streak	days_since_last_streak_end
u1	47	3	0
u2	12	0	5
u3	100	0	30

Why this works — concept by concept:

Split the pipeline into materialise + summarise — the expensive island detection runs nightly and lands in a table; the cheap per-user aggregation runs each morning and lands in the push-service-ready table. This split avoids re-computing islands every morning.
Idempotent materialisation — the user_streaks table can be rebuilt on any night without breaking downstream consumers; the island ids are deterministic given the input data. If the pipeline fails, you re-run and get the same output.
COALESCE for the "no current streak" case — the MAX(CASE WHEN ...) returns NULL for users without a current streak; COALESCE(current_streak, 0) fills in zero. The push notification templates check current_streak > 0 to decide whether to send.
days_since_last_streak_end for retention — the delta from today to the user's last streak end is a leading indicator of churn. Growth teams filter to days_since_last_streak_end BETWEEN 3 AND 7 and target re-engagement campaigns.
Cost — Stage 1 is O(N log N) per user for the window sort. On 10B rows with 50M users, this is ~1B operations per partition summed over partitions — 15-30 minutes on a well-clustered Snowflake medium warehouse. Stage 2 is O(unique users) — seconds. Overall, the pipeline hits the daily push cutoff comfortably.

SQL
Topic — window functions (hard)
Window function problems (SQL, hard)

Practice →

SQL Topic — sliding window Sliding window drills

Practice →

5. Run-length encoding + dialect matrix

`sql run length encoding` compresses a state series into runs — the `SUM(CASE WHEN prev <> curr) OVER` primitive, plus the MATCH_RECOGNIZE alternative for Snowflake and Oracle

The mental model in one line: RLE compresses a sequence of state values into one row per contiguous run using the same LAG + SUM-of-change-flag primitive as sessionization; when the warehouse ships MATCH_RECOGNIZE (Snowflake, Oracle), you get a declarative alternative that reads like a regex over rows. Once you say "RLE = flag change points + cumulative sum, or MATCH_RECOGNIZE PATTERN," the RLE interview surface reduces to a dialect lookup.

The RLE primitive.

-- Given (t, status) ordered by t, compress consecutive same-status rows into runs
WITH flagged AS (
  SELECT
    t,
    status,
    CASE
      WHEN status <> LAG(status) OVER (ORDER BY t)
        OR LAG(status) OVER (ORDER BY t) IS NULL
        THEN 1
      ELSE 0
    END AS is_new_run
  FROM series
),
grouped AS (
  SELECT
    t,
    status,
    SUM(is_new_run) OVER (ORDER BY t) AS run_id
  FROM flagged
)
SELECT
  status,
  MIN(t) AS run_start,
  MAX(t) AS run_end,
  COUNT(*) AS run_length
FROM grouped
GROUP BY status, run_id
ORDER BY run_start;

Three CTEs, one output row per run. This is the portable answer that works on every warehouse.

Where RLE lands in pipelines.

Status series compression. Sensor readings, service uptime, feature-flag states — sequences with many consecutive same-value rows. Storage cost drops 10-100×.
Price plateaus. Slow-moving prices (real-estate listings, insurance quotes) compress dramatically. Financial modelling often carries prices as runs.
Feature-flag audit logs. "This flag was on for user X from t1 to t2, then off from t2 to t3." Downstream A/B analysis reads this shape.
Slowly changing dimensions (SCD-2). Type-2 slowly-changing dimensions store one row per contiguous state — that's RLE on the SCD-2 output.
CDC compaction. Debezium and Fivetran ship one row per change event; downstream compaction collapses consecutive unchanged rows into RLE runs.

MATCH_RECOGNIZE — the declarative alternative.

-- Snowflake / Oracle syntax
SELECT * FROM series
MATCH_RECOGNIZE (
  ORDER BY t
  MEASURES
    FIRST(t) AS run_start,
    LAST(t) AS run_end,
    COUNT(*) AS run_length,
    A.status AS status
  ONE ROW PER MATCH
  PATTERN (A B*)
  DEFINE
    A AS TRUE,
    B AS B.status = A.status
);

Reads like a regex over the row sequence: match one row A, then zero or more rows B where B.status = A.status. Each match emits one output row.
MEASURES block declares the per-match columns.
ONE ROW PER MATCH — one row per run. ALL ROWS PER MATCH would preserve the raw rows.

MATCH_RECOGNIZE dialect matrix.

Warehouse	MATCH_RECOGNIZE support
Oracle 12c+	YES — full ANSI
Snowflake	YES — full ANSI (since 2021)
Postgres	NO (as of 2026) — use LAG + SUM(new_group) OVER
BigQuery	NO — use LAG + SUM
SQL Server	NO — use LAG + SUM
MySQL 8	NO — use LAG + SUM
Redshift	NO — use LAG + SUM

Choosing between the two.

Portable code. Use LAG + SUM OVER. Works on every warehouse; readable; easy to reason about.
Complex multi-row patterns. Use MATCH_RECOGNIZE. Patterns like "A then B+ then C" (one A, one or more B, one C) are trivial in MATCH_RECOGNIZE and painful in window functions.
Regex-like row patterns. Use MATCH_RECOGNIZE. Financial fraud detection, sequence pattern mining, click-stream funnel matching are the classic use cases.
Simple RLE. Either works. Prefer LAG + SUM for portability.

Postgres-specific extension: RANGE INCLUSIVE.

Postgres ships a range-type extension that can store (status, tsrange) directly. range_agg(tsrange(t, t + INTERVAL '1 day')) collapses consecutive same-status rows into a range set.
Concise for storage but non-portable — only Postgres and a few others support ranges.

Snowflake-specific extension: AUTOMATIC_CLUSTERING on the RLE output.

Materialise the RLE output as a table and cluster on the state column. Queries like "get all runs where status = 'error' in the last month" become micro-partition-pruned.
Common in reliability dashboards where 99% of runs are ok and 1% are error.

Common sql run length encoding interview probes.

"Explain the SUM(CASE WHEN prev <> curr) trick." — LAG produces the previous value; CASE flags the change point with a 1; SUM OVER accumulates the flags into a run id.
"Handle the first row where LAG is NULL." — extra branch in the CASE: OR LAG(...) IS NULL THEN 1. The first row always starts a run.
"Which warehouses support MATCH_RECOGNIZE?" — Oracle 12c+, Snowflake. Not Postgres, BigQuery, SQL Server, MySQL, Redshift.
"What's the compression ratio?" — depends on the value stability. A series where 90% of rows are duplicates compresses ~10×. Slowly-changing prices compress 50-100×.
"How do you handle nulls in the state column?" — treat NULL as its own value (IS NOT DISTINCT FROM comparison) or exclude NULLs entirely. Every downstream consumer's answer differs.

Worked example — feature-flag rollout compression

Detailed explanation. A feature-flag audit table logs one row per second per user showing the flag state. Most seconds have the same state as the previous second. RLE compresses to one row per (user, state, interval), cutting storage by 10-100×.

Question. Given flag_log(user_id, log_time, flag_state) with per-second rows, compress to one row per (user_id, flag_state, run) with run start / end / length.

Input.

user_id	log_time	flag_state
u1	10:00:01	off
u1	10:00:02	off
u1	10:00:03	on
u1	10:00:04	on
u1	10:00:05	on
u1	10:00:06	off

Code.

WITH flagged AS (
  SELECT
    user_id,
    log_time,
    flag_state,
    CASE
      WHEN flag_state <> LAG(flag_state) OVER (PARTITION BY user_id ORDER BY log_time)
        OR LAG(flag_state) OVER (PARTITION BY user_id ORDER BY log_time) IS NULL
        THEN 1
      ELSE 0
    END AS is_new_run
  FROM flag_log
),
grouped AS (
  SELECT
    user_id,
    log_time,
    flag_state,
    SUM(is_new_run) OVER (PARTITION BY user_id ORDER BY log_time) AS run_id
  FROM flagged
)
SELECT
  user_id,
  flag_state,
  MIN(log_time) AS run_start,
  MAX(log_time) AS run_end,
  COUNT(*) AS run_length_seconds
FROM grouped
GROUP BY user_id, flag_state, run_id
ORDER BY user_id, run_start;

Step-by-step explanation.

flagged CTE — flag is_new_run = 1 when the current flag_state differs from the previous row's, or when there's no previous row (first row per user). The PARTITION BY user_id scoped LAG ensures per-user comparison.
grouped CTE — cumulative-sum is_new_run per user to produce run_id. Every 1 increments the id; every 0 preserves it.
Outer SELECT — group by (user_id, flag_state, run_id). Including flag_state in the group is defensive; the run_id already implies the state, but including it keeps the query readable and avoids surprises if the CASE ever fires incorrectly.
Aggregations — MIN(log_time) is the run start, MAX(log_time) the end, COUNT(*) the run length in seconds. For minute-granularity, divide COUNT(*) by 60.
Compression ratio — the input has one row per second per user; the output has one row per state change per user. If the average state duration is 10 minutes, compression is ~600× per user.

Output.

user_id	flag_state	run_start	run_end	run_length_seconds
u1	off	10:00:01	10:00:02	2
u1	on	10:00:03	10:00:05	3
u1	off	10:00:06	10:00:06	1

Rule of thumb. Every slowly-changing state series is a RLE candidate. Compression ratio equals the average run length; storage cost drops proportionally. Materialise the compressed shape as a downstream table for cheap analytical reads.

Worked example — MATCH_RECOGNIZE on Snowflake

Detailed explanation. Snowflake ships full ANSI MATCH_RECOGNIZE. The RLE query above compresses to a shorter, more readable form using PATTERN and DEFINE clauses. This is the Snowflake-native answer to the same question.

Question. On Snowflake, write the same feature-flag RLE query using MATCH_RECOGNIZE.

Input. (Same as above.)

Code.

SELECT
  user_id,
  flag_state,
  run_start,
  run_end,
  run_length
FROM flag_log
MATCH_RECOGNIZE (
  PARTITION BY user_id
  ORDER BY log_time
  MEASURES
    FIRST(log_time)  AS run_start,
    LAST(log_time)   AS run_end,
    A.flag_state     AS flag_state,
    COUNT(*)         AS run_length
  ONE ROW PER MATCH
  PATTERN (A B*)
  DEFINE
    A AS TRUE,
    B AS B.flag_state = A.flag_state
);

Step-by-step explanation.

PARTITION BY user_id — sessionize each user independently, same as the window functions do.
ORDER BY log_time — establish the row order to match against.
PATTERN (A B*) — match one A row, then zero or more B rows. Every run starts with an A and continues with Bs.
DEFINE A AS TRUE, B AS B.flag_state = A.flag_state — A matches every row unconditionally; B matches the next row only if its flag_state equals A.flag_state — i.e., the run continues while the state is unchanged.
MEASURES — declare the columns for each match. FIRST(log_time) is the run start; LAST(log_time) the end; A.flag_state the state; COUNT(*) the length.

Output. (Same as the LAG + SUM query.)

Rule of thumb. MATCH_RECOGNIZE is the Snowflake-native RLE syntax; the LAG + SUM version is portable. For a Snowflake-only codebase, MATCH_RECOGNIZE reads more clearly. For a multi-warehouse dbt project, stick with LAG + SUM.

Worked example — price plateau compression across dialects

Detailed explanation. A price series with slow changes compresses dramatically under RLE. This example writes the same query in the two flavours (portable and Snowflake) so a candidate can compare readability.

Question. Given prices(symbol, ts, price) with per-minute rows and mostly-stable prices, compress to one row per (symbol, price, run) with run start / end / length. Show both the portable and Snowflake versions.

Input.

symbol	ts	price
AAPL	09:30	190
AAPL	09:31	190
AAPL	09:32	191
AAPL	09:33	191
AAPL	09:34	191
AAPL	09:35	190

Code.

-- Portable — LAG + SUM (works on every warehouse)
WITH flagged AS (
  SELECT
    symbol,
    ts,
    price,
    CASE
      WHEN price <> LAG(price) OVER (PARTITION BY symbol ORDER BY ts)
        OR LAG(price) OVER (PARTITION BY symbol ORDER BY ts) IS NULL
        THEN 1
      ELSE 0
    END AS is_new_run
  FROM prices
),
grouped AS (
  SELECT
    symbol,
    ts,
    price,
    SUM(is_new_run) OVER (PARTITION BY symbol ORDER BY ts) AS run_id
  FROM flagged
)
SELECT
  symbol,
  price,
  MIN(ts) AS run_start,
  MAX(ts) AS run_end,
  COUNT(*) AS run_length_minutes
FROM grouped
GROUP BY symbol, price, run_id
ORDER BY symbol, run_start;

-- Snowflake — MATCH_RECOGNIZE (shorter, less portable)
SELECT
  symbol,
  price,
  run_start,
  run_end,
  run_length_minutes
FROM prices
MATCH_RECOGNIZE (
  PARTITION BY symbol
  ORDER BY ts
  MEASURES
    FIRST(ts) AS run_start,
    LAST(ts)  AS run_end,
    A.price   AS price,
    COUNT(*)  AS run_length_minutes
  ONE ROW PER MATCH
  PATTERN (A B*)
  DEFINE
    A AS TRUE,
    B AS B.price = A.price
);

Step-by-step explanation.

Both queries produce the same output. The portable version reads any warehouse; the MATCH_RECOGNIZE version reads Snowflake or Oracle only.
The portable version has three CTEs and one GROUP BY. The MATCH_RECOGNIZE version has one query — shorter but requires familiarity with PATTERN syntax.
Plan quality — on Snowflake, both compile to similar physical plans (a window scan or a MATCH_RECOGNIZE-specific operator). MATCH_RECOGNIZE has a slight edge for complex multi-row patterns; for simple RLE, the plans are indistinguishable.
When to prefer MATCH_RECOGNIZE: patterns like "buy signal followed by three consecutive down-days followed by a stop-loss trigger" — regex-like. When to prefer LAG + SUM: simple RLE, sessionization, streaks — the classic patterns.
Team fluency — MATCH_RECOGNIZE requires training. If your team knows window functions cold and MATCH_RECOGNIZE not at all, choose the portable version. Every hour of training is a real cost.

Output.

symbol	price	run_start	run_end	run_length_minutes
AAPL	190	09:30	09:31	2
AAPL	191	09:32	09:34	3
AAPL	190	09:35	09:35	1

Rule of thumb. Write portable LAG + SUM for cross-warehouse code. Reach for MATCH_RECOGNIZE only on Snowflake or Oracle when the pattern is genuinely multi-row and regex-like.

Senior interview question on dialect-portable RLE

A senior interviewer might ask: "Your team maintains an analytics warehouse on Postgres and a data-science warehouse on Snowflake. Design the SCD-2 compaction pipeline that runs identically on both and produces the same (user_id, state, valid_from, valid_to) history — walk me through the code and why you'd not use MATCH_RECOGNIZE even though Snowflake supports it."

Solution Using a portable LAG + SUM SCD-2 compactor

-- SCD-2 compactor — portable Postgres + Snowflake + BigQuery + SQL Server + MySQL
CREATE OR REPLACE VIEW user_state_scd2 AS
WITH flagged AS (
  SELECT
    user_id,
    log_time,
    state,
    CASE
      WHEN state <> LAG(state) OVER (PARTITION BY user_id ORDER BY log_time)
        OR LAG(state) OVER (PARTITION BY user_id ORDER BY log_time) IS NULL
        THEN 1
      ELSE 0
    END AS is_new_run
  FROM user_state_log
),
grouped AS (
  SELECT
    user_id,
    log_time,
    state,
    SUM(is_new_run) OVER (PARTITION BY user_id ORDER BY log_time) AS run_id
  FROM flagged
)
SELECT
  user_id,
  state,
  MIN(log_time) AS valid_from,
  MAX(log_time) AS valid_to,
  COUNT(*) AS event_count,
  ROW_NUMBER() OVER (PARTITION BY user_id ORDER BY MIN(log_time)) AS scd_version
FROM grouped
GROUP BY user_id, state, run_id;

Step-by-step trace.

Step	What runs	Purpose
1	flagged CTE (LAG + CASE)	Flag change points as 1, else 0
2	grouped CTE (SUM OVER)	Cumulative-sum to produce run_id per user
3	Aggregate to valid_from / valid_to	One row per contiguous state
4	ROW_NUMBER() per user	Assign scd_version for change-tracking

The four steps produce a SCD-2-shaped history: one row per contiguous state per user, each row has valid_from, valid_to, and a monotonically-increasing scd_version.

Output:

user_id	state	valid_from	valid_to	event_count	scd_version
u1	trial	2026-06-01	2026-06-14	14	1
u1	paid	2026-06-15	2026-06-30	16	2
u1	churned	2026-07-01	2026-07-10	10	3

Why this works — concept by concept:

Portable primitive — LAG, CASE, SUM OVER, and ROW_NUMBER are ANSI-standard window functions. Every warehouse from Postgres 8.4 onwards ships them. The exact same view definition compiles on Postgres, Snowflake, BigQuery, SQL Server 2012+, MySQL 8, and Redshift.
No MATCH_RECOGNIZE even where supported — even though Snowflake supports MATCH_RECOGNIZE, using it would fork the code. The maintenance cost of two parallel implementations vastly exceeds the readability improvement.
scd_version via ROW_NUMBER — a monotonically-increasing per-user version number that downstream consumers use as a stable join key. Every dimensional model wants this column; not having it is an interview red flag.
valid_to is inclusive — the last state's valid_to is MAX(log_time), not NULL or 9999-12-31. Downstream consumers doing "what's the current state?" filter by WHERE CURRENT_TIMESTAMP BETWEEN valid_from AND valid_to. Adjust to your team's convention.
Cost — three window scans + one aggregate. On a 1B-row log, the pipeline runs in single-digit minutes on a modest cluster. Materialise as a table if downstream reads are frequent; keep as a view if you re-run on every batch. Both are valid; the choice depends on the read/refresh ratio.

SQL
Topic — window functions (SQL)
Window functions in SQL — full library

Practice →

SQL
Topic — SQL
SQL problem library — 450+ DE-focused questions

Practice →

Cheat sheet — Gaps & Islands recipe list

Universal template. Order rows within each partition. Assign an island id (Flavour A: dense subtraction; Flavour B: cumulative sum of change flag). Group by island id and aggregate. Every gaps-and-islands answer uses this exact three-step pattern.
Dense-date island id (Flavour A). login_date - ROW_NUMBER() OVER (PARTITION BY user_id ORDER BY login_date) * INTERVAL '1 day'. Constant inside an island; changes across islands. Tabibitosan trick.
Change-flag island id (Flavour B). SUM(CASE WHEN gap > threshold THEN 1 ELSE 0 END) OVER (PARTITION BY user_id ORDER BY ts). Increments at every boundary crossing. Works for sessionization, RLE, uptime.
Sessionization primitive. LAG(event_time) + CASE WHEN gap > 30 min THEN 1 ELSE 0 END + SUM(new_session) OVER (PARTITION BY user_id ORDER BY event_time). Threshold defaults to 30 min per GA / Segment convention.
Longest streak per user. Compute island_id, aggregate to per-island (start, end, length), rank per user by length desc, filter to ROW_NUMBER() = 1. Four CTEs; one row per user.
Current streak per user. Same island primitive plus filter WHERE island_end >= CURRENT_DATE - INTERVAL '1 day'. Includes yesterday for timezone tolerance.
Gap-tolerant streak. Switch to Flavour B with a boundary threshold > 1 day. Threshold of 2 days tolerates a 1-day miss.
Business-day streak. Pre-join a business_days(date, business_day_num) calendar table; use business_day_num as the ordered attribute so weekends and holidays don't break the streak.
RLE compression. LAG(state) + CASE WHEN state <> LAG(state) THEN 1 ELSE 0 END + SUM(is_new_run) OVER. Three CTEs, one aggregate. Compression ratio equals the average run length.
MATCH_RECOGNIZE syntax. MATCH_RECOGNIZE (PARTITION BY ... ORDER BY ... MEASURES FIRST(...), LAST(...), COUNT(*) ... ONE ROW PER MATCH PATTERN (A B*) DEFINE A AS TRUE, B AS B.state = A.state). Snowflake + Oracle only.
MATCH_RECOGNIZE dialect matrix. Oracle 12c+ YES · Snowflake YES · Postgres NO · BigQuery NO · SQL Server NO · MySQL NO · Redshift NO. Default to portable LAG + SUM on any multi-warehouse project.
PARTITION BY user_id is non-negotiable. Every window function in every gaps-and-islands query partitions by the entity — user, session, symbol, device. Skipping the partition mixes entities; the interviewer catches it every time.
First-and-last per island. Swap GROUP BY + MIN/MAX for MIN() OVER (PARTITION BY ..., island_id) window aggregates when you want to enrich each row with its island's boundaries without collapsing.
Top-1 per group. ROW_NUMBER() OVER (PARTITION BY user_id ORDER BY length DESC) = 1 is the portable "top-1 per user" idiom. Snowflake / BigQuery ship QUALIFY ROW_NUMBER() = 1 shorthand; Postgres has DISTINCT ON (user_id) ORDER BY length DESC.
FILTER (WHERE ...). Standard SQL clause for conditional aggregation — MIN(page_url) FILTER (WHERE event_name = 'page_view'). Postgres, Snowflake, BigQuery. SQL Server uses MIN(CASE WHEN ... THEN ... END) as the equivalent.
LAST_VALUE IGNORE NULLS forward-fill. For "carry the last non-null value forward" — LAST_VALUE(x IGNORE NULLS) OVER (ORDER BY t ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW). Postgres 16+, Snowflake, BigQuery, SQL Server 2022+.
SCD-2 compactor. RLE view over a CDC / state log with per-entity valid_from / valid_to and a scd_version. Portable across every warehouse.
Compression math. For a slowly-changing series, RLE cuts storage proportionally to the average run length. A 90% duplicate series compresses ~10×; slow-changing prices compress 50-100×.
When to reach for MATCH_RECOGNIZE. Multi-row patterns with strict ordering ("A then B+ then C"), regex-like row sequences, fraud detection, sequence pattern mining. Rare in day-to-day analytics; common in specialised finance / risk workloads.
When to stick with LAG + SUM. Portability, team fluency, simple RLE, sessionization, streaks. The default answer for 90% of gaps-and-islands questions on 90% of warehouses.

Frequently asked questions

What is the gaps and islands problem in SQL?

sql gaps and islands is a family of query patterns that identify runs of consecutive rows sharing some ordering property — for example, consecutive-day login streaks, sessionization of event streams, uptime windows, and price-plateau compression. The canonical answer assigns each row an "island id" and groups by that id: (a) dense-date sequences use login_date - ROW_NUMBER() OVER (PARTITION BY user_id ORDER BY login_date) to produce a constant per island; (b) computed-boundary sequences use SUM(CASE WHEN gap > threshold THEN 1 ELSE 0 END) OVER (...) to cumulative-sum a change flag. Once the island id is right, GROUP BY user_id, island_id and aggregate — MIN(ts) as start, MAX(ts) as end, COUNT(*) as length. Senior interviewers love this family because it tests window-function fluency without being about ranking, has both a portable answer (LAG + SUM) and a MATCH_RECOGNIZE alternative on Snowflake / Oracle, and generalises across sessionization, streaks, RLE, and SCD-2 with the same three-step template.

How do you sessionize an event stream in SQL?

Sessionization uses the change-flag flavour of the gaps-and-islands template. In three lines: (1) compute the gap to the previous event with LAG(event_time) OVER (PARTITION BY user_id ORDER BY event_time); (2) flag new_session = 1 when the gap exceeds a threshold (usually 30 minutes per Google Analytics / Segment convention) or when there's no previous event; (3) cumulative-sum the flag with SUM(new_session) OVER (PARTITION BY user_id ORDER BY event_time) to produce a per-user session id. Session-level aggregations then group by (user_id, session_id) and pull MIN(event_time) as session_start, MAX(event_time) as session_end, COUNT(*) as event_count, and per-event details via MIN(...) FILTER (WHERE event_name = 'page_view') for the landing page. The primitive extends to explicit boundary events (sign-out closes a session) by adding a LAG(event_name) = 'sign_out' branch to the CASE.

What is the row_number minus date trick?

The row_number - date trick (also known as the Tabibitosan method) turns a dense-date sequence into constant island ids by subtracting the per-partition ROW_NUMBER() from the ordered date attribute. When consecutive dates advance in lockstep with the row_number (2026-07-01 · rn=1, 2026-07-02 · rn=2, 2026-07-03 · rn=3), the subtraction yields the same constant (2026-06-30) inside the island — a natural GROUP BY key. When a gap appears (2026-07-06 · rn=4), the row_number moves ahead of the date and the subtraction produces a different constant (2026-07-02), correctly starting a new island. Every senior data engineer recognises this trick in ten seconds — practice writing it until you can produce the six-line CTE from memory. It's the fastest answer to LeetCode's "consecutive login days" problem and every senior data-engineering interview's dense-date island question.

How do you compute the longest login streak per user?

Longest-streak-per-user is a four-CTE query on top of the Tabibitosan trick: (1) assign per-user ROW_NUMBER() ordered by login_date; (2) compute island_id = login_date - rn * INTERVAL '1 day'; (3) group by (user_id, island_id) to aggregate per-island (streak_start, streak_end, streak_length); (4) rank per user by streak_length DESC, streak_end DESC and filter to ROW_NUMBER() = 1. The output is one row per user with their longest ever streak. Current-streak-per-user swaps the ranking step for a filter WHERE streak_end >= CURRENT_DATE - INTERVAL '1 day' — the -1 day tolerance handles timezones. Gap-tolerant streaks (allow one-day misses) switch to the change-flag flavour with a 2-day boundary threshold. Business-day streaks pre-join a calendar table with a dense business_day_num and use that as the ordered attribute. Every variation is a two-line change from the base recipe.

What is SQL run length encoding and when do you reach for it?

sql run length encoding compresses a sequence of state values into one row per contiguous run — (state, run_start, run_end, run_length) — using the same LAG + change-flag primitive as sessionization but ordered by a general timestamp instead of a session boundary. The canonical query flags change points with CASE WHEN state <> LAG(state) OVER (...) THEN 1 ELSE 0 END, cumulative-sums the flag into run_id, then groups by (state, run_id) to aggregate. Reach for RLE whenever a slowly-changing series has many consecutive same-value rows: feature-flag audit logs (10-100× compression), sensor readings, uptime pings, price plateaus, and SCD-2 slowly-changing dimensions (valid_from / valid_to history). Compression ratio equals the average run length; downstream analytical reads become dramatically cheaper. Snowflake and Oracle ship the declarative MATCH_RECOGNIZE alternative — PATTERN (A B*) DEFINE B AS B.state = A.state — but the portable LAG + SUM version is the safer default for cross-warehouse code.

Which warehouses support MATCH_RECOGNIZE?

MATCH_RECOGNIZE is the ANSI SQL/JSON row-pattern-matching clause that reads like a regex over ordered rows. Support in 2026: Oracle 12c+ (YES, the first ship in 2013), Snowflake (YES, since 2021 GA), Postgres (NO — no planned support as of 2026), BigQuery (NO), SQL Server (NO), MySQL 8 (NO), Redshift (NO). For cross-warehouse projects, default to the portable LAG + SUM(new_group) OVER pattern; it works everywhere and reads well once your team knows window functions cold. Reach for MATCH_RECOGNIZE only on Snowflake or Oracle when the pattern is genuinely multi-row and regex-like — sequences like "A then B+ then C" for fraud detection, click-stream funnel matching, or financial signal recognition. For simple RLE, sessionization, and streaks, the portable window-function recipe compiles to identical physical plans and doesn't fork your codebase across dialects.

Practice on PipeCode

Drill the gaps-and-islands practice library → for date-island detection, sessionization, streak counting, and RLE reps across every warehouse dialect.
Rehearse on window-function problems → — the ROW_NUMBER, LAG, LEAD, and cumulative SUM primitives that every gaps-and-islands answer builds on.
Sharpen SQL window function drills → for the dialect-portable half — Postgres, Snowflake, BigQuery, SQL Server, MySQL.
Push the difficulty ceiling with medium window-function problems → and the hard tier → for streak, sessionization, and RLE senior interview scenarios.
Practise sliding-window problems → for the rolling-aggregate cousin of gaps-and-islands.
Layer time-series drills → for the timestamp-heavy contexts where sessionization and streaks live.
Stack the interval-processing library → for uptime windows, overlap detection, and RLE variations.
For general SQL sharpening, work through the SQL problem library → which contains 450+ DE-focused questions.
For the broader SQL interview surface, take the SQL for Data Engineering course →.

Pipecode.ai is Leetcode for Data Engineering — every `sql gaps and islands` recipe above ships with hands-on practice rooms where you write the Tabibitosan island-id derivation, wire the 30-minute sessionization pattern, chase the longest-streak-per-user query, compress a state series with RLE, and rehearse the MATCH_RECOGNIZE alternative against real graded inputs. PipeCode pairs every reading with 450+ DE-focused problems and a real-time scoring engine, so you never have to wonder whether your `consecutive rows sql` answer holds up under a senior interviewer's depth probes.

Practice gaps and islands now →
Window function drills →

SQL JSON Functions: JSON_VALUE, JSON_EXTRACT, JSONB, JSON_TABLE Across Dialects

Gowtham Potureddi — Fri, 10 Jul 2026 17:32:28 +0000

sql json is the single most-searched semi-structured keyword in modern warehouse SQL — and the single most dialect-fractured one in 2026. The same "extract user.id out of an event payload and use it as a join key" ask has five different answers across the five engines most data teams live in: Postgres uses payload->'user'->>'id' or jsonb_path_query, MySQL 8 ships JSON_EXTRACT(payload, '$.user.id') and the ->> shortcut, SQL Server ships JSON_VALUE(payload, '$.user.id'), Snowflake reaches for its dot-notation payload:user.id::string on top of a VARIANT column, and BigQuery ships JSON_VALUE(payload, '$.user.id') against a native JSON type. There is no portable spelling of a JSON path; the interviewer wants to hear you say that out loud in the first sentence.

This guide is the mid-to-senior comparison you wished existed the first time an interviewer asked you to write a json_value sql expression on the whiteboard, sketch a json_extract sql join in MySQL, wire a jsonb postgres GIN index for a hot containment filter, flatten a nested payload with json_table sql, chase a schema-drift regression in bigquery json, or design a snowflake variant column for a fast-changing event stream. It walks through the JSONPath grammar every engine borrows ($ root, .key child, [i] index, [*] wildcard, ..desc recursive descent), the scalar-vs-sub-tree split that separates JSON_VALUE from JSON_QUERY and Postgres ->> from ->, the Postgres jsonb type with its jsonb_ops and jsonb_path_ops GIN indexes plus the @> containment operator, the ANSI JSON_TABLE clause with its NESTED PATH sub-block and its Snowflake LATERAL FLATTEN, BigQuery UNNEST(JSON_QUERY_ARRAY(...)), and Postgres jsonb_to_recordset equivalents, and the storage-plus-index performance matrix across all five engines including when a materialised generated column beats every path expression. Each section pairs a teaching block with a Solution-Tail interview answer — code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works.

When you want hands-on reps immediately after reading, drill the JSON practice library →, rehearse on parsing problems →, and sharpen the reshape axis with data-transformation drills →.

On this page

Why JSON in SQL matters in 2026
JSON_VALUE / JSON_EXTRACT — path expressions
JSONB in Postgres — index it or regret it
JSON_TABLE — flatten JSON to rows
Dialect matrix + performance
Cheat sheet — SQL JSON recipe list
Frequently asked questions
Practice on PipeCode

1. Why JSON in SQL matters in 2026

JSON columns are the default event-payload shape — and the mental model every senior data engineer needs to lock in on day one

The one-sentence invariant: a sql json column stores semi-structured data whose schema drifts faster than the surrounding relational model, and every senior interviewer expects you to name the dialect's storage type, its path syntax, its index options, and the flattening operator in the same breath. Once you internalise that "storage type + path syntax + index shape + flatten operator" is the whole design space, the JSON-in-SQL interview surface collapses into a four-cell mental card per dialect.

Four axes interviewers actually probe.

Storage type. Postgres has both json (raw text, ordered, no dedup) and jsonb (parsed, deduped, binary, indexable); MySQL 8 has one binary JSON type; SQL Server piggybacks on NVARCHAR(MAX) with ISJSON validation; Snowflake has VARIANT (columnar, self-describing); BigQuery has a native JSON type on top of the older STRING-of-JSON pattern. Every choice trades parse cost, storage size, and query cost against schema flexibility.
Path syntax. The five engines share the JSONPath grammar ($ root, .key child, [i] index, [*] wildcard) but disagree on the wrapper: Postgres exposes ->, ->>, #>, #>> arrow operators plus SQL/JSON jsonb_path_query; MySQL 8 ships JSON_EXTRACT and the ->> shortcut; SQL Server ships JSON_VALUE and JSON_QUERY; Snowflake extends the SQL grammar with col:key.sub.notation; BigQuery ships JSON_VALUE, JSON_QUERY, and a dot-syntax on the native JSON type.
Scalar vs sub-tree. Every dialect distinguishes "extract a scalar and coerce it to a SQL type" from "extract a JSON sub-tree that stays JSON." SQL Server draws the line at JSON_VALUE (scalar) vs JSON_QUERY (sub-tree); Postgres draws it at ->> (scalar text) vs -> (JSON child); MySQL draws it at ->> vs ->; BigQuery draws it at JSON_VALUE vs JSON_QUERY. Missing this split is the most-asked interview correction.
Flatten to rows. Nested arrays cannot be joined against a relational query without a flattening operator. The ANSI answer is JSON_TABLE (Oracle, MySQL 8, SQL Server 2022); Snowflake ships LATERAL FLATTEN; BigQuery pairs UNNEST with JSON_QUERY_ARRAY; Postgres uses jsonb_array_elements, jsonb_to_recordset, or a LATERAL join.

Where semi-structured data lands in your pipelines.

Event tracking payloads. Segment, Rudderstack, Snowplow, and PostHog all emit event(id, timestamp, event_type, payload jsonb) schemas. The event-type-specific columns live inside payload because they change too fast to materialise as columns.
Third-party API bodies. Webhooks from Stripe, Shopify, Zendesk, HubSpot, and 100 other SaaS integrations arrive as JSON. Landing them in a JSON column preserves the source-of-truth shape; downstream transforms extract typed columns as they stabilise.
CDC envelopes. Debezium, Fivetran, Airbyte, and Meltano wrap each row change in a JSON envelope with before, after, op, source, ts_ms keys. Consumer pipelines routinely JSON_EXTRACT(payload, '$.after.user_id') to project the changed row.
Config and feature flags. LaunchDarkly, GrowthBook, and internal feature-flag tables often store variant assignments as a JSON blob per user. Filtering on payload->>'variant' is a hot analytics path.
Semi-structured logs. Application logs (structured JSON) land as JSON columns for indexing; SELECT * FROM logs WHERE payload @> '{"level":"error"}' is a common Postgres pattern.

When JSON-in-SQL beats a separate document store.

One system, one backup, one auth model. Running MongoDB or DynamoDB alongside your warehouse doubles your operational surface. Modern warehouses ship JSON support strong enough that most read-heavy workloads never leave SQL.
Joins with the relational world. Every event has a user_id that joins against your users table. Doing that join across a warehouse and MongoDB is painful; doing it inside Postgres with jsonb extraction is a two-line query.
Analytics-friendly shape. Once you flatten with JSON_TABLE or LATERAL FLATTEN, you get a relational shape that every BI tool understands. Grafana, Metabase, Superset, Looker — every dashboard tool wants columns, not documents.
Cost. Snowflake VARIANT and BigQuery native JSON push semi-structured storage into columnar formats. Query cost on a 10TB event log is often lower than a document store's query cost on the same shape.

When a separate document store still wins.

Sub-10ms point-read latency. MongoDB and DynamoDB are optimised for OLTP-style single-document reads. Warehouses are not.
Deep nesting with heavy write-heavy update patterns. JSONB updates in Postgres rewrite the whole binary blob; MongoDB updates individual sub-documents. If your workload is 10× writes per read on a big nested doc, a document store fits better.
Write throughput above 100k QPS. Warehouses are batch-write friendly; document stores handle single-row streaming writes better.

What senior interviewers probe on JSON-in-SQL.

Path vs pointer. JSONPath ($.a.b.c) is the query grammar; JSON Pointer (/a/b/c) is the addressing grammar (RFC 6901). Both exist; pointers are what Postgres #> and #>> use.
Scalar vs sub-tree. Do you say "JSON_VALUE is scalar, JSON_QUERY is sub-tree" without prompting? Do you know Postgres draws the same line at ->> vs ->?
Storage cost. jsonb is ~10% smaller than json (binary encoded, deduped keys), Snowflake VARIANT is columnar-encoded, BigQuery native JSON beats STRING-of-JSON for both storage and query cost.
Index shape. GIN indexes with jsonb_ops (universal, larger) vs jsonb_path_ops (containment-only, smaller). When to add a functional expression index on a single path. When to materialise a generated column instead.
When to flatten. Every join key, filter key, and group-by key that comes out of JSON should be materialised as a column. Every read-only projection can stay in JSON. Naming this rule out loud is the senior signal.

Worked example — the same "extract user id" ask in five dialects

Detailed explanation. The canonical warm-up interview ask — given an events table with a JSON payload, extract user_id and count events per user — reads identically as an English requirement across every warehouse. The SQL to express it looks wildly different by dialect. Writing the same extraction five ways builds a mental dialect matrix that pays off in every JSON follow-up question.

Question. Given an events table with (event_id, ts, payload) where payload is a JSON column containing at minimum a user_id key, write a query that counts events per user in Postgres (jsonb), MySQL 8, SQL Server, Snowflake, and BigQuery.

Input.

event_id	ts	payload
1	2026-07-01	{"user_id": "u1", "kind": "click"}
2	2026-07-01	{"user_id": "u1", "kind": "view"}
3	2026-07-01	{"user_id": "u2", "kind": "click"}
4	2026-07-02	{"user_id": "u1", "kind": "purchase"}

Code.

-- Postgres (jsonb) — arrow operator shortcut
SELECT payload->>'user_id' AS user_id, COUNT(*) AS n
FROM events
GROUP BY payload->>'user_id';

-- MySQL 8 — JSON_EXTRACT + JSON_UNQUOTE (or the ->> shortcut)
SELECT payload->>'$.user_id' AS user_id, COUNT(*) AS n
FROM events
GROUP BY payload->>'$.user_id';

-- SQL Server — JSON_VALUE
SELECT JSON_VALUE(payload, '$.user_id') AS user_id, COUNT(*) AS n
FROM events
GROUP BY JSON_VALUE(payload, '$.user_id');

-- Snowflake — dot-notation VARIANT access + cast
SELECT payload:user_id::string AS user_id, COUNT(*) AS n
FROM events
GROUP BY payload:user_id::string;

-- BigQuery — JSON_VALUE on native JSON type
SELECT JSON_VALUE(payload, '$.user_id') AS user_id, COUNT(*) AS n
FROM events
GROUP BY JSON_VALUE(payload, '$.user_id');

Step-by-step explanation.

Postgres's payload->>'user_id' is the "arrow followed by arrow followed by greater-than" operator that reads: "extract the user_id key and return the value as text." The double-arrow ->> unwraps the JSON string to a SQL text — critical for GROUP BY.
MySQL 8's payload->>'$.user_id' is a syntactic shortcut for JSON_UNQUOTE(JSON_EXTRACT(payload, '$.user_id')). Same mechanics as Postgres's ->> but the path is JSONPath (with $. prefix) rather than a bare key name.
SQL Server's JSON_VALUE(payload, '$.user_id') returns a scalar SQL string. It errors if the target is a JSON object or array — JSON_VALUE is scalar-only. For sub-tree extraction you would reach for JSON_QUERY instead.
Snowflake's payload:user_id::string uses colon-notation for the key access and the SQL :: cast operator to force the return type. Without the cast, the value stays a VARIANT — which is fine for further chaining but often surprises consumers who expect a VARCHAR.
BigQuery's JSON_VALUE(payload, '$.user_id') looks like SQL Server's but runs against the native JSON type. If your column is STRING-of-JSON, use JSON_EXTRACT_SCALAR instead — a common bigquery json gotcha.

Output.

user_id	n
u1	3
u2	1

Rule of thumb. In every dialect, always cast or unwrap the extracted value to a SQL scalar type before GROUP BY, JOIN, or WHERE. Grouping on a raw JSON sub-tree produces different equality semantics per dialect and is almost always a bug.

Worked example — sub-tree vs scalar (JSON_VALUE vs JSON_QUERY)

Detailed explanation. The most common junior-to-senior correction: candidates reach for JSON_VALUE to grab a nested object and hit "the requested JSON path expression is not a scalar." The interviewer wants to see you know that JSON_VALUE returns a scalar, JSON_QUERY returns a sub-tree, and the same split exists in Postgres (->> vs ->), MySQL (->> vs ->), and Snowflake (::string cast vs raw VARIANT).

Question. Given a payload {"user": {"id": "u1", "name": "Ada"}, "kind": "click"}, write two queries per dialect (SQL Server + Postgres): one that returns the user sub-tree as JSON, and one that returns the user.id scalar as a SQL string. Explain the error you'd hit if you mis-used them.

Input.

id	payload
1	{"user": {"id": "u1", "name": "Ada"}, "kind": "click"}
2	{"user": {"id": "u2", "name": "Bo"}, "kind": "view"}

Code.

-- SQL Server — JSON_QUERY for sub-tree, JSON_VALUE for scalar
SELECT
  JSON_QUERY(payload, '$.user')       AS user_json,   -- returns JSON object
  JSON_VALUE(payload, '$.user.id')    AS user_id      -- returns scalar
FROM events;

-- WRONG — JSON_VALUE on a sub-tree returns NULL (or errors depending on option)
SELECT JSON_VALUE(payload, '$.user') FROM events;

-- Postgres (jsonb) — -> for sub-tree, ->> for scalar
SELECT
  payload -> 'user'              AS user_json,   -- returns jsonb
  payload -> 'user' ->> 'id'     AS user_id      -- returns text
FROM events;

-- Alt — Postgres path pointer #> (sub-tree), #>> (scalar text)
SELECT
  payload #>  '{user}'           AS user_json,
  payload #>> '{user,id}'        AS user_id
FROM events;

Step-by-step explanation.

JSON_QUERY and JSON_VALUE are complementary: one returns a JSON sub-tree (still JSON), the other returns a scalar coerced to a SQL type. Mixing them up is the most-asked correction.
In SQL Server, JSON_VALUE(payload, '$.user') on an object target returns NULL in lax mode (default) or errors in strict mode. Neither is what you want — reach for JSON_QUERY instead.
Postgres's -> returns a jsonb child; ->> returns the text representation of a scalar child. The double-arrow always unwraps quotes; the single-arrow leaves them.
Postgres's #> and #>> are the path-pointer variants that accept a text-array of keys. payload #>> '{user,id}' is equivalent to payload -> 'user' ->> 'id'. Path pointers are useful for programmatic construction; arrow chains are more readable for humans.
When the extracted value is used as a GROUP BY key or a JOIN target, always use the scalar form — JSON_VALUE in SQL Server, ->> in Postgres/MySQL, ::string in Snowflake, JSON_VALUE in BigQuery. The sub-tree form is only for further JSON chaining or for embedding in another JSON output.

Output.

id	user_json	user_id
1	{"id": "u1", "name": "Ada"}	u1
2	{"id": "u2", "name": "Bo"}	u2

Rule of thumb. Every extraction is either scalar (headed for a SQL type) or sub-tree (headed for further JSON work). Name the direction before you write the operator — the operator falls out of the direction.

Worked example — the JSONPath grammar every engine borrows

Detailed explanation. JSONPath is the ANSI-flavoured grammar every engine implements. The five must-know constructs are $ (root), .key (child by name), [i] (child by index), [*] (wildcard over an array), and ..desc (recursive descent). Once you know the grammar, the dialect surface reduces to "which function wraps the path?"

Question. Given a payload with a nested orders array {"user_id": "u1", "orders": [{"id": 1, "total": 10}, {"id": 2, "total": 25}]}, write the JSONPath expressions to extract (a) the user_id, (b) the first order's total, (c) all order totals, (d) any id anywhere in the payload. Show one dialect for each — Postgres jsonb_path_query, MySQL JSON_EXTRACT, SQL Server JSON_VALUE / JSON_QUERY, BigQuery JSON_VALUE / JSON_QUERY_ARRAY.

Input.

id	payload
1	{"user_id": "u1", "orders": [{"id": 1, "total": 10}, {"id": 2, "total": 25}]}

Code.

-- Postgres — SQL/JSON jsonb_path_query
SELECT
  jsonb_path_query_first(payload, '$.user_id')      AS user_id,
  jsonb_path_query_first(payload, '$.orders[0].total') AS first_total,
  jsonb_path_query(payload, '$.orders[*].total')    AS every_total,
  jsonb_path_query(payload, '$..id')                AS every_id
FROM events;

-- MySQL 8 — JSON_EXTRACT
SELECT
  JSON_EXTRACT(payload, '$.user_id')            AS user_id,
  JSON_EXTRACT(payload, '$.orders[0].total')    AS first_total,
  JSON_EXTRACT(payload, '$.orders[*].total')    AS every_total,
  JSON_EXTRACT(payload, '$..id')                AS every_id
FROM events;

-- SQL Server — JSON_VALUE (scalar), JSON_QUERY (sub-tree / array)
SELECT
  JSON_VALUE(payload, '$.user_id')              AS user_id,
  JSON_VALUE(payload, '$.orders[0].total')      AS first_total,
  JSON_QUERY(payload, '$.orders[*].total')      AS every_total  -- array
FROM events;
-- Note: SQL Server has no ..desc recursive-descent operator.

-- BigQuery — JSON_VALUE, JSON_QUERY_ARRAY
SELECT
  JSON_VALUE(payload, '$.user_id')                        AS user_id,
  JSON_VALUE(payload, '$.orders[0].total')                AS first_total,
  JSON_QUERY_ARRAY(payload, '$.orders')                   AS every_order,
  JSON_QUERY_ARRAY(payload, '$..id')                      AS every_id
FROM events;

Step-by-step explanation.

$ is the root — every JSONPath starts with it. $.user_id reads "from the root, take the user_id child."
[i] is index access. $.orders[0] picks the first element; $.orders[-1] picks the last (dialect-dependent — Postgres and BigQuery support negative indexing, SQL Server does not).
[*] is the wildcard — "every element of this array." Combined with a child key, $.orders[*].total returns every order's total. The return shape is an array when the wrapper function returns arrays (JSON_QUERY), or a set of rows when the wrapper is set-returning (jsonb_path_query).
..key is recursive descent — "find key anywhere in the sub-tree." Postgres, MySQL, and BigQuery support it; SQL Server does not. Great for pulling a value out of a deeply nested payload without knowing the exact path.
Every dialect maps the same JSONPath grammar to a different wrapper: Postgres jsonb_path_query (set-returning) or jsonb_path_query_first (scalar), MySQL JSON_EXTRACT (returns a JSON value that may be an array), SQL Server JSON_VALUE (scalar) or JSON_QUERY (sub-tree), BigQuery JSON_VALUE (scalar) or JSON_QUERY / JSON_QUERY_ARRAY (sub-tree/array).

Output (Postgres set-returning).

user_id	first_total	every_total	every_id
"u1"	10	10	1
"u1"	10	25	2

Rule of thumb. Memorise the five JSONPath constructs ($, .key, [i], [*], ..desc) once. After that, the dialect surface is just "which function wraps the path?" and "does the function return a scalar, a JSON, or a set of rows?"

Senior interview question on the JSON-in-SQL mental model

A senior interviewer often opens with: "You have a table events(id, ts, payload) where payload is a JSON column. Walk me through your decision tree for whether to keep a value inside the JSON, materialise it as a generated column, or copy it into a separate typed column. What's your rule of thumb?"

Solution Using the "hot path materialisation" decision framework

Decision framework — keep in JSON, materialise, or extract to a column

1. Is the key on the hot filter path (WHERE payload->>'key' = ...)?
   yes → materialise as a generated column + index it
   no  → keep in JSON

2. Is the key a JOIN target?
   yes → materialise as a generated column + index it
   no  → keep in JSON

3. Is the key a GROUP BY or ORDER BY key?
   yes → materialise as a generated column
   no  → keep in JSON

4. Does the key have a stable presence across all rows?
   yes → materialise (or promote to a plain column)
   no  → keep in JSON

5. Is the key part of a schema-drift-heavy sub-tree
   (rarely-queried event-type-specific fields)?
   yes → keep in JSON forever
   no  → consider promotion to a plain column

Step-by-step trace.

Key	Q1 hot filter	Q2 join	Q3 group/order	Q4 stable	Q5 drift	Verdict
`user_id`	yes	yes	yes	yes	no	Promote to a plain column
`event_type`	yes	no	yes	yes	no	Materialise + index
`session_id`	sometimes	no	no	yes	no	Materialise (no index)
`orders[*].total` (array agg)	no	no	no	no	yes	Keep in JSON, flatten at query time
`custom_flags` (freeform tags)	no	no	no	no	yes	Keep in JSON forever

The framework turns "should I extract this key?" from a taste question into a mechanical decision. Every key on the hot query path pays for materialisation with much better plan quality; every drifty key stays in the JSON payload where it belongs.

Output:

Strategy	When it wins
Promote to plain column	Stable + always-present + hot filter/join/group key
Materialise as generated column + index	Stable + hot filter path + still expressible as a JSON extract
Materialise as generated column (no index)	Hot projection but not filter/join key
Keep in JSON, flatten at query time	Nested arrays consumed by rare reports
Keep in JSON forever	Schema-drift heavy, low-frequency reads

Why this works — concept by concept:

Hot path materialisation — every warehouse has a query planner that struggles with expression-based filters. WHERE payload->>'user_id' = 'u1' triggers a full scan unless the expression is indexed. Materialising the extraction into a real column exposes it to the planner and unlocks index-based filtering.
Generated columns preserve the source of truth — a GENERATED ALWAYS AS (payload->>'user_id') STORED column stays in sync with the JSON automatically. You get the query-time win of a materialised column and the schema-drift resilience of the JSON payload in the same table.
Schema drift is a first-class concern — the reason data engineers reach for JSON at all is that the schema changes weekly. Freezing every field into a plain column defeats the purpose. The framework keeps the drifty fields in JSON and only promotes the stable, high-value ones.
Stable + hot signals both matter — a field that's stable but never queried doesn't need materialisation; a hot field that's rarely present doesn't fit as a plain column. The Q1×Q4 intersection is where materialisation earns its keep.
Cost — every materialised column costs some write-time CPU (evaluate the expression on insert/update) and some storage (the extracted value stored alongside the JSON). The trade-off is query-time speedup on the hot path; if the query fires 1000 times a day and the write fires 100 times, the trade almost always pays off.

SQL
Topic — JSON
JSON extraction and reshape problems

Practice →

SQL Topic — parsing Parsing drills

Practice →

2. JSON_VALUE / JSON_EXTRACT — path expressions

`json_value sql` returns a scalar, `json_extract sql` returns JSON — the same JSONPath grammar wraps them, but the dialect dresses the wrapper differently

The mental model in one line: every JSON-in-SQL path expression is a JSONPath ($.a.b.c) wrapped in a dialect-specific function that returns either a scalar SQL type (JSON_VALUE, ->>) or a JSON sub-tree (JSON_QUERY, ->), and the interviewer expects you to name both halves of the split without prompting. Once you say "scalar vs sub-tree, JSONPath vs pointer, arrow vs function," the path-expression interview surface is a lookup.

JSONPath grammar refresher.

$ — root of the JSON document.
.key — child by name (dot notation). Fails if the key contains hyphens, spaces, or reserved words — use bracket notation instead.
["key-with-hyphen"] — bracket notation. Always works, always safe.
[i] — child by array index (0-based).
[*] — wildcard over an array (every element).
..key — recursive descent (find key anywhere in the sub-tree). Not supported in SQL Server.
[?(@.total > 100)] — filter expression (SQL/JSON standard; Postgres and MySQL 8 support it).

Dot vs bracket notation.

Use dot when the key is a plain identifier: $.user.id.
Use bracket when the key has a hyphen, space, or reserved word: $["user-id"], $["order date"], $["$type"].
Bracket notation works everywhere dot notation works, so a code generator that emits pure bracket notation is bulletproof.

Postgres arrow operators.

-> — child by name, returns jsonb. payload -> 'user' returns the user sub-tree as jsonb.
->> — child by name, returns text. payload ->> 'user_id' returns the value as a SQL text.
#> — path by text-array, returns jsonb. payload #> '{user, id}' navigates to user.id and returns jsonb.
#>> — path by text-array, returns text. Same navigation but returns text.
These are shortcuts over jsonb_extract_path and jsonb_extract_path_text and are more concise for common queries.

Postgres SQL/JSON path API (Postgres 12+).

jsonb_path_query(payload, '$.orders[*].total') — set-returning; one row per match.
jsonb_path_query_first(payload, '$.user_id') — scalar; first match or NULL.
jsonb_path_exists(payload, '$.user_id') — boolean; true if any match.
jsonb_path_match(payload, '$.total > 100') — boolean; true if the SQL/JSON predicate matches.
These functions accept the full SQL/JSON path grammar including filter expressions and recursive descent.

MySQL 8 JSON_EXTRACT + shortcuts.

JSON_EXTRACT(payload, '$.user.id') — returns a JSON value.
payload -> '$.user.id' — shortcut for JSON_EXTRACT (returns JSON, keeps quotes).
payload ->> '$.user.id' — shortcut for JSON_UNQUOTE(JSON_EXTRACT(...)) (returns text, strips quotes).
JSON_VALUE(payload, '$.user.id') — added in MySQL 8.0.21; returns a scalar with optional type coercion (RETURNING INT).
JSON_UNQUOTE(x) — strips the JSON string quotes; often paired with JSON_EXTRACT when you need a plain SQL string.

SQL Server JSON_VALUE vs JSON_QUERY.

JSON_VALUE(payload, '$.user.id') — scalar. Returns NVARCHAR(4000) by default; use the optional RETURNING clause (SQL Server 2022+) for typed return.
JSON_QUERY(payload, '$.user') — sub-tree. Returns NVARCHAR(MAX) containing valid JSON.
Lax mode (default) — path errors return NULL. Strict mode ('strict $.user.id') — path errors throw.
ISJSON(x) — returns 1 if x is valid JSON; use it in CHECK constraints to validate a text column stores valid JSON.
OPENJSON(payload, '$.orders') — flattens an array to rows (see the JSON_TABLE section).

Snowflake VARIANT access.

payload:user_id — colon-notation child access. Returns VARIANT.
payload:user.id — chained dot after colon. Same as payload:user:id.
payload:orders[0].total — bracket for index, dot for child.
payload:user_id::string — cast operator forces the return type.
GET_PATH(payload, 'user.id') — function form, useful for programmatic access.
GET(payload, 'user_id') and TRY_PARSE_JSON(text) — helpers for defensive access and casting.

BigQuery JSON functions.

JSON_VALUE(payload, '$.user.id') — scalar; returns STRING.
JSON_QUERY(payload, '$.user') — sub-tree; returns STRING (JSON text) if payload is STRING, or JSON if payload is the native type.
JSON_EXTRACT_SCALAR(payload, '$.user.id') — legacy alias for JSON_VALUE on STRING-typed columns.
JSON_EXTRACT(payload, '$.user') — legacy alias for JSON_QUERY.
Dot notation on native JSON: payload.user.id — reads elegantly, only works on native JSON (not STRING).

Oracle JSON access.

JSON_VALUE(payload, '$.user.id') — scalar with RETURNING VARCHAR2(100) clause.
JSON_QUERY(payload, '$.user') — sub-tree.
Dot notation on JSON columns: t.payload.user.id (Oracle 12c+).
JSON_EXISTS(payload, '$.user.id') — boolean.
JSON_TABLE(payload, '$.orders[*]' COLUMNS ...) — the ANSI flattening operator; Oracle was the first to ship it.

When the extraction returns NULL.

Path doesn't match — the key doesn't exist. Every dialect returns NULL in lax mode.
Path matches but the value is null (the JSON literal null). Postgres returns SQL NULL; SQL Server returns SQL NULL; Snowflake returns SQL NULL after ::string cast; MySQL returns the JSON null which is not equal to SQL NULL (a common mysql json gotcha).
Path matches but the value is a sub-tree (object or array) and you asked for a scalar. SQL Server returns NULL in lax mode; strict mode throws.

Type coercion after extraction.

Every dialect returns the extracted value as text by default. To cast to a numeric type, use CAST(... AS INT) in ANSI-flavoured dialects or ::int in Postgres.
Snowflake requires an explicit ::type cast to leave the VARIANT universe.
BigQuery's JSON_VALUE returns STRING; use SAFE_CAST(JSON_VALUE(...) AS INT64) for safe coercion.
MySQL 8's JSON_VALUE accepts a RETURNING clause: JSON_VALUE(payload, '$.total' RETURNING SIGNED).

Worked example — Postgres arrow chain vs SQL/JSON path

Detailed explanation. Postgres offers two paths for extraction: the arrow chain (payload->'user'->>'id') and the SQL/JSON path (jsonb_path_query_first(payload, '$.user.id')). Both produce the same result. The arrow chain is more concise; the SQL/JSON path supports filter expressions and recursive descent. Knowing when to reach for each is a senior signal.

Question. Given an events table with payload jsonb, write two equivalent Postgres queries — one using arrow chains, one using jsonb_path_query_first — that project user_id, session_id, and the total of the first order. Explain when the SQL/JSON path form wins.

Input.

id	payload
1	{"user_id": "u1", "session_id": "s1", "orders": [{"id": 1, "total": 10}]}
2	{"user_id": "u2", "session_id": "s2", "orders": [{"id": 2, "total": 30}]}

Code.

-- Arrow-chain form
SELECT
  id,
  payload ->> 'user_id'                            AS user_id,
  payload ->> 'session_id'                         AS session_id,
  (payload -> 'orders' -> 0 ->> 'total')::int      AS first_total
FROM events;

-- SQL/JSON path form
SELECT
  id,
  jsonb_path_query_first(payload, '$.user_id')       #>> '{}' AS user_id,
  jsonb_path_query_first(payload, '$.session_id')    #>> '{}' AS session_id,
  (jsonb_path_query_first(payload, '$.orders[0].total') #>> '{}')::int AS first_total
FROM events;

Step-by-step explanation.

payload ->> 'user_id' returns the text value of the user_id key. Concise and readable — the canonical Postgres extraction for a top-level scalar.
payload -> 'orders' -> 0 ->> 'total' chains: extract orders (jsonb), index into position 0 (jsonb), extract total (text). The ::int cast forces numeric.
jsonb_path_query_first(payload, '$.orders[0].total') returns the first match as jsonb. To convert to SQL text, chain #>> '{}' — a bit awkward but ANSI-flavoured.
The SQL/JSON path form wins for filter expressions: jsonb_path_query(payload, '$.orders[*] ? (@.total > 20)') returns every order with total > 20 — impossible in a pure arrow chain.
The SQL/JSON path form also wins for recursive descent: jsonb_path_query(payload, '$..id') returns every id anywhere in the sub-tree — again impossible in a pure arrow chain.

Output.

id	user_id	session_id	first_total
1	u1	s1	10
2	u2	s2	30

Rule of thumb. Use arrow chains for simple top-level or single-branch extractions. Reach for jsonb_path_query when you need filter predicates, recursive descent, or want the full SQL/JSON grammar (Postgres 12+).

Worked example — MySQL JSON_EXTRACT with `JSON_UNQUOTE` and `->>`

Detailed explanation. MySQL's most-asked JSON gotcha: JSON_EXTRACT(payload, '$.user_id') returns "u1" — a JSON string with the quotes still on. To get the SQL string u1, you either wrap with JSON_UNQUOTE or use the ->> shortcut. Interviewers love this gotcha because it separates casual from careful MySQL users.

Question. Given a MySQL 8 events table with payload JSON, write two equivalent queries that project user_id as an unquoted SQL string. Show how a naive JSON_EXTRACT breaks a downstream string equality.

Input.

id	payload
1	{"user_id": "u1"}
2	{"user_id": "u2"}

Code.

-- BROKEN — quotes leak into the projected value
SELECT id, JSON_EXTRACT(payload, '$.user_id') AS user_id
FROM events;
-- Result: user_id = "u1" (a 4-char string with quotes)

-- FIXED (long form) — wrap with JSON_UNQUOTE
SELECT id, JSON_UNQUOTE(JSON_EXTRACT(payload, '$.user_id')) AS user_id
FROM events;

-- FIXED (shortcut) — the ->> operator is JSON_UNQUOTE(JSON_EXTRACT(...))
SELECT id, payload->>'$.user_id' AS user_id
FROM events;

-- ALT — MySQL 8.0.21+ ships JSON_VALUE with optional RETURNING
SELECT id, JSON_VALUE(payload, '$.user_id' RETURNING VARCHAR(64)) AS user_id
FROM events;

Step-by-step explanation.

JSON_EXTRACT returns a JSON value — for a string, that means the string with its quotes. Comparing JSON_EXTRACT(payload, '$.user_id') = 'u1' fails silently because "u1" (with quotes) is not equal to u1 (without).
JSON_UNQUOTE(JSON_EXTRACT(payload, '$.user_id')) strips the quotes and returns a plain SQL string. This is the canonical fix.
payload->>'$.user_id' is the syntactic shortcut for JSON_UNQUOTE(JSON_EXTRACT(payload, '$.user_id')). Reads cleaner and is the community-preferred form.
JSON_VALUE(payload, '$.user_id') (MySQL 8.0.21+) is a modern alternative that returns a scalar. The optional RETURNING VARCHAR(64) clause forces the return type — great when you know the target column type upfront.
In every case, always use the unquoted form when the extracted value is a WHERE, GROUP BY, JOIN, or ORDER BY target. The quote-leak bug is silent — the query returns 0 rows instead of erroring, which is the worst kind of bug.

Output (fixed variants).

id	user_id
1	u1
2	u2

Rule of thumb. In MySQL, always reach for ->> (or JSON_UNQUOTE(JSON_EXTRACT(...))) when the extracted value will be compared, grouped, joined, or ordered. Save JSON_EXTRACT alone for JSON sub-tree extraction where the quotes belong.

Worked example — Snowflake VARIANT chained access with casts

Detailed explanation. Snowflake's VARIANT type accepts colon notation for key access and returns another VARIANT — leaving the value untyped until you cast. The idiomatic Snowflake style is payload:user_id::string — colon for access, :: for cast. Missing the cast is one of the top three snowflake variant interview corrections.

Question. Given a Snowflake events(id, payload VARIANT) table, write a query that projects user_id as STRING, total as NUMBER, and is_paid as BOOLEAN. Explain what happens if you skip the casts.

Input.

id	payload
1	{"user_id": "u1", "total": 100, "is_paid": true}
2	{"user_id": "u2", "total": 250, "is_paid": false}

Code.

SELECT
  id,
  payload:user_id::string   AS user_id,
  payload:total::number     AS total,
  payload:is_paid::boolean  AS is_paid
FROM events;

-- Alternative — GET_PATH is the function form
SELECT
  id,
  GET_PATH(payload, 'user_id')::string  AS user_id,
  GET_PATH(payload, 'total')::number    AS total
FROM events;

Step-by-step explanation.

payload:user_id returns a VARIANT whose scalar content is the string "u1". Without a cast, downstream string equality (WHERE user_id = 'u1') still works because Snowflake auto-coerces, but the plan may be slower and column type semantics can surprise you.
payload:user_id::string casts to VARCHAR — the type shows up in the plan, downstream tools see a string column, and the plan is faster because the cast can be pushed down.
payload:total::number casts to NUMBER — necessary for arithmetic or numeric aggregation. Skipping the cast can produce silent JSON-to-string coercion that breaks SUM(total) in subtle ways.
GET_PATH(payload, 'user_id') is the function form of the colon operator. Useful for programmatic path construction; verbose for hand-written queries.
Every column with a materially-typed downstream consumer should be cast explicitly. The general Snowflake rule of thumb: cast at the earliest project, not the last.

Output.

id	user_id	total	is_paid
1	u1	100	true
2	u2	250	false

Rule of thumb. In Snowflake, always cast the extracted VARIANT to a concrete SQL type at the earliest projection. The ::string, ::number, ::boolean casts pay for themselves in plan quality and downstream compatibility.

Senior interview question on portable path extraction

A senior interviewer might ask: "You're building a dbt package that must run on Postgres, Snowflake, BigQuery, and SQL Server. Write a portable macro that extracts a top-level scalar key from a JSON column and returns it as a string. What's the mental split you use for the four dialects?"

Solution Using a dbt cross-dialect macro dispatch

-- macros/json_scalar.sql
{% macro json_scalar(col, key) %}
  {%- if target.type == 'postgres' or target.type == 'redshift' -%}
    ({{ col }}->>'{{ key }}')
  {%- elif target.type == 'snowflake' -%}
    ({{ col }}:{{ key }}::string)
  {%- elif target.type == 'bigquery' -%}
    (JSON_VALUE({{ col }}, '$.{{ key }}'))
  {%- elif target.type == 'sqlserver' -%}
    (JSON_VALUE({{ col }}, '$.{{ key }}'))
  {%- elif target.type == 'mysql' -%}
    (JSON_UNQUOTE(JSON_EXTRACT({{ col }}, '$.{{ key }}')))
  {%- else -%}
    {{ exceptions.raise_compiler_error('json_scalar not supported on ' ~ target.type) }}
  {%- endif -%}
{% endmacro %}

-- usage in a model
SELECT
  id,
  {{ json_scalar('payload', 'user_id') }} AS user_id,
  {{ json_scalar('payload', 'session_id') }} AS session_id
FROM {{ ref('events') }};

Step-by-step trace.

Adapter	Rendered expression
postgres	`(payload->>'user_id')`
snowflake	`(payload:user_id::string)`
bigquery	`(JSON_VALUE(payload, '$.user_id'))`
sqlserver	`(JSON_VALUE(payload, '$.user_id'))`
mysql	`(JSON_UNQUOTE(JSON_EXTRACT(payload, '$.user_id')))`

Once the macro is defined, every model that reads a JSON column calls {{ json_scalar('payload', 'user_id') }} and the compiled SQL is dialect-correct without manual edits. The macro is the analytics-engineer's answer to the JSON dialect fracture.

Output:

Dialect	Portability outcome
postgres	Uses the arrow shortcut; identical plan to `jsonb_extract_path_text`
snowflake	Uses colon notation + cast; plan-eligible for VARIANT columnar prune
bigquery	Uses `JSON_VALUE`; native on `JSON` type, works on `STRING` too
sqlserver	Uses `JSON_VALUE`; scalar-only, returns NVARCHAR(4000) default
mysql	Uses `JSON_UNQUOTE(JSON_EXTRACT(...))`; safe for equality and GROUP BY

Why this works — concept by concept:

Adapter-dispatch is the portable pattern — every warehouse has a different function for the same concept, but dbt's target.type variable lets you pick the right one at compile time. The compiled SQL is dialect-native; only the source is portable.
Compile-time dispatch beats runtime abstraction — you don't want a runtime CASE that picks the syntax; the query planner would see all branches. Compile-time dispatch produces one clean SQL statement per target.
Scalar-vs-sub-tree matters even in the macro — the macro name json_scalar documents that the output is a scalar. A separate json_query macro handles sub-tree extraction. Never mix the two.
Fallback error — the else branch raises a compiler error for unsupported adapters. Better to fail loudly at compile time than to silently produce wrong SQL for an unknown adapter.
Cost — the macro is compile-time only; there is zero query-time overhead. Every dialect gets its native fast path. This is the analytics-engineer's answer to the JSON dialect fracture.

SQL
Topic — JSON
JSON path extraction problems

Practice →

SQL Topic — string parsing String parsing and extraction drills

Practice →

3. JSONB in Postgres — index it or regret it

`jsonb postgres` is the binary, deduped, indexable JSON type — pair it with a GIN index on the containment operator or every query becomes a full scan

The mental model in one line: Postgres ships two JSON types (json and jsonb) with subtly different semantics; only jsonb is indexable; the canonical index is a GIN with either jsonb_ops (universal) or jsonb_path_ops (containment-only, ~30% smaller); and the @> containment operator is the JSON equivalent of = ANY(...) that every senior data engineer knows cold. Once you say "jsonb + GIN + @> = fast JSON queries in Postgres," the interview surface collapses to a recipe list.

example card at the bottom; on a light PipeCode card." width="800" height="447">

jsonb vs json in Postgres.

json — raw text storage. Preserves whitespace, key order, and duplicates. No parsing on read (which is cheap on write, expensive on read). No indexing.
jsonb — binary parsed storage. Keys are deduped, whitespace is dropped, key order is not preserved. Parsing happens once on insert; every read is cheap. Indexable via GIN.
Storage: jsonb is typically ~5-15% smaller than the equivalent json (dedup + binary encoding beat text).
Write cost: jsonb is ~20-40% slower on insert because parsing happens up-front. Usually a great trade if the row is read many times.
Query cost: jsonb is dramatically faster on any read that navigates the sub-tree (arrow operators, @>, ?).

When to use json (rare).

You need to preserve key order or whitespace (RFC 7396 patches, canonical form for signing).
You never query on the JSON content, only pass it through.
You measured and confirmed the write cost dominates the workload.
For 99% of cases, jsonb is the right default.

GIN operator classes.

jsonb_ops — DEFAULT. Indexes every key AND every value. Supports @>, ?, ?&, ?|, and the SQL/JSON path operators @?, @@.
jsonb_path_ops — SMALLER. Indexes only @> containment. Doesn't support ? existence. ~30% smaller and faster to build; picks the right choice for 90% of production containment workloads.
Choose jsonb_path_ops when your read pattern is 100% containment (WHERE payload @> '{"status": "active"}').
Choose jsonb_ops when you also need key-existence queries (WHERE payload ? 'user_id').

GIN index recipe.

-- Universal — supports @>, ?, ?&, ?|
CREATE INDEX ix_events_payload_gin
  ON events USING GIN (payload);

-- Containment-only — smaller, faster to build
CREATE INDEX ix_events_payload_gin_path
  ON events USING GIN (payload jsonb_path_ops);

The @> containment operator.

payload @> '{"status": "active"}' — true if payload contains the given JSON as a sub-object.
Works recursively: payload @> '{"user": {"id": "u1"}}' matches any payload with a user sub-object containing id: u1.
Works for arrays: payload @> '{"tags": ["premium"]}' matches any payload where tags contains premium (among other elements).
GIN-indexable, unlike most JSON extraction predicates.
The canonical WHERE clause for JSON filtering in Postgres.

Existence operators (jsonb_ops only).

payload ? 'user_id' — true if payload has a top-level key user_id.
payload ?& array['user_id','ts'] — true if all listed keys exist.
payload ?| array['a','b'] — true if any listed key exists.

SQL/JSON path operators (Postgres 12+).

payload @? '$.user.id' — true if any value matches the path (existence check).
payload @@ '$.total > 100' — true if the SQL/JSON predicate matches.
Both are GIN-indexable via jsonb_ops.

jsonb_path_query for extraction with predicates.

jsonb_path_query(payload, '$.orders[*] ? (@.total > 100)') — returns every order with total > 100.
Set-returning; useful in LATERAL joins for row expansion.
Predicate expressions inside ? (...) — the SQL/JSON standard filter form.

jsonb_set and || for updates.

UPDATE events SET payload = jsonb_set(payload, '{status}', '"paid"') WHERE id = 1; — set a nested key.
UPDATE events SET payload = payload || '{"reviewed": true}'; — merge two objects (right side wins on key collision).
UPDATE events SET payload = payload - 'temp_field'; — delete a top-level key.
UPDATE events SET payload = payload #- '{user,temp}'; — delete a nested key.

Functional expression indexes for a single hot path.

-- Index a single extracted value — great for one hot filter
CREATE INDEX ix_events_user_id
  ON events ((payload->>'user_id'));

Use this when you only ever filter on one key. Cheaper than GIN, faster to plan.
The trade-off: only supports the exact expression indexed. WHERE payload->>'user_id' = 'u1' uses it; WHERE payload @> '{"user_id": "u1"}' does not.

Generated column materialisation.

-- Materialise a stable field as a plain column
ALTER TABLE events
  ADD COLUMN user_id text GENERATED ALWAYS AS (payload->>'user_id') STORED;

CREATE INDEX ix_events_user_id ON events (user_id);

The generated column stays in sync with the JSON automatically.
Query planner sees a plain column and picks the BTREE index without any expression magic.
Best pattern when the extracted field is a hot join key, group-by key, or sort key.

When plain TEXT beats jsonb.

The JSON is never queried, only passed through.
The write throughput is critical and the parse cost matters more than the read cost.
You need bit-identical JSON preservation (whitespace, key order, duplicates).
All three cases are rare; jsonb is the safe default.

Common jsonb postgres interview probes.

"When would you use json over jsonb?" — almost never; only when key order or whitespace preservation matters.
"Which GIN operator class supports the existence operator ??" — jsonb_ops (default); jsonb_path_ops does not.
"What's the containment operator, and why is it interesting?" — @>; the JSON equivalent of = ANY(...), GIN-indexable, the fastest filter in Postgres JSON.
"How would you index a single hot filter key without a GIN?" — a functional expression index on (payload->>'key') or a generated column with a BTREE.
"What does jsonb_set do that || doesn't?" — jsonb_set writes to a nested path; || only merges top-level keys.

Worked example — GIN index + containment filter for event routing

Detailed explanation. The most common Postgres JSON workload: an events table with 100M rows, filtered by an event-type-specific containment predicate (WHERE payload @> '{"kind": "purchase"}'). Without a GIN, this is a full scan. With a GIN, it's milliseconds.

Question. Given events(id, ts, payload jsonb) with 100M rows, write the DDL to add a GIN index on payload and the query that filters to purchase events. Explain the plan difference before and after the index.

Input.

id	ts	payload
1	2026-07-01	{"kind": "click", "user_id": "u1"}
2	2026-07-01	{"kind": "purchase", "user_id": "u1", "total": 100}
3	2026-07-01	{"kind": "view", "user_id": "u2"}

Code.

-- DDL — add a containment-only GIN for smaller footprint
CREATE INDEX CONCURRENTLY ix_events_payload_gin
  ON events USING GIN (payload jsonb_path_ops);

-- Query — filter to purchase events
SELECT id, ts, payload->>'user_id' AS user_id, (payload->>'total')::int AS total
FROM events
WHERE payload @> '{"kind": "purchase"}';

-- Before the index: Seq Scan on events (cost=0.00..3000000.00)
-- After the index:  Bitmap Heap Scan on events (cost=100.00..500.00)
--                     -> Bitmap Index Scan on ix_events_payload_gin (cost=0.00..100.00)

Step-by-step explanation.

CREATE INDEX CONCURRENTLY builds the GIN without blocking writes. On a 100M-row table, this takes minutes to hours; run it during low-load windows.
The jsonb_path_ops operator class is ~30% smaller than the default jsonb_ops and supports only @>. That's the right trade for this workload — we only filter with containment.
payload @> '{"kind": "purchase"}' is the containment predicate. The GIN maps every JSON path fragment to the row ids that contain it; the planner uses a bitmap of matching rows.
Plan before the index: sequential scan reads all 100M rows, evaluates the predicate on each. Cost is millions of tuple reads.
Plan after the index: bitmap index scan produces a set of ~100k candidate row ids (assuming 0.1% are purchases), then a bitmap heap scan visits each. Cost is 3-4 orders of magnitude lower.

Output.

id	ts	user_id	total
2	2026-07-01	u1	100

Rule of thumb. For any Postgres JSON column that will be queried by containment, add a GIN with jsonb_path_ops as the default. Upgrade to jsonb_ops only when you also need existence (?, ?&, ?|) operators.

Worked example — functional expression index for a single hot key

Detailed explanation. When the read pattern is 100% on one extracted key — say WHERE payload->>'user_id' = 'u1' — a full GIN is overkill. A functional expression index on the extraction is smaller and faster. Interviewers love this micro-optimisation because it separates "reads docs" from "reads plans."

Question. Given events(id, payload jsonb) and a workload that only filters on payload->>'user_id', write the DDL for a functional expression index and confirm it beats a GIN for this specific query.

Input.

id	payload
1	{"user_id": "u1", "kind": "click"}
2	{"user_id": "u2", "kind": "purchase"}
3	{"user_id": "u1", "kind": "view"}

Code.

-- Functional expression index on the extracted key
CREATE INDEX ix_events_user_id
  ON events ((payload->>'user_id'));

-- Query — uses the functional index
SELECT id, payload->>'kind' AS kind
FROM events
WHERE payload->>'user_id' = 'u1';

-- Plan: Index Scan using ix_events_user_id (cost=0.42..8.44 rows=2)

Step-by-step explanation.

The functional expression index stores payload->>'user_id' for every row and points at the row id. It's a BTREE, so range scans and equality both work at BTREE speed.
The index is much smaller than a GIN because it stores exactly one key per row (versus GIN's many entries per row for every JSON path fragment).
The query planner recognises the exact expression payload->>'user_id' in the WHERE clause and picks the functional index automatically. If you write the WHERE clause differently (e.g., payload @> '{"user_id": "u1"}'), the functional index does NOT match — a common gotcha.
Trade-off: this index only supports the one hot expression. If your workload later adds a second predicate (WHERE payload->>'session_id' = 's1'), you'd need a second functional index, or fall back to a GIN.
In practice, the choice comes down to workload cardinality. If you have 1-2 hot keys, use functional indexes. If you have 5+ query patterns, use a GIN.

Output.

id	kind
1	click
3	view

Rule of thumb. For a workload with one or two hot extraction keys, functional expression indexes beat GIN on size, build time, and plan quality. For a workload with many extraction patterns or containment queries, GIN wins.

Worked example — generated column + BTREE for a stable join key

Detailed explanation. When a JSON key is a stable join target (present in every row, drives the query planner's join order), materialising it as a generated column with a BTREE is the fastest option. This is Postgres 12+ and is the analytics-engineer's canonical answer for hot join keys.

Question. Given events(id, payload jsonb) where payload->>'user_id' is a hot join key against users(id), write the DDL to materialise user_id as a generated column with a BTREE. Show a JOIN query that benefits.

Input.

id	payload
1	{"user_id": "u1", "kind": "click"}
2	{"user_id": "u2", "kind": "purchase"}
3	{"user_id": "u1", "kind": "view"}

Code.

-- Materialise the join key
ALTER TABLE events
  ADD COLUMN user_id text GENERATED ALWAYS AS (payload->>'user_id') STORED;

CREATE INDEX ix_events_user_id ON events (user_id);

-- JOIN — planner sees a plain column
SELECT e.id, u.name, e.payload->>'kind' AS kind
FROM events e
JOIN users u ON u.id = e.user_id;

Step-by-step explanation.

GENERATED ALWAYS AS (payload->>'user_id') STORED creates a physical column that materialises the extraction. Every insert or update to payload re-evaluates the expression and stores the result.
The STORED keyword is required (Postgres only supports STORED, not VIRTUAL). It pays a small write cost and gives you a full-fidelity column at query time.
A plain BTREE index on user_id behaves exactly like any BTREE. The planner sees a normal column and picks the index for equality, range, and sort operations without any expression magic.
The JOIN ON u.id = e.user_id is a plain equi-join on two indexed columns. The planner picks a hash join or nested-loop join depending on cardinality — either way, it's optimal.
The trade-off is disk usage: the extracted value is stored twice (once in JSON, once in the generated column). For hot join keys, this is almost always worth it.

Output.

id	name	kind
1	Ada	click
2	Bo	purchase
3	Ada	view

Rule of thumb. For any JSON key that's stable, always present, and on a hot join / group-by / sort path, materialise it as a GENERATED ALWAYS AS (...) STORED column with a plain BTREE. This is the fastest pattern in Postgres 12+ and the analytics-engineer's default answer.

Senior interview question on Postgres JSONB tuning

A senior interviewer might ask: "You've inherited a Postgres table events(id, ts, payload jsonb) with 500M rows. The team has one query pattern: WHERE payload @> '{"kind": "..."}' AND ts > NOW() - INTERVAL '7 days'. Walk me through your indexing plan and why."

Solution Using a composite plan of BTREE + GIN

-- Step 1 — BTREE on the time dimension (partition-eligible)
CREATE INDEX ix_events_ts
  ON events (ts DESC);

-- Step 2 — GIN on the JSON payload for containment
CREATE INDEX CONCURRENTLY ix_events_payload_gin
  ON events USING GIN (payload jsonb_path_ops);

-- Step 3 — optional partial index for the hot event types
CREATE INDEX ix_events_purchase
  ON events (ts DESC)
  WHERE payload @> '{"kind": "purchase"}';

-- Step 4 — analyze so the planner has fresh stats
ANALYZE events;

Step-by-step trace.

Step	What it does	Why it matters
1	BTREE on `ts DESC`	Fast time-window pruning; enables merge/append/index scans
2	GIN(jsonb_path_ops) on payload	Fast containment; planner combines with BTREE via BitmapAnd
3	Partial BTREE on `ts DESC WHERE payload @> ...`	Ultra-fast hot-event scans; skips containment entirely
4	ANALYZE	Refreshes stats so the planner picks the right combination

The composite plan uses BTREE for time and GIN for containment; the planner combines the two via a BitmapAnd. For the ultra-hot event kinds where 90% of queries hit, the partial BTREE eliminates the containment step entirely.

Output:

Query pattern	Plan the planner picks
`ts > NOW() - 7 days`	BitmapAnd(BTREE(ts), GIN(payload)) — fastest general case
`payload @> '{"kind":"purchase"}' AND ts > ...`	Partial BTREE — skips containment
`payload @> '{"kind":"rare"}' AND ts > ...`	BitmapAnd — general case

Why this works — concept by concept:

BTREE on the time dimension — every OLTP + analytics workload with a ts column benefits from a descending BTREE. The DESC ordering matches the common "latest N days" pattern; the planner picks it for range scans and merge joins.
GIN on the JSON payload — the jsonb_path_ops operator class supports only @>, but that's the entire containment surface. Storing exactly the containment fragments makes the index ~30% smaller and faster than the default.
Partial indexes for hot filters — when 90% of queries hit one event kind, a partial index that indexes only those rows is dramatically smaller than a full index. The planner picks it automatically when the WHERE clause matches the partial's predicate.
BitmapAnd combines multiple indexes — Postgres's planner can intersect BTREE and GIN bitmaps to find rows that satisfy both predicates. This is why you don't need one giant multi-column index; two smaller indexes plus BitmapAnd is often faster.
Cost — the composite plan is O(log N) for the BTREE + O(log N) for the GIN. On a 500M row table, that's tens of thousands of index reads instead of 500M sequential scans — a 4-5 order-of-magnitude speedup.

SQL
Topic — JSON
JSONB indexing and containment drills

Practice →

SQL Topic — data manipulation Data manipulation problems

Practice →

4. JSON_TABLE — flatten JSON to rows

`json_table sql` is the ANSI SQL/JSON operator that turns a nested array into rows — Snowflake reaches for `LATERAL FLATTEN`, BigQuery for `UNNEST(JSON_QUERY_ARRAY(...))`, Postgres for `jsonb_to_recordset`

The mental model in one line: a nested JSON array cannot be joined against a relational query without a flattening operator that emits one row per array element, and every warehouse ships a different keyword for the same operation. Once you say "ANSI = JSON_TABLE, Snowflake = LATERAL FLATTEN, BigQuery = UNNEST(JSON_QUERY_ARRAY), Postgres = jsonb_array_elements / jsonb_to_recordset," the flattening interview surface reduces to a syntax lookup.

ANSI JSON_TABLE — the standard.

Shipped by Oracle (first, 12c), MySQL 8.0.4+, SQL Server 2022+, IBM Db2, and PostgreSQL 17+.
Syntax: JSON_TABLE(payload, '$.orders[*]' COLUMNS (id INT PATH '$.id', total DECIMAL(10,2) PATH '$.total')) AS orders.
Reads: "for each element of $.orders[*], emit a row whose id column comes from $.id and total column comes from $.total (relative to the array element)."
Used in a FROM clause with a LATERAL semantic (implicit) — the flattening is per-row.
Supports NESTED PATH sub-blocks for deeper flattening (see below).
Supports ERROR / NULL / DEFAULT ... ON EMPTY and ERROR / NULL / DEFAULT ... ON ERROR for missing/invalid values.

Snowflake LATERAL FLATTEN — the flagship for VARIANT.

Syntax: SELECT e.id, o.value:id::int AS order_id, o.value:total::number AS total FROM events e, LATERAL FLATTEN(input => e.payload:orders) o.
FLATTEN returns a table with columns SEQ, KEY, PATH, INDEX, VALUE, THIS. The interesting one is VALUE (the element).
Add OUTER => true to preserve rows where the flattened array is empty or NULL (like a LEFT JOIN).
Add RECURSIVE => true to recurse into nested objects/arrays automatically.
Great for semi-structured Snowflake VARIANT columns; the canonical Snowflake pattern.

BigQuery UNNEST(JSON_QUERY_ARRAY(...)).

Syntax: SELECT e.id, JSON_VALUE(o, '$.id') AS order_id, JSON_VALUE(o, '$.total') AS total FROM events e, UNNEST(JSON_QUERY_ARRAY(e.payload, '$.orders')) AS o.
JSON_QUERY_ARRAY returns an ARRAY<STRING> (or ARRAY<JSON> for native JSON) of the array elements.
UNNEST flattens the array to rows in a FROM clause.
Per-element extraction still uses JSON_VALUE / JSON_QUERY on the unnested element.
BigQuery also supports JSON_EXTRACT_ARRAY as a legacy alias for JSON_QUERY_ARRAY.

Postgres flattening options.

jsonb_array_elements(payload -> 'orders') — set-returning; emits one row per array element, each as jsonb. Used in a LATERAL join.
jsonb_to_recordset(payload -> 'orders') — set-returning; emits typed rows given a column list. SELECT * FROM jsonb_to_recordset(payload -> 'orders') AS o(id int, total numeric).
jsonb_array_elements_text(payload -> 'tags') — for string arrays; emits text.
Postgres 17+ ships full ANSI JSON_TABLE on jsonb columns.

Nested arrays — NESTED PATH.

ANSI JSON_TABLE supports a NESTED PATH '$.sub_array[*]' COLUMNS (...) sub-block for deeper flattening.
Reads: "for each element of the outer array, also flatten the inner array named sub_array, and emit one row per (outer, inner) pair."
Great for hierarchical payloads like {"user": "u1", "sessions": [{"session_id": "s1", "events": [{"kind": "click"}, {"kind": "view"}]}]}.

Null handling clauses.

NULL ON EMPTY — the default; missing paths produce NULL values in the output.
ERROR ON EMPTY — throw an error when the path doesn't match. Useful in strict-mode ETLs where a missing key is a bug.
DEFAULT 'x' ON EMPTY — substitute a default value.
ERROR ON ERROR — throw if the extracted value can't be coerced to the declared column type. Default is NULL ON ERROR.

When to flatten vs when to keep in JSON.

Flatten when the array elements feed downstream joins, aggregates, or filters that would benefit from a relational shape.
Keep in JSON when the array is a read-only projection that ships back to the client.
Never flatten a large array eagerly if the query only needs a few elements — use JSON_EXTRACT or jsonb_path_query with a predicate to pull only what you need.

Common json_table sql interview probes.

"Which dialects ship ANSI JSON_TABLE?" — Oracle, MySQL 8.0.4+, SQL Server 2022+, PostgreSQL 17+.
"What's the Snowflake equivalent?" — LATERAL FLATTEN(input => col).
"What's the BigQuery equivalent?" — UNNEST(JSON_QUERY_ARRAY(col, '$.path')).
"How would you handle a doubly-nested array?" — NESTED PATH sub-block in ANSI; chained LATERAL FLATTEN in Snowflake; chained UNNEST in BigQuery; chained LATERAL jsonb_array_elements in Postgres.
"What's the difference between JSON_TABLE and OPENJSON?" — SQL Server ships both; OPENJSON is Microsoft's older non-standard flattener, JSON_TABLE is the ANSI one (2022+). Prefer JSON_TABLE on new work.

Worked example — SQL Server JSON_TABLE for a nested orders array

Detailed explanation. The canonical ANSI flattening ask: given an events table where each row has a payload with a nested orders array, produce one row per (event, order) pair with typed columns. SQL Server 2022+ ships JSON_TABLE; older versions use OPENJSON.

Question. Given events(id, payload NVARCHAR(MAX)) where payload contains {"user": "u1", "orders": [{"id": 1, "total": 10}, {"id": 2, "total": 25}]}, write a SQL Server 2022 JSON_TABLE that produces (event_id, user, order_id, total). Explain the PATH clause and the COLUMNS list.

Input.

id	payload
100	{"user": "u1", "orders": [{"id": 1, "total": 10}, {"id": 2, "total": 25}]}
101	{"user": "u2", "orders": [{"id": 3, "total": 40}]}

Code.

SELECT
  e.id      AS event_id,
  JSON_VALUE(e.payload, '$.user') AS user,
  o.order_id,
  o.total
FROM events e
CROSS APPLY JSON_TABLE(
  e.payload,
  '$.orders[*]'
  COLUMNS (
    order_id INT           PATH '$.id',
    total    DECIMAL(10,2) PATH '$.total'
  )
) AS o;

Step-by-step explanation.

JSON_TABLE(e.payload, '$.orders[*]' COLUMNS (...)) reads: "for each element of $.orders[*] inside e.payload, emit a row with the declared columns."
The row-generating path $.orders[*] selects every element of the orders array. Each element becomes one output row.
Inside COLUMNS (...), each column has a name, a SQL type, and a PATH relative to the current array element. order_id INT PATH '$.id' pulls .id from the element, coerces to INT.
CROSS APPLY (SQL Server keyword) makes JSON_TABLE behave like a per-row function — for each event, the flattener runs against that event's payload.
The outer SELECT projects both the event columns (via e. alias) and the flattened columns (via o. alias). The result is one row per (event, order) pair.

Output.

event_id	user	order_id	total
100	u1	1	10.00
100	u1	2	25.00
101	u2	3	40.00

Rule of thumb. For any SQL Server 2022+ flattening task, prefer JSON_TABLE over OPENJSON — it's ANSI, portable to Oracle / MySQL / PostgreSQL 17+, and reads more clearly. Keep OPENJSON for legacy SQL Server 2017-2019 codebases.

Worked example — Snowflake LATERAL FLATTEN for VARIANT arrays

Detailed explanation. The canonical Snowflake flattening ask: given an events table with a VARIANT payload containing a nested orders array, produce one row per (event, order) pair. LATERAL FLATTEN is the flagship Snowflake pattern.

Question. Given a Snowflake events(id, payload VARIANT) table where each payload has an orders array, write a query using LATERAL FLATTEN that produces (event_id, user, order_id, total). Explain the input => argument and the OUTER => true option.

Input.

id	payload
100	{"user": "u1", "orders": [{"id": 1, "total": 10}, {"id": 2, "total": 25}]}
101	{"user": "u2", "orders": []}

Code.

-- INNER — drops rows where the array is empty or NULL
SELECT
  e.id                    AS event_id,
  e.payload:user::string  AS user,
  o.value:id::int         AS order_id,
  o.value:total::number   AS total
FROM events e,
     LATERAL FLATTEN(input => e.payload:orders) o;

-- OUTER — preserves rows with empty arrays (order_id / total NULL)
SELECT
  e.id                    AS event_id,
  e.payload:user::string  AS user,
  o.value:id::int         AS order_id,
  o.value:total::number   AS total
FROM events e,
     LATERAL FLATTEN(input => e.payload:orders, outer => true) o;

Step-by-step explanation.

LATERAL FLATTEN(input => e.payload:orders) reads: "for each event, flatten the orders array into rows." The input => argument is the array to flatten.
Each output row from FLATTEN has columns SEQ, KEY, PATH, INDEX, VALUE, THIS. VALUE is the array element as VARIANT; that's what you extract from.
o.value:id::int navigates into the element with colon-notation and casts to INT. Same colon syntax as any Snowflake VARIANT access.
The default is an inner join — events with empty orders arrays disappear from the output. To preserve them (like a LEFT JOIN), add outer => true.
LATERAL before FLATTEN is Snowflake syntax for "correlate this table function with the current outer row." Without LATERAL, FLATTEN can't reference e.payload.

Output (INNER variant).

event_id	user	order_id	total
100	u1	1	10
100	u1	2	25

Output (OUTER variant).

event_id	user	order_id	total
100	u1	1	10
100	u1	2	25
101	u2	NULL	NULL

Rule of thumb. In Snowflake, LATERAL FLATTEN is the default answer for any VARIANT flattening. Use outer => true when the outer row should survive empty arrays; use recursive => true when you want to explode nested sub-objects automatically.

Worked example — BigQuery UNNEST(JSON_QUERY_ARRAY) with per-element extraction

Detailed explanation. BigQuery's flattening pattern: use JSON_QUERY_ARRAY to get the array as an ARRAY<STRING> (or ARRAY<JSON> for native JSON), then UNNEST it in the FROM clause. Per-element extraction uses JSON_VALUE on the unnested element.

Question. Given a BigQuery events(id STRING, payload JSON) table where each payload has an orders array, write a query that produces (event_id, user, order_id, total). Show both the native-JSON form and the STRING-of-JSON form.

Input.

id	payload
100	{"user": "u1", "orders": [{"id": 1, "total": 10}, {"id": 2, "total": 25}]}
101	{"user": "u2", "orders": [{"id": 3, "total": 40}]}

Code.

-- Native JSON type
SELECT
  e.id                          AS event_id,
  JSON_VALUE(e.payload, '$.user') AS user,
  JSON_VALUE(o, '$.id')         AS order_id,
  JSON_VALUE(o, '$.total')      AS total
FROM events e,
     UNNEST(JSON_QUERY_ARRAY(e.payload, '$.orders')) AS o;

-- STRING-of-JSON — same shape, legacy alias
SELECT
  e.id                                 AS event_id,
  JSON_EXTRACT_SCALAR(e.payload, '$.user') AS user,
  JSON_EXTRACT_SCALAR(o, '$.id')       AS order_id,
  JSON_EXTRACT_SCALAR(o, '$.total')    AS total
FROM events e,
     UNNEST(JSON_EXTRACT_ARRAY(e.payload, '$.orders')) AS o;

Step-by-step explanation.

JSON_QUERY_ARRAY(e.payload, '$.orders') returns an ARRAY<JSON> (native) or ARRAY<STRING> (legacy) containing the array elements.
UNNEST(array) flattens the array to rows in a FROM clause. Each row is one element, aliased as o.
The implicit correlation between events e and UNNEST is what BigQuery calls a "correlated cross join" — it runs the UNNEST per outer row automatically. No LATERAL keyword needed.
Per-element extraction uses JSON_VALUE(o, '$.id') on the unnested element. Because the element is native JSON, we still need to path into it.
The STRING-of-JSON path (JSON_EXTRACT_ARRAY + JSON_EXTRACT_SCALAR) is the legacy form for STRING-typed columns. Use the native JSON path for new work; it has better plan characteristics on BigQuery's columnar engine.

Output.

event_id	user	order_id	total
100	u1	1	10
100	u1	2	25
101	u2	3	40

Rule of thumb. For BigQuery, always prefer native JSON columns + JSON_VALUE / JSON_QUERY_ARRAY on new work. Only use STRING-of-JSON + JSON_EXTRACT_SCALAR when you're stuck with a legacy schema — the migration cost is often worth the query-plan improvement.

Worked example — Postgres jsonb_to_recordset for typed flattening

Detailed explanation. Postgres's jsonb_to_recordset is the concise cousin of ANSI JSON_TABLE — it emits typed rows given a column list, without the PATH clauses. Great when the array elements are flat objects with a known schema.

Question. Given a Postgres events(id int, payload jsonb) table with a nested orders array of flat objects, write a query using jsonb_to_recordset that produces (event_id, user, order_id, total).

Input.

id	payload
100	{"user": "u1", "orders": [{"id": 1, "total": 10}, {"id": 2, "total": 25}]}
101	{"user": "u2", "orders": [{"id": 3, "total": 40}]}

Code.

-- jsonb_to_recordset — typed rows, no PATH clauses
SELECT
  e.id                      AS event_id,
  e.payload ->> 'user'      AS user,
  o.id                      AS order_id,
  o.total
FROM events e,
     LATERAL jsonb_to_recordset(e.payload -> 'orders') AS o(id int, total numeric);

Step-by-step explanation.

jsonb_to_recordset(jsonb_array) is a set-returning function that emits one row per array element. Each element must be a flat object whose keys match the requested column names.
The AS o(id int, total numeric) clause names the columns and their types. Postgres reads the JSON object per element and maps id → id int, total → total numeric.
LATERAL allows the function to reference e.payload — the outer row. Without LATERAL, Postgres can't correlate.
The outer SELECT projects both event columns (e.) and per-order columns (o.). The result is one row per (event, order) pair.
jsonb_to_recordset requires flat objects — nested keys don't work. For nested arrays or nested objects, fall back to jsonb_array_elements + LATERAL or PostgreSQL 17's ANSI JSON_TABLE.

Output.

event_id	user	order_id	total
100	u1	1	10
100	u1	2	25
101	u2	3	40

Rule of thumb. For Postgres < 17, jsonb_to_recordset is the concise pattern when the array elements are flat. For nested arrays, chain LATERAL jsonb_array_elements. For Postgres 17+, JSON_TABLE is the ANSI answer and the future default.

Senior interview question on nested-array flattening

A senior interviewer might ask: "You have events with a doubly-nested payload: each session has a list of events, and each event has a list of tags. Write a portable flattener across ANSI dialects that produces one row per (session, event, tag) — and explain why the flat product matters."

Solution Using NESTED PATH with JSON_TABLE

-- ANSI JSON_TABLE with NESTED PATH — Oracle, MySQL 8, SQL Server 2022, Postgres 17
SELECT
  s.id            AS session_id,
  e.event_id,
  e.event_kind,
  t.tag
FROM sessions s
CROSS APPLY JSON_TABLE(
  s.payload,
  '$.events[*]'
  COLUMNS (
    event_id   INT       PATH '$.id',
    event_kind VARCHAR(32) PATH '$.kind',
    NESTED PATH '$.tags[*]'
      COLUMNS (tag VARCHAR(64) PATH '$')
  )
) AS e
JOIN tags_registry t ON t.tag = e.tag;
-- CROSS APPLY == LATERAL in ANSI dialects; syntax varies per engine.

Step-by-step trace.

Step	What runs	Cardinality
1	Outer `sessions` scan	N sessions
2	Row-generator path `$.events[*]` fires per session	N × avg(events/session)
3	`NESTED PATH '$.tags[*]'` fires per event	N × avg(events) × avg(tags)
4	Each (event, tag) pair emits a row	Cartesian per level
5	Outer join to `tags_registry` filters/joins tags	Final result set

The nested-path form emits one row per leaf (session, event, tag) tuple. Every nesting level multiplies the cardinality — this is the "flat product" and is exactly what a relational join wants.

Output:

session_id	event_id	event_kind	tag
s1	1	click	premium
s1	1	click	mobile
s1	2	view	premium
s2	3	purchase	premium
s2	3	purchase	high-value

Why this works — concept by concept:

NESTED PATH nests inside COLUMNS — the outer PATH '$.events[*]' generates one row per event; the inner NESTED PATH '$.tags[*]' generates one row per (event, tag) pair. The two combine into the Cartesian product per session.
PATH '$' refers to the current scalar — inside NESTED PATH, the column mapping tag VARCHAR(64) PATH '$' reads: "the current tag element is a scalar; take its value as-is." Useful for arrays of strings, not objects.
Cartesian expansion is desired — a row per leaf tuple is what enables relational joins downstream. Every session, every event, every tag becomes one row that can join, filter, aggregate.
NULL handling on empty nested arrays — the default is NULL ON EMPTY; sessions with no tags still emit event rows with tag = NULL. Add ERROR ON EMPTY if you want strict schemas.
Cost — cost is O(sessions × avg(events) × avg(tags)). For big multipliers, this can be a lot of rows — the interviewer expects you to recognise the cardinality risk and consider filtering before flattening.

SQL
Topic — data transformation
Data transformation problems

Practice →

SQL Topic — JSON JSON flattening drills

Practice →

5. Dialect matrix + performance

The five-engine JSON matrix: `bigquery json` vs `snowflake variant` vs Postgres jsonb vs SQL Server JSON vs MySQL JSON — storage, index, and pruning tell the whole story

The mental model in one line: every engine has a JSON storage type, an index shape, and a pruning story — Postgres pairs jsonb with GIN, MySQL pairs JSON with a functional index on a generated column, SQL Server pairs NVARCHAR(MAX) with a computed column + index, Snowflake pairs VARIANT with automatic clustering + micro-partition prune, BigQuery pairs the native JSON type with partitioning + clustering on a materialised column, and the interviewer wants to see you name the pair for each dialect. Once you say the pair out loud, the performance interview surface reduces to a lookup.

Storage cost across five engines.

Postgres jsonb — binary encoded, deduped keys. Typically 5-15% smaller than raw JSON. Fixed per-row overhead of ~4 bytes; body scales with content.
Postgres json — raw text. Ordered, no dedup. ~10-15% larger than jsonb.
MySQL JSON — binary encoded, similar dedup story to jsonb. Slightly larger than jsonb due to different binary format.
SQL Server — no dedicated JSON type; stored as NVARCHAR(MAX). Raw text. ~2× larger than jsonb (Unicode) unless compressed.
Snowflake VARIANT — columnar-encoded semi-structured storage. Each JSON path becomes a virtual sub-column; storage is per-path columnar. Typically 20-50% smaller than raw JSON due to columnar compression.
BigQuery native JSON — columnar storage similar to Snowflake VARIANT. Per-path pruning at query time. Much smaller than STRING-of-JSON.

Index strategies.

Postgres jsonb + GIN — the canonical pair. jsonb_path_ops for containment-only; jsonb_ops for the full operator set.
Postgres jsonb + functional expression index — for one hot path: CREATE INDEX ON events ((payload->>'user_id')).
Postgres jsonb + generated column + BTREE — the analytics-engineer's pattern; materialise the extraction, index the column.
MySQL JSON + generated column + BTREE — MySQL doesn't support functional indexes on JSON expressions; the workaround is a GENERATED ... VIRTUAL / STORED column + a normal BTREE.
SQL Server + computed column + index — ALTER TABLE events ADD user_id AS JSON_VALUE(payload, '$.user_id') PERSISTED; CREATE INDEX ix_user_id ON events(user_id);.
Snowflake VARIANT + automatic clustering — Snowflake clusters micro-partitions on any JSON path automatically; no explicit index required. ALTER TABLE events CLUSTER BY (payload:user_id::string); for explicit clustering.
BigQuery + partitioning + clustering — partition by ts, cluster by a materialised JSON extraction. Native JSON columns support direct clustering on JSON_VALUE(payload, '$.user_id').

Pruning stories per engine.

Postgres — GIN doesn't prune whole pages; it produces a bitmap of candidate rows. BTREE indexes on materialised columns prune at the page level.
MySQL — BTREE on a generated column prunes pages at BTREE speed. No JSON-native pruning.
SQL Server — computed column index prunes pages; JSON functions themselves don't prune.
Snowflake VARIANT — micro-partition prune on any clustered JSON path. If a query filters on payload:user_id::string = 'u1' and the table is clustered on that expression, Snowflake reads only the micro-partitions containing u1. Massive win for large tables.
BigQuery native JSON — partition prune on ts; cluster prune on any clustered materialised JSON extraction. Columnar storage means only the relevant JSON path is read — often 90% cheaper than STRING-of-JSON.

When a materialised column beats JSON extraction.

Hot filter path. WHERE JSON_VALUE(payload, '$.status') = 'active' — every planner struggles with this. Materialise status as a column, index the column, done.
Join key. JOIN users ON JSON_VALUE(payload, '$.user_id') = users.id — never faster than a plain column join. Materialise.
Aggregate key. GROUP BY JSON_VALUE(payload, '$.plan') — materialise for planner-friendly hash grouping.
Sort key. ORDER BY JSON_VALUE(payload, '$.created_at') — materialise + index.
Anything hit by dashboards. BI tools generate queries that assume plain columns; materialise every column a dashboard filters on.

When to keep the field inside JSON.

Rare projections. A field only surfaced in one report per month; the JSON extraction overhead is negligible.
Schema drift. Fields that change across event types (event-specific payloads); materialising each variant creates schema explosion.
Deep sub-trees. Objects that are always consumed as sub-trees (never as scalars) — extract the sub-tree at query time.

Snowflake VARIANT + AUTOMATIC_CLUSTERING deep dive.

Snowflake's storage engine indexes JSON paths automatically as it writes. Each micro-partition tracks min/max stats per JSON path.
Query planner uses these stats to skip micro-partitions that can't match the predicate.
For a query like WHERE payload:user_id::string = 'u1', Snowflake reads only the micro-partitions where user_id includes u1 in its min/max range.
Explicit CLUSTER BY (payload:user_id::string) forces the physical layout to concentrate matching rows in fewer micro-partitions — accelerates automatic clustering.
Great for wide tables with many JSON paths and workloads that filter on many different paths across queries.

BigQuery native JSON vs STRING-of-JSON.

Native JSON type — columnar per-path storage. Only the queried paths are read from disk. Pruning is native.
STRING-of-JSON — the whole column is read as text, then parsed on the fly per row. Massively more expensive.
BigQuery bills by bytes read; native JSON bills only for the paths you extract. A WHERE JSON_VALUE(payload, '$.status') query on a 10TB table might read 100GB (native) vs 10TB (STRING). 100× cost difference.
Migration path: CREATE TABLE events_native AS SELECT id, ts, PARSE_JSON(payload) AS payload FROM events_string; DROP TABLE events_string; (with appropriate care for pipelines).

Common bigquery json performance probes.

"How would you migrate a 10TB STRING-of-JSON table to native JSON?" — PARSE_JSON in a CREATE TABLE AS SELECT, verify with a dry-run cost estimate, cut over with a view rename.
"What's the cost difference between JSON_VALUE on native vs STRING?" — often 10-100× because native only reads the queried path.
"How do you cluster a BigQuery JSON table?" — cluster on a materialised JSON_VALUE(payload, '$.hot_key') column; the JSON column itself is not clusterable.

Common snowflake variant performance probes.

"What is AUTOMATIC_CLUSTERING?" — Snowflake's background service that reorganises micro-partitions to keep them well-clustered; runs continuously.
"How does micro-partition prune work on a JSON path?" — Snowflake tracks min/max per path per micro-partition; the planner skips partitions whose stats can't match the predicate.
"When would you ALTER TABLE CLUSTER BY a VARIANT column?" — when queries filter heavily on one path and automatic clustering isn't converging fast enough.

Worked example — Postgres jsonb vs generated column + BTREE benchmarking

Detailed explanation. The most common tuning debate: "should I index the JSON column with GIN, or materialise the hot path as a generated column with a BTREE?" The answer depends on query patterns. Show the interviewer a concrete comparison.

Question. Given a Postgres events(id, ts, payload jsonb) table with 100M rows, compare the plan and speed of three options for WHERE payload->>'user_id' = 'u1': (a) no index, (b) GIN on payload, (c) generated column + BTREE.

Input.

id	ts	payload
1	2026-07-01	{"user_id": "u1", "kind": "click"}
...	...	...

Code.

-- Option A — no index
-- Plan: Seq Scan on events (cost=0.00..3000000.00 rows=100k)

-- Option B — GIN on payload (jsonb_ops for @>)
CREATE INDEX ix_events_gin ON events USING GIN (payload jsonb_path_ops);

-- Query rewrite for GIN to be used
SELECT * FROM events WHERE payload @> '{"user_id": "u1"}';
-- Plan: Bitmap Heap Scan on events (cost=100..500 rows=100k)

-- Option C — generated column + BTREE
ALTER TABLE events
  ADD COLUMN user_id text GENERATED ALWAYS AS (payload->>'user_id') STORED;

CREATE INDEX ix_events_user_id ON events (user_id);

SELECT * FROM events WHERE user_id = 'u1';
-- Plan: Index Scan using ix_events_user_id (cost=0.42..12.44 rows=100k)

Step-by-step explanation.

Option A (no index): a full sequential scan on 100M rows. Runtime scales linearly with table size — unacceptable for interactive queries.
Option B (GIN + rewrite to @>): the GIN indexes containment fragments; the query planner picks a bitmap heap scan. Fast, but requires rewriting the query from ->> to @>.
Option C (generated column + BTREE): the plain BTREE is the fastest option for equality. The generated column stays in sync automatically. Query is unchanged from the "plain column" mental model.
Trade-offs: GIN supports many query patterns (containment, existence) with one index. Generated column + BTREE supports one pattern per column but is faster and smaller.
Rule of thumb: for 1-2 hot keys, generated column + BTREE wins. For many query patterns or deep containment, GIN wins.

Output (plan-and-timing comparison).

Option	Plan	Runtime (100M rows, sel=0.1%)
No index	Seq Scan	30-60s
GIN + `@>` rewrite	Bitmap Heap Scan	50-200ms
Generated column + BTREE	Index Scan	5-20ms

Rule of thumb. When the workload is dominated by 1-2 hot extractions with plain equality, always materialise as generated columns with BTREEs. Fall back to GIN when you need containment (@>) or existence (?) semantics.

Worked example — Snowflake VARIANT clustering for a hot user_id filter

Detailed explanation. Snowflake's flagship performance pattern: cluster a VARIANT column on a hot JSON path. Micro-partition prune reads only the relevant partitions, cutting query cost dramatically.

Question. Given a Snowflake events(id, ts, payload VARIANT) table with 10TB and a hot query pattern WHERE payload:user_id::string = 'u1', write the DDL to cluster on the user_id path and estimate the pruning gain.

Input.

id	ts	payload
1	2026-07-01	{"user_id": "u1", "kind": "click"}
...	...	...

Code.

-- Cluster the table on the hot JSON path
ALTER TABLE events CLUSTER BY (payload:user_id::string);

-- Run the query
SELECT id, ts, payload:kind::string AS kind
FROM events
WHERE payload:user_id::string = 'u1'
  AND ts > CURRENT_TIMESTAMP - INTERVAL '7 days';

-- Snowflake will automatically re-cluster in the background;
-- explicit CLUSTER BY declaration accelerates convergence.

Step-by-step explanation.

ALTER TABLE events CLUSTER BY (payload:user_id::string) declares the clustering key. Snowflake's AUTOMATIC_CLUSTERING service will start reorganising micro-partitions in the background.
Once well-clustered, matching user_id values are concentrated in a small subset of micro-partitions. Snowflake's per-partition min/max stats let the planner skip the rest.
For a table with 10TB in 100k micro-partitions and a hot user with 0.01% of rows, well-clustered reads scan ~10-100 micro-partitions (1-10GB), not the full 10TB.
The query also filters on ts > NOW() - 7 days. Snowflake tables are implicitly partitioned by insert order, so time-based filters also prune. Combining both cuts cost by 5-6 orders of magnitude.
Cost model: Snowflake bills by bytes scanned. A well-clustered query on this pattern is often <$0.01; the same query on an unclustered table is $10-100.

Output.

id	ts	kind
1	2026-07-01	click
...	...	...

Rule of thumb. For any Snowflake VARIANT table with a hot JSON-path filter, declare CLUSTER BY (payload:path::type). Automatic clustering handles the rest. The pruning gain is often 100-10000× on large tables.

Worked example — BigQuery native JSON migration cost analysis

Detailed explanation. Migrating a 10TB STRING-of-JSON table to native JSON is often a 10-100× cost win. The math is simple: BigQuery only reads the queried JSON paths on native columns; on STRING-of-JSON, it reads the whole column and parses on the fly.

Question. Given a BigQuery events_string(id, ts, payload STRING) table with 10TB and daily queries that extract 3 out of 20 payload keys, estimate the cost saving of migrating to a native JSON column and show the migration DDL.

Input.

id	ts	payload (STRING)
1	2026-07-01	{"user_id":"u1","kind":"click","other_18_keys":"..."}

Code.

-- Step 1 — create the native JSON table via CTAS
CREATE TABLE `project.dataset.events_native`
PARTITION BY DATE(ts)
CLUSTER BY user_id_mat  -- optional materialised column
AS
SELECT
  id,
  ts,
  PARSE_JSON(payload) AS payload,
  JSON_VALUE(payload, '$.user_id') AS user_id_mat
FROM `project.dataset.events_string`;

-- Step 2 — verify with a dry-run query
SELECT COUNT(*) FROM `project.dataset.events_native`
WHERE JSON_VALUE(payload, '$.user_id') = 'u1';
-- Dry run reports bytes processed — expect ~5% of the original.

-- Step 3 — swap via view
CREATE OR REPLACE VIEW `project.dataset.events` AS
SELECT * FROM `project.dataset.events_native`;

Step-by-step explanation.

PARSE_JSON(payload) converts the STRING column into native JSON. BigQuery stores the parsed representation columnar — one virtual column per JSON path.
The CTAS also creates a materialised user_id_mat column that's cluster-eligible. Native JSON columns can't be clustered directly; the workaround is a materialised extraction.
Cost model: STRING-of-JSON queries scan the entire payload column (say 8TB out of 10TB). Native JSON queries scan only the queried paths (say 400GB — 5% of total). 20× cost saving.
Migration cost: one-time CTAS scans 10TB (~$50 at $5/TB), the ongoing query cost drops 20×.
Downstream compatibility: existing queries that use JSON_EXTRACT_SCALAR on a STRING column need to be rewritten to JSON_VALUE on the native column. Use a view during the transition to minimise breakage.

Output (cost estimate).

Query pattern	STRING-of-JSON cost	Native JSON cost	Saving
Filter on 1 key, project 3 keys	10TB scan	500GB scan	95%
Filter on 5 keys	10TB scan	2TB scan	80%
Aggregate over the full payload	10TB scan	10TB scan	~0%

Rule of thumb. For any BigQuery table above 100GB where JSON queries touch a small subset of keys, migrate to the native JSON type. The cost saving pays for the migration in weeks or days, and the query plans improve dramatically.

Senior interview question on choosing the right JSON stack

A senior interviewer might ask: "You're greenfielding a data platform that will ingest 100M events/day, each with a 2KB JSON payload. You need to serve ad-hoc analytics with sub-10s latency. Which warehouse do you pick, which storage type, which index / cluster strategy, and why?"

Solution Using a 5-question warehouse-picking framework

Warehouse + JSON storage picker — 5 questions

1. Read pattern — ad-hoc analytics or point reads?
   ad-hoc     → columnar warehouse (Snowflake / BigQuery / Redshift)
   point read → OLTP (Postgres + jsonb)

2. Volume — is it above 10TB in the JSON column?
   yes → Snowflake VARIANT or BigQuery native JSON (columnar per-path)
   no  → Postgres jsonb (GIN + generated cols is enough)

3. Filter selectivity — do most queries filter on 1-2 hot keys?
   yes → cluster / partition on materialised extractions
   no  → let automatic clustering handle it (Snowflake) or add many gen cols

4. Schema drift — how fast do new keys arrive?
   fast (weekly) → VARIANT / native JSON — schema-on-read
   slow (monthly) → promote hot keys to plain columns

5. Portability — one warehouse or multi-cloud?
   one warehouse → pick the native stack (VARIANT / native JSON / jsonb)
   multi         → dbt macro dispatch, minimise dialect-specific SQL

Step-by-step trace.

Requirement	Q1	Q2	Q3	Q4	Q5	Pick
100M events/day, ad-hoc, sub-10s	ad-hoc	20TB/mo	2 hot	fast	one	Snowflake VARIANT + cluster on user_id
100M events/day, OLTP + BI	mixed	20TB/mo	5 hot	fast	one	BigQuery native JSON + clustering
1M rows/day, transactional	point	1GB	1 hot	slow	one	Postgres jsonb + generated col + BTREE
Multi-cloud, portable analytics	ad-hoc	5TB/mo	3 hot	fast	multi	Postgres jsonb (self-hosted) + dbt macros

The framework turns "which warehouse and how do I store JSON?" from a taste question into a mechanical decision. The answers pop out from the volume + read pattern + schema drift axes.

Output:

Pick	Storage	Index / cluster	Why
Snowflake VARIANT	VARIANT columnar	cluster on hot path	best columnar + auto-clustering
BigQuery native JSON	native JSON	partition + cluster on materialised col	best cost model for path-heavy queries
Postgres jsonb	jsonb binary	GIN + generated col + BTREE	best OLTP + operational simplicity
SQL Server JSON	NVARCHAR(MAX)	computed col + BTREE	best on Windows/enterprise stacks
MySQL JSON	JSON binary	generated col + BTREE	best on MySQL-only shops

Why this works — concept by concept:

Read pattern drives the warehouse choice — point reads are OLTP (Postgres); ad-hoc analytics are OLAP (Snowflake/BigQuery). Everything else follows.
Volume drives the storage type — above 10TB, columnar per-path storage wins on cost by orders of magnitude. Below 10TB, jsonb is fine and often simpler.
Hot keys drive the index strategy — one hot key wants a plain generated column + BTREE; five wants clustering; many wants a GIN.
Schema drift decides what to materialise — fast-drifting fields stay in the JSON payload forever; slow-drifting fields eventually get promoted to plain columns.
Cost — every path has a query-plan cost and a storage cost that scale differently. The framework picks the stack where the cost curve matches the workload — the definition of a good design decision.

SQL
Topic — SQL
SQL interview problem library

Practice →

SQL
Topic — JSON
JSON storage and indexing drills

Practice →

Cheat sheet — SQL JSON recipe list

Path extraction across five dialects. Postgres: payload->>'key' or jsonb_path_query_first(payload, '$.key'). MySQL 8: payload->>'$.key' or JSON_VALUE(payload, '$.key' RETURNING VARCHAR(64)). SQL Server: JSON_VALUE(payload, '$.key'). Snowflake: payload:key::string. BigQuery: JSON_VALUE(payload, '$.key').
Scalar vs sub-tree split. JSON_VALUE / ->> returns a scalar SQL type; JSON_QUERY / -> returns a JSON sub-tree. Never mix them — the interviewer will catch it.
JSONPath primitives. $ (root), .key (child), [i] (index), [*] (wildcard), ..key (recursive descent — not SQL Server), [?(@.total > 100)] (filter — Postgres 12+ and MySQL 8).
Postgres jsonb GIN index recipe. CREATE INDEX ix ON t USING GIN (payload jsonb_path_ops); for containment-only (smaller, faster). Use jsonb_ops when you also need ? existence.
Postgres containment operator. WHERE payload @> '{"kind":"purchase"}' — GIN-indexable, the fastest JSON filter in Postgres.
Postgres functional expression index. CREATE INDEX ON t ((payload->>'user_id')); for one hot key. Query must use the exact same expression.
Postgres generated column + BTREE. ADD COLUMN user_id text GENERATED ALWAYS AS (payload->>'user_id') STORED; + CREATE INDEX ON t (user_id);. Best for hot join / group-by keys.
JSON_TABLE flatten recipe. SELECT ... FROM t CROSS APPLY JSON_TABLE(payload, '$.orders[*]' COLUMNS (id INT PATH '$.id', total DECIMAL PATH '$.total')) AS o;. Oracle, MySQL 8, SQL Server 2022+, Postgres 17+.
Snowflake VARIANT + FLATTEN recipe. SELECT o.value:id::int, o.value:total::number FROM t, LATERAL FLATTEN(input => t.payload:orders) o;. Add OUTER => true for LEFT-JOIN semantics.
BigQuery UNNEST(JSON_QUERY_ARRAY) recipe. SELECT JSON_VALUE(o, '$.id'), JSON_VALUE(o, '$.total') FROM t, UNNEST(JSON_QUERY_ARRAY(payload, '$.orders')) AS o;.
Postgres jsonb_to_recordset recipe. SELECT o.id, o.total FROM t, LATERAL jsonb_to_recordset(t.payload -> 'orders') AS o(id int, total numeric);. Flat objects only.
NESTED PATH. ANSI JSON_TABLE sub-block that flattens a second-level array: NESTED PATH '$.tags[*]' COLUMNS (tag VARCHAR PATH '$'). Emits one row per (outer, inner) pair.
Null-safe extraction pattern. COALESCE(payload->>'key', 'default') in Postgres; IFNULL(JSON_VALUE(payload, '$.key'), 'default') in MySQL / Snowflake; ISNULL(JSON_VALUE(payload, '$.key'), 'default') in SQL Server; IFNULL(JSON_VALUE(payload, '$.key'), 'default') in BigQuery.
Materialised column indexing recipe. Any hot JSON key that participates in filter, join, group-by, or sort should be materialised as a generated / computed column and indexed with a BTREE. Universal across dialects.
Snowflake clustering recipe. ALTER TABLE t CLUSTER BY (payload:hot_key::string);. Automatic clustering handles the rest; monitor with SYSTEM$CLUSTERING_INFORMATION.
BigQuery native JSON migration recipe. CREATE TABLE new AS SELECT id, ts, PARSE_JSON(payload) AS payload FROM old;. Then swap via view for zero-downtime cutover.
Containment vs equality. @> (Postgres) tests JSON sub-object containment; = '{"key":"val"}' tests full-object equality including key order (broken by jsonb dedup). Use @> for filtering, never =.
When to use json over jsonb in Postgres. Almost never — only when key order or whitespace preservation matters (RFC 7396 patches, signature contexts).
dbt cross-dialect dispatch. Write a json_scalar(col, key) macro that dispatches on target.type to emit Postgres ->> vs Snowflake ::string vs BigQuery JSON_VALUE. One source, dialect-native compiled SQL.
MySQL JSON_UNQUOTE gotcha. JSON_EXTRACT returns quoted strings; always wrap with JSON_UNQUOTE or use ->> when the value feeds a WHERE, GROUP BY, or JOIN.

Frequently asked questions

What is the difference between `JSON_VALUE` and `JSON_EXTRACT`?

JSON_VALUE returns a scalar SQL value (usually VARCHAR or a typed value via RETURNING) — designed for equality, aggregation, and typed downstream use; it errors or returns NULL when the target is a JSON object or array. JSON_EXTRACT is dialect-specific — MySQL and SQLite ship it, and it returns a JSON value that may be a scalar, an object, or an array with quotes preserved on strings. In MySQL 8, JSON_EXTRACT(payload, '$.name') returns "Ada" (with quotes) but JSON_VALUE(payload, '$.name') returns Ada (unquoted). The interviewer expects you to name the split — scalar vs sub-tree, quoted vs unquoted — in the first sentence. In SQL Server, BigQuery, and Oracle, the standard split is JSON_VALUE (scalar) vs JSON_QUERY (sub-tree); in Postgres, the arrow operators ->> (scalar text) vs -> (JSON child) do the same job.

When should I use `jsonb` over `json` in Postgres?

Almost always use jsonb. The json type stores raw text — preserving whitespace, key order, and duplicate keys — but is not indexable and requires re-parsing on every read. The jsonb type stores a binary parsed representation — deduped keys, sorted internally, indexable with GIN — and is ~5-15% smaller. Write cost is ~20-40% higher for jsonb (parse on insert), but read cost is dramatically lower because navigation is O(1) per key rather than O(n) text-parse. Reach for json only in narrow cases: preserving RFC 7396 patch documents, canonical signing forms, or when the JSON is never queried and only passed through. For every OLTP or analytics workload where the JSON is read more than once, jsonb is the correct default.

How do I flatten a JSON array into rows across dialects?

Each engine ships a different keyword for the same operation. ANSI SQL/JSON JSON_TABLE — supported by Oracle 12c+, MySQL 8.0.4+, SQL Server 2022+, and Postgres 17+ — is the standard: JSON_TABLE(payload, '$.orders[*]' COLUMNS (id INT PATH '$.id', total DECIMAL PATH '$.total')). Snowflake uses LATERAL FLATTEN(input => payload:orders) and per-element extraction via value:field::type. BigQuery pairs UNNEST(JSON_QUERY_ARRAY(payload, '$.orders')) with per-element JSON_VALUE. Older Postgres (< 17) uses jsonb_array_elements or jsonb_to_recordset in a LATERAL join. For doubly-nested arrays, ANSI JSON_TABLE supports a NESTED PATH '$.sub[*]' COLUMNS (...) sub-block; Snowflake chains LATERAL FLATTEN; BigQuery chains UNNEST; Postgres chains LATERAL jsonb_array_elements. The output shape is always the same — one row per leaf-level tuple — because that's what a relational query wants.

Does BigQuery support the native JSON type or do I use STRING?

BigQuery ships a native JSON type since 2022, and it's the strongly preferred storage for JSON payloads on new tables. Native JSON stores data columnar per JSON path — meaning a query that extracts one key only reads the bytes for that path, not the whole payload. On a 10TB bigquery json table, a hot-key filter query typically bills 100-500GB scanned (native) versus 10TB scanned (STRING-of-JSON) — a 20-100× cost improvement plus a proportional speed improvement. The migration path is a CREATE TABLE ... AS SELECT id, ts, PARSE_JSON(payload) AS payload FROM old_table;, followed by a view rename for zero-downtime cutover. Legacy STRING-of-JSON columns with JSON_EXTRACT_SCALAR still work, but every new table and every migration budget should target the native type.

What is Snowflake VARIANT and how does it index JSON paths?

snowflake variant is Snowflake's self-describing semi-structured type — a single column that holds JSON, Avro, or Parquet-like nested data. Storage is columnar per JSON path (a "virtual sub-column" per path), which means selective extraction only reads the queried paths from disk, not the whole payload. Snowflake's AUTOMATIC_CLUSTERING service tracks min/max stats per JSON path per micro-partition; when a query filters on payload:user_id::string = 'u1', the planner skips micro-partitions whose stats can't match. For hot filter paths, declare an explicit ALTER TABLE t CLUSTER BY (payload:hot_key::string) to accelerate physical layout convergence — automatic clustering will do it eventually, but explicit declarations converge faster. The typical result on a 10TB table with a well-clustered VARIANT column: a hot-user filter reads 1-10GB instead of 10TB, a 3-4 order-of-magnitude speedup.

Is JSON in SQL fast enough to replace a document store?

For read-heavy analytics workloads with warehouse-scale volumes, the modern answer is yes — Snowflake VARIANT, BigQuery native JSON, and Postgres jsonb all outperform document stores on ad-hoc analytical queries, largely because columnar storage and GIN-indexed containment beat MongoDB's default point-read profile at scale. Where document stores still win: sub-10ms OLTP point reads at high QPS (DynamoDB, MongoDB), heavy in-place sub-document updates (MongoDB's atomic operators), and write throughput above 100k QPS. The right question isn't "warehouse vs document store" but "what's the read/write ratio and latency budget?" For 90% of data-engineering workloads — event log analytics, CDC pipelines, feature stores, config tables — JSON-in-SQL is the correct default because it collocates the semi-structured data with the relational world, reducing your operational surface to one system, one backup, one auth model. Document stores earn their keep in the remaining 10% of point-read-heavy OLTP.

Practice on PipeCode

Drill the JSON practice library → for path extraction, containment, and JSON_TABLE flattening reps across all five dialects.
Rehearse on parsing problems → when the interviewer wants a string-to-structure conversion.
Sharpen data-transformation drills → for wide-to-long, long-to-wide, and JSON-to-relational reshapes.
Layer the string-parsing library → for the tokenisation primitives that underpin every JSON path.
Stack the data manipulation library → for UPDATE, MERGE, and jsonb_set interview probes.
For general SQL sharpening, work through the SQL problem library → which contains 450+ DE-focused questions.
For the broader SQL interview surface, take the SQL for Data Engineering course →.

Pipecode.ai is Leetcode for Data Engineering — every `sql json` recipe above ships with hands-on practice rooms where you write the Postgres `jsonb` GIN filter, wire the Snowflake `LATERAL FLATTEN`, chase the BigQuery native-JSON migration, and rehearse the `JSON_TABLE` interview walk-through against real graded inputs. PipeCode pairs every reading with 450+ DE-focused problems and a real-time scoring engine, so you never have to wonder whether your `json_value sql` answer holds up under a senior interviewer's depth probes.

Practice JSON patterns now →
Data transformation drills →

SQL PIVOT / UNPIVOT / CROSSTAB Across Dialects (Postgres, Snowflake, BigQuery, SQL Server)

Gowtham Potureddi — Fri, 10 Jul 2026 17:29:59 +0000

sql pivot is the single most-searched reshaping keyword in analytics SQL — and the single most dialect-fractured one in 2026. The same "turn a long monthly-sales table into a wide one column per month" ask has four different answers across the four warehouses most data teams live in: Postgres uses tablefunc.crosstab, Snowflake ships a full ANSI PIVOT with a dynamic ANY ORDER BY form, BigQuery ships PIVOT as a table-function on a subquery, and SQL Server ships the OG PIVOT with an alias table and bracketed labels. There is no portable spelling; the interviewer wants to hear you say that out loud in the first sentence.

This guide is the mid-to-senior comparison you wished existed the first time an interviewer asked you to write snowflake pivot on the whiteboard, sketch a sql server pivot for a monthly reporting table, wrangle a bigquery pivot for a survey dataset, run an sql unpivot to normalise a wide extract, or roll a dynamic pivot sql where the column set is unknown at compile time. It walks through the long-to-wide reshape via postgres crosstab and PIVOT keywords (Postgres tablefunc + Snowflake / BigQuery / SQL Server PIVOT), the wide-to-long reshape via UNPIVOT and its dialect equivalents (SQL Server / Snowflake / BigQuery UNPIVOT, Postgres via UNION ALL / VALUES / jsonb_each_text, MySQL emulation), dynamic pivots when the column set is unknown at compile time (PREPARE / EXECUTE, Snowflake ANY ORDER BY, dbt macros, Python-generated SQL), and the universal SUM(CASE WHEN …) plus ANSI FILTER (WHERE …) conditional-aggregation fallback that works in every dialect. Each section pairs a teaching block with a Solution-Tail interview answer — code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works.

When you want hands-on reps immediately after reading, drill the unpivoting practice library →, rehearse on conditional-aggregation problems →, and sharpen the grouping axis with grouping-sets drills →.

On this page

Why pivot patterns matter in 2026
Postgres crosstab + Snowflake / BigQuery PIVOT
UNPIVOT and dialect equivalents
Dynamic pivots — unknown columns at compile time
Conditional-aggregation fallback
Cheat sheet — pivot and unpivot recipes
Frequently asked questions
Practice on PipeCode

1. Why pivot patterns matter in 2026

PIVOT and UNPIVOT are the two halves of table reshaping — every dialect spells them differently

The one-sentence invariant: a sql pivot turns a long-form table (one row per observation) into a wide-form table (one column per observation category), and a sql unpivot reverses the operation — but no two major warehouses spell the operation the same way, so every senior data engineer keeps a mental dialect matrix in their head. Once you internalise that "reshape direction + dialect keyword" is the whole design space, the entire pivot interview surface collapses to a lookup: what's the input shape, what's the target shape, and what's the warehouse?

Four axes interviewers actually probe.

Direction. Long-to-wide (PIVOT) or wide-to-long (UNPIVOT). Analysts usually want long-to-wide for reporting; ETL pipelines usually want wide-to-long for normalisation. The mental model split is: PIVOT collapses rows into columns; UNPIVOT explodes columns into rows.
Known vs unknown column set. If the labels are known at compile time (Jan, Feb, Mar), any dialect's PIVOT handles it. If they are unknown at compile time (a new product category can arrive at any time), you need either dynamic pivot sql (PREPARE / EXECUTE), Snowflake's ANY ORDER BY form, or a code-generator (dbt macro, Python + SQLAlchemy).
Aggregate function. PIVOT requires an aggregate — SUM, MAX, MIN, AVG, COUNT, LIST_AGG. The choice matters: MAX vs SUM produces different answers when there are duplicate rows. Interviewers love to probe "what if two rows share the same (product, month) key?" — the answer depends on the aggregate.
Portability. SUM(CASE WHEN …) and the ANSI FILTER (WHERE …) clause both work as universal fallbacks, and both plan identically to PIVOT in every modern warehouse. When your team ships one query across Postgres + Snowflake + BigQuery, conditional aggregation is often the right call.

The dialect split — same shape, different keywords.

Postgres — no native PIVOT keyword; use the tablefunc extension's crosstab(text, text) function, or fall back to SUM(CASE WHEN …). The crosstab function has known ergonomic warts (output column names must be pre-declared, category ordering matters).
Snowflake — full ANSI PIVOT(agg FOR col IN (v1, v2, …)) since 2019, plus a flagship ANY ORDER BY dynamic form since 2023 that eliminates the need for EXECUTE IMMEDIATE for most cases.
BigQuery — table-function PIVOT(agg FOR col IN (v1, v2, …)) applied to a subquery. Requires explicit column list; no ANY form.
SQL Server — the original PIVOT(agg FOR col IN ([v1], [v2], …)) p syntax with an alias table. Labels are [bracketed] because they become identifiers.
MySQL / MariaDB — no PIVOT keyword at all. Every pivot is either SUM(CASE WHEN …) or a stored procedure that builds dynamic SQL from a metadata query.

When conditional aggregation still wins over PIVOT.

Mixed aggregates in one row. SUM(CASE WHEN month = 'Jan' THEN sales END) AS jan_sales, COUNT(CASE WHEN month = 'Jan' THEN 1 END) AS jan_orders. PIVOT gives you one aggregate for all columns; CASE WHEN lets you mix SUM, COUNT, MAX, and AVG freely.
Tiny column sets. If you have 3 pivoted columns, SUM(CASE WHEN …) is 6 lines vs PIVOT's 4 lines and is instantly portable across dialects — no readability loss.
Dialect portability. One query that works identically on Postgres, Snowflake, BigQuery, and SQL Server. Conditional aggregation is the only mechanism that satisfies that constraint.
Debuggable. The plan is a plain GROUP BY with a projection list. Every runner explains it the same way. PIVOT in some dialects hides the plan inside a syntactic sugar layer that is harder to read.

Common interview asks.

"Reshape a monthly sales table (year, month, product, sales) into a wide table with one column per month." — the classic long-to-wide.
"Given a wide survey table (respondent_id, q1, q2, q3, q4, q5), turn it into a long (respondent_id, question, answer) shape." — the classic wide-to-long.
"New product categories are added to a source table every week. Write a report that pivots by category — how do you handle the unknown column set?" — the dynamic pivot sql probe.
"Same reshape, but the pipeline runs on both Snowflake and Postgres. Write one query that works in both." — the portability probe (answer: SUM(CASE WHEN …)).
"What is the difference between PIVOT in SQL Server and crosstab in Postgres?" — the dialect-comparison probe.

What interviewers listen for.

Do you say "PIVOT is long-to-wide, UNPIVOT is wide-to-long" in the first sentence? — required answer.
Do you mention "Postgres has no PIVOT keyword — use crosstab or CASE WHEN" unprompted? — senior signal.
Do you push back on "just use PIVOT everywhere" with "MySQL and Postgres have no first-class PIVOT"? — senior signal.
Do you reach for SUM(CASE WHEN …) as the portable fallback? — senior signal.
Do you know FILTER (WHERE …) as the ANSI cleaner form? — senior signal.

Worked example — same monthly-sales reshape, four dialects

Detailed explanation. The classic reporting ask — turn a long (year, month, product, sales) table into a wide table with one column per month per year — reads identically as an English requirement across every warehouse. The SQL to express it looks wildly different by dialect. Writing the same reshape four ways is the fastest way to build a mental dialect matrix.

Question. Given a sales table with columns (year, month, product, sales), write a query that produces one row per product with columns jan, feb, mar containing the sum of sales for each month. Show the query in Postgres (crosstab), Snowflake, BigQuery, and SQL Server.

Input.

year	month	product	sales
2026	Jan	Widget	100
2026	Feb	Widget	150
2026	Mar	Widget	200
2026	Jan	Gadget	80
2026	Feb	Gadget	90
2026	Mar	Gadget	110

Code.

-- Postgres — tablefunc crosstab (extension required)
CREATE EXTENSION IF NOT EXISTS tablefunc;

SELECT * FROM crosstab(
  $$ SELECT product, month, SUM(sales)::int
     FROM sales
     GROUP BY product, month
     ORDER BY 1, 2 $$,
  $$ VALUES ('Jan'), ('Feb'), ('Mar') $$
) AS ct(product text, jan int, feb int, mar int);

-- Snowflake — native ANSI PIVOT
SELECT product, jan, feb, mar
FROM (
  SELECT product, month, sales FROM sales
)
PIVOT (SUM(sales) FOR month IN ('Jan', 'Feb', 'Mar'))
  AS p (product, jan, feb, mar);

-- BigQuery — table-function PIVOT on a subquery
SELECT *
FROM (
  SELECT product, month, sales FROM sales
)
PIVOT (SUM(sales) FOR month IN ('Jan', 'Feb', 'Mar'));

-- SQL Server — PIVOT with alias table and bracketed labels
SELECT product, [Jan], [Feb], [Mar]
FROM (
  SELECT product, month, sales FROM sales
) src
PIVOT (SUM(sales) FOR month IN ([Jan], [Feb], [Mar])) AS p;

Step-by-step explanation.

Postgres's tablefunc.crosstab takes two SQL strings: the first must return three columns (row_key, category, value) and be ORDER BY row_key, category; the second is a VALUES list that pins the category order. The AS ct(…) alias declares the output column names — the ergonomic wart is that you must type them.
Snowflake's PIVOT operates on a subquery (or table). Every column of the subquery that is NOT named in FOR or aggregated is treated as an implicit grouping key. ('Jan', 'Feb', 'Mar') is a quoted-string label list — Snowflake infers the column names from those labels.
BigQuery's PIVOT is nearly identical to Snowflake's but requires the subquery form (FROM (SELECT …)). The output column names are Jan, Feb, Mar — no bracketing.
SQL Server's PIVOT also uses a subquery + alias table, but labels go in [square brackets] because they become identifiers. Any label with a space or a reserved word MUST be bracketed.
Every dialect's output is identical: one row per product, three columns per month, sums populated. The interviewer wants to see you write two or three of these from memory.

Output.

product	jan	feb	mar
Gadget	80	90	110
Widget	100	150	200

Rule of thumb. For any pivot in a Snowflake or BigQuery pipeline, reach for native PIVOT first — it is the most readable. For Postgres, use crosstab only if you already have the extension; otherwise SUM(CASE WHEN …) is cleaner. For SQL Server, always bracket the labels even when they don't strictly require it — future you will insert a label with a space and you will be glad.

Worked example — direction reversal (wide-to-long)

Detailed explanation. The mirror ask: given a wide table with one column per month, turn it into a long shape with (product, month, sales). The interviewer often frames this as "unpivot," "normalise," or "stack." Every dialect except MySQL has an UNPIVOT keyword; the shapes differ subtly.

Question. Given a wide sales_wide table with columns (product, jan, feb, mar), write the wide-to-long reshape to (product, month, sales) in Snowflake, BigQuery, SQL Server, and Postgres.

Input.

product	jan	feb	mar
Widget	100	150	200
Gadget	80	90	110

Code.

-- Snowflake — UNPIVOT keyword
SELECT product, month, sales
FROM sales_wide
UNPIVOT (sales FOR month IN (jan, feb, mar));

-- BigQuery — UNPIVOT keyword (same shape)
SELECT product, month, sales
FROM sales_wide
UNPIVOT (sales FOR month IN (jan, feb, mar));

-- SQL Server — UNPIVOT keyword, requires alias table
SELECT product, month, sales
FROM (SELECT product, jan, feb, mar FROM sales_wide) src
UNPIVOT (sales FOR month IN (jan, feb, mar)) AS p;

-- Postgres — no UNPIVOT keyword; use UNION ALL
SELECT product, 'jan' AS month, jan AS sales FROM sales_wide
UNION ALL
SELECT product, 'feb' AS month, feb AS sales FROM sales_wide
UNION ALL
SELECT product, 'mar' AS month, mar AS sales FROM sales_wide;

Step-by-step explanation.

Snowflake's UNPIVOT(sales FOR month IN (jan, feb, mar)) reads as: "emit one row per input row per named column; put the column name in month and the value in sales." No aggregate is needed because unpivoting is a 1-to-N explosion.
BigQuery's UNPIVOT is syntactically identical. The one option worth knowing is INCLUDE NULLS — by default UNPIVOT drops rows where the unpivoted value is NULL; adding INCLUDE NULLS keeps them.
SQL Server also uses UNPIVOT, with an alias table like PIVOT. Labels don't need brackets here because you're using them as column references, not identifier literals.
Postgres has no UNPIVOT keyword. The idiomatic fallback is UNION ALL — one SELECT per unpivoted column. This is the most explicit form: you name the month label and the value column at every branch.
The output shape is identical in every dialect: (product, month, sales) in long form. UNPIVOT is the reverse operation of PIVOT — apply both and you get back the original table.

Output.

product	month	sales
Widget	jan	100
Widget	feb	150
Widget	mar	200
Gadget	jan	80
Gadget	feb	90
Gadget	mar	110

Rule of thumb. For 3–5 columns to unpivot, UNION ALL is fine even on Snowflake or SQL Server. For 20+ columns, always use UNPIVOT (or Postgres's jsonb_each_text trick) — the boilerplate savings pay for the marginal complexity.

Worked example — the aggregate function matters

Detailed explanation. A common interview trap: your input has multiple rows per (product, month) because the source is a transaction log, not a monthly total. The PIVOT aggregate matters — SUM gives you totals, MAX gives you the largest single transaction, COUNT gives you the number of transactions. The interviewer wants to see you name the aggregate deliberately.

Question. A transactions table has multiple rows per (product, month) — one per transaction. Write a PIVOT that reports monthly sums, then modify it to report monthly transaction counts, then modify it to report the maximum single transaction per month. Use Snowflake syntax.

Input.

product	month	amount
Widget	Jan	30
Widget	Jan	70
Widget	Feb	150
Gadget	Jan	80
Gadget	Feb	40
Gadget	Feb	50

Code.

-- Monthly sum per product
SELECT * FROM (SELECT product, month, amount FROM transactions)
PIVOT (SUM(amount) FOR month IN ('Jan', 'Feb'));

-- Monthly transaction count per product
SELECT * FROM (SELECT product, month, amount FROM transactions)
PIVOT (COUNT(amount) FOR month IN ('Jan', 'Feb'));

-- Monthly max single transaction per product
SELECT * FROM (SELECT product, month, amount FROM transactions)
PIVOT (MAX(amount) FOR month IN ('Jan', 'Feb'));

Step-by-step explanation.

SUM(amount) treats the input as monthly totals to be aggregated. Widget in Jan has two rows (30 + 70), so the pivoted 'Jan' column for Widget shows 100.
COUNT(amount) counts rows per (product, month). Widget in Jan again has two rows, so the pivoted 'Jan' column for Widget shows 2 (not 100).
MAX(amount) picks the largest single value per group. Widget in Jan has rows (30, 70), so the pivoted 'Jan' column for Widget shows 70.
The interviewer wants to see you name the aggregate deliberately. "I'm using SUM because we want monthly totals; if we wanted transaction counts, I'd swap to COUNT" is the answer that shows depth.
Every dialect's PIVOT requires exactly one aggregate function. To emit both sum and count in the same query, you need conditional aggregation (SUM(CASE WHEN month = 'Jan' THEN amount END) AS jan_sum, COUNT(CASE WHEN month = 'Jan' THEN 1 END) AS jan_count) — which is why the conditional-aggregation fallback is the senior go-to for mixed aggregates.

Output (SUM variant).

product	Jan	Feb
Widget	100	150
Gadget	80	90

Rule of thumb. Always name the aggregate function in the first sentence of your interview answer: "I'll use SUM because we want monthly totals." When the interviewer asks a follow-up ("what if we wanted counts?"), swap SUM to COUNT — the shape doesn't change, only the aggregate does.

Senior interview question on pivot / unpivot mental model

A senior interviewer often opens with: "Walk me through when you reach for PIVOT vs UNPIVOT vs SUM(CASE WHEN …). What is the mental split, and when do you pick one over the others?"

Solution Using a 4-question decision framework

Decision framework — PIVOT / UNPIVOT / conditional aggregation

1. Direction?
   - Long-to-wide  → PIVOT (or SUM CASE WHEN as portable fallback)
   - Wide-to-long  → UNPIVOT (or UNION ALL as portable fallback)

2. Column set known at compile time?
   - Yes → static PIVOT / UNPIVOT
   - No  → dynamic PIVOT (Snowflake ANY ORDER BY, or EXECUTE IMMEDIATE)

3. Single aggregate or mixed?
   - Single (all SUM, all COUNT, all MAX) → PIVOT
   - Mixed (SUM + COUNT + MAX in one row) → conditional aggregation

4. Portability required?
   - One dialect         → native PIVOT / UNPIVOT
   - Multiple dialects   → conditional aggregation (works everywhere)

Step-by-step trace.

Ask	Q1 direction	Q2 known cols	Q3 mixed	Q4 portable	Picked
Monthly sales matrix (Snowflake only)	long-to-wide	yes (12 months)	single SUM	no	Snowflake PIVOT
Monthly matrix (Postgres + Snowflake)	long-to-wide	yes	single SUM	yes	SUM(CASE WHEN …)
Report SUM + COUNT + MAX per month	long-to-wide	yes	mixed	no	conditional aggregation
New product category added weekly	long-to-wide	no	single	no	Snowflake ANY ORDER BY
Wide survey (q1..q10) to long	wide-to-long	yes	n/a	no	UNPIVOT
Wide survey, portable across warehouses	wide-to-long	yes	n/a	yes	UNION ALL

After the 4-question pass, the reshape strategy is unambiguous. The remaining 5% — where multiple answers work — defaults to whatever the team's style guide already picks.

Output:

Strategy	When it wins
Native PIVOT	Single dialect, known columns, single aggregate, best readability
Snowflake ANY ORDER BY	Snowflake, unknown columns, no need to write dynamic SQL
SUM(CASE WHEN …)	Multi-dialect portability, mixed aggregates, tiny column sets
UNPIVOT	Wide-to-long, single dialect, known columns
UNION ALL	Wide-to-long, Postgres or MySQL, or portability required

Why this works — concept by concept:

Direction is the first split — PIVOT and UNPIVOT are not variants; they are opposite operations. Every reshape either compresses rows into columns (PIVOT) or explodes columns into rows (UNPIVOT). Answering the direction question first eliminates half of the option space.
Known vs unknown columns — the column set determines whether static SQL is enough or whether you need a code-generation layer. Interviewers care about this because it separates "reads docs" from "designs pipelines that survive schema drift."
Single vs mixed aggregates — PIVOT gives you one aggregate for every output column; conditional aggregation lets you mix. This is the axis that decides whether the natural syntax is PIVOT or SUM(CASE WHEN …).
Portability is a hard constraint — one query that runs on Postgres AND Snowflake AND BigQuery cannot use native PIVOT in Postgres. Conditional aggregation is the only mechanism that satisfies "runs everywhere without translation."
Cost — every strategy plans to the same GROUP BY scan under the hood. The choice is a readability / portability trade-off, not a performance one. Pick the shape that reads best in your codebase's dialect.

SQL
Topic — unpivoting
Unpivot / reshape problems

Practice →

SQL Topic — conditional aggregation Conditional aggregation problems

Practice →

2. Postgres crosstab + Snowflake / BigQuery PIVOT

`postgres crosstab` is the tablefunc form; `snowflake pivot` and `bigquery pivot` ship the ANSI PIVOT keyword; `sql server pivot` is the OG dialect ancestor

The mental model in one line: Postgres has no PIVOT keyword — it exposes the tablefunc.crosstab function; Snowflake, BigQuery, and SQL Server all ship an ANSI-style PIVOT clause, but the label syntax and required aliases differ. Once you say "Postgres uses crosstab, everyone else uses PIVOT with different label syntaxes," the entire long-to-wide interview surface becomes a syntax lookup rather than a design problem.

Postgres tablefunc.crosstab — the two-argument form.

CREATE EXTENSION IF NOT EXISTS tablefunc; — one-time install per database.
crosstab('SELECT row_key, category, value FROM t ORDER BY 1, 2', 'VALUES (''a''), (''b''), (''c'')') — first arg produces the long-form input, second arg pins the category order.
Output column list must be declared in AS ct(row_key type, cat_a type, cat_b type, cat_c type).
Cardinality guarantee: category order in output = row order in the second VALUES list.
Common gotcha: the first argument MUST be ORDER BY row_key, category — otherwise crosstab silently returns wrong results.

Postgres tablefunc.crosstab — the single-argument form.

crosstab('SELECT row_key, category, value FROM t ORDER BY 1, 2') — automatic category detection, but the first N distinct categories in the ordered input become the columns.
Rarely the right choice — you almost always want to name the categories explicitly to make ordering and missing values deterministic.

Snowflake PIVOT.

SELECT * FROM t PIVOT(SUM(sales) FOR month IN ('Jan', 'Feb', 'Mar')) — quoted string labels; the output column names are 'Jan', 'Feb', 'Mar'.
The IN clause supports comma-separated literal values, a subquery (FOR month IN (SELECT DISTINCT month FROM t)), or the flagship ANY ORDER BY month dynamic form (Snowflake exclusive, GA 2023).
DEFAULT ON NULL (0) option — Snowflake-specific extension that replaces NULL cells with a default value.
Aliasing: PIVOT(…) AS p (product, jan, feb, mar) renames both the implicit group columns and the pivoted columns.

BigQuery PIVOT.

SELECT * FROM (SELECT product, month, sales FROM t) PIVOT(SUM(sales) FOR month IN ('Jan', 'Feb', 'Mar')) — mandatory subquery form.
Output column names are the literal string values without quotes (Jan, Feb, Mar).
Supports AS alias after each IN value: IN ('Jan' AS jan_2026, 'Feb' AS feb_2026).
No ANY form — the label list is always static. Dynamic pivots require EXECUTE IMMEDIATE.
Non-string labels (integers, dates) work but produce numeric column names like 2026, which must be back-ticked in downstream queries.

SQL Server PIVOT.

SELECT product, [Jan], [Feb] FROM (SELECT product, month, sales FROM t) src PIVOT(SUM(sales) FOR month IN ([Jan], [Feb])) AS p — bracketed labels because they become identifiers.
Alias table (AS p) is required; the src alias on the inner subquery is required.
The NULL value is displayed as NULL — no default-value option; use COALESCE on the outer projection.
Labels with spaces or reserved words MUST be bracketed; even without special characters, brackets are the safe default.

The 4-dialect matrix at a glance.

Postgres: crosstab function + AS ct(…) declared output types.
Snowflake: PIVOT(agg FOR col IN ('a', 'b')) + optional ANY ORDER BY for dynamic.
BigQuery: PIVOT(agg FOR col IN ('a', 'b')) on subquery + no ANY form.
SQL Server: PIVOT(agg FOR col IN ([a], [b])) AS p + bracketed labels.
MySQL / MariaDB: no keyword; use SUM(CASE WHEN …) or build dynamic SQL.

Common interview probes on PIVOT syntax.

"How does Postgres do PIVOT?" — it doesn't; use tablefunc.crosstab or SUM(CASE WHEN …).
"What is the second argument of crosstab?" — the category order pin list; ensures deterministic column order.
"What is ANY ORDER BY in Snowflake PIVOT?" — dynamic pivot where the label set is discovered at query time; no dynamic SQL required.
"Does BigQuery support dynamic PIVOT?" — not natively; use EXECUTE IMMEDIATE on a string-built column list.
"Why does SQL Server PIVOT need brackets on labels?" — labels become identifiers; brackets escape reserved words and spaces.

Worked example — Postgres tablefunc crosstab, two-argument form

Detailed explanation. The most common postgres crosstab interview question: given a sparse (product, month, sales) table, reshape to a wide monthly matrix. Some product-month cells have no rows — the interviewer wants to know that the two-argument form handles missing cells cleanly, and that you know to ORDER BY 1, 2 in the first argument.

Question. Given a sales table with sparse (product, month, sales) rows, write a Postgres query using tablefunc.crosstab that produces one row per product with columns jan, feb, mar. Explain what happens for missing product-month combinations.

Input.

product	month	sales
Widget	Jan	100
Widget	Mar	200
Gadget	Feb	90

Code.

CREATE EXTENSION IF NOT EXISTS tablefunc;

SELECT * FROM crosstab(
  $$
    SELECT product, month, SUM(sales)::int
    FROM sales
    GROUP BY product, month
    ORDER BY 1, 2
  $$,
  $$ VALUES ('Jan'), ('Feb'), ('Mar') $$
) AS ct (
  product text,
  jan int,
  feb int,
  mar int
);

Step-by-step explanation.

The first argument must produce three columns: the row key (product), the category (month), and the value (SUM(sales)). It must be ORDER BY row_key, category — otherwise crosstab may assign values to the wrong output cells.
The second argument is a VALUES list that pins the output column order. 'Jan' is column 1, 'Feb' is column 2, 'Mar' is column 3. Categories in the first arg not present in this list are dropped from the output.
Missing product-month combinations produce NULL in the output cell. Widget has no February row, so feb is NULL for Widget. To display 0 instead, wrap the outer projection with COALESCE(jan, 0).
The AS ct(…) alias declares the output column names AND their types. You cannot omit this — crosstab is a SETOF record function, and Postgres requires explicit type declarations for record-returning functions in the FROM clause.
The ::int cast on SUM(sales) matters because SUM returns bigint by default. The output column types in AS ct(…) must match — a mismatch throws structure of query does not match function result type.

Output.

product	jan	feb	mar
Gadget	NULL	90	NULL
Widget	100	NULL	200

Rule of thumb. Always use the two-argument form of crosstab. The one-argument form auto-detects categories from the input, which makes column order non-deterministic and breaks when a category is missing from a batch. Pin the categories explicitly with VALUES (...).

Worked example — Snowflake PIVOT with an inline subquery

Detailed explanation. Snowflake's PIVOT is the cleanest of the four — no alias table required, no bracketing, and the operand can be an inline subquery. The interviewer often uses this to test whether you understand implicit grouping: any column in the input NOT named in FOR and NOT aggregated becomes an implicit GROUP BY key.

Question. Given a orders table with (order_id, product, region, month, amount), write a Snowflake PIVOT that produces monthly sums per (product, region). Explain the implicit grouping and how to avoid an accidental cross-product from extra columns.

Input.

order_id	product	region	month	amount
1	Widget	US	Jan	100
2	Widget	US	Feb	120
3	Widget	EU	Jan	60
4	Gadget	US	Jan	80

Code.

-- Correct — implicit grouping on (product, region)
SELECT * FROM (
  SELECT product, region, month, amount FROM orders
)
PIVOT (SUM(amount) FOR month IN ('Jan', 'Feb'))
  AS p (product, region, jan, feb);

-- WRONG — order_id makes every row unique, defeats the pivot
SELECT * FROM orders
PIVOT (SUM(amount) FOR month IN ('Jan', 'Feb'));

Step-by-step explanation.

PIVOT treats every column of the input NOT named in FOR and NOT the aggregated column as an implicit GROUP BY key. In the correct version, product and region are the implicit groups.
In the wrong version, order_id is unique per row — so the implicit GROUP BY (order_id, product, region) produces one output row per input row. The pivot degenerates because every group has exactly one (product, region, month) combination.
Always project only the columns you want as grouping keys + the pivot column + the value column, using an inner subquery: SELECT product, region, month, amount FROM orders. Never let PIVOT operate on a table with extra columns you don't want as grouping keys.
The AS p (product, region, jan, feb) clause renames the implicit and explicit columns in one shot. It is optional — without it, Snowflake keeps the implicit column names and generates 'Jan', 'Feb' as the pivoted column names.
To sanity-check, run the subquery first: SELECT product, region, month, amount FROM orders. If those four columns are exactly what you want to pivot, wrap with PIVOT.

Output.

product	region	jan	feb
Gadget	US	80	NULL
Widget	EU	60	NULL
Widget	US	100	120

Rule of thumb. Always use the inline-subquery form of PIVOT. Never point PIVOT at a raw table — the implicit GROUP BY on every column is the most common snowflake pivot bug. The subquery gives you control over exactly which columns become grouping keys.

Worked example — BigQuery PIVOT with column aliases

Detailed explanation. BigQuery's PIVOT produces output column names from the label literals verbatim. 'Jan' becomes a column literally named Jan, which is fine for identifiers but breaks when the label is a number, a date, or contains special characters. The AS alias in the IN list is the fix.

Question. Given an events table with (user_id, year, event_count), write a BigQuery PIVOT that produces one column per year (2024, 2025, 2026). Handle the fact that numeric column names must be back-ticked downstream.

Input.

user_id	year	event_count
u1	2024	10
u1	2025	15
u1	2026	20
u2	2025	8
u2	2026	12

Code.

-- Without alias — numeric column names, requires back-ticks downstream
SELECT * FROM (
  SELECT user_id, year, event_count FROM events
)
PIVOT (SUM(event_count) FOR year IN (2024, 2025, 2026));

-- With alias — human-friendly column names
SELECT * FROM (
  SELECT user_id, year, event_count FROM events
)
PIVOT (SUM(event_count) FOR year IN (
  2024 AS y2024,
  2025 AS y2025,
  2026 AS y2026
));

Step-by-step explanation.

Without the AS alias, BigQuery produces columns literally named 2024, 2025, 2026. In subsequent queries, you must back-tick them: SELECT \2024FROM pivoted_table.
Back-ticked numeric column names are legal BigQuery but visually noisy and error-prone. Downstream tooling (Looker, dbt) often chokes on them.
The AS y2024 alias in the IN clause makes the output column name a plain identifier. This is the recommended pattern for any non-string label (numbers, dates, quoted strings with special characters).
The same trick works in Snowflake: IN (2024 AS y2024, 2025 AS y2025). SQL Server doesn't need it because bracket labels are already identifier-safe.
Postgres crosstab dodges this whole class by making you declare column names explicitly in AS ct(user_id text, y2024 int, y2025 int, y2026 int) — the trade-off is verbosity in exchange for total control.

Output.

user_id	y2024	y2025	y2026
u1	10	15	20
u2	NULL	8	12

Rule of thumb. Any time your PIVOT labels are not plain string identifiers, alias them with AS. Numeric years, dates, quoted strings with hyphens, and anything else non-alphanumeric will produce hard-to-use column names downstream. The alias costs three characters and buys you a downstream identifier that behaves like every other column.

Worked example — SQL Server PIVOT with COALESCE and bracketed labels

Detailed explanation. SQL Server's PIVOT produces NULL for missing cells and has no built-in default-value option. The idiomatic fix is to wrap the outer projection with COALESCE. The interviewer wants to see you handle NULL cells explicitly.

Question. Given a sales table with (product, month, sales), write a SQL Server PIVOT for the three months (Jan, Feb, Mar) that reports 0 for missing months. Show the alias-table + bracket syntax.

Input.

product	month	sales
Widget	Jan	100
Widget	Feb	150
Gadget	Jan	80

Code.

SELECT
  product,
  COALESCE([Jan], 0) AS jan,
  COALESCE([Feb], 0) AS feb,
  COALESCE([Mar], 0) AS mar
FROM (
  SELECT product, month, sales FROM sales
) src
PIVOT (
  SUM(sales) FOR month IN ([Jan], [Feb], [Mar])
) AS p;

Step-by-step explanation.

The inner subquery (src) alias is mandatory in SQL Server PIVOT — the parser requires it.
The alias table AS p after the PIVOT clause is also mandatory — SQL Server does not accept an unaliased pivot.
Labels in the IN clause are bracketed: [Jan], [Feb], [Mar]. Brackets make them identifier-safe; without them, any label with a space or reserved word would break.
Cells with no matching input rows come out as NULL. COALESCE([Jan], 0) in the outer projection replaces NULL with 0. This is the SQL Server-idiomatic way — there is no DEFAULT ON NULL clause like Snowflake.
The AS jan on each COALESCE renames the output column from [Jan] (bracketed identifier) to jan (plain identifier). This is optional but recommended — downstream tooling reads plain names more easily.

Output.

product	jan	feb	mar
Gadget	80	0	0
Widget	100	150	0

Rule of thumb. Always wrap sql server pivot output with COALESCE(col, 0) or COALESCE(col, '') on the outer projection when the schema expects zero-instead-of-NULL. Do it in the outer SELECT, not inside the pivot — the pivot itself has no NULL-substitution mechanism.

Senior interview question on Postgres crosstab specifics

A senior interviewer might ask: "You work on Postgres. You have a daily_metrics table with (date, metric_name, value) and you want a wide report with one column per metric. Walk me through the crosstab call and the two common pitfalls people hit."

Solution Using tablefunc.crosstab with explicit category order

-- Postgres tablefunc.crosstab — canonical pattern
CREATE EXTENSION IF NOT EXISTS tablefunc;

SELECT * FROM crosstab(
  $$
    SELECT date::text, metric_name, value
    FROM daily_metrics
    WHERE date >= CURRENT_DATE - INTERVAL '7 days'
    ORDER BY date, metric_name       -- MUST be ordered
  $$,
  $$
    VALUES ('cpu_pct'), ('mem_mb'), ('disk_pct'), ('conn_count')
  $$
) AS ct (
  date text,
  cpu_pct   numeric,
  mem_mb    numeric,
  disk_pct  numeric,
  conn_count numeric
);

Step-by-step trace.

Step	What happens	Why it matters
1	First arg runs, returns rows `(date, metric, value)` ordered by `(date, metric)`	crosstab expects the natural row order
2	Second arg establishes column order: cpu_pct, mem_mb, disk_pct, conn_count	Pins output shape; missing metrics → NULL
3	For each distinct `date`, one output row is emitted	Each row_key becomes one output row
4	Each metric's `value` lands in its matching column	Cell = the value where `metric_name` matches the category
5	AS ct(…) declares the column list + types	Postgres requires typed record output for FROM clause functions

The two most common pitfalls interviewers probe:

Pitfall 1: forgetting ORDER BY 1, 2 in the first argument. Crosstab does NOT re-sort internally; if the input is unordered, values can land in the wrong output cells silently. Always ORDER BY row_key, category in the first argument.
Pitfall 2: category-name type mismatch. The second-argument VALUES are matched to the category column of the first argument by textual equality. If metric_name is text but you write VALUES (1) (integer), the match fails and every cell is NULL. Cast both sides to text if in doubt.

Output:

date	cpu_pct	mem_mb	disk_pct	conn_count
2026-07-04	55	1200	60	42
2026-07-05	58	1250	61	45
2026-07-06	60	1300	63	44

Why this works — concept by concept:

Two-argument form pins column order — the VALUES list is the source of truth for which categories appear and in what order. Categories in the input but not the list are dropped; categories in the list but not the input become NULL columns.
ORDER BY 1, 2 is non-negotiable — crosstab uses the row order to detect row-key boundaries. Skipping the ORDER BY produces silent data corruption. Always sort by (row_key, category) in the first argument.
AS ct(…) declares record types — Postgres's SETOF record return type requires the caller to name output columns and their types. The types must match the first argument's projection exactly.
NULL cells are the missing-value contract — if a (date, metric_name) combination has no row, the cell is NULL. Wrap with COALESCE(cpu_pct, 0) in the outer projection when the schema expects zeros.
Cost — crosstab is a wrapper around the underlying GROUP BY row_key scan plus a hash of categories. Cost is O(rows) plus O(categories) per row_key. Suitable for any reasonable metric-count workload; not a bottleneck.

SQL
Topic — aggregation
Aggregation and pivot problems

Practice →

SQL Topic — case-when CASE WHEN pivot fallback drills

Practice →

3. UNPIVOT and dialect equivalents

`sql unpivot` is the mirror of PIVOT — SQL Server, Snowflake, BigQuery ship the keyword; Postgres and MySQL use UNION ALL or JSON helpers

The mental model in one line: UNPIVOT explodes named columns into (name, value) pairs — SQL Server, Snowflake, and BigQuery ship a first-class UNPIVOT keyword, while Postgres uses UNION ALL, unnest(ARRAY[...]), or jsonb_each_text(row_to_json(t)) to achieve the same shape, and MySQL falls back to UNION ALL or JSON_TABLE. Once you say "wide-to-long has three named implementations and three portable fallbacks," the entire unpivot interview surface becomes a keyword lookup.

SQL Server UNPIVOT — the OG dialect.

Syntax: SELECT keys, name_col, value_col FROM src UNPIVOT (value_col FOR name_col IN (col1, col2, col3)) AS p.
Requires an alias table (AS p), same as PIVOT.
The value_col and name_col are new column names you invent; IN (…) lists the source columns to unpivot.
All unpivoted columns must have the same data type — SQL Server does not coerce. If your source has int and varchar mixed, cast to a common type in a subquery first.
Drops NULL cells by default; no INCLUDE NULLS option — use UNION ALL if you need every cell preserved.

Snowflake UNPIVOT.

Syntax: SELECT * FROM t UNPIVOT (value_col FOR name_col IN (col1, col2, col3)).
Cleaner than SQL Server — no alias table required.
INCLUDE NULLS option: UNPIVOT INCLUDE NULLS (value_col FOR name_col IN (…)) keeps NULL cells; default is EXCLUDE NULLS.
Column aliases in the IN clause: IN (jan AS 'January', feb AS 'February') renames the label values.
Type coercion is stricter than PIVOT — the unpivoted columns must share a compatible type; mixing NUMBER and VARCHAR errors out.

BigQuery UNPIVOT.

Syntax matches Snowflake almost line-for-line: SELECT * FROM t UNPIVOT (value_col FOR name_col IN (col1, col2, col3)).
Also supports INCLUDE NULLS and EXCLUDE NULLS (default).
The IN clause can also accept groups of columns for multi-value unpivots: IN ((jan_sales, jan_cost) AS 'Jan', (feb_sales, feb_cost) AS 'Feb') — a Snowflake and BigQuery exclusive.
BigQuery does NOT allow arbitrary subqueries in the IN list (unlike PIVOT's IN); it must be a static column list.

Postgres — three portable equivalents.

UNION ALL — the classical form. One SELECT per unpivoted column: SELECT keys, 'jan' AS month, jan AS value FROM t UNION ALL SELECT keys, 'feb', feb FROM t …. Verbose for many columns but explicit.
unnest(ARRAY[...], ARRAY[...]) — the terse form using paired arrays. SELECT keys, m.month, m.value FROM t, unnest(ARRAY['jan','feb','mar'], ARRAY[t.jan, t.feb, t.mar]) AS m(month, value). One row per array element, per input row.
jsonb_each_text(to_jsonb(t)) — the "unpivot every column" one-liner. Converts each row to a JSON object, then explodes each key-value pair. Great for wide tables where you don't want to type the column names.

MySQL emulation.

No UNPIVOT keyword. Options: UNION ALL (works on every MySQL version), JSON_TABLE (MySQL 8.0.4+, joins against a JSON array), stored procedure with dynamic SQL.
The typical answer: UNION ALL for 3-10 columns, JSON_TABLE for wider tables.

When to pick which.

≤ 5 columns to unpivot, portability required — UNION ALL. Every dialect. Verbose but bulletproof.
5-50 columns, single dialect (SQL Server / Snowflake / BigQuery) — UNPIVOT keyword. Boilerplate savings pay off.
Postgres, wide table (20+ columns), you want "unpivot everything" — jsonb_each_text(to_jsonb(t)). One line, no column list to maintain.
BigQuery, multi-value unpivot (jan_sales + jan_cost together) — grouped IN clause. Snowflake and BigQuery exclusive.

Common interview probes on UNPIVOT.

"How do you UNPIVOT in Postgres?" — Postgres has no keyword; use UNION ALL, unnest(ARRAY[...], ARRAY[...]), or jsonb_each_text(to_jsonb(t)).
"What is the difference between INCLUDE NULLS and EXCLUDE NULLS?" — EXCLUDE NULLS (default in Snowflake / BigQuery / SQL Server) drops rows where the unpivoted value is NULL; INCLUDE NULLS keeps them.
"How do you UNPIVOT in MySQL?" — UNION ALL, JSON_TABLE, or a stored procedure with dynamic SQL. No first-class keyword.
"What is a multi-value unpivot?" — grouping multiple source columns to a single label: Snowflake / BigQuery IN ((jan_sales, jan_cost) AS 'Jan', …). Useful when your wide table has repeated column groups per period.
"Why must UNPIVOT columns share a type?" — because the output value_col has exactly one type; source columns are stacked into it, and SQL is strict about type unification.

Worked example — SQL Server UNPIVOT with mixed types

Detailed explanation. A wide metrics table has (server_id, cpu_pct decimal, mem_mb bigint, disk_pct decimal). Direct UNPIVOT fails because bigint and decimal don't unify. The interviewer wants to see you cast to a common type in an inner subquery.

Question. Given a wide metrics table with mixed numeric types, write a SQL Server UNPIVOT that produces (server_id, metric_name, value) in long form. Handle the type-mismatch error.

Input.

server_id	cpu_pct	mem_mb	disk_pct
s1	55.0	8192	60.5
s2	62.0	16384	45.2

Code.

-- BROKEN — mixed types cause "operand type clash" error
-- SELECT server_id, metric_name, value
-- FROM metrics
-- UNPIVOT (value FOR metric_name IN (cpu_pct, mem_mb, disk_pct)) AS p;

-- Fixed — cast all source columns to a common type in inner subquery
SELECT server_id, metric_name, value
FROM (
  SELECT server_id,
         CAST(cpu_pct  AS decimal(18,4)) AS cpu_pct,
         CAST(mem_mb   AS decimal(18,4)) AS mem_mb,
         CAST(disk_pct AS decimal(18,4)) AS disk_pct
  FROM metrics
) src
UNPIVOT (value FOR metric_name IN (cpu_pct, mem_mb, disk_pct)) AS p;

Step-by-step explanation.

UNPIVOT requires every column in the IN list to have the same data type — the resulting value column can only be one type.
Without the inner cast, SQL Server throws The type of column "cpu_pct" conflicts with the type of other columns specified in the UNPIVOT list. The error message is unusually clear here.
The inner subquery casts all three source columns to decimal(18,4) — a common type that holds both the integer mem_mb and the decimal cpu_pct / disk_pct values without loss.
UNPIVOT (value FOR metric_name IN (…)) reads: "emit one row per input row per named column; put the column name as text in metric_name and the column value in value."
NULL cells are dropped silently (no INCLUDE NULLS in SQL Server). If a server has NULL disk_pct, that row is missing from the output — you have to add UNION ALL for the missing rows if you need them.

Output.

server_id	metric_name	value
s1	cpu_pct	55.0000
s1	mem_mb	8192.0000
s1	disk_pct	60.5000
s2	cpu_pct	62.0000
s2	mem_mb	16384.0000
s2	disk_pct	45.2000

Rule of thumb. Whenever the columns you want to UNPIVOT don't share a type, cast them to a common one in an inner subquery. decimal(18,4) is a safe superset for most numeric mixes; varchar(4000) or nvarchar(max) for text mixes.

Worked example — Snowflake UNPIVOT with INCLUDE NULLS

Detailed explanation. A wide attendance table records student attendance across 5 days. Some cells are NULL because the student wasn't enrolled that day. The interviewer wants a long-form view where NULL rows are preserved (so the count of enrolled students matches).

Question. Given a wide attendance table with (student_id, mon, tue, wed, thu, fri), write a Snowflake UNPIVOT that preserves NULL cells. Explain when to use INCLUDE NULLS vs EXCLUDE NULLS.

Input.

student_id	mon	tue	wed	thu	fri
s1	present	present	absent	present	present
s2	NULL	present	present	present	NULL

Code.

-- Default EXCLUDE NULLS — drops NULL cells
SELECT student_id, day, status
FROM attendance
UNPIVOT (status FOR day IN (mon, tue, wed, thu, fri));

-- INCLUDE NULLS — preserves every cell, useful for enrolment analysis
SELECT student_id, day, status
FROM attendance
UNPIVOT INCLUDE NULLS (status FOR day IN (mon, tue, wed, thu, fri));

Step-by-step explanation.

Default UNPIVOT behavior in Snowflake is EXCLUDE NULLS. Cells where the source column is NULL do not produce a row in the output.
INCLUDE NULLS reverses that — every (student_id, day) pair produces a row, with status = NULL if the source cell was NULL.
When to use EXCLUDE NULLS: analytics workloads where NULL means "not applicable" and shouldn't inflate row counts. Attendance-percentage calculations often use this to exclude non-enrolled days.
When to use INCLUDE NULLS: audit workloads where you need to know every source cell was considered. Enrolment-vs-attendance reports need this to differentiate "not enrolled" from "absent."
BigQuery has identical INCLUDE NULLS / EXCLUDE NULLS semantics. SQL Server does not — it always excludes; use UNION ALL for the include case.

Output (INCLUDE NULLS variant).

student_id	day	status
s1	mon	present
s1	tue	present
s1	wed	absent
s1	thu	present
s1	fri	present
s2	mon	NULL
s2	tue	present
s2	wed	present
s2	thu	present
s2	fri	NULL

Rule of thumb. Name the NULL-handling explicitly in every Snowflake UNPIVOT — either INCLUDE NULLS or EXCLUDE NULLS. Relying on the default is a bug magnet when a downstream query counts rows and misses the NULL cases.

Worked example — Postgres jsonb_each_text for wide-table unpivot

Detailed explanation. A wide daily_stats table has 30 metric columns. Typing them into a UNION ALL would be 30 branches. Postgres has a one-liner: convert each row to a JSON object with to_jsonb(t), then explode each key-value pair with jsonb_each_text. Zero column-name maintenance.

Question. Given a wide daily_stats table with columns (date, m1, m2, m3, m4, m5, …), write a Postgres query that produces long-form (date, metric_name, value) without listing every column. Explain the mechanics of to_jsonb + jsonb_each_text.

Input.

date	m1	m2	m3
2026-07-01	10	20	30
2026-07-02	15	25	35

Code.

SELECT date,
       key   AS metric_name,
       value AS value_text
FROM daily_stats t,
     LATERAL jsonb_each_text(to_jsonb(t) - 'date') AS kv (key, value);

Step-by-step explanation.

to_jsonb(t) converts the entire row t into a JSON object where each column name is a key and each value is the JSON representation of the cell. For row ('2026-07-01', 10, 20, 30), it produces {"date": "2026-07-01", "m1": 10, "m2": 20, "m3": 30}.
The - 'date' operator removes the date key from the JSON object so the date doesn't end up as a metric row. Standard set-minus for jsonb.
jsonb_each_text(json_obj) returns a set of (key text, value text) rows, one per key-value pair in the object. For the row above, it emits (m1, 10), (m2, 20), (m3, 30).
LATERAL allows the function to reference the outer row t — required whenever a table-returning function depends on the current row. Without LATERAL, Postgres wouldn't allow t inside jsonb_each_text(to_jsonb(t)).
The output is long-form (date, metric_name, value_text) for every metric column in the row. Adding a new metric column to daily_stats requires zero changes to this query — that's the whole reason to use this pattern.

Output.

date	metric_name	value_text
2026-07-01	m1	10
2026-07-01	m2	20
2026-07-01	m3	30
2026-07-02	m1	15
2026-07-02	m2	25
2026-07-02	m3	35

Rule of thumb. For any Postgres unpivot with more than 5 columns, reach for jsonb_each_text(to_jsonb(t) - 'excluded_col'). It's declarative, requires zero column-name maintenance, and survives schema changes. Cast the value back to the target type (value::int, value::numeric) in an outer projection if you need numeric aggregation.

Worked example — Postgres unnest for a fixed column set

Detailed explanation. When you know the exact columns to unpivot but don't want the UNION ALL boilerplate, Postgres's paired-array unnest is the terse form. It stays declarative and produces a compact SQL that reads well.

Question. Given a weekly_sales table with (product, mon, tue, wed, thu, fri), write a Postgres query using unnest(ARRAY[...], ARRAY[...]) that produces (product, day, sales) in long form.

Input.

product	mon	tue	wed	thu	fri
Widget	10	20	15	25	30
Gadget	5	15	10	20	25

Code.

SELECT t.product, d.day, d.sales
FROM weekly_sales t,
     LATERAL unnest(
       ARRAY['mon', 'tue', 'wed', 'thu', 'fri'],
       ARRAY[t.mon, t.tue, t.wed, t.thu, t.fri]
     ) AS d (day, sales);

Step-by-step explanation.

unnest(ARRAY[...], ARRAY[...]) is a two-argument variant that pairs elements: element 1 of the first array pairs with element 1 of the second, and so on. The arrays MUST have the same length.
The first array holds the day labels ('mon', 'tue', …). The second holds the values from the current row (t.mon, t.tue, …).
AS d (day, sales) names the two output columns from the paired unnest.
LATERAL is required because the second array references t — the outer row. Postgres executes unnest once per outer row, producing 5 rows per input row (one per day label).
Compared to the UNION ALL variant, this is one SELECT statement instead of five. Compared to jsonb_each_text, it's more explicit about the label set — great when you want to name and order the categories.

Output.

product	day	sales
Widget	mon	10
Widget	tue	20
Widget	wed	15
Widget	thu	25
Widget	fri	30
Gadget	mon	5
Gadget	tue	15
Gadget	wed	10
Gadget	thu	20
Gadget	fri	25

Rule of thumb. Use unnest(ARRAY[...], ARRAY[...]) when the label list is small and fixed and you want a single-statement Postgres unpivot. Use jsonb_each_text(to_jsonb(t)) when the column set is large or evolving. Both are more idiomatic than UNION ALL for anything beyond 3-4 columns.

Senior interview question on portable UNPIVOT

A senior interviewer might ask: "You're writing a data-pipeline library that needs to work on Postgres, MySQL, Snowflake, and SQL Server. Write a wide-to-long reshape that runs identically on all four. What syntax do you pick and why?"

Solution Using UNION ALL as the portable canonical form

-- Portable wide-to-long — works on every dialect
SELECT product, 'jan' AS month, jan AS sales FROM sales_wide
UNION ALL
SELECT product, 'feb' AS month, feb AS sales FROM sales_wide
UNION ALL
SELECT product, 'mar' AS month, mar AS sales FROM sales_wide
UNION ALL
SELECT product, 'apr' AS month, apr AS sales FROM sales_wide;

-- For "include NULLs" everywhere, don't filter — WHERE sales IS NOT NULL
-- if you want the drop-nulls behavior of default UNPIVOT.

Step-by-step trace.

Step	What runs	Output
1	First SELECT emits (product, 'jan', jan) for every row	N rows for Jan
2	UNION ALL appends without dedup	N rows preserved
3	Second SELECT emits (product, 'feb', feb)	N rows for Feb
4	Repeat for every month	4×N rows total
5	Optionally WHERE sales IS NOT NULL on outer	Emulates EXCLUDE NULLS

The pattern trades N branches of typing for total portability. For 4 columns it's a 4-branch query; for 30 columns it's a 30-branch query — at that point, you build the query in Python or dbt instead of writing it by hand.

Output:

product	month	sales
Widget	jan	100
Widget	feb	150
Widget	mar	200
Widget	apr	175
Gadget	jan	80
Gadget	feb	90
Gadget	mar	110
Gadget	apr	95

Why this works — concept by concept:

UNION ALL is the ANSI SQL-92 fallback — every dialect since the 90s supports it. If your query needs to run on Postgres AND MySQL AND SQL Server AND Snowflake, UNION ALL is the only mechanism guaranteed to work.
One SELECT per unpivoted column — the syntax is verbose but explicit. Every column is named, every label is named, every projection is explicit. Code reviewers can spot missing columns immediately.
WHERE clauses on outer SELECT emulate NULL-handling — wrap the whole UNION ALL in a subquery and add WHERE sales IS NOT NULL to emulate EXCLUDE NULLS. Omit the filter for INCLUDE NULLS behavior.
Code-generation friendly — if you have 30 columns, a 10-line Python loop or a dbt Jinja for block writes the 30 branches. The generated SQL is boring and readable.
Cost — every branch scans the table once. Cost is O(N × rows) where N is the branch count. For a 30-column unpivot on a 100M row table, that's 3B row-touches — at that scale, switch to warehouse-native UNPIVOT or materialise the long form once.

SQL
Topic — unpivoting
UNION ALL unpivot problems

Practice →

SQL Topic — case-expression CASE expression reshape drills

Practice →

4. Dynamic pivots — unknown columns at compile time

`dynamic pivot sql` — Snowflake ANY ORDER BY, PREPARE / EXECUTE elsewhere, dbt macros for the analytics layer

The mental model in one line: a dynamic pivot happens when the label set is unknown at query-authoring time (new product categories, unknown month range, custom user attributes) — Snowflake handles it declaratively with IN (ANY ORDER BY col), everyone else builds a SQL string from a metadata query and runs it with PREPARE / EXECUTE or EXECUTE IMMEDIATE, and analytics engineers side-step the whole class with dbt macros or Python code generators. Once you say "know the labels? static PIVOT. Don't know them? dynamic answer differs by dialect," the entire dynamic-pivot interview surface collapses to a four-way pattern selection.

The dynamic-pivot problem statement.

Static pivot: PIVOT(SUM(sales) FOR month IN ('Jan', 'Feb', 'Mar')). The labels are hard-coded.
Dynamic pivot: PIVOT(SUM(sales) FOR month IN (?)) where ? is filled from SELECT DISTINCT month FROM sales. A new month next week? The pivot picks it up automatically.
The general recipe (non-Snowflake): (1) query the metadata for distinct labels, (2) build the pivot SQL string with the labels interpolated, (3) execute the string.

Snowflake ANY ORDER BY — the flagship dynamic form.

Syntax: SELECT * FROM t PIVOT(SUM(sales) FOR month IN (ANY ORDER BY month)).
ANY tells Snowflake: "discover the label set at query time from the input."
ORDER BY month pins the column order in the output. Without it, order is undefined.
Works for any label type (string, number, date). Output columns are named exactly as the label values.
No EXECUTE IMMEDIATE required — this is Snowflake's declarative dynamic-pivot answer, GA in 2023.

SQL Server dynamic pivot with PREPARE / EXECUTE.

Build a comma-separated bracketed-label string with STRING_AGG(QUOTENAME(month), ',').
Concatenate into a full PIVOT SQL string.
Run with EXEC(@sql) or sp_executesql @sql.
The classic three-step: metadata → string build → execute.

BigQuery EXECUTE IMMEDIATE for dynamic pivot.

Same pattern as SQL Server: build the label string, concatenate into a pivot statement, run with EXECUTE IMMEDIATE @sql USING @params.
BigQuery has no ANY form; you must build the string.
The @params USING clause supports parameter binding for safety.

dbt macro pattern for analytics engineers.

At dbt compile time, run a run_query('SELECT DISTINCT month FROM {{ ref("sales") }}') inside a Jinja block.
Loop over the returned values to emit SUM(CASE WHEN month = '{{ col }}' THEN sales END) AS {{ col }} for each label.
The compiled SQL is static from the warehouse's perspective — no EXECUTE IMMEDIATE needed.
This is the analytics-engineer canonical answer because it works on every dialect.

Python-generated SQL.

When the analytics-engineer answer doesn't fit (out-of-cycle report, one-off query), a small Python script queries the labels, builds the SQL, and runs it against the warehouse.
pandas.DataFrame.pivot_table does the pivot client-side after pd.read_sql; useful when the pivot is downstream of a small warehouse query.
SQLAlchemy's Query API lets you build the pivot expression programmatically for stronger typing.

Common interview probes on dynamic pivots.

"How do you handle a pivot when the column set is unknown at compile time?" — Snowflake: ANY ORDER BY; SQL Server: PREPARE + EXECUTE; BigQuery: EXECUTE IMMEDIATE; dbt: Jinja {% for %} macro; else: build the SQL in Python.
"What is ANY ORDER BY in Snowflake?" — the declarative dynamic-pivot form; auto-discovers labels, produces one column per distinct label.
"How do you avoid SQL injection in dynamic pivots?" — use QUOTENAME in SQL Server, format('%I') in Postgres, EXECUTE IMMEDIATE … USING @params in BigQuery. Never string-concatenate untrusted input.
"When would you use a dbt macro instead of dynamic SQL?" — every time you can. Compiled SQL is static, cacheable, and testable; dynamic SQL is opaque to the query planner.
"What's the failure mode of dynamic pivots?" — a new label the day the report runs causes a schema drift downstream (dashboards break, dbt tests fail, BI models need updates). Always document the discovered label set.

Worked example — Snowflake ANY ORDER BY for a growing product catalog

Detailed explanation. A sales table grows new product categories weekly. Static PIVOT would need updating every week; ANY ORDER BY discovers the label set at query time. This is the pattern to reach for on Snowflake — it's declarative, fast, and requires zero dynamic SQL.

Question. Given a sales(month, product, amount) table where new products are added over time, write a Snowflake query that produces a monthly matrix with one column per product, discovered at query time. Show the ANY ORDER BY syntax and its trade-offs.

Input.

month	product	amount
Jan	Widget	100
Jan	Gadget	80
Jan	Gizmo	50
Feb	Widget	150
Feb	Gadget	90
Feb	Gizmo	60
Feb	Doohickey	40

Code.

-- Snowflake dynamic pivot — ANY ORDER BY, no dynamic SQL
SELECT * FROM (
  SELECT month, product, amount FROM sales
)
PIVOT (SUM(amount) FOR product IN (ANY ORDER BY product));

Step-by-step explanation.

The subquery SELECT month, product, amount FROM sales establishes the input shape. month becomes the implicit grouping key.
PIVOT (SUM(amount) FOR product IN (ANY ORDER BY product)) reads: "pivot on the product column; discover every distinct product value; sort the output columns alphabetically by product."
ANY triggers Snowflake's dynamic-label discovery. At execution time, Snowflake scans the input to find distinct product values, then produces one output column per value.
ORDER BY product pins the output column order to alphabetical by product name. Without it, the order is undefined and could change between runs — that's a downstream-breaking bug waiting to happen.
When a new product (e.g., 'Doohickey') appears in Feb, it automatically gets a column in the output. Old months without that product show NULL in the new column. Downstream consumers must handle schema drift explicitly.

Output.

month	Doohickey	Gadget	Gizmo	Widget
Feb	40	90	60	150
Jan	NULL	80	50	100

Rule of thumb. On Snowflake, always use ANY ORDER BY <col> for dynamic pivots. Skip EXECUTE IMMEDIATE entirely — the declarative form is faster, testable, and can't be SQL-injected. Warn downstream consumers about schema drift when new labels can appear.

Worked example — SQL Server dynamic pivot with PREPARE / EXECUTE

Detailed explanation. SQL Server has no ANY form. The canonical pattern is three-step: query distinct labels, build a comma-separated bracketed list with STRING_AGG(QUOTENAME(col), ','), and execute the concatenated pivot string.

Question. Given a SQL Server sales(month, product, amount) table, write a dynamic pivot that discovers products at query time. Show the metadata query, string build, and execution steps.

Input.

month	product	amount
Jan	Widget	100
Jan	Gadget	80
Feb	Widget	150
Feb	Gadget	90
Feb	Gizmo	60

Code.

-- Step 1 — build the bracketed label list from a metadata query
DECLARE @cols NVARCHAR(MAX);

SELECT @cols = STRING_AGG(QUOTENAME(product), ',')
FROM (SELECT DISTINCT product FROM sales) AS d;
-- @cols now = '[Gadget],[Gizmo],[Widget]'

-- Step 2 — build the full pivot SQL string
DECLARE @sql NVARCHAR(MAX);

SET @sql =
  N'SELECT month, ' + @cols + N'
    FROM (SELECT month, product, amount FROM sales) src
    PIVOT (SUM(amount) FOR product IN (' + @cols + N')) AS p;';

-- Step 3 — execute the dynamic SQL
EXEC sp_executesql @sql;

Step-by-step explanation.

STRING_AGG(QUOTENAME(product), ',') produces the comma-separated bracketed label list. QUOTENAME wraps each label in brackets AND handles any special characters (spaces, reserved words) — critical for SQL injection safety.
The SELECT DISTINCT product FROM sales inside the subquery is the metadata query; it produces one row per unique product. STRING_AGG collapses those rows into a single string.
The @sql variable is built by string concatenation: the outer projection lists the bracketed labels, and the PIVOT clause repeats them in the IN list. Note that @cols is used twice.
sp_executesql @sql is preferred over EXEC(@sql) because it caches the query plan and supports parameter binding for further safety. For this specific case, the plan cache benefit is small because each run's @sql string is different by design.
To wrap this in a stored procedure for reuse, parameterise the aggregate function and the source table name. Use sp_executesql @sql, N'@table NVARCHAR(128)', @table = 'sales' for parameter binding.

Output.

month	Gadget	Gizmo	Widget
Jan	80	NULL	100
Feb	90	60	150

Rule of thumb. Always use QUOTENAME in STRING_AGG to build the label list — it's the SQL-injection-safe way to interpolate identifiers. Never string-concatenate raw user input into dynamic pivot SQL; a hostile category name like 'Widget]) SELECT * FROM users; --' would break the query without proper escaping.

Worked example — BigQuery EXECUTE IMMEDIATE dynamic pivot

Detailed explanation. BigQuery has no ANY form and requires the string-build approach. EXECUTE IMMEDIATE is the mechanism; the label discovery uses STRING_AGG inside a scripting block.

Question. Given a BigQuery sales(month, product, amount) table, write a dynamic pivot that discovers products at query time. Show the scripting block that builds and executes the pivot.

Input.

month	product	amount
Jan	Widget	100
Feb	Widget	150
Feb	Gadget	90

Code.

-- BigQuery scripting block — dynamic pivot with EXECUTE IMMEDIATE
DECLARE cols STRING;
DECLARE sql STRING;

-- Step 1 — build the label list
SET cols = (
  SELECT STRING_AGG(DISTINCT CONCAT("'", product, "'") ORDER BY product)
  FROM `project.dataset.sales`
);
-- cols now = "'Gadget','Widget'"

-- Step 2 — build the pivot statement
SET sql = FORMAT("""
  SELECT * FROM (
    SELECT month, product, amount FROM `project.dataset.sales`
  )
  PIVOT (SUM(amount) FOR product IN (%s))
""", cols);

-- Step 3 — run it
EXECUTE IMMEDIATE sql;

Step-by-step explanation.

DECLARE introduces script-level variables. cols will hold the label string; sql will hold the full pivot statement.
STRING_AGG(DISTINCT CONCAT("'", product, "'") ORDER BY product) builds a quoted-string list 'Gadget','Widget'. The DISTINCT deduplicates; the ORDER BY fixes column order.
FORMAT("... %s ...", cols) is BigQuery's printf-style string interpolation. Cleaner than string concatenation with ||.
EXECUTE IMMEDIATE sql runs the built SQL. The result set is emitted as if you'd written the pivot statically.
For parameterised queries with user input, use EXECUTE IMMEDIATE sql USING @param1, @param2 — BigQuery's binding syntax that avoids injection.

Output.

month	Gadget	Widget
Jan	NULL	100
Feb	90	150

Rule of thumb. On BigQuery, wrap dynamic pivots in a scripting block with DECLARE + EXECUTE IMMEDIATE. For labels sourced from user input, always use the USING @param binding form — string concatenation is the classic SQL-injection vector.

Worked example — dbt macro pivot for analytics engineers

Detailed explanation. In an analytics-engineering workflow, dbt compiles Jinja to SQL at build time. A pivot macro can query the label list at compile time, then emit static SUM(CASE WHEN …) expressions for each label. The compiled SQL is portable across every dialect that supports basic SUM and CASE WHEN.

Question. Write a dbt macro pivot(source_model, column, pivot_column, agg) that emits a static pivot at compile time. Show its use for a monthly-sales report.

Input. The sales model has (month, product, amount). The macro will produce one column per distinct product.

Code.

-- macros/pivot.sql
{% macro pivot(source_model, column, pivot_column, agg='sum') %}
  {% set labels_query %}
    select distinct {{ pivot_column }} as label
    from {{ ref(source_model) }}
    order by 1
  {% endset %}
  {% set labels = run_query(labels_query).columns[0].values() %}

  {% for label in labels %}
    {{ agg }}(case when {{ pivot_column }} = '{{ label }}' then {{ column }} end)
      as {{ label | replace(' ', '_') | lower }}
    {%- if not loop.last -%},{%- endif %}
  {% endfor %}
{% endmacro %}

-- models/monthly_pivot.sql
select
  month,
  {{ pivot('sales', 'amount', 'product') }}
from {{ ref('sales') }}
group by month
order by month;

Step-by-step explanation.

The pivot macro takes four arguments: the source model name, the value column, the label column, and the aggregate function (defaulting to sum).
At compile time, run_query() executes the labels_query against the warehouse to fetch every distinct product. This is a one-shot metadata call.
The Jinja {% for label in labels %} loop emits one SUM(CASE WHEN product = 'X' THEN amount END) AS x expression per label. loop.last controls the trailing comma.
The replace(' ', '_') | lower filter normalises label names into snake-case identifiers safe for output column names.
The compiled result is plain static SQL — every warehouse can execute it. No dynamic SQL, no EXECUTE IMMEDIATE, no injection risk. The macro handles the dynamic label discovery at compile time and emits static output.

Output (compiled SQL for two labels — Widget, Gadget).

select
  month,
  sum(case when product = 'Widget' then amount end) as widget,
  sum(case when product = 'Gadget' then amount end) as gadget
from sales
group by month
order by month;

Rule of thumb. For any dbt project that needs a dynamic pivot, write a pivot macro once and reuse it. The compiled SQL is static, testable with dbt tests, and portable — you can migrate the same model from Snowflake to BigQuery without changing the macro call. Analytics engineers should reach for compile-time code generation over runtime EXECUTE IMMEDIATE whenever possible.

Senior interview question on dynamic pivot design

A senior interviewer might ask: "You're building a reporting system where the pivot column set can grow monthly. How do you architect the dynamic pivot, and how do you protect downstream consumers from schema drift?"

Solution Using compile-time codegen + explicit label contract

Dynamic pivot architecture — compile-time codegen + schema-drift discipline
===========================================================================

Design decision 1 — where does the pivot happen?
  - dbt macro (compile time)  → BEST, works everywhere
  - Snowflake ANY ORDER BY    → good on Snowflake, still drifts
  - EXECUTE IMMEDIATE         → last resort; hard to test

Design decision 2 — how do downstream consumers handle drift?
  - Contract table: every new label must be added to
    `label_registry(label, first_seen_date, retired_date)`
    with a code review before the pivot picks it up.
  - Materialise the pivot output; downstream reads a stable schema.
  - Emit an event on new-label detection so the BI team is notified.

Design decision 3 — what happens on label removal?
  - Retired labels stay in the output as zero-filled columns until
    the retention window ends, then are dropped.
  - Never silently drop a label — dashboards will break.

Step-by-step trace.

Step	Action	Output
1	Query metadata for distinct labels	Live label list
2	Join against `label_registry` — keep only registered labels	Approved label list
3	Compile pivot with approved list	Static SQL
4	Materialise pivot output to a stable table	Stable schema
5	Nightly job: detect new labels, emit event to registry team	Drift alert
6	On approval, next compile picks up the new label	Controlled rollout

The architecture turns "silent schema drift" into "explicit review workflow." Downstream dashboards and BI models see a stable schema; new labels go through a change-management step before they hit production.

Output:

Layer	Component	Purpose
Metadata	`label_registry` table	Contract of approved labels
Compile	dbt pivot macro	Emits static SQL, respects registry
Warehouse	Materialised pivot table	Stable output for BI
Ops	Nightly drift detector	Alerts on unregistered labels

Why this works — concept by concept:

Compile-time codegen — dbt macros run once per model build, emit static SQL, and are testable with dbt tests. Runtime EXECUTE IMMEDIATE is opaque to the query planner and the test framework.
Label registry as a contract — a label_registry table converts "new label appeared in source" from a silent event into an explicit workflow. Engineers see it, review it, and approve it before it hits production.
Materialisation stabilises downstream — writing the pivot output to a table (or dbt incremental model) means BI consumers read a fixed schema. Their dashboards don't break when the pivot picks up a new label mid-day.
Explicit retirement — retired labels stay as zero-filled columns for a retention window. Deleting a column on the day it stops appearing in source data breaks every downstream that references it.
Cost — the compile-time metadata query is one round trip per dbt run. The registry table is negligible. The materialisation is one full-scan-plus-write per refresh. Total cost is comparable to a static pivot; the value is in the drift-control workflow, not throughput.

SQL
Topic — aggregation
Dynamic pivot aggregation drills

Practice →

SQL Topic — grouping sets Grouping sets and multi-level reports

Practice →

5. Conditional-aggregation fallback

`SUM(CASE WHEN …)` is the universal portable pivot; ANSI `FILTER (WHERE …)` is the cleaner form where supported

The mental model in one line: SUM(CASE WHEN col = 'X' THEN val END) AS x is the pivot recipe that works in every SQL dialect ever shipped, and the ANSI FILTER (WHERE …) clause is the cleaner Postgres / SQLite / DuckDB form that plans to the exact same execution — so conditional aggregation is the senior go-to whenever portability, mixed aggregates, or tiny column sets matter more than syntactic sugar. Once you internalise "PIVOT is sugar over SUM(CASE WHEN …)," every dialect gap collapses to a template you fill in.

SUM(CASE WHEN …) — the universal recipe.

Template: SUM(CASE WHEN pivot_col = 'label' THEN value_col END) AS label. Repeat for each label.
THEN value_col (no ELSE) makes non-matching rows contribute NULL, which SUM ignores. Do NOT write ELSE 0 — that inflates counts and, more subtly, breaks AVG if you swap the aggregate.
The outer query needs a GROUP BY on any column NOT in the SUM expression (the row keys). Any column that appears bare in SELECT must be in GROUP BY.
Works in every SQL dialect ever shipped: Postgres, MySQL, MariaDB, SQL Server, Snowflake, BigQuery, Oracle, DB2, SQLite, DuckDB, ClickHouse. Portability is unmatched.

FILTER (WHERE …) — the ANSI cleaner form.

Syntax: SUM(value_col) FILTER (WHERE pivot_col = 'label') AS label. The FILTER clause post-filters rows for that specific aggregate.
ANSI-standard since SQL:2003.
Supported in: Postgres 9.4+, SQLite 3.30+, DuckDB, Firebird.
NOT supported in: Snowflake, BigQuery, SQL Server, MySQL, MariaDB, Oracle.
The query plan is identical to SUM(CASE WHEN …) — this is pure syntactic sugar for readability.

When conditional aggregation wins over PIVOT.

Mixed aggregates in one row. SUM(CASE WHEN month = 'Jan' THEN amount END) AS jan_sum, COUNT(CASE WHEN month = 'Jan' THEN 1 END) AS jan_count, MAX(CASE WHEN month = 'Jan' THEN amount END) AS jan_max. PIVOT gives one aggregate per output column set; CASE WHEN lets you mix freely.
Tiny column sets. For 3–5 pivoted columns, the boilerplate saved by PIVOT is small. The debuggability of a plain SELECT + GROUP BY often wins.
Dialect portability. One query, four warehouses. Only conditional aggregation runs identically everywhere.
Debuggability. Every runner explains the plan the same way. PIVOT in some warehouses hides the plan behind a syntactic sugar layer that's harder to inspect.
Predicate integration. Every label branch is a plain SQL expression, so you can add arbitrary conditions: SUM(CASE WHEN month = 'Jan' AND region = 'US' THEN amount END) AS jan_us.

When PIVOT wins over conditional aggregation.

Many output columns (10+). PIVOT(SUM(amount) FOR month IN ('Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec')) reads as one line. The equivalent SUM(CASE WHEN …) is 12 lines.
Code brevity in a Snowflake / BigQuery / SQL Server single-dialect codebase. If you're never going to port the query, use the sugar.
Dynamic column ergonomics. Snowflake ANY ORDER BY is far cleaner than any hand-coded CASE WHEN generator for unknown label sets.
Analyst readability. BI teams often read PIVOT more easily than a stack of CASE WHEN expressions.

Performance parity.

Every warehouse compiles PIVOT down to the same plan as SUM(CASE WHEN …): a GROUP BY scan with per-column expressions in the projection.
Empirically confirmed on Postgres, Snowflake, BigQuery, SQL Server — same plan, same runtime.
The one exception: some warehouses may vectorise PIVOT slightly better on wide pivots because they can pre-compute the label bitmap. This is a micro-optimisation; ignore it unless you profile and see a difference.

Common interview probes on conditional aggregation.

"How do you pivot in a dialect with no PIVOT keyword?" — SUM(CASE WHEN pivot_col = 'label' THEN val END) AS label, repeated per label, with GROUP BY on the row-key columns.
"What is FILTER (WHERE …) and where does it work?" — the ANSI-standard alternative to CASE WHEN inside an aggregate; supported in Postgres, SQLite, DuckDB.
"Why not write ELSE 0 in the CASE?" — because non-matching rows should contribute NULL (which SUM ignores). ELSE 0 inflates COUNT and breaks AVG if you swap the aggregate.
"Same plan as PIVOT?" — yes. PIVOT is syntactic sugar over SUM(CASE WHEN …) under the hood in every modern warehouse.
"How do you do mixed aggregates (SUM + COUNT)?" — conditional aggregation lets you: SUM(CASE WHEN …) AS s, COUNT(CASE WHEN …) AS c. PIVOT cannot.

Worked example — SUM(CASE WHEN …) monthly pivot

Detailed explanation. The canonical portable pivot: given a long (product, month, sales) table, emit one row per product with columns for each month. Written with SUM(CASE WHEN …), this query runs identically on every dialect.

Question. Write a monthly-sales pivot in SUM(CASE WHEN …) form that produces one row per product with columns for January, February, and March. Explain why ELSE 0 should NOT be used.

Input.

product	month	sales
Widget	Jan	100
Widget	Feb	150
Widget	Mar	200
Gadget	Jan	80
Gadget	Mar	110

Code.

SELECT
  product,
  SUM(CASE WHEN month = 'Jan' THEN sales END) AS jan,
  SUM(CASE WHEN month = 'Feb' THEN sales END) AS feb,
  SUM(CASE WHEN month = 'Mar' THEN sales END) AS mar
FROM sales
GROUP BY product
ORDER BY product;

Step-by-step explanation.

CASE WHEN month = 'Jan' THEN sales END returns sales when the row's month is 'Jan' and NULL otherwise. SUM ignores NULLs by definition, so the total is the sum of only the 'Jan' rows.
GROUP BY product is required because product appears bare in the projection. Every column not inside an aggregate must be in GROUP BY.
The three SUM(CASE WHEN …) expressions each pick a subset of the input and sum them. Same table scan for all three — the planner reads the table once and evaluates the three expressions per row.
Non-matching cells come out as NULL (Gadget in February has no row). To display 0 instead, wrap with COALESCE(SUM(CASE WHEN …), 0).
Do NOT write ELSE 0 inside the CASE. SUM(CASE WHEN month = 'Jan' THEN sales ELSE 0 END) would work for SUM but breaks two ways: (a) COUNT(CASE WHEN month = 'Jan' THEN sales ELSE 0 END) counts every row (including 0s), inflating the count; (b) AVG(CASE WHEN month = 'Jan' THEN sales ELSE 0 END) averages the zeros in, producing the wrong mean.

Output.

product	jan	feb	mar
Gadget	80	NULL	110
Widget	100	150	200

Rule of thumb. Always write CASE WHEN pred THEN val END (with no ELSE). Wrap the whole aggregate with COALESCE(…, 0) if you need zero-instead-of-null in the output. This one habit prevents 90% of accidental aggregation bugs.

Worked example — mixed aggregates in one row

Detailed explanation. A retail report needs monthly sum, count, max, and avg per product, all in one row. PIVOT cannot express this — it gives you one aggregate per output column set. Conditional aggregation nails it.

Question. Given the same sales table, produce one row per product with columns jan_sum, jan_count, jan_max, feb_sum, feb_count, feb_max. Show that conditional aggregation makes this trivial and PIVOT cannot.

Input.

product	month	sales
Widget	Jan	30
Widget	Jan	70
Widget	Feb	150
Gadget	Jan	80
Gadget	Feb	40
Gadget	Feb	50

Code.

SELECT
  product,

  -- January metrics
  SUM  (CASE WHEN month = 'Jan' THEN sales END) AS jan_sum,
  COUNT(CASE WHEN month = 'Jan' THEN 1     END) AS jan_count,
  MAX  (CASE WHEN month = 'Jan' THEN sales END) AS jan_max,

  -- February metrics
  SUM  (CASE WHEN month = 'Feb' THEN sales END) AS feb_sum,
  COUNT(CASE WHEN month = 'Feb' THEN 1     END) AS feb_count,
  MAX  (CASE WHEN month = 'Feb' THEN sales END) AS feb_max
FROM sales
GROUP BY product
ORDER BY product;

Step-by-step explanation.

Every metric per month is its own CASE WHEN inside its own aggregate. SUM sums the sales, COUNT counts the 1s, MAX takes the largest value.
COUNT(CASE WHEN month = 'Jan' THEN 1 END) counts rows where month = 'Jan'. The 1 is arbitrary — any non-NULL value works. This is equivalent to COUNT(*) filtered to Jan rows.
Widget in Jan has two rows (30, 70) → sum = 100, count = 2, max = 70. The query computes all three in one pass.
PIVOT(SUM(sales) FOR month IN ('Jan', 'Feb')) gives only the sums — you can't ask for sum, count, and max in one PIVOT. To get all three, you'd need three separate PIVOT subqueries joined together, or you'd have to concede and use SUM(CASE WHEN …).
This is the killer case for conditional aggregation. Every senior data engineer knows: when you need mixed aggregates, PIVOT is out and CASE WHEN is in.

Output.

product	jan_sum	jan_count	jan_max	feb_sum	feb_count	feb_max
Gadget	80	1	80	90	2	50
Widget	100	2	70	150	1	150

Rule of thumb. Any time an interviewer says "and also count / max / avg per month," reach for conditional aggregation. PIVOT is a one-aggregate tool; CASE WHEN is the multi-aggregate tool. Learning to recognise the mixed-aggregate cue in the interview question is a senior signal.

Worked example — FILTER (WHERE …) on Postgres

Detailed explanation. Postgres, SQLite, and DuckDB all support the ANSI FILTER (WHERE …) clause. It reads more like natural language — "SUM of sales filtered to January" — and plans identically to the CASE WHEN form. Postgres teams should default to it.

Question. Rewrite the previous mixed-aggregate query using FILTER (WHERE …). Show the plan is identical to the CASE WHEN version.

Input. Same as the previous example.

Code.

SELECT
  product,

  -- January metrics with FILTER
  SUM  (sales) FILTER (WHERE month = 'Jan') AS jan_sum,
  COUNT(*)     FILTER (WHERE month = 'Jan') AS jan_count,
  MAX  (sales) FILTER (WHERE month = 'Jan') AS jan_max,

  -- February metrics with FILTER
  SUM  (sales) FILTER (WHERE month = 'Feb') AS feb_sum,
  COUNT(*)     FILTER (WHERE month = 'Feb') AS feb_count,
  MAX  (sales) FILTER (WHERE month = 'Feb') AS feb_max
FROM sales
GROUP BY product
ORDER BY product;

Step-by-step explanation.

SUM(sales) FILTER (WHERE month = 'Jan') reads: "sum the sales column, but only for rows where month = 'Jan'." The FILTER clause post-filters rows for that specific aggregate.
COUNT(*) FILTER (WHERE month = 'Jan') counts rows matching the filter. Cleaner than COUNT(CASE WHEN month = 'Jan' THEN 1 END).
The Postgres query planner compiles this to the same executor plan as the CASE WHEN version — you can confirm with EXPLAIN ANALYZE. Both show a single GroupAggregate node with N aggregate expressions.
Postgres also supports FILTER on window functions: SUM(sales) FILTER (WHERE month = 'Jan') OVER (PARTITION BY region). This is a Postgres exclusive as of 2026.
Snowflake, BigQuery, SQL Server, and MySQL do NOT support FILTER. Trying to use it there produces a syntax error — always confirm your target dialect before adopting FILTER in a codebase.

Output. Identical to the CASE WHEN version — same rows, same columns, same values.

Rule of thumb. On any Postgres, SQLite, or DuckDB project, adopt FILTER (WHERE …) as the house style for conditional aggregates. It's more readable, plans identically, and follows the SQL standard. Just remember it doesn't port to the four biggest cloud warehouses.

Worked example — conditional aggregation for a portable multi-dialect pipeline

Detailed explanation. A data-pipeline library needs a monthly pivot that runs identically on Postgres, Snowflake, BigQuery, and SQL Server. SUM(CASE WHEN …) is the only mechanism that satisfies portability. Show the recipe with COALESCE for missing-cell defaults.

Question. Write a portable monthly pivot for (product, month, sales) that runs on Postgres, Snowflake, BigQuery, and SQL Server without modification. Handle missing cells with COALESCE(0).

Input.

product	month	sales
Widget	Jan	100
Widget	Feb	150
Gadget	Jan	80

Code.

-- Portable across Postgres, Snowflake, BigQuery, SQL Server, MySQL
SELECT
  product,
  COALESCE(SUM(CASE WHEN month = 'Jan' THEN sales END), 0) AS jan,
  COALESCE(SUM(CASE WHEN month = 'Feb' THEN sales END), 0) AS feb,
  COALESCE(SUM(CASE WHEN month = 'Mar' THEN sales END), 0) AS mar
FROM sales
GROUP BY product
ORDER BY product;

Step-by-step explanation.

The SUM(CASE WHEN …) core is the portable primitive. Every dialect ships SUM, CASE WHEN, GROUP BY, and ORDER BY — no dialect-specific syntax needed.
COALESCE(SUM(…), 0) replaces the NULL cells (product-month combinations with no rows) with 0. COALESCE is ANSI-standard and works on every dialect. (Postgres, MySQL, SQL Server all also support IFNULL / ISNULL — but COALESCE is the portable choice.)
The query has no dialect-specific features: no PIVOT, no crosstab, no FILTER, no bracketed identifiers. Runs identically on every warehouse the pipeline touches.
To add a fourth month, add a fourth COALESCE(SUM(CASE WHEN …), 0) line. To swap the aggregate to MAX, change all three SUMs to MAX. No structural rework.
If the library maintainer needs to adapt this to a dialect that has PIVOT for readability, the swap is mechanical. But the reverse — PIVOT code that needs to become portable — requires a full rewrite.

Output.

product	jan	feb	mar
Gadget	80	0	0
Widget	100	150	0

Rule of thumb. For any pipeline that runs across multiple warehouses, write conditional aggregation as the canonical form. Add native PIVOT variants only when a specific dialect's readability wins a code-review argument. The portable form is the "single source of truth"; native forms are optimisations.

Senior interview question on picking the right shape

A senior interviewer might ask: "You're reviewing a pull request that uses Snowflake PIVOT for a monthly-sales report. When would you push back and ask for SUM(CASE WHEN …) instead, and what would the review comment say?"

Solution Using the "portability + mixed-aggregate + column-count" trilemma

Review checklist — PIVOT vs SUM(CASE WHEN …)

1. Portability
   - Will this query ever run on a non-Snowflake warehouse?
     yes → push back on PIVOT; require SUM(CASE WHEN …)
     no  → PIVOT OK for readability

2. Mixed aggregates
   - Does the report emit SUM + COUNT + MAX + AVG per label?
     yes → PIVOT cannot express this; require SUM(CASE WHEN …)
     no  → PIVOT works

3. Column count
   - Fewer than 6 output columns?
     yes → SUM(CASE WHEN …) is roughly the same length; either OK
     no  → PIVOT wins on brevity

4. Dynamic labels?
   - Unknown label set at compile time?
     yes → Snowflake ANY ORDER BY, or dbt macro emitting CASE WHEN
     no  → static form of either

5. Dialect-specific features (COALESCE, DEFAULT ON NULL)?
   - Need zero-fill on missing cells?
     yes → wrap SUM(CASE WHEN …) with COALESCE; PIVOT needs
           dialect-specific option (Snowflake DEFAULT ON NULL,
           SQL Server outer COALESCE)

Step-by-step trace.

PR under review	Q1 portability	Q2 mixed aggs	Q3 col count	Verdict
Sales report, Snowflake-only, 12-month PIVOT	no	no	12	Approve PIVOT
Sales report, Postgres + Snowflake	yes	no	4	Push back → SUM CASE WHEN
Ops report, SUM + COUNT + MAX per month	no	yes	12	Push back → SUM CASE WHEN
Dashboard, unknown categories	no	no	dynamic	Approve ANY ORDER BY
Small ad-hoc, 3 columns, one-off	no	no	3	Either OK

The review checklist turns "PIVOT or not?" from a taste question into a mechanical decision. Portability and mixed-aggregate concerns force CASE WHEN; column count is a tie-breaker.

Output:

Verdict	Reasoning
Approve PIVOT	Single dialect, single aggregate, many columns
Push back → CASE WHEN	Portability required OR mixed aggregates OR tiny column count
Approve ANY ORDER BY	Snowflake dynamic labels, single aggregate
Push back → dbt macro	Dynamic labels across multiple warehouses

Why this works — concept by concept:

Trilemma of PIVOT trade-offs — portability, mixed aggregates, and column count are the three axes that decide whether PIVOT fits. Fix any two axes and the third is determined.
PIVOT is syntactic sugar — it compiles to SUM(CASE WHEN …) in every warehouse. So the choice between them is a readability trade-off, not a performance one. Reviews should focus on maintainability.
Mixed aggregates force CASE WHEN — PIVOT cannot mix SUM + COUNT + MAX in one output row. Any report that needs those alongside each other must use conditional aggregation. This is the killer question in the review checklist.
Portability is a hard constraint — one query on multiple warehouses cannot use PIVOT (Postgres has none, MySQL has none). The moment cross-dialect matters, PIVOT is out and SUM(CASE WHEN …) is in.
Cost — every strategy is O(rows) with one table scan. Column count multiplies the projection cost linearly. Wide pivots (30+ columns) may benefit from PIVOT's vectorisation on some warehouses; this is a micro-optimisation, not a design driver.

SQL
Topic — conditional aggregation
Conditional aggregation problems

Practice →

SQL
Topic — SQL
SQL interview problem library

Practice →

Cheat sheet — pivot and unpivot recipes

Monthly-sales pivot in Snowflake. SELECT * FROM (SELECT product, month, sales FROM t) PIVOT(SUM(sales) FOR month IN ('Jan', 'Feb', 'Mar')). Inline subquery + quoted labels.
Monthly-sales pivot in BigQuery. Same shape as Snowflake but the subquery form is mandatory. PIVOT(SUM(sales) FOR month IN ('Jan', 'Feb', 'Mar')).
Monthly-sales pivot in SQL Server. PIVOT(SUM(sales) FOR month IN ([Jan], [Feb], [Mar])) AS p. Bracketed labels + mandatory alias table.
Monthly-sales pivot in Postgres. SELECT * FROM crosstab($$…$$, $$VALUES ('Jan'), ('Feb'), ('Mar')$$) AS ct(product text, jan int, feb int, mar int). Two-argument form with declared output columns.
Portable monthly pivot. SELECT product, SUM(CASE WHEN month = 'Jan' THEN sales END) AS jan, SUM(CASE WHEN month = 'Feb' THEN sales END) AS feb FROM t GROUP BY product. Runs on every dialect.
UNPIVOT in Snowflake / BigQuery. SELECT * FROM t UNPIVOT (sales FOR month IN (jan, feb, mar)). Add INCLUDE NULLS when you need to preserve NULL cells.
UNPIVOT in SQL Server. SELECT keys, month, sales FROM src UNPIVOT (sales FOR month IN (jan, feb, mar)) AS p. All source columns must share a type — cast in inner subquery if mixed.
UNPIVOT in Postgres via UNION ALL. SELECT product, 'jan' AS month, jan AS sales FROM t UNION ALL SELECT product, 'feb', feb FROM t …. Verbose but explicit; portable to MySQL too.
UNPIVOT in Postgres via jsonb_each_text. SELECT keys, key AS name, value FROM t, LATERAL jsonb_each_text(to_jsonb(t) - 'keys'). Zero column-name maintenance; great for wide tables.
UNPIVOT in Postgres via unnest. SELECT product, d.month, d.sales FROM t, LATERAL unnest(ARRAY['jan','feb','mar'], ARRAY[t.jan, t.feb, t.mar]) AS d(month, sales). Fixed label set, single statement.
Dynamic pivot in Snowflake. PIVOT(SUM(sales) FOR product IN (ANY ORDER BY product)). Native, declarative, no dynamic SQL.
Dynamic pivot in SQL Server. Build a STRING_AGG(QUOTENAME(product), ',') label list, concatenate into the pivot SQL, run with sp_executesql @sql. Always use QUOTENAME for injection safety.
Dynamic pivot in BigQuery. Build the label string with STRING_AGG inside a scripting block, run with EXECUTE IMMEDIATE sql USING @params.
Dynamic pivot in dbt. Macro that calls run_query() at compile time, loops labels in Jinja, emits static SUM(CASE WHEN col = 'X' THEN val END) AS x. Portable output, testable, no runtime dynamic SQL.
Conditional aggregation template. SUM(CASE WHEN pivot_col = 'label' THEN val END) AS label. No ELSE 0 — non-matching rows should contribute NULL. Wrap with COALESCE(…, 0) for zero-fill.
FILTER (WHERE …) form. SUM(sales) FILTER (WHERE month = 'Jan') AS jan. Postgres, SQLite, DuckDB only. Plans identically to CASE WHEN.
Missing-cell defaults. Snowflake: PIVOT(…) DEFAULT ON NULL (0). SQL Server: COALESCE([Jan], 0) in outer projection. BigQuery: COALESCE(Jan, 0) in outer projection. Portable: COALESCE(SUM(CASE WHEN …), 0).
When to use PIVOT. 6+ output columns, single dialect, single aggregate, static labels. When to use CASE WHEN: portability, mixed aggregates, tiny column sets, debuggability.
MySQL emulation. No PIVOT, no UNPIVOT. Use SUM(CASE WHEN …) for pivot, UNION ALL or JSON_TABLE (8.0.4+) for unpivot. Stored procedures with dynamic SQL for compile-time-unknown labels.

Frequently asked questions

Does Postgres support `PIVOT`?

Not with the PIVOT keyword — Postgres has no native PIVOT clause. Postgres teams have three options: (1) the tablefunc extension's crosstab(query, categories) function, which is the classical answer but requires declaring output column names and types explicitly in AS ct(…); (2) SUM(CASE WHEN month = 'Jan' THEN sales END) AS jan, the universal portable recipe that works on every dialect; (3) the ANSI FILTER (WHERE …) clause supported by Postgres 9.4+, which reads slightly more naturally than CASE WHEN and plans identically. For any Postgres query you might port to another warehouse, use conditional aggregation. For Postgres-only reports where the column count is high, crosstab remains readable.

What is the difference between `PIVOT` and `crosstab`?

PIVOT is a SQL clause in a FROM context that shapes a subquery — it treats every implicit column as a grouping key and expands one column into many. crosstab is a Postgres tablefunc function that takes two SQL strings: one for the long-form input (must be ORDER BY row_key, category) and one for the category order pin list. crosstab requires you to declare the output column list and types in an AS ct(row_key type, cat_a type, …) alias; PIVOT infers them from the label list. Functionally the two produce identical output for the same input; syntactically crosstab is more verbose and has stricter ordering requirements. If you're on Snowflake, BigQuery, or SQL Server, use PIVOT; if you're on Postgres with tablefunc installed, crosstab works but many teams still prefer SUM(CASE WHEN …) for portability.

Does BigQuery `PIVOT` work with dynamic column lists?

Not directly — bigquery pivot requires a static IN list in the pivot clause. To handle unknown labels at query time, wrap the pivot in a BigQuery scripting block: declare a STRING variable, populate it by running STRING_AGG(DISTINCT CONCAT("'", pivot_col, "'") ORDER BY pivot_col) against the source, then EXECUTE IMMEDIATE FORMAT("SELECT * FROM t PIVOT(SUM(val) FOR col IN (%s))", cols). Always use USING @param binding when the label list depends on user input to avoid SQL injection. The compile-time alternative — a dbt macro that discovers labels with run_query() and emits static SUM(CASE WHEN …) — is preferred for analytics workflows because the resulting SQL is testable, cacheable, and dialect-portable.

How do I `UNPIVOT` in MySQL?

MySQL has no UNPIVOT keyword. The two idiomatic answers are UNION ALL (works on every MySQL version) and JSON_TABLE (MySQL 8.0.4+). For UNION ALL, write one SELECT per unpivoted column: SELECT product, 'jan' AS month, jan AS sales FROM t UNION ALL SELECT product, 'feb', feb FROM t …. For JSON_TABLE, build a JSON array of label-value pairs and join against it: SELECT t.product, j.month, j.sales FROM t, JSON_TABLE(JSON_ARRAY(JSON_OBJECT('month', 'jan', 'sales', t.jan), JSON_OBJECT('month', 'feb', 'sales', t.feb)), '$[*]' COLUMNS (month VARCHAR(3) PATH '$.month', sales INT PATH '$.sales')) AS j. UNION ALL reads more clearly for 3–5 columns; JSON_TABLE scales better beyond that. For any wider unpivot, generate the SQL in application code — the boilerplate savings pay for the templating complexity.

When should I use conditional aggregation instead of `PIVOT`?

Three cases push you to SUM(CASE WHEN …) over PIVOT. First, portability — one query that runs identically on Postgres AND Snowflake AND BigQuery AND SQL Server AND MySQL cannot use PIVOT because Postgres and MySQL have no such keyword. Second, mixed aggregates in a single output row — SUM + COUNT + MAX + AVG per label combination is not expressible with PIVOT (which allows only one aggregate) but is a one-line pattern with conditional aggregation. Third, tiny column sets (3–5 outputs) where PIVOT's boilerplate savings are small and the debuggability of a plain SELECT + GROUP BY wins. Reach for PIVOT when the output has 6+ columns and readability is the main win, or when you're using Snowflake's flagship ANY ORDER BY dynamic form. Every dialect compiles PIVOT to the same plan as SUM(CASE WHEN …), so performance is never the deciding factor.

What is the `FILTER (WHERE …)` clause and where does it work?

FILTER (WHERE …) is the ANSI SQL:2003 clause that post-filters rows for a specific aggregate. Syntax: SUM(sales) FILTER (WHERE month = 'Jan') AS jan. It reads more naturally than SUM(CASE WHEN month = 'Jan' THEN sales END) and plans identically — the query optimiser produces the exact same execution plan for both forms. Supported dialects in 2026: Postgres 9.4+, SQLite 3.30+, DuckDB, and Firebird. NOT supported: Snowflake, BigQuery, SQL Server, MySQL, MariaDB, Oracle. On Postgres, FILTER also works with window functions: SUM(sales) FILTER (WHERE month = 'Jan') OVER (PARTITION BY region) — a niche superpower. For any Postgres-only or SQLite-only codebase, adopt FILTER as the house style; for a portable library, fall back to SUM(CASE WHEN …) because it works everywhere.

Practice on PipeCode

Drill the unpivoting practice library → for the wide-to-long reshape family — UNPIVOT, UNION ALL, and the Postgres jsonb one-liner.
Rehearse on conditional-aggregation problems → when the interviewer wants a portable SUM(CASE WHEN …) pivot.
Sharpen grouping-sets drills → for ROLLUP, CUBE, and multi-level aggregation reports adjacent to pivot patterns.
Stack the aggregation library → for SUM / COUNT / MAX / MIN / AVG interview probes that underpin every pivot.
Layer the CASE expression library → for the conditional-aggregation primitive that powers portable pivots.
Complement the reshape grind with the CASE WHEN topic collection → for shorter warm-up drills.
For general SQL sharpening, work through the SQL problem library →.
For the broader data-engineering surface, read top data engineering interview questions →.
Stack the prerequisites with the only 5 skills you need to become a data engineer →.
For the broader SQL interview surface, take the SQL for Data Engineering course →.

Pipecode.ai is Leetcode for Data Engineering — every PIVOT, UNPIVOT, and conditional-aggregation recipe above ships with hands-on practice rooms where you write the Snowflake `PIVOT`, wire the Postgres `crosstab`, unpivot with `UNION ALL`, and pick the portable `SUM(CASE WHEN …)` against real graded inputs. PipeCode pairs every reading with 450+ DE-focused problems and a real-time scoring engine, so you never have to wonder whether your `sql pivot` answer holds up under a senior interviewer's depth probes.

Practice pivot patterns now →
Conditional aggregation drills →

SQL Recursive CTEs: Hierarchies, Trees, Graph Traversal, Bill-of-Materials

Gowtham Potureddi — Fri, 10 Jul 2026 17:20:37 +0000

sql recursive cte is the one piece of SQL that turns "impossible without a loop" into "one query with a WITH RECURSIVE." Org-chart traversal, category-tree explosion, friendship-graph BFS, multi-level bill-of-materials rollup — every one of these used to be a Python script pulling rows in batches until a manager column was NULL. Since 2018 or so, every serious dialect has shipped a standards-compliant recursive CTE. The engineering trade-off is no longer "can I do this in SQL?" but "which anchor, which recursive term, and where does the termination live?"

This guide is the mid-to-senior data-engineer walkthrough you wished existed the first time an interviewer asked you to write a recursive query sql on the whiteboard, sketch a sql hierarchy query for an org chart, roll up a bill of materials sql, or run a graph traversal sql with cycle detection. It walks through the recursive CTE anatomy (anchor + recursive term + UNION ALL + termination + dialect keyword differences), single-parent hierarchies (employee-manager tree, depth level column, path array, ORDER BY sibling column, category tree), multi-parent graph traversal (undirected friendship graph BFS, visited-path arrays, cycle detection with NOT ... = ANY(path), Postgres SEARCH BREADTH FIRST / SEARCH DEPTH FIRST and CYCLE clauses), bill-of-materials rollup with quantity multiplication, and the 5-dialect max-recursion matrix (Postgres / MySQL 8 / SQL Server / Snowflake / BigQuery). Each section pairs a teaching block with a Solution-Tail interview answer — code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works.

When you want hands-on reps immediately after reading, drill the CTE practice library →, rehearse on hard-difficulty CTE problems →, and sharpen the SQL axis with the SQL CTE drills →.

On this page

Why recursive CTEs matter in 2026
Recursive CTE anatomy
Hierarchies and trees — org chart / category tree
Graph traversal and cycle detection
Bill-of-materials and dialect quirks
Cheat sheet — recursive CTE recipes
Frequently asked questions
Practice on PipeCode

1. Why recursive CTEs matter in 2026

Recursive CTEs replace app-side loops for hierarchies, trees, graphs, and BOMs — one query, no round-trips

The one-sentence invariant: a sql recursive cte walks a self-referential structure inside the database by repeatedly joining a "frontier" to the parent table, so the shape of the answer is not "loop in Python, one query per level" but "one query, one plan, one pass through the buffer pool". Every other consequence — depth column, path array, cycle detection, BOM rollup — is a specific application of the same anchor-plus-recursive-term shape. Once you internalise the shape, recursive query sql becomes a template you fill in rather than an algorithm you invent.

Three axes interviewers actually probe.

Structure. A sql hierarchy query walks a single-parent tree (manager_id points to at most one manager). A graph traversal sql walks a multi-parent DAG or cyclic graph (a friendship edge table where any node can point to many). A recursive CTE handles both; the second requires cycle detection and the first does not.
Termination. The recursive term must produce zero rows eventually, or the plan runs until the dialect's max-recursion setting fires. For a tree, termination is automatic (leaves have no children). For a graph, termination is engineered — a visited-path array plus a NOT visited predicate.
Rollup. BOM rollup multiplies quantities along every edge: child_accumulated = parent_accumulated × this_edge_qty. The recursive term does the arithmetic; the outer aggregate sums per component. Interviewers love to see the multiplication happen inside the CTE, not after.

Hierarchies vs graphs — the structural split.

Single-parent tree. Employee-manager, filesystem, category tree, comment thread. Each row points to at most one parent. Cycle detection is not required because a tree by definition cannot have cycles.
Multi-parent DAG. BOM (a wheel appears in a bike and a scooter), permission chain (a role inherits from multiple parent roles), lineage graph (a dataset has multiple upstream sources). Cycles are impossible by construction but paths can revisit the same node — dedup at the outer query.
Cyclic graph. Social network, biological pathway, dependency graph with feedback. Cycles are possible; cycle detection is mandatory to prevent infinite recursion.

Dialect status matrix — 2026 reality.

Postgres — first-class since 8.4 (2009). WITH RECURSIVE, SEARCH BREADTH FIRST / DEPTH FIRST, CYCLE clause. Deepest feature set of any mainstream engine.
MySQL 8+ — recursive CTE since 8.0 (2018). WITH RECURSIVE required. cte_max_recursion_depth default 1000. No SEARCH or CYCLE clauses — roll your own.
SQL Server — recursive CTE since 2005. WITH (recursion inferred from the self-reference). Default MAXRECURSION 100, override per-query with OPTION (MAXRECURSION N).
Snowflake — supported. WITH RECURSIVE. Effectively unbounded depth; watch cost.
BigQuery — supported since 2022. WITH RECURSIVE. Default 500 iterations, no per-query override — pre-flatten deeper structures or move to a graph DB.
DuckDB — first-class since 0.7. WITH RECURSIVE and effective for local analytics.

Where a recursive CTE replaces app-side code.

Python loop that hits the DB N times. Any code that walks a parent chain by issuing "SELECT * FROM t WHERE id = ?" in a while loop is a recursive CTE waiting to happen. One query, one round trip.
ORM select_related chains. Django / SQLAlchemy select_related cannot follow a self-referential foreign key of unknown depth. Recursive CTE fixes it.
Materialised transitive closure. A "closure table" (one row per ancestor-descendant pair) can be regenerated from a recursive CTE nightly, without maintaining triggers or app-side denormalisation.
Graph DB for shallow depth. Neo4j / TigerGraph shine at depth > 5 with millions of nodes. For depth ≤ 4 or nodes ≤ 100K, a recursive CTE against Postgres is usually cheaper and simpler.

What interviewers listen for.

Do you say "anchor + recursive term + UNION ALL + termination" in the first sentence? — senior signal.
Do you distinguish "single-parent tree (no cycle guard needed)" vs "multi-parent or cyclic graph (cycle guard mandatory)"? — required answer.
Do you mention max-recursion overrides per dialect (MySQL cte_max_recursion_depth, SQL Server MAXRECURSION) unprompted? — senior signal.
Do you push back on "just use a graph database" with "recursive CTE is the right answer for shallow depth and small node count"? — senior signal.

Worked example — a single query that would take a Python loop

Detailed explanation.

The problem shape. Given an employees table with (id, name, manager_id), list every employee under CEO Alice with their depth from the top. The naive shape is a Python loop: fetch direct reports of Alice, then their direct reports, then their direct reports, joining a level column each time. A recursive CTE does the same walk in one query.
Why one query wins. Each Python-loop iteration pays a full network round-trip. The recursive CTE runs entirely inside the executor with a single plan, materialising the intermediate frontier in a worktable. Latency drops from O(depth × round-trip) to O(single query).
What the recursive CTE looks like. Anchor selects Alice (WHERE manager_id IS NULL). Recursive term joins employees to the CTE by matching employees.manager_id = cte.id, adding a level + 1. Terminates when a level has no direct reports.

Question. Rewrite the Python "walk the org chart" loop as a single recursive CTE. Return employee_id, name, level (root = 0), and sort depth-first.

Input.

id	name	manager_id
1	Alice	NULL
2	Bob	1
3	Carol	1
4	Dan	2
5	Eve	2
6	Frank	3
7	Grace	4

Code.

WITH RECURSIVE org_chart AS (
    -- anchor: the CEO
    SELECT id, name, manager_id, 0 AS level, ARRAY[id] AS path
    FROM employees
    WHERE manager_id IS NULL

    UNION ALL

    -- recursive term: children of the current frontier
    SELECT e.id, e.name, e.manager_id, oc.level + 1, oc.path || e.id
    FROM employees e
    JOIN org_chart oc ON e.manager_id = oc.id
)
SELECT id, name, level
FROM org_chart
ORDER BY path;

Step-by-step explanation.

Anchor. Runs once. Selects Alice — the only row where manager_id IS NULL. Materialises the initial frontier {Alice} with level = 0 and path = [1].
Recursive step 1. Joins the frontier {Alice} to employees on e.manager_id = oc.id (i.e. manager_id = 1). Emits Bob and Carol with level = 1 and paths [1, 2], [1, 3].
Recursive step 2. New frontier is {Bob, Carol}. Joining on manager_id IN (2, 3) emits Dan (level=2, [1,2,4]), Eve (level=2, [1,2,5]), Frank (level=2, [1,3,6]).
Recursive step 3. Frontier {Dan, Eve, Frank} produces Grace (level=3, [1,2,4,7]). No other children exist.
Recursive step 4. Frontier is {Grace}. Nobody reports to Grace → recursive term returns zero rows → CTE terminates.
Outer query. Selects every accumulated row and orders by path, which renders the tree in depth-first order (Alice → Bob → Dan → Grace → Eve → Carol → Frank).

Output.

id	name	level
1	Alice	0
2	Bob	1
4	Dan	2
7	Grace	3
5	Eve	2
3	Carol	1
6	Frank	2

Rule of thumb. Any time you catch yourself writing a while row.parent_id is not None loop in application code against a table with a self-referential FK, stop and write a recursive CTE. One query beats N round-trips every time.

Worked example — hierarchies vs graphs, side by side

Detailed explanation.

A hierarchy is a special graph. A tree is a graph where every node has at most one parent and the reachability from the root has no cycle. A recursive CTE against a tree never revisits a node.
A graph can have cycles. A friendship edge table can encode alice → bob, bob → carol, carol → alice. Without a cycle guard, the recursive CTE loops forever until MAXRECURSION fires.
The syntactic split is small. The only difference between a tree walker and a graph walker is one extra WHERE clause and one accumulated array. The mental model is identical.

Question. Given the same recursive CTE skeleton, show the delta between walking a tree and walking a potentially cyclic graph. Highlight the cycle-guard predicate.

Input — tree (employees).

id	name	manager_id
1	Alice	NULL
2	Bob	1

Input — graph (friendships, undirected).

a	b
1	2
2	3
3	1

Code.

-- Tree walker — no cycle guard needed
WITH RECURSIVE org_chart AS (
    SELECT id, name, 0 AS level FROM employees WHERE manager_id IS NULL
    UNION ALL
    SELECT e.id, e.name, oc.level + 1
    FROM employees e
    JOIN org_chart oc ON e.manager_id = oc.id
)
SELECT * FROM org_chart;

-- Graph walker — CYCLE GUARD IS MANDATORY
WITH RECURSIVE friends AS (
    SELECT 1 AS start_id, 1 AS node_id, 0 AS hop, ARRAY[1] AS path
    UNION ALL
    SELECT f.start_id,
           CASE WHEN fr.a = f.node_id THEN fr.b ELSE fr.a END AS node_id,
           f.hop + 1,
           f.path || CASE WHEN fr.a = f.node_id THEN fr.b ELSE fr.a END
    FROM friends f
    JOIN friendships fr ON f.node_id IN (fr.a, fr.b)
    WHERE NOT (CASE WHEN fr.a = f.node_id THEN fr.b ELSE fr.a END = ANY(f.path))  -- cycle guard
)
SELECT * FROM friends;

Step-by-step explanation.

Tree. The recursive term joins children of the frontier via manager_id = oc.id. Because every child has exactly one manager, no row is ever emitted twice. Termination is automatic when the frontier hits leaves.
Graph. The recursive term joins any edge incident to the frontier node. Without a guard, once we visit 1 → 2 → 3, the next step matches 3 → 1 and we start the cycle again. The WHERE NOT (... = ANY(f.path)) predicate blocks re-visiting a node.
The path array is the memory. It records every node visited from the starting point along this particular walk. Two different walks from the same start can share prefixes but the path is per-row, not per-CTE-wide.
Cost of the guard. Adding = ANY(f.path) is an O(depth) linear scan per recursive step. For deep graphs, this is negligible compared to the join cost.
Termination. With the guard in place, the recursive term returns zero rows once every reachable node has been visited from every reachable start. Same as the tree — the frontier eventually drains.

Output — tree walker.

id	name	level
1	Alice	0
2	Bob	1

Output — graph walker (BFS from node 1).

start_id	node_id	hop	path
1	1	0	[1]
1	2	1	[1, 2]
1	3	2	[1, 2, 3]

Rule of thumb. If the structure guarantees a single-parent tree, skip the cycle guard. If the structure has any multi-parent or bidirectional edges, add the guard as a matter of course — even if you "know" the data is cycle-free today, tomorrow's bug will insert one.

Worked example — when NOT to use a recursive CTE

Detailed explanation.

Depth > 1000 with millions of nodes. Recursive CTE cost is O(edges × depth × node_count in the worktable). At millions of nodes and depth > 10, the plan gets expensive fast. Move to a graph DB or a materialised closure table.
Read-heavy workload with rare structural changes. If the tree changes weekly but the query runs every second, materialise a closure table once and query it directly. O(1) lookup vs O(depth) recursion.
BigQuery hard limits. BigQuery's 500-iteration cap and per-slot cost model make deep recursive CTEs pricy; pre-flatten with a dbt materialised transitive closure model.
You need shortest path, not any path. A recursive CTE finds paths; it does not naturally give the shortest. Cypher / Gremlin do; a recursive CTE with ORDER BY hop LIMIT 1 works but wastes IO.

Question. Given a 20M-node, average-depth-15 hierarchy queried 500 times per second, would you use a recursive CTE? What's the alternative?

Input.

Constraint	Value
Nodes	20 M
Average depth	15
Read QPS	500
Structural change QPS	~1 per day

Code.

-- Closure table — one row per (ancestor, descendant, depth) pair
CREATE TABLE org_closure (
    ancestor_id BIGINT,
    descendant_id BIGINT,
    depth INT,
    PRIMARY KEY (ancestor_id, descendant_id)
);

-- Rebuild from the recursive CTE nightly
INSERT INTO org_closure
WITH RECURSIVE walk AS (
    SELECT id AS ancestor_id, id AS descendant_id, 0 AS depth FROM employees
    UNION ALL
    SELECT w.ancestor_id, e.id, w.depth + 1
    FROM walk w
    JOIN employees e ON e.manager_id = w.descendant_id
)
SELECT * FROM walk;

-- The 500 QPS read now hits a single index lookup, not a recursive CTE
SELECT descendant_id
FROM org_closure
WHERE ancestor_id = :root_id;

Step-by-step explanation.

The closure table stores one row per (ancestor, descendant, depth) pair — the transitive closure of the hierarchy. For a tree with N nodes and average depth D, the table has roughly N × D rows.
Rebuilding is a single recursive CTE, run once per day (or triggered on structural change). The write cost is amortised.
Reads become a single-index lookup: WHERE ancestor_id = :root_id returns all descendants without any recursion at query time.
For 500 QPS, the closure table is O(1) per read; the recursive CTE is O(depth × frontier size), roughly 100x slower.
The trade-off is disk space (N × D rows) and write amplification on structural changes. For a nightly-rebuilt table with 15-average depth, this is 300M rows for 20M nodes — a couple of GB, trivial on modern storage.

Output.

Approach	Read latency	Write cost	Storage
Recursive CTE at query time	30-200 ms per read	zero	zero
Closure table + nightly rebuild	1-5 ms per read	one recursive CTE / day	~N × D rows
Graph DB (Neo4j)	1-10 ms per read	continuous	native graph store

Rule of thumb. Use a recursive CTE at query time for depth ≤ 10 with reads ≤ 10 QPS. Materialise a closure table for depth > 10 or reads > 100 QPS. Move to a graph DB only when both depth and QPS are high and the structural change rate is comparable to the read rate.

Senior interview question on recursive CTE fit

A senior interviewer often opens with: "When would you reach for a recursive CTE, and when would you push back and say 'not in SQL'? Walk me through the 3-4 questions you ask, in order, and what answers push you one way or the other."

Solution Using a 4-question decision framework

Recursive CTE fit — decision framework
======================================

1. Is the structure self-referential?
   - yes (parent_id, edges table)  → recursive CTE eligible
   - no (flat facts)               → not applicable

2. What is the max depth?
   - ≤ 10 (org chart, category tree) → recursive CTE fine
   - 10–100 (deep BOM)               → recursive CTE + max-recursion override
   - > 100 (very deep DAG)           → closure table or graph DB

3. What is the read QPS?
   - ≤ 10 QPS  → recursive CTE at query time
   - 10–100    → recursive CTE + result cache (Redis / materialised view)
   - > 100     → closure table (rebuilt nightly) or graph DB

4. Are cycles possible?
   - no (tree by construction)  → no cycle guard needed
   - yes (graph, feedback)      → cycle guard + path array mandatory

Step-by-step trace.

Use case	Q1 self-ref	Q2 depth	Q3 read QPS	Q4 cycles	Picked
Small org chart (500 people)	yes	5	2	no	recursive CTE at query time
Product category tree (5k rows)	yes	4	200	no	closure table + rebuild on change
Friendship graph (10M users)	yes	∞	100	yes	graph DB or 2-hop closure
Multi-level BOM (aerospace)	yes	30	20	no (DAG)	recursive CTE + MAXRECURSION override
Dataset lineage graph	yes	20	10	possible	recursive CTE + cycle guard

After the 4-question pass, the fit is usually unambiguous. The remaining edge cases — where recursive CTE, closure table, and graph DB all work — default to whatever the team already operates.

Output:

Approach	When it wins
Recursive CTE at query time	Shallow depth, low-to-medium QPS, tree or bounded DAG
Closure table + nightly rebuild	Medium depth, high QPS, low structural change rate
Graph database (Neo4j, TigerGraph)	Deep, cyclic, and high-QPS at the same time

Why this works — concept by concept:

Self-referential structure — the gate — recursive CTE is only relevant when the data model has a self-reference (a foreign key back to the same table or an edges table). Flat facts do not need recursion.
Depth budget — every dialect has a default max recursion. Postgres is effectively unbounded, MySQL is 1000, SQL Server is 100, BigQuery is 500. Depth greater than the default forces an override or a redesign.
QPS budget — recursive CTEs are O(depth × frontier size) per query. High-QPS reads amortise better against a materialised closure table or a graph DB.
Cycle possibility — trees do not need a guard; graphs always do. Add the guard whenever the structure has multi-parent or bidirectional edges — even if today's data is clean.
Cost — recursive CTE at query time is zero infrastructure; closure table adds one nightly job and disk space; graph DB adds a new system to operate. The three tiers scale with the complexity of the workload, not the elegance of the model.

SQL
Topic — CTE
Recursive CTE problems (all difficulty)

Practice →

SQL Topic — CTE · SQL CTE problems in SQL

Practice →

2. Recursive CTE anatomy

`WITH RECURSIVE cte AS (anchor UNION ALL recursive_term) SELECT ... FROM cte` — the only shape you ever need

The mental model in one line: a recursive CTE has exactly two SELECTs stitched by UNION ALL — the anchor runs once and seeds a "frontier"; the recursive term runs repeatedly, each time joining cte to a base table until it returns zero rows. Once you say that out loud, every sql cte interview question about recursion becomes a template you fill in.

The three moving parts.

Anchor SELECT. The base case. Runs once. Materialises the first "generation" of rows into the CTE's worktable. The anchor is the only place where the CTE's column list is established — the recursive term's column list must match exactly (types included).
UNION ALL connector. Fuses the anchor and the recursive term. Almost always UNION ALL — you want every generation appended, not deduplicated at the CTE level. UNION (implicit DISTINCT) is legal in the standard but rare in practice.
Recursive term SELECT. Joins cte (self-reference) to a base table. On every iteration, the executor treats the previous iteration's output as the "current" CTE contents; the recursive term joins that frontier to the base table and emits the next generation. Termination happens when the recursive term returns zero rows.

The stream-vs-set duality.

Conceptually, the CTE contains the union of all generations by the time the outer query reads from it.
Operationally, the executor materialises the anchor's rows into a worktable, then iterates: read from the worktable, join to base, insert result back into the worktable, repeat.
The outer query sees the union — same data as the conceptual model, materialised via the operational one.

UNION ALL vs UNION — the dedup trade-off.

UNION ALL — preserves all rows. Faster (no sort/hash). The right default.
UNION — implicit SELECT DISTINCT on every step. Silently dedupes generations. Sometimes useful for cyclic graphs but a cycle guard is usually better.
Practical rule. Always start with UNION ALL. If you need dedup, do it in the outer query with SELECT DISTINCT where you can see it, not hidden inside the recursion.

Termination — the recursive term must eventually return zero rows.

Tree termination. Automatic. Leaves have no children; the frontier drains.
Bounded numeric termination. A WHERE n < 10 predicate on the recursive term forces zero rows once the counter passes the bound.
Graph termination. The cycle guard (path predicate) removes already-visited nodes; the frontier drains once every reachable node is visited.
What happens without termination. The dialect's max-recursion setting fires: MySQL errors at 1000, SQL Server at 100, Postgres runs until stack overflow. Never rely on the max-recursion to save you — it is a safety net, not a design.

Running numbers 1..N — the "hello world" that proves the shape.

The simplest possible recursive CTE. Anchor emits 1. Recursive term emits n + 1 WHERE n < N. Terminates at n = N.
Reveals the shape without any joins. Useful for generating date series, ID gaps analysis, cross-joining a schedule to a calendar.

Dialect keyword differences.

Postgres, MySQL 8+, Snowflake, BigQuery, DuckDB — WITH RECURSIVE cte AS (...). The RECURSIVE keyword is required.
SQL Server — WITH cte AS (...). Recursion is inferred from the self-reference; no RECURSIVE keyword. The rest of the shape is identical.
Oracle 11g+ — supports the standard WITH cte(col) AS (anchor UNION ALL recursive_term) since 11gR2. Older Oracle uses CONNECT BY syntax (proprietary; do not use in new code).

The generic template — memorise this.

WITH RECURSIVE cte AS (
    -- ANCHOR: base case, runs once
    SELECT ... FROM base_table WHERE <root_condition>

    UNION ALL

    -- RECURSIVE TERM: joins CTE to base table, runs until 0 rows
    SELECT ... FROM base_table b JOIN cte ON <link>
    WHERE <termination_predicate>
)
SELECT ... FROM cte
ORDER BY ...;

Common interview probes on anatomy.

"What is the difference between the anchor and the recursive term?" — anchor runs once and seeds the CTE; recursive term joins CTE to a base table and runs until zero rows.
"Why UNION ALL and not UNION?" — UNION ALL preserves rows; UNION dedupes. Dedup at the CTE level is almost always the wrong place; do it in the outer query.
"How do you terminate a recursive CTE?" — the recursive term must return zero rows. For trees, automatic. For counters, a WHERE n < bound. For graphs, a cycle guard.
"What if I omit RECURSIVE in Postgres?" — you get a syntax error on the self-reference; Postgres does not infer recursion.

Worked example — running numbers 1..N

Detailed explanation.

Why start here. Running numbers is the smallest recursive CTE — no joins, no path arrays, just a counter. It shows the shape (anchor + recursive + termination) with nothing else in the way.
The termination predicate. WHERE n < 10 on the recursive term. On the 10th iteration, n = 10, so n < 10 is false → no rows → CTE terminates. The final CTE contains n = 1, 2, ..., 10.
Where this is useful. Generating a date series (WITH RECURSIVE dates AS (...) SELECT current_date + n * interval '1 day' FROM dates), building a slot table for calendar joins, generating a range for gaps-and-islands prep work.

Question. Write a recursive CTE that generates the numbers 1 through 10 as a single-column result set. Show it in Postgres, MySQL 8, and SQL Server syntax.

Input.

(none — the CTE generates its own input)

Code.

-- Postgres / MySQL 8 / Snowflake / BigQuery — WITH RECURSIVE required
WITH RECURSIVE nums AS (
    SELECT 1 AS n                        -- anchor
    UNION ALL
    SELECT n + 1 FROM nums WHERE n < 10   -- recursive term
)
SELECT n FROM nums;

-- SQL Server — recursion inferred, no RECURSIVE keyword
WITH nums AS (
    SELECT 1 AS n
    UNION ALL
    SELECT n + 1 FROM nums WHERE n < 10
)
SELECT n FROM nums;

Step-by-step explanation.

Iteration 0 (anchor). Executes SELECT 1. Worktable now has {n=1}. This is the initial "current CTE contents."
Iteration 1 (recursive). Executes SELECT n + 1 FROM nums WHERE n < 10 against the current CTE contents. n = 1 < 10, so it emits n = 2. Worktable accumulates {n=1, n=2}.
Iteration 2 (recursive). The "current CTE" for this step is just the previous iteration's output ({n=2}), not the entire worktable. This is the SQL-standard semantics for recursive CTE — the recursive term joins the incremental frontier, not the union. Emits n = 3. Worktable now {1, 2, 3}.
Iterations 3-9. Same shape. Emits n = 4, 5, 6, 7, 8, 9, 10. Worktable accumulates through n = 10.
Iteration 10. Current frontier is {n = 10}. 10 < 10 is false → recursive term emits zero rows → CTE terminates.
Outer SELECT. Reads all 10 rows from the accumulated worktable. Returns 1 through 10.

Output.

n
1
2
3
4
5
6
7
8
9
10

Rule of thumb. The running-numbers pattern is the fastest way to sanity-check that a dialect supports recursive CTEs and that your syntax is right. If this 4-line query fails, you know to check the RECURSIVE keyword and the anchor+recursive shape before touching anything more complex.

Worked example — `UNION ALL` vs `UNION`

Detailed explanation.

UNION ALL preserves every emitted row. The worktable grows monotonically; the recursive term joins the current step's output to the base table.
UNION performs an implicit SELECT DISTINCT between the anchor and the recursive term. In practice, this means the executor dedupes across generations. If a generation would emit a row that already appeared in the worktable, that row is silently dropped.
The trap. In a cyclic graph, UNION can sometimes accidentally save you (the cycle produces a duplicate row that gets deduped), but it hides the bug rather than fixing it. Use a cycle guard, not UNION, for graph traversal.

Question. Show the difference between UNION ALL and UNION when the recursive term can re-emit the same row. Use a tiny 3-node cyclic graph.

Input — edges (directed cycle).

src	dst
1	2
2	3
3	1

Code.

-- UNION ALL — will run until MAXRECURSION fires
WITH RECURSIVE walk_all AS (
    SELECT 1 AS start_id, 1 AS node_id, 0 AS hop
    UNION ALL
    SELECT w.start_id, e.dst, w.hop + 1
    FROM walk_all w
    JOIN edges e ON w.node_id = e.src
)
SELECT * FROM walk_all;
-- ERROR: max recursion 100 exhausted (SQL Server) / 1000 (MySQL)

-- UNION — implicit DISTINCT dedupes and (accidentally) terminates
WITH RECURSIVE walk_dist AS (
    SELECT 1 AS start_id, 1 AS node_id, 0 AS hop
    UNION
    SELECT w.start_id, e.dst, w.hop + 1
    FROM walk_dist w
    JOIN edges e ON w.node_id = e.src
)
SELECT * FROM walk_dist;

Step-by-step explanation.

UNION ALL — infinite loop. Anchor emits {1}. Recursive step 1 emits {2} (via edge 1→2). Step 2 emits {3} (via 2→3). Step 3 emits {1} (via 3→1) — but with a different hop (3) than the anchor (0). Since UNION ALL compares all columns, (1, 1, 0) and (1, 1, 3) are different rows; the executor keeps recursing forever.
UNION — silent dedup. Same steps 1-3. Step 3 emits (1, 1, 3). UNION compares — wait, is (1, 1, 3) equal to (1, 1, 0)? No, hop differs. So the dedup still lets it through. If you drop hop from the SELECT, then UNION would collapse the cycle to a single row and terminate. But that also loses information you probably wanted.
The lesson. UNION only terminates a cycle if the columns you SELECT collapse to already-seen values. Any hop column, timestamp, or path array defeats the accidental termination.
The correct fix. Add a cycle guard. Track visited nodes in a path array and filter with NOT node_id = ANY(path). Do not rely on UNION for termination.
Cost. UNION costs an extra hash/sort per step to dedupe. On non-cyclic recursion, that overhead is pure waste — always use UNION ALL unless you have measured UNION and it is cheaper.

Output — UNION ALL on 3-cycle.

start_id	node_id	hop
1	1	0
1	2	1
1	3	2
1	1	3
...	...	... (runs until MAXRECURSION)

Rule of thumb. Default to UNION ALL. Reach for UNION only when you have measured a specific dedup win and can articulate exactly which duplicate you are dropping. For graph traversal, use a cycle guard — never rely on UNION to save you.

Worked example — dialect keyword differences

Detailed explanation.

Postgres, MySQL 8+, Snowflake, BigQuery, DuckDB — the WITH RECURSIVE keyword is mandatory. Omit it and you get a syntax error at parse time.
SQL Server — no RECURSIVE keyword exists. The parser detects recursion from the self-reference. The rest of the shape (anchor + UNION ALL + recursive term) is identical.
Oracle — supports the standard WITH ... AS (anchor UNION ALL recursive_term) since 11g Release 2. Legacy code uses CONNECT BY — do not use in new code, it is not portable.

Question. Write the same running-numbers 1..5 CTE in the four dominant dialects. Highlight the keyword differences.

Input.

(none — the CTE generates its own input)

Code.

-- Postgres
WITH RECURSIVE nums AS (
    SELECT 1 AS n
    UNION ALL
    SELECT n + 1 FROM nums WHERE n < 5
)
SELECT n FROM nums;

-- MySQL 8+
WITH RECURSIVE nums AS (
    SELECT 1 AS n
    UNION ALL
    SELECT n + 1 FROM nums WHERE n < 5
)
SELECT n FROM nums;

-- SQL Server (no RECURSIVE keyword)
WITH nums AS (
    SELECT 1 AS n
    UNION ALL
    SELECT n + 1 FROM nums WHERE n < 5
)
SELECT n FROM nums
OPTION (MAXRECURSION 1000);   -- optional; default 100

-- Snowflake
WITH RECURSIVE nums AS (
    SELECT 1 AS n
    UNION ALL
    SELECT n + 1 FROM nums WHERE n < 5
)
SELECT n FROM nums;

-- BigQuery
WITH RECURSIVE nums AS (
    SELECT 1 AS n
    UNION ALL
    SELECT n + 1 FROM nums WHERE n < 5
)
SELECT n FROM nums;

Step-by-step explanation.

Every dialect ships the same anchor + UNION ALL + recursive-term shape. The syntactic delta is exactly one keyword (RECURSIVE) and, for SQL Server, an optional OPTION (MAXRECURSION N) at the outer query.
Postgres, MySQL, Snowflake, BigQuery, DuckDB all require RECURSIVE. Omitting it gives a syntax error, not a silent non-recursive interpretation.
SQL Server's parser infers recursion when the CTE's FROM clause references itself. This is a legacy design choice (SQL Server 2005 pre-dated the SQL standard's RECURSIVE keyword); it works but is easy to miss when porting.
Oracle 11gR2+ supports the standard shape too, without RECURSIVE (Oracle also infers). Legacy Oracle used CONNECT BY PRIOR, which is proprietary and less flexible than the standard CTE — do not use in new code.
When porting between dialects, adding or removing the RECURSIVE keyword is usually the only change required. If you also need a max-recursion override, the syntax differs per dialect (see H2-5).

Output.

n
1
2
3
4
5

Rule of thumb. Write portable code by defaulting to WITH RECURSIVE. On SQL Server, the keyword is accepted (as of 2019+) but harmless. This keeps you code-compatible across every mainstream engine except very old Oracle.

Senior interview question on recursive CTE anatomy

A senior interviewer might ask: "Walk me through exactly what the database does when it executes a recursive CTE. What is the anchor doing, what is the recursive term doing, and how does the executor know when to stop? Where can it go wrong?"

Solution Using the anchor + iterative-frontier execution model

Recursive CTE execution — inside the executor
=============================================

Setup
-----
1. Parser detects the WITH RECURSIVE cte AS (...) shape.
2. The CTE's column list is fixed from the anchor SELECT.
3. Executor allocates two worktables:
   - "accumulated" — every row ever emitted
   - "current" — rows emitted in the last iteration

Anchor (runs once)
------------------
4. Execute the anchor SELECT.
5. Insert rows into BOTH "accumulated" and "current".

Recursive term (iterates)
-------------------------
6. While "current" is not empty:
   a. Execute the recursive term with cte := "current".
   b. Insert emitted rows into "accumulated".
   c. Replace "current" with the emitted rows.
7. Loop terminates when the recursive term emits zero rows.

Outer query
-----------
8. The outer SELECT reads "accumulated" as if it were a regular table.

Failure modes
-------------
9. No termination → max-recursion setting fires → error.
10. Column list mismatch between anchor and recursive term → error.
11. Recursive term references the CTE more than once → error in some dialects.

Step-by-step trace.

For running-numbers 1..3 in Postgres:

Iteration	Current worktable	Recursive term emits	Accumulated
0 (anchor)	—	`{1}`	`{1}`
1	`{1}`	`{2}`	`{1, 2}`
2	`{2}`	`{3}`	`{1, 2, 3}`
3	`{3}`	`{}` (3 < 3 false)	`{1, 2, 3}`
4	terminate	—	`{1, 2, 3}`

The current worktable is the incremental frontier — the last iteration's output, not the whole accumulated set. This is the semantic that lets the executor be O(edges) per step rather than O(accumulated).

Output:

Executor concept	What it does
Anchor SELECT	Runs once; seeds the CTE
Current worktable	Holds the last iteration's output
Accumulated worktable	Holds every emitted row
Recursive term	Joins base to current; emits next generation
Termination	Recursive term returns zero rows
Outer SELECT	Reads accumulated as if it were a regular table

Why this works — concept by concept:

Anchor + recursive term — the shape is a standards-defined two-part CTE. The anchor supplies the base case and the column types; the recursive term supplies the induction step. Interviewers listen for this vocabulary.
Incremental frontier — the recursive term joins to the last iteration's output, not the whole accumulated set. This is why the executor is O(edges × depth) not O(edges × accumulated²).
UNION ALL semantics — every recursive-term row is appended without dedup. Dedup at the CTE level is expensive and rarely correct; use UNION ALL and dedup in the outer query if needed.
Termination is your job — the recursive term must produce zero rows; the executor does not infer termination. Max-recursion settings are a safety net for accidental infinite loops.
Cost — O(anchor + Σ recursive-term over generations). For a tree of N nodes at depth D, roughly O(N) since each edge is visited once. For a graph without a cycle guard, O(∞).

SQL
Topic — CTE
Recursive CTE anatomy problems

Practice →

SQL Topic — CTE · hard Hard CTE interview problems

Practice →

3. Hierarchies and trees — org chart / category tree

`sql hierarchy query` walks a single-parent tree — anchor selects the root, recursive term joins children, depth column and path array make the tree navigable

The mental model in one line: a hierarchy is a tree where every row has at most one parent; the recursive CTE anchors on the roots (WHERE manager_id IS NULL), the recursive term joins children of the current frontier (WHERE e.manager_id = t.id), a level + 1 column tracks depth, and a path array (or concatenated string) records ancestry. Once you say that out loud, every sql hierarchy query interview question about org charts, category trees, comment threads, and filesystems becomes the same skeleton with the label swapped.

The canonical shape.

Anchor. Selects the roots — rows where the parent column is NULL, or a specific starting row when you want a sub-tree.
Recursive term. Joins the base table to the CTE where base.parent_id = cte.id. Emits the direct reports of every current-frontier row.
Level column. cte.level + 1 on every recursive step. Root is level 0 (or 1 — pick one and stick to it).
Path array. cte.path || base.id on every recursive step. Records the full ancestry chain from root to this row.
Termination. Automatic — leaves have no children; the recursive term drains.

Two-column identity: (level, path).

Combined, (level, path) uniquely identifies each row's position in the tree.
ORDER BY path renders the tree in depth-first order (siblings appear together, then their children, then the next sibling).
ORDER BY level, name renders the tree in breadth-first level order (all level-0, then all level-1, then all level-2).

Path array vs concatenated string.

Postgres. Native array support: ARRAY[id], path || e.id. Comparisons are natural (= ANY, <@).
MySQL. CONCAT(cte.path, '/', e.id) returns a string. Watch for lexicographic sort order ('2' > '10').
SQL Server / Snowflake / BigQuery. Use CONCAT or string helpers; padded integers (FORMAT(id, '000000')) keep sort order sane.

Where the anchor changes shape.

Whole tree. WHERE parent_id IS NULL.
Descendants of a specific row. WHERE id = :root_id.
Ancestors of a specific row. Flip the join direction — anchor is WHERE id = :leaf_id, recursive term is JOIN employees e ON cte.manager_id = e.id. Walks up instead of down.

Depth-first vs breadth-first ordering.

Depth-first (DFS). Emit each sub-tree in full before moving to the next sibling. ORDER BY path on a tuple-encoded or padded-string path produces DFS.
Breadth-first (BFS). Emit all level-0, then level-1, then level-2. ORDER BY level, name produces BFS.
Postgres shortcut. SEARCH DEPTH FIRST BY id SET seq or SEARCH BREADTH FIRST BY id SET seq adds a seq column that sorts correctly without hand-coding the path.

Category tree — the e-commerce twin.

Products belong to categories that belong to categories: Electronics → Computers → Laptops → Gaming Laptops.
Same shape as the org chart, just different labels. Anchor is WHERE parent_category_id IS NULL; recursive term joins children.
Common interview probe: "given a category id, list every product in that category or any descendant category." Solved with a recursive CTE on categories plus a join to products.

Common interview probes on hierarchies.

"How do you list every employee under a specific manager?" — recursive CTE with the manager as the anchor, recursive term joins direct reports.
"How do you compute the depth of every node in an org chart?" — recursive CTE anchored on manager_id IS NULL, with level + 1 per recursive step.
"How do you find the CEO of an arbitrary employee?" — flip the direction — anchor on that employee, recursive term walks up.
"How do you sort the tree depth-first?" — carry a path array (or padded-string) and ORDER BY path in the outer query.

Worked example — employees under a specific manager

Detailed explanation.

The interview probe. "Given manager_id = 2 (Bob), list every employee under Bob with their depth and full path from Bob." This is the "subtree query" — the anchor is a specific row rather than the whole root set.
The delta from the whole-tree walk. Only the anchor changes. Instead of WHERE manager_id IS NULL, it becomes WHERE id = 2. The recursive term is identical.
What the path represents. path = [2, 4, 7] means Grace's chain from Bob (the query root) is Bob → Dan → Grace. Note that this path starts at Bob, not at the tree root (Alice) — it is a subtree path.

Question. Write a recursive CTE that returns every employee under manager_id = 2 (Bob), with level (Bob = 0) and path (starts at Bob).

Input — employees.

id	name	manager_id
1	Alice	NULL
2	Bob	1
3	Carol	1
4	Dan	2
5	Eve	2
6	Frank	3
7	Grace	4

Code.

WITH RECURSIVE subtree AS (
    -- anchor: Bob himself
    SELECT id, name, manager_id, 0 AS level, ARRAY[id] AS path
    FROM employees
    WHERE id = 2

    UNION ALL

    -- recursive: children of the frontier
    SELECT e.id, e.name, e.manager_id, s.level + 1, s.path || e.id
    FROM employees e
    JOIN subtree s ON e.manager_id = s.id
)
SELECT id, name, level, path
FROM subtree
ORDER BY path;

Step-by-step explanation.

Anchor. Selects Bob (id=2). Emits (2, Bob, 1, 0, [2]). Frontier is {Bob}.
Recursive step 1. Joins children of Bob. Emits Dan (level=1, path=[2, 4]) and Eve (level=1, path=[2, 5]). Frontier now {Dan, Eve}.
Recursive step 2. Children of Dan: Grace (level=2, path=[2, 4, 7]). Children of Eve: none. Frontier now {Grace}.
Recursive step 3. Children of Grace: none. Recursive term emits zero rows → CTE terminates.
Outer query. ORDER BY path gives depth-first traversal from Bob: Bob → Dan → Grace → Eve.

Output.

id	name	level	path
2	Bob	0	[2]
4	Dan	1	[2, 4]
7	Grace	2	[2, 4, 7]
5	Eve	1	[2, 5]

Rule of thumb. For subtree queries, change only the anchor. The recursive term stays identical across "whole tree," "subtree under X," and "ancestors of X" (though for ancestors, the join direction flips).

Worked example — ancestors of an employee

Detailed explanation.

The direction flip. For descendants, the recursive term joins e.manager_id = cte.id. For ancestors, it joins cte.manager_id = e.id — cte holds the child, e is the parent.
The termination. Automatic. The root has manager_id IS NULL; the recursive term joining a NULL manager_id emits zero rows.
The path. Prepends parents as you walk up: [7, 4, 2, 1] for Grace's ancestry to Alice. path[1] is Grace, path[-1] (or reversed) is the CEO.

Question. Given employee_id = 7 (Grace), return every ancestor in the chain up to the CEO, with depth (Grace = 0, CEO = 3 in this tree).

Input — same employees table as above.

Code.

WITH RECURSIVE ancestors AS (
    -- anchor: Grace herself
    SELECT id, name, manager_id, 0 AS depth_up
    FROM employees
    WHERE id = 7

    UNION ALL

    -- recursive: walk UP via manager_id
    SELECT e.id, e.name, e.manager_id, a.depth_up + 1
    FROM employees e
    JOIN ancestors a ON a.manager_id = e.id
)
SELECT id, name, depth_up
FROM ancestors
ORDER BY depth_up;

Step-by-step explanation.

Anchor. Selects Grace (id=7). Frontier {Grace}, depth_up = 0.
Recursive step 1. Joins Grace's manager: Dan (id=4). Emits (4, Dan, 2, 1). Frontier {Dan}.
Recursive step 2. Joins Dan's manager: Bob (id=2). Emits (2, Bob, 1, 2). Frontier {Bob}.
Recursive step 3. Joins Bob's manager: Alice (id=1). Emits (1, Alice, NULL, 3). Frontier {Alice}.
Recursive step 4. Alice's manager_id IS NULL. The recursive term's join ON a.manager_id = e.id has NULL on the left side → no match → zero rows emitted → CTE terminates.
Outer query. Orders by depth_up — starts at Grace (0), ends at the CEO Alice (3).

Output.

id	name	depth_up
7	Grace	0
4	Dan	1
2	Bob	2
1	Alice	3

Rule of thumb. The direction of the recursive term's join determines "up" vs "down." Descendants use e.manager_id = cte.id; ancestors use cte.manager_id = e.id. Everything else stays the same.

Worked example — category tree with product rollup

Detailed explanation.

The e-commerce twin. A categories table has (id, name, parent_category_id). A products table has (id, name, category_id). Interview probe: "list every product in category 3 or any descendant category of 3."
The two-CTE chain. First recursive CTE walks the category tree from id = 3. Second (regular) CTE joins products to that walk.
Why not a single query. You can inline the join, but the two-step form is more readable and often easier for the optimiser to plan.

Question. Given category_id = 3 (Computers), return every product in that category or any descendant. Show the category tree walk and the product join.

Input — categories.

id	name	parent_category_id
1	Everything	NULL
2	Books	1
3	Computers	1
4	Laptops	3
5	Gaming	4
6	Desktops	3

Input — products.

id	name	category_id
100	Novel	2
200	Textbook	2
300	MacBook	4
400	Alienware	5
500	iMac	6

Code.

WITH RECURSIVE cat_tree AS (
    -- anchor: Computers
    SELECT id, name, parent_category_id
    FROM categories
    WHERE id = 3

    UNION ALL

    -- recursive: descendants
    SELECT c.id, c.name, c.parent_category_id
    FROM categories c
    JOIN cat_tree ct ON c.parent_category_id = ct.id
)
SELECT p.id, p.name, c.name AS category
FROM products p
JOIN cat_tree c ON p.category_id = c.id
ORDER BY p.id;

Step-by-step explanation.

Anchor. Selects Computers (id=3). Frontier {Computers}.
Recursive step 1. Children of Computers: Laptops (id=4), Desktops (id=6). Frontier {Laptops, Desktops}.
Recursive step 2. Children of Laptops: Gaming (id=5). Children of Desktops: none. Frontier {Gaming}.
Recursive step 3. Children of Gaming: none. Recursive term emits zero rows → CTE terminates. cat_tree now contains {Computers, Laptops, Desktops, Gaming}.
Outer query. Joins products to cat_tree. Every product whose category_id matches any row in the tree passes through. MacBook (Laptops), Alienware (Gaming), iMac (Desktops) match; Novel and Textbook (both Books) do not.

Output.

id	name	category
300	MacBook	Laptops
400	Alienware	Gaming
500	iMac	Desktops

Rule of thumb. For "products in a category or any descendant category," write the recursive CTE on the category tree first, then join products. The two-step shape reads naturally and gives the planner clear boundaries to reason about.

Senior interview question on org-chart traversal

A senior interviewer might ask: "Write a SQL query that returns, for every employee, their name, their level in the org chart (CEO = 0), their manager's name, and the total number of employees under them (direct + indirect reports). One query, no application code."

Solution Using a recursive CTE plus an aggregate join

WITH RECURSIVE org AS (
    -- anchor: the CEO
    SELECT id, name, manager_id, 0 AS level, ARRAY[id] AS path
    FROM employees
    WHERE manager_id IS NULL

    UNION ALL

    -- recursive: children
    SELECT e.id, e.name, e.manager_id, o.level + 1, o.path || e.id
    FROM employees e
    JOIN org o ON e.manager_id = o.id
),
reports_count AS (
    -- for every row, count how many other rows have this row's id in their path
    SELECT o1.id AS emp_id,
           COUNT(o2.id) - 1 AS total_reports   -- minus self
    FROM org o1
    JOIN org o2 ON o1.id = ANY(o2.path)
    GROUP BY o1.id
)
SELECT o.name AS employee,
       o.level,
       m.name AS manager,
       r.total_reports
FROM org o
LEFT JOIN employees m ON o.manager_id = m.id
JOIN reports_count r ON o.id = r.emp_id
ORDER BY o.path;

Step-by-step trace.

Using the seven-row employees table:

Step	What it does
1	Anchor: emit `(Alice, 0, [1])`
2	Recursive: children of Alice → Bob (level 1, `[1,2]`), Carol (level 1, `[1,3]`)
3	Recursive: children of Bob, Carol → Dan `[1,2,4]`, Eve `[1,2,5]`, Frank `[1,3,6]`
4	Recursive: children of Dan/Eve/Frank → Grace `[1,2,4,7]`
5	Recursive: zero rows → CTE `org` terminates
6	`reports_count` CTE joins `org` to itself where `o1.id = ANY(o2.path)` — for every ancestor, count descendants
7	Outer SELECT joins `org` to `employees` (via manager_id) and to `reports_count`
8	`ORDER BY path` renders DFS

The self-join in reports_count uses o1.id = ANY(o2.path): row o1 is "in the path of" row o2 if and only if o1 is an ancestor of o2. Grouping by o1.id and counting o2.id (minus 1 for self) gives the count of descendants.

Output:

employee	level	manager	total_reports
Alice	0	(NULL)	6
Bob	1	Alice	3
Dan	2	Bob	1
Grace	3	Dan	0
Eve	2	Bob	0
Carol	1	Alice	1
Frank	2	Carol	0

Why this works — concept by concept:

Path array as ancestry chain — carrying the full path from root to this row makes every ancestor addressable. o1.id = ANY(o2.path) is a set-membership test that returns TRUE if and only if o1 is an ancestor of o2.
Self-join on org — the CTE is used twice in the same query: once to enumerate rows, once to count descendants via path membership. A regular CTE is fine here because the second usage does not itself recurse.
LEFT JOIN for the manager name — the CEO's manager_id is NULL; a LEFT JOIN preserves the row with a NULL manager name rather than dropping it.
ORDER BY path — arrays compare element-wise; [1] < [1,2] < [1,2,4] < [1,2,4,7] < [1,2,5] < [1,3] < [1,3,6]. That is exactly the DFS pre-order.
Cost — recursive CTE is O(N) for a tree of N nodes. The reports_count self-join is O(N × avg_depth); on org charts with depth ≤ 10 and N ≤ 100K, that's a millisecond query.

SQL
Topic — tree
Tree traversal problems

Practice →

SQL Topic — CTEs CTE hierarchy problems

Practice →

4. Graph traversal and cycle detection

`graph traversal sql` walks a multi-parent (possibly cyclic) edge table — the path array is your memory, the `NOT visited` predicate is your cycle guard, and Postgres has built-in `SEARCH` and `CYCLE` clauses

The mental model in one line: a graph is an edges table (src, dst) where any node can point to many; the recursive CTE anchors on the starting node, the recursive term joins every incident edge to the frontier, an accumulated path array records visited nodes, and a NOT node = ANY(path) predicate blocks cycles. Once you say that out loud, every friendship-BFS, permission-chain, or dependency-graph interview question becomes the same recipe with different labels.

Directed vs undirected edges.

Directed. edges(src, dst) means src → dst only. Recursive term joins edges.src = cte.node_id; emits edges.dst.
Undirected. Same table, but src ↔ dst (both ways). Recursive term joins where edges.src = cte.node_id or edges.dst = cte.node_id; emits the other endpoint.
Symmetric edge trick. Store undirected edges as two rows ((a, b) and (b, a)) and treat as directed. Trades storage for query simplicity.

BFS via recursive CTE.

Anchor emits the starting node with hop = 0, path = [start].
Recursive term joins each frontier node to edges, filtered by NOT dst = ANY(path). Increments hop, appends dst to path.
Terminates when every reachable-from-start node has been visited (frontier drains).

Cycle detection — three flavours.

Path array with = ANY (Postgres, DuckDB) — WHERE NOT child = ANY(cte.path). O(depth) per row, but O(depth) is small.
Concatenated string with LIKE (MySQL, SQL Server) — WHERE cte.path NOT LIKE CONCAT('%,', child, ',%'). Same idea; slower on very deep paths.
Postgres CYCLE clause — declarative: CYCLE id SET is_cycle USING path. Adds is_cycle and path columns automatically; you filter WHERE NOT is_cycle in the outer query.

Postgres SEARCH and CYCLE — the built-ins.

SEARCH BREADTH FIRST BY col SET seq — adds a seq column that, when you ORDER BY seq, gives BFS order.
SEARCH DEPTH FIRST BY col SET seq — same, but DFS order.
CYCLE col SET is_cycle USING path — adds is_cycle (bool) and path (array) columns. Sets is_cycle = true the moment a cycle is detected; the recursive term stops on cycle-flagged rows. Combines cleanly with SEARCH.

Shortest path — recursive CTE + aggregation.

Recursive CTE walks all paths from a start.
Outer query: SELECT MIN(hop) FROM cte WHERE node_id = :target.
Not efficient for large graphs (recursive term enumerates every path), but correct.
For large graphs, use a graph DB or an app-side BFS with proper visited state.

Common interview probes on graphs.

"How do you detect a cycle in a graph in SQL?" — accumulate a path array in the recursive CTE; filter NOT child = ANY(path) on the recursive term.
"How do you find the shortest path between two nodes?" — recursive CTE that walks all paths; MIN(hop) FROM cte WHERE node_id = :target. Slow but correct.
"What does Postgres' SEARCH BREADTH FIRST clause do?" — adds an ordering column that sorts BFS-first without hand-coding path arithmetic.
"When would you not use a recursive CTE for a graph?" — high-QPS shortest-path queries on large graphs → graph DB.

Worked example — undirected friendship graph BFS

Detailed explanation.

The setup. A friendships(a, b) table storing undirected edges. To make the join symmetric, we treat both friendships.a = cte.node_id and friendships.b = cte.node_id as matches.
The visited-path guard. Without a guard, the recursive term keeps re-visiting nodes and looping through the triangle 1-2-3 forever. WHERE NOT other_endpoint = ANY(path) stops it.
The output shape. For each starting node, we get one row per reachable node with hop (distance) and path (the sequence).

Question. Given an undirected friendship graph, write a recursive CTE that returns every friend-of-a-friend (up to 3 hops) from user 1, with the hop distance and the full path.

Input — friendships (undirected).

a	b
1	2
1	3
2	4
3	4
4	5

Code.

WITH RECURSIVE bfs AS (
    -- anchor: user 1
    SELECT 1 AS node_id, 0 AS hop, ARRAY[1] AS path
    UNION ALL
    SELECT other AS node_id, b.hop + 1, b.path || other
    FROM bfs b
    JOIN LATERAL (
        SELECT CASE WHEN f.a = b.node_id THEN f.b ELSE f.a END AS other
        FROM friendships f
        WHERE b.node_id IN (f.a, f.b)
    ) e ON true
    WHERE NOT other = ANY(b.path)
      AND b.hop < 3
)
SELECT node_id, hop, path
FROM bfs
ORDER BY hop, node_id;

Step-by-step explanation.

Anchor. Emits user 1 with hop = 0, path = [1]. Frontier {1}.
Recursive step 1. For frontier node 1, joins friendships where 1 IN (a, b). Matches edges (1,2) and (1,3). The LATERAL picks the other endpoint. Filter NOT other = ANY(path) is true for both. Filter hop < 3 is true. Emits nodes 2 and 3 with hop = 1. Paths [1, 2] and [1, 3].
Recursive step 2. Frontier is {2, 3}. From node 2: edges (1,2) and (2,4). Other endpoints: 1 (blocked by path guard), 4 (allowed). From node 3: edges (1,3) and (3,4). Other endpoints: 1 (blocked), 4 (allowed via different path [1, 3, 4]). Emits (4, 2, [1, 2, 4]) and (4, 2, [1, 3, 4]).
Recursive step 3. Frontier is {4 (from path [1,2,4]), 4 (from path [1,3,4])}. From node 4: edges (2,4), (3,4), (4,5). Other endpoints: 2, 3, 5. Path guards block 2 and 3 (already in each path). 5 is allowed. Emits (5, 3, [1, 2, 4, 5]) and (5, 3, [1, 3, 4, 5]).
Recursive step 4. Frontier is {5, 5}. But hop < 3 is false (would be 4) → recursive term emits zero rows → CTE terminates.
Outer query. Orders by (hop, node_id). Returns 1 at hop 0, 2/3 at hop 1, 4 at hop 2 (two paths), 5 at hop 3 (two paths).

Output.

node_id	hop	path
1	0	[1]
2	1	[1, 2]
3	1	[1, 3]
4	2	[1, 2, 4]
4	2	[1, 3, 4]
5	3	[1, 2, 4, 5]
5	3	[1, 3, 4, 5]

Rule of thumb. BFS in SQL emits every reachable path, not every unique node. For "unique nodes at each hop," dedup with SELECT DISTINCT node_id, MIN(hop) in the outer query. Two paths to the same node at the same hop is a signal of shortest-path ties.

Worked example — cycle detection with a directed graph

Detailed explanation.

The setup. A directed edges(src, dst) table containing a cycle 1 → 2 → 3 → 1. Without a cycle guard, the recursive term loops forever.
The path predicate. WHERE NOT dst = ANY(path) prevents re-visiting. When dst = 1 and path = [1, 2, 3], the guard blocks the emission.
Detecting the cycle explicitly. If the interviewer wants "list the cycles," track the cycle with an additional flag: is_cycle := dst = ANY(path) on the emit; then WHERE is_cycle in the outer query.

Question. Given a directed graph with a cycle 1 → 2 → 3 → 1, walk from node 1 and detect the cycle. Return the path that closes the loop.

Input — edges (directed).

src	dst
1	2
2	3
3	1
3	4

Code.

WITH RECURSIVE walk AS (
    SELECT 1 AS node_id, ARRAY[1] AS path, false AS is_cycle
    UNION ALL
    SELECT e.dst,
           w.path || e.dst,
           e.dst = ANY(w.path) AS is_cycle
    FROM walk w
    JOIN edges e ON w.node_id = e.src
    WHERE NOT w.is_cycle                       -- stop expanding once cycle found on this path
)
SELECT node_id, path, is_cycle
FROM walk
ORDER BY array_length(path, 1);

Step-by-step explanation.

Anchor. (1, [1], false). Frontier {1}.
Recursive step 1. Edge 1 → 2. dst = 2 not in [1] → is_cycle = false. Emits (2, [1, 2], false).
Recursive step 2. Edge 2 → 3. dst = 3 not in [1, 2] → is_cycle = false. Emits (3, [1, 2, 3], false).
Recursive step 3. From node 3: edges 3 → 1 and 3 → 4.
- 3 → 1: dst = 1 is in [1, 2, 3] → is_cycle = true. Emits (1, [1, 2, 3, 1], true).
- 3 → 4: dst = 4 not in [1, 2, 3] → is_cycle = false. Emits (4, [1, 2, 3, 4], false).
Recursive step 4. Frontier row (1, [1, 2, 3, 1], true) is blocked from expanding by WHERE NOT w.is_cycle. Frontier row (4, [1, 2, 3, 4], false): no outgoing edges from 4 → emits nothing.
Recursive step 5. Zero rows → CTE terminates.
Outer query. WHERE is_cycle returns exactly the row(s) that closed a cycle.

Output.

node_id	path	is_cycle
1	[1]	false
2	[1, 2]	false
3	[1, 2, 3]	false
1	[1, 2, 3, 1]	true
4	[1, 2, 3, 4]	false

Rule of thumb. For every graph traversal, add a path column and a cycle guard even if you "know" the data is cycle-free. The overhead is O(depth) per row — negligible — and the safety is invaluable. For explicit cycle reporting, add an is_cycle boolean and stop expanding on true.

Worked example — Postgres `SEARCH` + `CYCLE` built-ins

Detailed explanation.

What SEARCH does. Adds an ordering column that, when you ORDER BY it, gives DFS or BFS order without hand-coding.
What CYCLE does. Adds is_cycle (bool) and path (array) columns automatically. Sets is_cycle = true the moment a cycle is detected; the recursive term stops expanding cycle-flagged rows.
Why prefer them. Declarative, less error-prone, harder to introduce a bug. And often the planner can optimise a CYCLE-clause CTE more aggressively than a hand-coded path predicate.

Question. Rewrite the cycle-detection query using Postgres' SEARCH DEPTH FIRST and CYCLE clauses. Compare readability.

Input — same directed edges table with a 1 → 2 → 3 → 1 cycle.

Code.

WITH RECURSIVE walk (node_id) AS (
    SELECT 1
    UNION ALL
    SELECT e.dst
    FROM walk w
    JOIN edges e ON w.node_id = e.src
)
SEARCH DEPTH FIRST BY node_id SET seq
CYCLE node_id SET is_cycle USING path
SELECT node_id, path, is_cycle, seq
FROM walk
ORDER BY seq;

Step-by-step explanation.

Postgres synthesises the path column automatically because of the CYCLE ... USING path clause. No manual ARRAY[node_id] in the anchor.
The CYCLE clause tracks visited nodes for us. When the recursive term is about to emit a row whose node_id is already in path, Postgres marks is_cycle = true and does not expand that row further.
The SEARCH DEPTH FIRST BY node_id SET seq adds a seq integer column. Rows sorted by seq are in DFS pre-order.
The outer query orders by seq. Postgres has already done the visited-set bookkeeping.
Compared to the hand-coded version: no explicit path array, no explicit NOT ... = ANY(path) predicate, no explicit is_cycle boolean. Postgres does all three declaratively.

Output.

node_id	path	is_cycle	seq
1	{1}	false	1
2	{1, 2}	false	2
3	{1, 2, 3}	false	3
1	{1, 2, 3, 1}	true	4
4	{1, 2, 3, 4}	false	5

Rule of thumb. On Postgres 14+, prefer SEARCH and CYCLE for any graph traversal — cleaner code, fewer bugs, and often a better plan. On other dialects, hand-code the path array and predicate. The mental model is identical.

Senior interview question on graph traversal

A senior interviewer might ask: "Given a directed graph edges table, write a recursive CTE that finds the shortest path between two specific nodes, protects against cycles, and returns the path as an array. Walk me through how it works and where it can go wrong."

Solution Using a path-tracking recursive CTE + MIN aggregation

WITH RECURSIVE paths AS (
    SELECT :start_id AS node_id, ARRAY[:start_id]::int[] AS path, 0 AS hop
    UNION ALL
    SELECT e.dst,
           p.path || e.dst,
           p.hop + 1
    FROM paths p
    JOIN edges e ON p.node_id = e.src
    WHERE NOT e.dst = ANY(p.path)              -- cycle guard
      AND p.hop < 20                            -- depth cap for safety
)
SELECT path, hop
FROM paths
WHERE node_id = :target_id
ORDER BY hop
LIMIT 1;

Step-by-step trace.

Given edges = {(1,2), (1,3), (2,4), (3,4), (4,5), (2,5)}, :start_id = 1, :target_id = 5:

Iteration	Frontier	Emitted rows
0 (anchor)	—	`(1, [1], 0)`
1	`{1}`	`(2, [1,2], 1)`, `(3, [1,3], 1)`
2	`{2, 3}`	`(4, [1,2,4], 2)`, `(5, [1,2,5], 2)`, `(4, [1,3,4], 2)`
3	`{4, 4, 5}`	`(5, [1,2,4,5], 3)`, `(5, [1,3,4,5], 3)`
4	`{5, 5}`	none (5 has no outgoing)
5	zero rows → terminate

Outer query WHERE node_id = 5 ORDER BY hop LIMIT 1 returns ([1, 2, 5], 2).

Output:

path	hop
{1, 2, 5}	2

Why this works — concept by concept:

Path array as memory — the recursive CTE has no other place to record "which nodes have I already visited on this branch." The path array is per-row, not per-CTE, so parallel branches keep independent memory.
Cycle guard as pre-emit filter — WHERE NOT e.dst = ANY(p.path) runs before the row is emitted. That way, we never insert a cycle-closing row, so the recursive term drains naturally.
Depth cap as belt-and-braces — AND p.hop < 20 is a safety net for graphs with unexpectedly long paths or subtle bugs in the cycle guard. Never rely on the dialect's max-recursion — engineer termination.
ORDER BY hop LIMIT 1 for shortest — the recursive CTE enumerates every path; the outer aggregate picks the minimum. Correct but not efficient — for high-QPS shortest-path queries, use a graph DB.
Cost — for a graph with average branching factor b and depth cap d, worst-case is O(b^d). Path arrays and cycle guards keep it near O(edges) in practice, but skewed graphs can blow up. For millions of edges, move to a graph DB.

SQL
Topic — graph
Graph traversal problems

Practice →

SQL Topic — recursion Recursion practice problems

Practice →

5. Bill-of-materials and dialect quirks

`bill of materials sql` rolls up child quantities into parent quantities, multiplying along every edge — and every dialect has its own max-recursion cap

The mental model in one line: a multi-level BOM is a DAG where each parent points to child components with a per-edge quantity weight; the recursive CTE anchors on the top-level product, joins children in the recursive term, multiplies accumulated_qty = parent_accumulated × edge_qty, and terminates when a component has no further sub-components; every dialect has its own default cap on recursion depth (Postgres effectively unbounded, MySQL 1000, SQL Server 100, Snowflake unbounded, BigQuery 500). Once you internalise "multiply along the edge, sum at the leaf," every bill of materials sql interview surface collapses to a template.

The BOM data model.

bom(parent_id, child_id, qty) — a "child_id appears qty times inside parent_id."
The graph is a DAG — a wheel appears in a bike and a scooter; there is no cycle by construction.
Roll-up is the natural operation: "for a bike, how many spokes total?" — walk down, multiply along every edge.

The multiplication trick.

Anchor: SELECT product_id, ARRAY[product_id] AS path, 1 AS accumulated_qty, 0 AS level FROM bom_top WHERE product_id = :root.
Recursive term: SELECT b.child_id, cte.path || b.child_id, cte.accumulated_qty * b.qty, cte.level + 1 FROM bom b JOIN cte ON b.parent_id = cte.product_id.
Termination: automatic — a leaf component has no bom rows as parent.

The rollup.

The CTE enumerates every path from root to leaf, carrying the accumulated quantity.
To answer "how much of each component in total," sum the accumulated quantity per component: SELECT component_id, SUM(accumulated_qty) FROM cte GROUP BY component_id.
This handles the DAG shape: if a wheel appears in the bike and a wheel appears in the trailer (which is also inside the bike as a sub-assembly), summing at the outer query aggregates both occurrences.

Dialect max-recursion matrix.

Dialect	Default cap	Override
Postgres	effectively unbounded (stack depth)	`SET LOCAL max_stack_depth = '4MB'`
MySQL 8+	`cte_max_recursion_depth = 1000`	`SET cte_max_recursion_depth = 5000`
SQL Server	`MAXRECURSION 100`	`OPTION (MAXRECURSION 5000)` per query
Snowflake	effectively unbounded	none needed
BigQuery	500 iterations	none per-query; redesign
DuckDB	effectively unbounded	none needed

When the max recursion fires.

The engine raises a specific error: MySQL says "Recursive query aborted after N iterations", SQL Server says "Maximum recursion depth exceeded", BigQuery says "Recursive query iteration limit reached."
The error is a hard stop — no partial results. Wrap your CTE in a LIMIT on the outer query does not prevent recursion from running to the cap.
The right fix depends on the shape:
- Legitimate deep recursion — override the cap (SQL Server OPTION (MAXRECURSION 5000), MySQL SET cte_max_recursion_depth = 5000).
- Infinite loop — the recursive term never returns zero rows; find the bug (missing termination, missing cycle guard).
- BigQuery hard 500 cap — pre-flatten the closure with a materialised model.

When to switch to a graph database.

Depth > 100 with millions of nodes. Recursive CTE cost grows with edges × depth. At depth > 100, a native graph engine's index-free adjacency wins by an order of magnitude.
High-QPS shortest-path queries. Graph DBs (Neo4j, TigerGraph) are built for this; recursive CTEs enumerate all paths and pick min, which is O(paths) not O(edges).
Frequent structural change plus deep queries. Materialised closure tables become a maintenance burden; graph DBs handle live updates natively.
Path expressiveness matters. Cypher's MATCH (a)-[:KNOWS*1..3]-(b) is more readable than a recursive CTE for people who write graph queries daily.

Common interview probes on BOM.

"How do you sum the total quantity of each raw material for a bike?" — recursive CTE + GROUP BY component_id, SUM(accumulated_qty).
"What happens if a component appears in two sub-assemblies?" — the CTE emits two rows with the same component_id; the outer SUM adds them correctly.
"What's the difference between MAXRECURSION in SQL Server and cte_max_recursion_depth in MySQL?" — same concept; different names. SQL Server is per-query hint; MySQL is a session variable.
"When does a BOM query hit the recursion cap?" — very deep sub-assemblies (aerospace, complex electronics) or a buggy self-reference that never terminates.

Worked example — multi-level BOM rollup

Detailed explanation.

The setup. A bom(parent_id, child_id, qty) table. A bike has 1 frame and 2 wheels. A frame has 3 aluminium bars and 8 steel bolts. A wheel has 1 rubber tyre and 32 spokes.
The multiplication. Walking down, accumulated_qty is multiplied by each edge's qty. A spoke's accumulated qty is 1 (bike) × 2 (wheels per bike) × 32 (spokes per wheel) = 64.
The output. For each component, the total quantity needed to build one bike.

Question. Given the BOM below, compute the total quantity of every raw material required to build 1 bike.

Input — bom.

parent_id	child_id	qty
bike	frame	1
bike	wheel	2
frame	aluminium_bar	3
frame	steel_bolt	8
wheel	rubber_tyre	1
wheel	spoke	32

Code.

WITH RECURSIVE rollup AS (
    -- anchor: the top-level product
    SELECT 'bike'::text AS component_id,
           1::bigint    AS accumulated_qty,
           0            AS level
    UNION ALL

    -- recursive: children of the frontier, quantity multiplied along the edge
    SELECT b.child_id,
           r.accumulated_qty * b.qty,
           r.level + 1
    FROM rollup r
    JOIN bom b ON r.component_id = b.parent_id
)
SELECT component_id,
       SUM(accumulated_qty) AS total_qty,
       MAX(level)           AS max_depth
FROM rollup
WHERE component_id != 'bike'      -- exclude the top-level product from the tally
GROUP BY component_id
ORDER BY total_qty DESC;

Step-by-step explanation.

Anchor. Emits ('bike', 1, 0). Frontier {bike}.
Recursive step 1. From bike: children frame (qty 1) and wheel (qty 2). Emits ('frame', 1*1 = 1, 1) and ('wheel', 1*2 = 2, 1). Frontier {frame, wheel}.
Recursive step 2. From frame: children aluminium_bar (qty 3) and steel_bolt (qty 8). Emits ('aluminium_bar', 1*3 = 3, 2) and ('steel_bolt', 1*8 = 8, 2). From wheel: children rubber_tyre (qty 1) and spoke (qty 32). Emits ('rubber_tyre', 2*1 = 2, 2) and ('spoke', 2*32 = 64, 2).
Recursive step 3. No leaf component has a bom row as parent → recursive term emits zero rows → CTE terminates.
Outer query. Groups by component_id, sums accumulated_qty. Every leaf has exactly one row here (DAG has no shared component), so the sum equals the accumulated. Filters out bike (the top-level product itself).

Output.

component_id	total_qty	max_depth
spoke	64	2
steel_bolt	8	2
aluminium_bar	3	2
wheel	2	1
rubber_tyre	2	2
frame	1	1

Rule of thumb. BOM rollup multiplies along every edge and sums at the outer query. The multiplication happens inside the recursive term (never after); the summing happens outside (never inside). Get this split right and every BOM question follows.

Worked example — dialect max-recursion overrides

Detailed explanation.

Why the caps exist. Every dialect has an implicit safety cap to prevent runaway recursion from burning CPU forever. The cap is intentionally low so that a buggy CTE fails fast.
When to override. Deep BOMs (aerospace: 30+ levels), deep organisational structures (multinationals: 20+), deep dataset lineage graphs. The default cap is not enough for these workloads.
How to override per dialect. MySQL uses a session variable. SQL Server uses a per-query hint. Postgres has no explicit cap — you tune max_stack_depth at the server level. BigQuery has no per-query override — you must pre-flatten.

Question. Show the max-recursion override syntax in MySQL 8, SQL Server, and Postgres. What happens if the cap fires in each dialect?

Input.

(none — configuration syntax varies)

Code.

-- MySQL 8+ — session variable, applies to subsequent CTEs in this session
SET SESSION cte_max_recursion_depth = 5000;

WITH RECURSIVE deep AS (
    SELECT 1 AS n
    UNION ALL
    SELECT n + 1 FROM deep WHERE n < 2000
)
SELECT COUNT(*) FROM deep;

-- SQL Server — per-query hint, no session change needed
WITH deep AS (
    SELECT 1 AS n
    UNION ALL
    SELECT n + 1 FROM deep WHERE n < 2000
)
SELECT COUNT(*) FROM deep
OPTION (MAXRECURSION 5000);

-- Postgres — no explicit cap; tune server-level max_stack_depth if needed
-- (typically only needed for depth > 10K)
SET LOCAL max_stack_depth = '4MB';   -- session-scoped

WITH RECURSIVE deep AS (
    SELECT 1 AS n
    UNION ALL
    SELECT n + 1 FROM deep WHERE n < 2000
)
SELECT COUNT(*) FROM deep;

-- BigQuery — no per-query override; 500-iteration cap is hard
-- If depth > 500, pre-flatten with a materialised model

Step-by-step explanation.

MySQL. cte_max_recursion_depth is a session variable, default 1000. Setting it to 5000 lets deeper CTEs run. When the cap fires, MySQL errors with "Recursive query aborted after N iterations".
SQL Server. OPTION (MAXRECURSION N) is a per-query hint appended to the outer SELECT. Default 100. MAXRECURSION 0 means unbounded (dangerous — use only when you have engineered termination in the recursive term). When the cap fires, SQL Server errors with "Maximum recursion depth of N has been exhausted before statement completion".
Postgres. No dedicated cap. Recursion is bounded by max_stack_depth (server-level, typically 2 MB) — you rarely need to touch this for reasonable CTEs. When stack overflow occurs, Postgres crashes the query (not the server) with "stack depth limit exceeded". In practice, Postgres CTEs recurse into the tens of thousands without issue.
BigQuery. 500 iterations, hard-coded. No override. If your workload needs deeper recursion, materialise a closure table incrementally via scheduled query or dbt.
General pattern. Always specify termination in the recursive term (WHERE n < bound, cycle guard). Rely on the dialect cap as a safety net, never as an intended stopping mechanism.

Output — configuration matrix.

Dialect	Default cap	Override syntax	Error message when fired
Postgres	stack (~10-100K)	`SET max_stack_depth`	"stack depth limit exceeded"
MySQL 8+	1 000	`SET cte_max_recursion_depth = N`	"Recursive query aborted after N iterations"
SQL Server	100	`OPTION (MAXRECURSION N)`	"Maximum recursion depth of N has been exhausted"
Snowflake	unbounded	—	(watch credit cost)
BigQuery	500	none per-query	"Recursive query iteration limit reached"
DuckDB	unbounded	—	(watch memory)

Rule of thumb. For deep BOMs or lineage graphs, override the cap and engineer termination in the recursive term. Never bump the cap to hide an infinite loop — fix the loop instead.

Worked example — when to switch to a graph database

Detailed explanation.

The signal to switch. Deep + wide + high-QPS + frequent updates → the four ingredients that make recursive CTEs stop being the right answer.
Depth alone is not enough. A 30-level BOM queried once a day runs fine on a recursive CTE. Add 500 QPS and the CPU becomes the bottleneck; add continuous updates and the closure-table approach becomes lossy.
Where graph DBs win. Index-free adjacency (traversing an edge is O(1) instead of a B-tree lookup), path expressions ((a)-[:KNOWS*1..3]-(b) in Cypher), and native shortest-path algorithms.

Question. For a 50M-node dependency graph with continuous updates and 200 shortest-path queries per second, would you use recursive CTE, closure table, or graph DB? Justify.

Input.

Constraint	Value
Nodes	50 M
Edges	250 M
Update rate	1000 edge changes / sec
Query rate	200 shortest-path / sec
Depth range	1-15

Code.

Decision:
  - Recursive CTE at query time?
    ✗ 200 QPS × O(edges × depth) = CPU-bound; every query is 100-500 ms.

  - Closure table + nightly rebuild?
    ✗ 1000 updates / sec invalidates the closure faster than we can rebuild.
      Even an incremental maintainer would lag behind.

  - Graph DB (Neo4j / TigerGraph / Amazon Neptune)?
    ✓ Index-free adjacency: each hop is O(1).
    ✓ Native shortest-path algos (Dijkstra, A*) run in ms.
    ✓ Live updates handled natively; no closure-table catchup lag.
    ✓ Path expression is 1 line of Cypher, not 30 lines of SQL.

Verdict: graph DB. Migrate the graph out of the RDBMS; keep the OLTP tables that generate edges in Postgres, ETL edges into Neo4j via a Kafka topic.

Step-by-step explanation.

Recursive CTE at query time. Each query walks up to depth 15 with a fanout that could be hundreds of neighbours per hop. At 200 QPS, we need to serve 200 × O(edges × depth) queries per second. Even at 100 ms per query, that's 20 CPU-seconds per wallclock second — several dedicated CPUs, per node in the cluster.
Closure table + nightly rebuild. A closure table for 50M nodes with depth 15 could be tens of billions of rows. Rebuilding it once a day is expensive. And with 1000 updates/sec, the closure is stale seconds after the rebuild.
Closure table + incremental maintenance. An incremental closure maintainer runs on every edge change: for each new edge (u, v), find all rows (x, u) in the closure and add (x, v). Feasible for lower update rates; at 1000/sec, the maintainer becomes the bottleneck.
Graph DB. Native graph engines (Neo4j, TigerGraph, Amazon Neptune) store edges as pointers between node records — traversing an edge is one memory dereference, not a B-tree lookup. 200 QPS with depth-15 shortest-path is a normal load.
Migration path. Keep OLTP tables in Postgres. Emit every edge change to a Kafka topic. Consume in the graph DB. Serve reads from the graph DB. This is the standard pattern; it avoids rewriting the OLTP layer.

Output.

Approach	Feasibility
Recursive CTE at 200 QPS on 50M-node graph	✗ CPU-bound
Closure table + nightly rebuild	✗ stale within seconds
Closure table + incremental maintainer	✗ maintainer is the bottleneck
Graph DB with edge stream from Postgres	✓ scales; ms-latency reads

Rule of thumb. When any two of (depth > 20, nodes > 10M, QPS > 100, updates > 100/sec) hold at once, plan a graph DB migration. Recursive CTEs are the right hammer for shallow and low-QPS; deep and hot workloads need a different toolbox.

Senior interview question on end-to-end BOM roll-up

A senior interviewer might frame this as: "Design a BOM roll-up that supports multi-level assemblies, tolerates a 30-level max depth on a MySQL 8 instance, and reports total quantity plus maximum depth per raw component. Show the SQL, the dialect config, and the failure modes."

Solution Using a recursive CTE with dialect override and quantity multiplication

-- MySQL 8+ — session config for deep BOMs
SET SESSION cte_max_recursion_depth = 100;

WITH RECURSIVE rollup AS (
    SELECT :root_id AS component_id,
           1        AS accumulated_qty,
           0        AS level,
           CAST(:root_id AS CHAR(255)) AS path
    UNION ALL
    SELECT b.child_id,
           r.accumulated_qty * b.qty,
           r.level + 1,
           CONCAT(r.path, '>', b.child_id)
    FROM rollup r
    JOIN bom b ON r.component_id = b.parent_id
    WHERE r.level < 50                                        -- engineered safety cap
      AND FIND_IN_SET(b.child_id, REPLACE(r.path, '>', ',')) = 0   -- guard against accidental cycles
)
SELECT component_id,
       SUM(accumulated_qty) AS total_qty,
       MAX(level)           AS max_depth,
       COUNT(*)             AS occurrences
FROM rollup
WHERE component_id != :root_id
GROUP BY component_id
ORDER BY total_qty DESC;

Step-by-step trace.

Given a 3-level BOM (bike → frame → aluminium/steel; bike → wheel → rubber/spoke):

Iteration	Frontier	Emitted rows
0 (anchor)	—	`(bike, 1, 0, 'bike')`
1	`{bike}`	`(frame, 1, 1, 'bike>frame')`, `(wheel, 2, 1, 'bike>wheel')`
2	`{frame, wheel}`	`(aluminium, 3, 2, ...)`, `(steel, 8, 2, ...)`, `(rubber, 2, 2, ...)`, `(spoke, 64, 2, ...)`
3	leaves	zero rows → terminate

Outer query aggregates: spoke → 64, steel → 8, aluminium → 3, wheel → 2, rubber → 2, frame → 1.

The FIND_IN_SET guard uses the string path (MySQL) to detect accidental cycles; the level < 50 cap is a belt-and-braces safety net well under the session's cte_max_recursion_depth.

Output:

component_id	total_qty	max_depth	occurrences
spoke	64	2	1
steel_bolt	8	2	1
aluminium_bar	3	2	1
wheel	2	1	1
rubber_tyre	2	2	1
frame	1	1	1

Why this works — concept by concept:

Quantity multiplied inside the recursive term — accumulated_qty * b.qty runs on every recursive step. This is the entire point of BOM rollup; skipping it gives you a component list without amounts.
Session config for MySQL cap — SET SESSION cte_max_recursion_depth = 100 bumps the MySQL default from 1000 down (or up) to a value that matches the workload. Sensible caps prevent runaway CTEs.
Path column as cycle guard — even for BOMs (which are DAGs by design), the FIND_IN_SET guard catches accidental data-entry cycles that would otherwise cook the query. Cheap insurance.
Engineered level cap — WHERE r.level < 50 gives you a predictable max depth that is independent of the dialect's implicit cap. If the workload's real depth is 30, the level cap is a safety net; if it fires unexpectedly, you know something is wrong before the dialect's error message surfaces.
Cost — O(edges) per traversal since each edge is walked once. For a 100K-part BOM at depth 30, that's < 100ms on modern Postgres/MySQL/SQL Server. For BigQuery, the 500-iteration cap prevents very deep BOMs; use a materialised closure table instead.

SQL
Topic — CTE · hard
Hard recursive CTE problems

Practice →

SQL
Topic — SQL
SQL interview problem library

Practice →

Cheat sheet — recursive CTE recipes

Minimum viable recursive CTE. WITH RECURSIVE cte AS (anchor SELECT UNION ALL recursive SELECT joining cte) SELECT ... FROM cte;. Two SELECTs, one UNION ALL, one termination condition. That is the whole shape.
Running-numbers series generator. WITH RECURSIVE nums AS (SELECT 1 AS n UNION ALL SELECT n + 1 FROM nums WHERE n < :N) SELECT n FROM nums;. The "hello world" — sanity-check dialect support and syntax in 4 lines.
Ancestors of a row. Anchor WHERE id = :leaf; recursive term JOIN employees e ON cte.manager_id = e.id. Walks up until the root's manager_id IS NULL.
All descendants of a row. Anchor WHERE id = :root; recursive term JOIN employees e ON e.manager_id = cte.id. Add level + 1 and path || e.id for depth and ancestry.
BFS from a start node. Anchor emits (start, 0, [start]). Recursive term joins edges to the frontier, filters NOT other = ANY(path), appends to path, increments hop. ORDER BY hop in the outer query.
Cycle detection idiom. Carry a path array; filter WHERE NOT dst = ANY(path) on the recursive term. Explicit cycle reporting: emit is_cycle := dst = ANY(path) and filter in the outer query.
Multi-level BOM rollup with quantity multiplication. Recursive term multiplies parent.accumulated_qty * edge.qty. Outer query groups by component and sums accumulated quantities.
Postgres SEARCH clause. SEARCH DEPTH FIRST BY col SET seq or SEARCH BREADTH FIRST BY col SET seq. Adds a seq column that sorts DFS/BFS without hand-coding a path.
Postgres CYCLE clause. CYCLE col SET is_cycle USING path. Auto-detects cycles, stops expanding cycle-flagged rows, exposes is_cycle and path columns.
MySQL max-recursion override. SET SESSION cte_max_recursion_depth = 5000; before your CTE. Default is 1000. Session-scoped.
SQL Server max-recursion override. OPTION (MAXRECURSION 5000) appended to the outer SELECT. Default is 100. Per-query. MAXRECURSION 0 is unbounded — dangerous.
BigQuery hard cap. 500 iterations. No per-query override. Deep workloads must pre-flatten via materialised closure model.
postgres recursive cte deep tuning. Postgres has no per-query cap; the practical limit is max_stack_depth. SET LOCAL max_stack_depth = '4MB' handles depths into the tens of thousands.
When NOT to use a recursive CTE. Depth > 100 with high QPS → closure table or graph DB. Cyclic graph with shortest-path queries at scale → graph DB. Simple 2-level lookup → plain join, no recursion needed.
Closure-table pattern. Rebuild once daily from a recursive CTE; store (ancestor_id, descendant_id, depth) with a composite PK. Reads become a single-index lookup.

Frequently asked questions

What is a SQL recursive CTE?

A sql recursive cte is a Common Table Expression that references itself in its own definition, enabling iterative traversal of hierarchies, trees, graphs, and multi-level structures without application-side loops. The shape is fixed: WITH RECURSIVE cte AS (anchor SELECT UNION ALL recursive SELECT joining cte) SELECT ... FROM cte. The anchor runs once and seeds the CTE with base rows; the recursive term joins the CTE to a base table and runs repeatedly until it emits zero rows. Every dialect except SQL Server (which infers recursion) requires the RECURSIVE keyword. Common uses: sql hierarchy query for org charts, graph traversal sql for friendship networks, bill of materials sql for multi-level assemblies, running-number series for date-range generation, and materialised closure-table rebuilds.

Does MySQL support recursive CTEs?

Yes — MySQL added first-class recursive CTE support in MySQL 8.0 (2018). The syntax is WITH RECURSIVE cte AS (anchor UNION ALL recursive_term) SELECT ..., identical to Postgres. Two MySQL-specific things to know: the default max recursion is cte_max_recursion_depth = 1000 (bump it with SET SESSION cte_max_recursion_depth = 5000 if your BOM or lineage graph exceeds 1000 levels), and MySQL does not have first-class SEARCH or CYCLE clauses like Postgres — you roll your own path array and FIND_IN_SET-style cycle guard using string paths (or JSON arrays on 8.0.17+). Older MySQL 5.7 and earlier do not support recursive CTEs — you either upgrade or fall back to application-side loops.

When should I use `UNION ALL` vs `UNION` in a recursive CTE?

Almost always UNION ALL. It preserves every row emitted by the recursive term, which is what you want. UNION (implicit SELECT DISTINCT) silently dedupes across generations, adding sort/hash overhead and hiding intent. The one place UNION sometimes "helps" is a cyclic graph where deduping accidentally terminates the recursion — but that is a happy accident, not an engineered termination. The correct fix for a cyclic graph is a cycle guard (path array + NOT node = ANY(path)), not UNION. Default to UNION ALL; reach for UNION only after measuring a specific dedup win and articulating exactly which duplicates you are dropping.

How do I detect cycles in a recursive CTE?

Accumulate a path array on every recursive step, then filter WHERE NOT next_node = ANY(path) before emitting. On Postgres, use array operators natively (ARRAY[node_id], path || node_id, = ANY). On MySQL, use CONCAT for a string path and FIND_IN_SET for the guard. On SQL Server, use CONCAT and a LIKE or a JSON array. Postgres 14+ ships a declarative CYCLE col SET is_cycle USING path clause that does all the bookkeeping for you and stops the recursion the moment a cycle is detected. Always add a cycle guard for anything with multi-parent or bidirectional edges, even if today's data is cycle-free — tomorrow's bug will insert one.

How do I compute the depth of every node in an org chart?

Anchor the recursive CTE on the root (WHERE manager_id IS NULL) with level = 0. In the recursive term, join employees e ON e.manager_id = cte.id and select cte.level + 1 as the new level. Termination is automatic — leaves have no children, so the frontier drains. To get depth-first traversal order, carry a path array (cte.path || e.id) and ORDER BY path in the outer query. Depth-first is what you want when rendering a tree in the UI; breadth-first (ORDER BY level, name) is what you want when reporting "all employees at level N." This is the canonical sql hierarchy query and it works identically across Postgres, MySQL 8+, SQL Server (drop the RECURSIVE keyword), Snowflake, BigQuery, and DuckDB.

What is the difference between a recursive CTE and a graph database query?

A recursive CTE runs inside your OLTP database against a normal table with a self-referential FK — no new system to operate, no ETL to maintain, and depth-≤-10 workloads are typically sub-second. A graph database (Neo4j, TigerGraph, Amazon Neptune) stores edges as native pointers between node records — traversing an edge is O(1) memory dereference rather than a B-tree lookup, path expressions like Cypher's (a)-[:KNOWS*1..3]-(b) are one line instead of thirty, and native shortest-path algorithms (Dijkstra, A*) run in milliseconds even on multi-million-node graphs. Reach for a recursive CTE for shallow depth (≤ 10) and moderate QPS (≤ 100), reach for a graph DB when two or more of "deep + wide + high-QPS + high-update-rate" hold at once. The postgres recursive cte covers 80% of production hierarchy and BOM workloads; the remaining 20% is where graph DBs earn their operational cost.

Practice on PipeCode

Drill the CTE practice library → for the recursive-CTE shape, anchor + recursive-term probes, and dialect-specific overrides.
Rehearse on hard-difficulty CTE problems → when the interviewer wants deep BOM rollups and multi-level ancestry queries.
Sharpen tree traversal problems → for the org-chart / category-tree / filesystem family.
Stack the graph problems library → for BFS, cycle detection, and shortest-path variants.
Layer the recursion library → for anchor + recursive-step reasoning that shows up outside CTEs too.
For general SQL sharpening, work through the SQL problem library →.
Complement the CTE grind with the CTEs topic collection → for shorter warm-up drills.
For the broader SQL interview surface, take the SQL for Data Engineering course →.

Pipecode.ai is Leetcode for Data Engineering — every recursive CTE recipe above ships with hands-on practice rooms where you write the anchor + recursive term, wire the cycle guard, and roll up a multi-level BOM against real graded inputs. PipeCode pairs every reading with 450+ DE-focused problems and a real-time scoring engine, so you never have to wonder whether your `sql recursive cte` answer holds up under a senior interviewer's depth probes.

Practice recursive CTEs now →
Graph traversal drills →

LLM Evaluation Pipelines: Golden Sets, Cosine Similarity, LLM-as-Judge for Data Teams

Gowtham Potureddi — Fri, 10 Jul 2026 16:37:24 +0000

llm evaluation is the discipline that separates a demoable prototype from an LLM feature you can put in front of paying customers — and, in 2026, it is squarely a data-engineering problem, not a research one. Every product team ships at least one language-model feature, every prompt change silently reshapes the answer distribution, and every retrieval index update touches the same measured quality surface. Without a versioned golden set, a numeric metric, and a llm eval pipeline that runs on every pull request, "did we make it worse?" becomes a Slack argument instead of a graph. The data-engineering signal an interviewer looks for is exactly the one that keeps a product shippable: rigorous inputs, deterministic scoring, cost-aware judge design, and CI hooks that block a regression before it ever reaches production.

This guide is the senior-DE walkthrough you wished existed the first time an interviewer asked "walk me through how you'd design a llm as judge scorer without letting the judge outvote your labels" or "when do you trust cosine similarity and when do you escalate to a rubric-scoring judge?" or "how do you keep a rag evaluation harness green through a text-embedding upgrade?" It walks through why LLM evaluation is a data-eng problem (golden-set discipline, drift monitoring, metric hygiene), how to curate and version a golden set the way you version code, the mechanics of cosine similarity scoring with text-embedding-3 and threshold calibration, the LLM-as-judge rubric with structured JSON output and cost control, and the CI + production monitoring loop that catches regressions before they ship. Each section pairs a teaching block with a Solution-Tail interview answer — code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works. ragas, deepeval, and promptfoo show up where they earn their place; the rest is the plumbing you'll build yourself.

When you want hands-on reps immediately after reading, drill the ETL practice library →, rehearse on the data-analysis practice library →, and sharpen the tuning axis with the optimization practice library →.

On this page

Why LLM evaluation is a data-eng problem
Golden sets — curation and versioning
Cosine similarity + embedding-based scoring
LLM-as-judge — GPT-4-class rubric scorer
CI integration + production monitoring
Cheat sheet — LLM eval recipes
Frequently asked questions
Practice on PipeCode

1. Why LLM evaluation is a data-eng problem

Evaluation, not modeling, is what keeps LLM features shippable — and it is pipelines, not papers

The one-sentence invariant: an LLM feature without a versioned golden set and a nightly llm eval pipeline is a research prototype pretending to be a product, and every prompt change is a silent regression risk until the evaluation harness proves otherwise. In 2026, most teams do not train models — they compose a base model with a prompt, a retrieval index, and a set of tools. The engineering surface that decides whether the product stays good is not "which model did you fine-tune," it is "which inputs did you preserve, which metric did you compute, and did the CI harness block yesterday's bad PR." That is a data-engineering problem end to end: curated inputs, deterministic scoring, versioned artefacts, and observable outputs.

The four "must-answer" axes interviewers actually probe.

Golden set. A curated, versioned set of (input, expected_output, metadata) tuples that the harness scores against on every change. The senior signal is how you curate it: SME-labelled, class-balanced, git-hashed, growing every week. "We use the same 20 examples we tested with in the demo" is a fail.
Metric. Which numeric score are you computing per row and how do you aggregate it? Exact match is too strict; free-form BLEU is meaningless; cosine similarity on embeddings is a strong baseline; llm as judge rubric scores are the modern default. The senior signal is knowing which metric for which task and how they fail.
LLM-as-judge. A GPT-4-class model scoring the prediction against the expected answer using a rubric, returning a structured score plus a rationale. Cheap enough to run per PR, expensive enough that you cache and sample intelligently. The senior signal is understanding judge bias, prompt sensitivity, and how you'd guard against them.
CI integration. The harness runs on every pull request, blocks the merge if the aggregate score drops by more than a configured threshold (typically 5 percent), and posts the diff into the PR comment. In production, you sample live traffic through the same harness and alert on drift. The senior signal is treating quality as a build gate, not a lagging indicator.

Why this is a data-engineering problem, not a modeling one.

The scarce resource is data, not compute. The bottleneck to shipping a better LLM feature is not a bigger model — it is a bigger, cleaner, better-labelled golden set plus a metric you trust. Both are data problems.
The pipeline shape is DAG-native. Load golden set → run predictions → embed → compare to expected → score → aggregate → alert. That is a DAG. It runs on the same Airflow / Dagster / Prefect scheduler you already have. The primitives (idempotent tasks, retry-on-failure, backfill by run_id) are exactly the data-engineering primitives you use for every other pipeline.
The failure modes are drift, freshness, and schema. Embedding-model version changes silently reshape cosine distributions; retrieval index refreshes drop coverage; prompt edits change the answer surface. These are drift and schema-evolution problems — the same problems you handle in every production data platform.
The observability story is metric surfaces. Per-row scores go into a warehouse table; nightly aggregates roll into a dashboard; alerts fire on regression. This is the "metric with a runbook" pattern you already run for every SLO. LLM-as-judge is a new signal source; the observability shape is the same.

Why "eval on demo examples" is the default failure mode.

The demo set is 20 rows. It is what the founder used to prove the product to the first ten customers. It has no class balance, no edge cases, no adversarial inputs. Scoring against it produces a green graph that promises nothing about production behaviour.
The demo set is not versioned. Rows get added ad hoc, expected answers get edited when a stakeholder pushes back, provenance is lost. The metric on Tuesday is not comparable to the metric on Friday. You cannot regress against a moving target.
The demo set has no metadata. No difficulty tag, no topic label, no "who wrote this expected answer." When the aggregate score drops, you cannot slice by anything and cannot find the regression's centre of mass.
The demo set is not calibrated. If cosine similarity of 0.87 counts as "correct," the team invented that threshold once, in a Colab notebook, three months ago, and never revisited. The threshold no longer matches the underlying embedding distribution — the metric drifts silently.

What senior interviewers listen for.

Do you frame LLM evaluation as "versioned inputs plus a deterministic metric plus a CI hook" rather than "we manually spot-check outputs"? — senior signal.
Do you name cosine similarity, LLM-as-judge, and exact/regex as the metric family and know which for which task? — required answer.
Do you push back on "eval on demo examples" with the class-balance and version-control argument? — required answer.
Do you treat evaluation cost (judge dollars per row × golden-set size × runs per week) as a first-class engineering constraint? — senior signal.

Worked example — the "demo set" failure mode quantified

Detailed explanation. The archetypal LLM-eval failure: a team ships a RAG chatbot backed by a 20-row demo set. Six weeks later they roll out a prompt tweak, the demo score stays green at 0.94 cosine, and a customer support flood reveals the tweak has cratered the answer quality on a whole family of questions the demo set never covered. Walk an interviewer through why the 20-row set could not catch this and what the correct golden-set size looks like.

The symptom. Support tickets spike from 5/day to 45/day within 24 hours of a prompt deploy. Metric dashboards are green.
The naive assumption. "The demo passes, therefore the prompt is fine."
The actual failure. The demo covers "what are your business hours" style questions; the prompt regression touched "how do I cancel a subscription" style questions. Zero overlap.
The fix. A golden set of at least 200–500 rows, class-balanced across question topics, with per-row topic metadata so slicing by class catches localized regressions.

Question. A team has a 20-row demo set with cosine 0.94 average on the current prompt. They plan to roll out a prompt change. Quantify the statistical power of scoring the change against 20 rows vs 200 rows, and derive the minimum golden-set size for a 5 percent regression to be detectable.

Input.

Parameter	Demo set	Golden set	Notes
Rows	20	200	10× fanout
Cosine standard deviation per row	0.08	0.08	Empirically measured
Standard error of mean	0.018	0.006	σ / √N
Detectable regression (2σ)	0.036	0.011	95% CI half-width
5% regression threshold	0.047	0.047	Business-defined
Detectable at 5% threshold?	barely	comfortably	2× vs 4× margin

Code.

import numpy as np

def detectable_regression(sigma_per_row: float, n_rows: int, confidence: float = 0.95) -> float:
    """
    Return the smallest regression (in the same units as sigma_per_row) that a
    golden set of size n_rows can reliably detect at the given confidence level.

    Uses a two-sided normal approximation: half-width = z * sigma / sqrt(n).
    """
    z = 1.96 if confidence == 0.95 else 2.58   # 95% or 99%
    sem = sigma_per_row / np.sqrt(n_rows)
    return z * sem


# The team's numbers
sigma = 0.08                # per-row cosine std dev
threshold = 0.05            # 5% business-defined regression threshold

for n in (20, 100, 200, 500, 1000):
    delta = detectable_regression(sigma, n)
    verdict = "yes" if delta < threshold else "no"
    print(f"n={n:>5d}  detectable={delta:.4f}  <threshold? {verdict}")

# Output:
# n=   20  detectable=0.0350  <threshold? yes  (barely, uses full CI budget)
# n=  100  detectable=0.0157  <threshold? yes
# n=  200  detectable=0.0111  <threshold? yes
# n=  500  detectable=0.0070  <threshold? yes
# n= 1000  detectable=0.0050  <threshold? yes

Step-by-step explanation.

The statistical setup: score deltas are approximately normal by CLT; the two-sided 95 percent confidence half-width on the mean is 1.96 × σ / √N. That half-width is the smallest regression the harness can distinguish from noise.
At N=20, the half-width is 0.035 — meaning any regression under 3.5 percent is indistinguishable from run-to-run variance. A 5 percent regression is barely detectable and only against a single quiet baseline; a 3 percent regression sails through unnoticed.
At N=200, the half-width drops to 0.011 — the harness can distinguish a 1.1 percent regression from noise. A 5 percent regression is caught with a 4× margin, and per-class slices (topic buckets) each have enough rows to be statistically meaningful.
The bigger reason 200 rows beats 20 is coverage. Statistical power is the floor; class coverage is the actual protection. A 200-row set with 10 topic buckets × 20 rows each catches localized regressions the 20-row demo never sees. A 20-row set has no coverage at all.
The team's failure was both: statistical power too low to distinguish a real change from noise, and coverage too narrow to see the class where the regression actually landed. Both problems dissolve at 200+ rows with balanced classes.

Output.

Golden-set size	Half-width (95% CI)	Detects 5% regression?	Detects 1% regression?	Practical use
20	0.035	barely	no	Demo only
100	0.016	yes	no	Development
200	0.011	yes	barely	Production baseline
500	0.007	yes	yes	Multi-slice production
1000	0.005	yes	yes	Regression-forensics grade

Rule of thumb. Golden-set size ≥ 200 rows, class-balanced across the top 10 question topics, is the minimum threshold for shipping an LLM feature to real users. Below 200 rows, the harness is decoration — the metric graph is green because the noise is louder than the signal.

Worked example — the four axes on a single interview slide

Detailed explanation. A senior interviewer often asks "walk me through your LLM evaluation architecture in one minute." The four-axis framing is the tightest possible response: golden set, metric, judge, CI. Practise deriving them from a concrete task — "customer support RAG chatbot" — so the answer is a mini design doc, not a list of buzzwords.

The four axes as questions. Which inputs? Which metric? Who judges? When does it run?
The concreteness test. Every axis should have a numeric answer for the task at hand.
The senior signal. Naming the metric family (cosine + judge, or judge + regex, or embedding + regex), not just "we evaluate."

Question. Fill in the four-axis architecture for a customer-support RAG chatbot: 500 daily support conversations, historical ticket-plus-resolution corpus, GPT-4o-mini as production model, GPT-4o as available judge, PR CI budget of 5 minutes total. Show the numbers.

Input.

Parameter	Value
Production model	GPT-4o-mini
Judge model (available)	GPT-4o
Daily live conversations	500
Historical ticket-plus-resolution corpus	~40,000 rows
PR CI budget	5 minutes
Judge cost per row	~$0.006 (GPT-4o at ~1500 tokens)
Judge budget per run	$2.00

Code.

# eval-config.yaml — the four axes for the support chatbot
golden_set:
  path: golden/support_v3.jsonl
  version: v3.2-sha-e1f2c9
  rows: 300
  class_balance:
    billing: 60
    account: 60
    product: 60
    shipping: 60
    other: 60
  curation: SME-labelled + LLM-augmented, weekly refresh
  growth_rate: +20 rows/week

metric:
  primary: llm_judge_correctness   # 0.0 to 1.0
  secondary: cosine_similarity      # sanity check, threshold 0.85
  aggregation: mean_over_rows

judge:
  model: gpt-4o
  prompt_hash: c93f11
  rubric:
    - correctness
    - completeness
    - style
  output_schema: {score: float, rationale: string, breakdown: {correctness: float, completeness: float, style: float}}
  cost_per_row_usd: 0.006
  runs_per_pr: 1
  budget_per_run_usd: 2.00

ci:
  trigger: on pull_request
  gate: aggregate_score >= previous_baseline - 0.05
  timeout_minutes: 5
  parallelism: 20
  slack_channel: '#llm-quality'

Step-by-step explanation.

The golden set is 300 rows split evenly across 5 topic classes — the class-balance chip means a topic regression cannot be masked by aggregate improvement elsewhere. Version stamp v3.2-sha-e1f2c9 is the git hash of the golden-set repo commit; every metric row is joined to it in the warehouse.
The metric mix is llm_judge_correctness (primary) plus cosine_similarity (secondary sanity check). The judge is expensive but authoritative; cosine is cheap and catches "the model hallucinated a URL that has zero embedding overlap." Together they cover both semantic and structural failure.
The judge is GPT-4o with a rubric of correctness / completeness / style. Prompt hash c93f11 versions the judge prompt itself — a prompt edit bumps the hash and the harness knows to re-run baseline scores rather than compare across judge-prompt versions.
CI cost math: 300 rows × $0.006 per row = $1.80 per PR run, comfortably under the $2.00 budget. With 20-way parallelism and per-row latency around 2 seconds, wall-clock time is (300 / 20) × 2s = 30s — well within the 5-minute PR budget.
The gate blocks the merge if the aggregate correctness drops by more than 5 percent below the previous baseline. "Previous baseline" is the last main-branch merge, not an all-time high — otherwise a one-time high score locks the team out of every future PR.

Output.

Axis	Numeric answer
Golden set	300 rows, v3.2-sha-e1f2c9, 5 classes × 60 rows
Metric	LLM-judge correctness (primary) + cosine 0.85 threshold (secondary)
Judge	GPT-4o with 3-criterion rubric, $0.006/row, $1.80/run
CI	PR gate, block on ≥ 5% aggregate drop vs previous baseline, 5-min budget

Rule of thumb. If any of the four axes has a qualitative answer ("we spot-check," "we use vibes," "we run it weekly"), the architecture is not ready to ship. Every axis should have a number an interviewer can quote back at you.

Worked example — evaluation cost as a first-class constraint

Detailed explanation. A common mistake in the first evaluation build-out: no one runs the numbers on what the judge actually costs at scale. The team ships a 2000-row golden set with an LLM-as-judge that costs $0.03 per row, and the CI bill lands at $60 per PR. At 40 PRs per week, that is $10,000 per year on eval alone — and the team has zero budget for the production monitoring layer that is the actual point. Walk through the cost model and the levers that keep it sane.

Cost per PR. golden_set_size × judge_cost_per_row.
Cost per year. cost_per_PR × PRs_per_week × 52.
The three levers. Reduce set size, reduce judge cost, reduce runs per PR.

Question. Given a 2000-row golden set, $0.03 per-row judge cost, and 40 PRs per week, quantify the annual eval bill and propose three optimizations that cut it by at least 5×.

Input.

Parameter	Baseline	After optimization
Golden set size	2000	400 (core) + 1600 (sample)
Judge cost per row	$0.03	$0.006 (smaller judge on 90% of rows)
Runs per PR	1 full	1 sample (400 rows) + 1 full nightly
PRs per week	40	40
Cost per full run	$60	$12 (nightly)
Cost per PR run	$60	$2.40 (sample only)
Cost per week	$2,400	$96 (PRs) + $84 (nightly) = $180
Cost per year	$124,800	$9,360

Code.

from dataclasses import dataclass

@dataclass
class EvalCostModel:
    golden_set_size: int
    judge_cost_per_row_usd: float
    prs_per_week: int
    runs_per_pr: int = 1
    nightly_full_runs: int = 0
    nightly_row_multiplier: float = 1.0   # 1.0 = full set; 0.2 = 20% sample
    weeks_per_year: int = 52

    def cost_per_pr_run(self) -> float:
        rows = int(self.golden_set_size * self.nightly_row_multiplier)
        return rows * self.judge_cost_per_row_usd

    def annual_cost_usd(self) -> float:
        pr_cost = self.cost_per_pr_run() * self.runs_per_pr * self.prs_per_week
        nightly_cost = self.golden_set_size * self.judge_cost_per_row_usd * self.nightly_full_runs * 7
        return (pr_cost + nightly_cost) * self.weeks_per_year


baseline = EvalCostModel(
    golden_set_size=2000,
    judge_cost_per_row_usd=0.03,
    prs_per_week=40,
)
print(f"Baseline: ${baseline.annual_cost_usd():,.0f}/yr")   # $124,800

# Optimization 1 — sample 20% on PR, full run nightly
opt1 = EvalCostModel(
    golden_set_size=2000,
    judge_cost_per_row_usd=0.03,
    prs_per_week=40,
    nightly_row_multiplier=0.2,
    nightly_full_runs=1,
)
print(f"Opt 1 (sample + nightly): ${opt1.annual_cost_usd():,.0f}/yr")

# Optimization 2 — cheaper judge (GPT-4o-mini instead of GPT-4)
opt2 = EvalCostModel(
    golden_set_size=2000,
    judge_cost_per_row_usd=0.006,
    prs_per_week=40,
    nightly_row_multiplier=0.2,
    nightly_full_runs=1,
)
print(f"Opt 2 (+cheap judge): ${opt2.annual_cost_usd():,.0f}/yr")

Step-by-step explanation.

The baseline math is brutal: 2000 rows × $0.03 × 40 PRs × 52 weeks = $124,800 per year, all on evaluation before a single dollar of production monitoring. The finance team will ask why.
Optimization 1 splits the harness into a fast PR run (20 percent sample = 400 rows, still statistically powerful) and a thorough nightly run over the full 2000. PR cost drops 5×; the nightly cost is 7 runs per week at full size, but only one team pays for it, not every PR author.
Optimization 2 stacks a cheaper judge on top. GPT-4o-mini is roughly 5× cheaper than GPT-4o for judge tasks and, for well-designed rubric prompts, correlates with GPT-4o judgements at r > 0.9 (measured with a 200-row A/B). Use the cheap judge on the PR sample; keep the expensive judge on the nightly full run for calibration.
The combined effect: $124,800 → $9,360, a 13× reduction, without cutting the golden set. Coverage is preserved via the nightly full run; PR sensitivity is preserved via the sample-based gate; the judge-cost sensitivity is spent where it matters (calibration), not where it does not (every PR).
The pattern that generalizes: sample for speed, calibrate on the full set, use a cheaper judge for the sample. It is the exact same pattern as "sample for AB tests, batch for correctness reports" — an old data-engineering pattern, applied to eval.

Output.

Configuration	Cost per PR	Cost per year	Coverage
Baseline (full set, expensive judge)	$60	$124,800	Full every PR
Opt 1 (sample 20% + nightly full)	$12	$16,320	Full nightly, sample per PR
Opt 2 (opt 1 + cheap judge)	$2.40	$9,360	Full nightly (expensive), sample per PR (cheap)
Opt 3 (opt 2 + reduce PRs to weekly nightly only)	n/a	$2,184	Nightly only, no PR gate

Rule of thumb. The judge bill is a data-engineering budget item, not a research afterthought. Model it before you build the harness; sample-on-PR + full-nightly + cheap-judge-with-expensive-calibration is the default architecture that scales.

Senior interview question on the LLM evaluation architecture

A senior interviewer often opens with: "Design an end-to-end LLM evaluation system for a mid-size product team shipping a RAG chatbot. Cover golden set, metric, judge, CI, production monitoring, and cost. Walk me through the concrete numbers and the trade-offs you'd make."

Solution Using a four-axis pipeline with golden set + cosine + judge + PR gate

# eval-arch.yaml — the reference architecture
version: v1.0

golden_set:
  storage: git-versioned JSONL
  path: golden/support_v3.jsonl
  rows: 300
  balance: 5 classes × 60 rows
  growth: +20 rows/week from production sampling
  provenance: SME-labelled + LLM-augmented
  version_stamp: v3.2-sha-e1f2c9

metrics:
  primary:
    name: llm_judge_correctness
    scale: 0.0-1.0
    judge_model: gpt-4o-mini      # sample judge
    calibration_model: gpt-4o     # nightly full-set judge
  secondary:
    name: cosine_similarity
    embedding_model: text-embedding-3-large
    threshold: 0.85               # calibrated per class
    role: sanity check + hallucination detector

harness:
  execution: DAG (Dagster / Airflow / Prefect)
  parallelism: 20
  timeout: 5 minutes
  storage: warehouse table eval_runs (run_id, golden_set_hash, prompt_hash, model, row_id, score, rationale)

ci:
  trigger: on pull_request touching prompts/, retrieval/, or model_config/
  sample: 400 rows (20% of full set)
  gate: mean_correctness >= previous_baseline - 0.05
  comment: post per-class delta table to PR
  budget: $2.40 per PR

nightly:
  trigger: cron 02:00 UTC daily
  scope: full 2000-row set with expensive judge (gpt-4o)
  gate: none (reports only, feeds baseline)
  budget: $12 per run

production_monitor:
  sampler: 1% of live conversations
  frequency: hourly aggregation
  metric: llm_judge_correctness (cheap judge)
  alert: 24h rolling mean drops > 5% vs last-week baseline
  cost: ~$50/week

Step-by-step trace.

Layer	Input	Output	Cost
Golden set	git commit v3.2-sha-e1f2c9	300 rows loaded	$0
Model inference	300 predictions	prediction column	~$0.30
Cosine scoring	prediction + expected	cosine column	$0.03 (embeddings)
Judge scoring (sample)	400 rows	judge score + rationale	$2.40
Aggregation	per-class + overall means	dashboard row + PR comment	$0
CI gate	mean vs baseline	pass or block	$0
Nightly	full 2000	reports	$12
Production sampler	1% traffic	continuous score	~$50/week

After the rollout, every PR touching prompts or retrieval runs a 400-row sample eval in ~30 seconds, posts a per-class delta table into the PR, and blocks the merge if aggregate correctness drops by more than 5 percent. The nightly full-set run recalibrates the baseline and catches slow drift the PR sample cannot see. Production monitoring samples 1 percent of live traffic through the same judge and alerts on 24-hour drops.

Output:

Metric	Value
PR eval wall-clock	~30 seconds (parallelism = 20)
PR eval cost	$2.40
Nightly eval wall-clock	3 minutes
Nightly eval cost	$12
Production monitor cost	~$50/week
Annual total	~$9,400
Detectable regression at PR gate	~1% (400-row sample)
Detectable regression nightly	~0.5% (2000-row full)

Why this works — concept by concept:

Golden set as the versioned input — every score is joined to golden_set_hash so the metric is comparable across runs. Editing the golden set bumps the hash and the harness knows to re-baseline rather than compare across incompatible inputs. This is the same "immutable-input, versioned-artifact" pattern that dbt uses for models.
Cosine as the cheap sanity check — cosine similarity on text-embedding-3-large costs about $0.0001 per row and catches structural failures (hallucinated URLs, wrong entities, missing named entities) at essentially zero cost. It is the cheap first line of defence before the expensive judge fires.
Judge with sample-on-PR + full-nightly — the sample keeps PR feedback fast and cheap; the nightly keeps the baseline anchored on the full set. The pattern is exactly the CI-vs-CD split from every mature data-engineering platform: fast on the write path, thorough on the batch path.
CI gate at 5% below previous baseline — the gate is relative to the previous baseline, not an all-time high, so the team can ratchet quality upward without accidentally trapping themselves under an unreproducible peak. The 5 percent tolerance leaves room for judge noise while still catching real regressions.
Cost — total annual eval cost lands at ~$9,400 for a mid-size product, dominated by production monitoring rather than PR gating. The alternative (unmeasured regressions shipping to prod) costs a single incident — the eval stack pays for itself the first time it catches a bad prompt.

ETL
Topic — etl
ETL problems on LLM evaluation pipelines

Practice →

Optimization Topic — optimization Optimization problems on eval cost and coverage

Practice →

2. Golden sets — curation and versioning

A golden set is versioned code, not a spreadsheet — curation discipline is the single biggest lever on eval quality

The mental model in one line: a golden set is (input, expected_output, metadata) tuples treated with the same versioning, review, and observability rigour you apply to database migrations — SME-labelled, class-balanced, git-hashed, growing weekly, joined to every downstream metric row by its content hash. Everything else in the LLM eval stack is downstream of the golden set; a bad golden set produces green metrics that promise nothing. This is the single highest-leverage discipline in llm evaluation.

The four axes of a golden set worth trusting.

Tuples. Every row is (input, expected_output, metadata). input is the exact string that would go into your production model. expected_output is the SME-labelled canonical answer. metadata is the rich sidecar: topic, difficulty, source ticket id, expected style, forbidden patterns, retrieval-index snapshot at label time.
Versioned. The golden set lives in a git repo. Every edit is a PR, gets code review, gets a content hash. Downstream metric rows join to the hash — comparing scores across hashes is disallowed by construction.
Balanced. Rows are distributed across the classes your product actually serves: topic buckets, difficulty tiers, language / locale / persona. Class imbalance in the golden set means the aggregate score is dominated by whichever class has the most rows, and localized regressions in under-represented classes vanish into the mean.
Growing. Golden sets are not static. Every week, sample fresh production conversations, SME-label them, add to the set. A stagnant golden set drifts away from the live traffic distribution and stops being predictive within months.

The curation loop.

Source. Sample from live production traffic — the golden set must reflect the actual input distribution, not an idealized version of it. For a chatbot: sample real user turns. For extraction: sample real documents.
Label. Route to an SME queue. For customer support: a senior support engineer. For medical Q&A: a clinician. For code review: a staff engineer. The SME writes the expected answer; a peer reviews it; the reviewed row is merged.
Augment. LLM-augmented golden sets — where a strong model generates the expected answer and an SME reviews it — are the compromise for scale. Purely LLM-generated golden sets are self-referential garbage; SME-reviewed LLM-generated sets are 5–10× cheaper than SME-authored ones while retaining most of the quality.
Balance. After every batch, re-check the class distribution. If billing questions are now 40 percent of the set and shipping questions are 5 percent, rebalance before the next merge.
Hash + release. The final artefact is a JSONL file with a content hash. v3.2-sha-e1f2c9 means "golden set version 3.2, content hash e1f2c9." Every downstream metric row carries this hash.

Version control patterns that work.

Storage. JSONL in a git repo (fast to diff, streams row-by-row into the harness). Alternative: parquet with an accompanying MANIFEST.json. Never a Google Sheet.
Review. PRs against the golden-set repo. Every edit is reviewed by at least one SME. Bulk merges (>50 rows) get double review.
Hash. Content hash computed on canonicalized JSON (sorted keys, no trailing whitespace) — SHA-256 first 6 hex is enough. Store in a MANIFEST.json alongside the file; harness verifies at load time.
Rollback. Git tags per release (golden-v3.2). If a bad batch is merged, revert the merge commit and re-tag. Downstream metric rows keyed by hash never confuse pre- and post-revert.

Common curation failures and their fixes.

"We use production traffic samples directly." Fails on labeler bias — the model's own output becomes the expected answer. Fix: SME must write the expected answer independently, without seeing the model's response.
"We only add rows when the model gets something wrong." Fails on class balance — the golden set becomes an adversarial edge-case set with no representative easy rows, and the aggregate metric is meaningless. Fix: add rows from all outcome buckets (correct, partially-correct, wrong) at fixed ratios.
"We edit expected answers when the SME disagrees with the score." Fails on immutability — the metric is no longer comparable across time. Fix: expected-answer edits go through the same PR-and-hash-bump process as adding new rows.
"We don't rebalance." Fails on drift — organic growth is never balanced, and the metric quietly gets dominated by whichever class the SMEs happen to label most. Fix: automatic imbalance detection with a warning below 20% deviation from target ratios.

What senior interviewers listen for.

Do you talk about the golden set as a first-class artefact with version control and PR review, rather than as a file someone maintains? — senior signal.
Do you volunteer the class-balance chip without prompting? — required answer.
Do you distinguish SME-labelled from LLM-augmented and defend both use cases? — senior signal.
Do you name content-hash + downstream-join as the mechanism that keeps the metric comparable across time? — senior signal.

Worked example — schema for a 300-row customer-support RAG golden set

Detailed explanation. Design the JSONL schema for a customer-support RAG golden set. Every row must carry enough metadata to slice by topic, difficulty, language, expected style, and provenance. Show the actual JSON with three representative rows and walk through why every field earns its place.

Row shape. input, expected_output, metadata.
Metadata fields. topic, difficulty, language, source_ticket_id, sme_labeler, reviewer, retrieval_snapshot_id, forbidden_patterns, style_tag.
The senior test. Every field must have a downstream consumer (dashboard slice, regression forensic, harness gate).

Question. Produce the JSONL schema plus three example rows for a customer-support RAG golden set covering billing, account, and shipping topics, with an SME-labeler chain and a retrieval-snapshot pin.

Input.

Field	Type	Purpose
input	string	The user turn as it enters the RAG chatbot
expected_output	string	SME-authored canonical answer
metadata.topic	enum	Class bucket for balance + slice
metadata.difficulty	enum	Bucket for slice + power analysis
metadata.language	enum	Locale slice
metadata.source_ticket_id	string	Provenance to the original support ticket
metadata.sme_labeler	string	The SME who wrote the expected answer
metadata.reviewer	string	The peer who approved
metadata.retrieval_snapshot_id	string	Retrieval index at label time
metadata.forbidden_patterns	list	Regexes that must not appear in prediction
metadata.style_tag	enum	Expected tone (formal, empathetic, terse)

Code.

{"input": "How do I update my billing address on file?", "expected_output": "You can update your billing address by opening Settings → Billing → Address, entering the new address, and clicking Save. The change takes effect on your next invoice. If a payment is currently processing, contact support before saving.", "metadata": {"topic": "billing", "difficulty": "easy", "language": "en", "source_ticket_id": "T-8821", "sme_labeler": "alice@support", "reviewer": "bob@support", "retrieval_snapshot_id": "idx-2026-06-15", "forbidden_patterns": ["credit card", "bank account"], "style_tag": "formal"}}
{"input": "My charge from last month is wrong — I was billed twice.", "expected_output": "I'm sorry about the duplicate charge — that's frustrating. Please email billing@company.com with your account email and the two transaction IDs. Our team can reverse one of the charges within 3 business days. If you'd like, I can also open a ticket for you now.", "metadata": {"topic": "billing", "difficulty": "hard", "language": "en", "source_ticket_id": "T-8834", "sme_labeler": "alice@support", "reviewer": "carol@support", "retrieval_snapshot_id": "idx-2026-06-15", "forbidden_patterns": ["we'll refund", "guaranteed"], "style_tag": "empathetic"}}
{"input": "Where is my order? Tracking says shipped but I haven't received it.", "expected_output": "I understand — a shipped order that hasn't arrived is stressful. Please check the tracking link in your shipment email for the latest carrier update. If the tracking has been stagnant for more than 5 business days, reply here with your order number and I'll open a carrier claim on your behalf.", "metadata": {"topic": "shipping", "difficulty": "medium", "language": "en", "source_ticket_id": "T-8912", "sme_labeler": "dave@support", "reviewer": "alice@support", "retrieval_snapshot_id": "idx-2026-06-15", "forbidden_patterns": ["lost forever", "impossible"], "style_tag": "empathetic"}}

# harness/load_golden.py — canonicalized load + hash verification
import hashlib
import json
from pathlib import Path


def content_hash(rows: list[dict]) -> str:
    """SHA-256 first 6 hex over canonicalized JSON."""
    canonical = "\n".join(json.dumps(r, sort_keys=True, ensure_ascii=False) for r in rows)
    return hashlib.sha256(canonical.encode("utf-8")).hexdigest()[:6]


def load_golden(path: Path, expected_hash: str | None = None) -> list[dict]:
    rows = [json.loads(line) for line in path.read_text().splitlines() if line.strip()]
    actual_hash = content_hash(rows)
    if expected_hash and actual_hash != expected_hash:
        raise ValueError(f"golden set hash mismatch: expected {expected_hash}, got {actual_hash}")
    return rows


rows = load_golden(Path("golden/support_v3.jsonl"), expected_hash="e1f2c9")
print(f"loaded {len(rows)} rows, hash e1f2c9 verified")

Step-by-step explanation.

Every row carries input, expected_output, and a rich metadata block. The metadata is not optional — it is what turns a golden set into a diagnostic tool. Without topic and difficulty, regressions can only be described at the aggregate; with them, you can point to "billing / hard rows dropped 12 percent" in the PR comment.
forbidden_patterns is the row-level guardrail: regexes the prediction must not contain. Row 1's ["credit card", "bank account"] means the model should never invite the user to email their credit card number — an easily-detectable structural failure that pure cosine similarity might miss.
style_tag distinguishes expected tone. Row 2's "empathetic" means the judge rubric should score empathy explicitly; row 1's "formal" means the same phrase would score lower. The metric responds to the semantic and the pragmatic axis.
retrieval_snapshot_id pins the retrieval index state at label time — critical for RAG evaluation. When the retrieval index refreshes and the expected answer no longer matches what a correct retrieval would return, the row is flagged for re-labelling rather than silently failing.
The content_hash function computes SHA-256 over canonicalized JSON (sorted keys, one row per line). Downstream metric rows join to this hash; the harness verifies the hash on load. If the file was edited without a PR review, the hash mismatches and the run aborts.

Output.

Field	Purpose	Downstream consumer
input / expected_output	Core tuple	Model + judge inputs
topic	Class balance + slice	Dashboard slice
difficulty	Slice + power	Regression forensics
language	Locale slice	Multilingual dashboards
source_ticket_id	Provenance	Row-level audit
sme_labeler / reviewer	Trust chain	SME QA reports
retrieval_snapshot_id	RAG pinning	Index-refresh guard
forbidden_patterns	Row-level guardrail	Structural failure check
style_tag	Style discipline	Judge rubric input

Rule of thumb. Every metadata field must have a downstream consumer. If you cannot name what a field is used for in the dashboard, harness, or forensics report, drop it — noise metadata makes the schema harder to evolve without making the metric better.

Worked example — the class-balance rebalancer

Detailed explanation. Golden sets grow organically as SMEs add rows. Without discipline, the distribution drifts toward whatever the SMEs happen to work on that week. Build the class-balance rebalancer that runs on every merge to the golden-set repo, warns on drift, and refuses to merge if drift exceeds a threshold.

Target ratios. 5 topic classes at 20 percent each (0.2 target per class).
Warning threshold. ±5 percent deviation from target — flags the row but allows the merge.
Blocking threshold. ±10 percent deviation — CI check fails.
Fix. Rebalance by removing over-represented rows or by generating replacements in the under-represented classes.

Question. Build the class-balance check as a pre-merge CI script that reads the JSONL, computes per-class ratios, prints a warning or fails, and outputs the recommended rebalance action.

Input.

Class	Current rows	Target ratio	Current ratio	Delta
billing	78	0.20	0.26	+0.06
account	55	0.20	0.18	-0.02
product	60	0.20	0.20	0.00
shipping	45	0.20	0.15	-0.05
other	62	0.20	0.21	+0.01
total	300	1.00	1.00	—

Code.

# harness/check_balance.py
import json
import sys
from collections import Counter
from pathlib import Path


TARGET = {"billing": 0.20, "account": 0.20, "product": 0.20, "shipping": 0.20, "other": 0.20}
WARN = 0.05
BLOCK = 0.10


def check_balance(path: Path) -> int:
    rows = [json.loads(line) for line in path.read_text().splitlines() if line.strip()]
    n = len(rows)
    counts = Counter(r["metadata"]["topic"] for r in rows)

    exit_code = 0
    print(f"golden set has {n} rows")
    print(f"{'topic':<10} {'rows':>5} {'ratio':>7} {'target':>7} {'delta':>7}")
    for topic, target in TARGET.items():
        c = counts.get(topic, 0)
        ratio = c / n if n else 0.0
        delta = ratio - target
        marker = ""
        if abs(delta) >= BLOCK:
            marker = "  BLOCK"
            exit_code = 1
        elif abs(delta) >= WARN:
            marker = "  WARN"
        print(f"{topic:<10} {c:>5d} {ratio:>7.3f} {target:>7.3f} {delta:>+7.3f}{marker}")

    # Suggested action
    over = [t for t, target in TARGET.items() if counts.get(t, 0) / n - target > 0.05]
    under = [t for t, target in TARGET.items() if target - counts.get(t, 0) / n > 0.05]
    if over or under:
        print()
        print("Suggested rebalance:")
        for t in under:
            need = int((TARGET[t] - counts.get(t, 0) / n) * n)
            print(f"  add {need:+d} rows for {t}")
        for t in over:
            drop = int((counts.get(t, 0) / n - TARGET[t]) * n)
            print(f"  drop {drop:+d} rows for {t}")

    return exit_code


if __name__ == "__main__":
    sys.exit(check_balance(Path(sys.argv[1])))

# .github/workflows/golden-check.yml
name: golden-set-check
on:
  pull_request:
    paths: ['golden/**']
jobs:
  balance:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.12'
      - run: python harness/check_balance.py golden/support_v3.jsonl

Step-by-step explanation.

The check iterates over the 5 target classes and computes the current ratio versus the target. Deviations under ±5 percent are silent, ±5–10 percent print a WARN marker (informational), ±10 percent or worse print BLOCK and exit non-zero.
In the input scenario, billing is at 26 percent (+6 percent over target) — WARN. shipping is at 15 percent (-5 percent) — WARN. No class hits the BLOCK threshold, so the check exits 0 and the merge is allowed, but the PR author sees the warnings inline.
The suggested-rebalance block computes concrete row counts: shipping needs +15 more rows to hit target; billing needs to drop 18 rows. The suggestion drives the next batch of SME curation work.
The check runs on every PR that touches golden/** via GitHub Actions. It is a 30-second job on a 300-row set; even a 10,000-row set runs in under 5 seconds. The cost is negligible; the guardrail is absolute.
The pattern extends: add checks for difficulty distribution, language distribution, and sme_labeler diversity (no single SME should label more than 40 percent of any class — labeller bias). All of them are the same shape.

Output.

golden set has 300 rows
topic       rows   ratio  target   delta
billing       78   0.260   0.200  +0.060  WARN
account       55   0.183   0.200  -0.017
product       60   0.200   0.200  +0.000
shipping      45   0.150   0.200  -0.050  WARN
other         62   0.207   0.200  +0.007

Suggested rebalance:
  add +15 rows for shipping
  drop -18 rows for billing

Rule of thumb. Automate class-balance checks the same way you automate lint. The check is cheap, the failure mode is expensive; the moment a golden set is not automatically balanced, drift wins and the metric decays silently.

Worked example — versioning golden sets like code

Detailed explanation. Walk through the git workflow for a golden-set repo: PRs, review, hash, tag, harness pin. Show what happens when a batch of 20 new rows is added — from the SME's local edit through the merged tag to the downstream metric join.

Repo shape. golden/support_v3.jsonl, golden/MANIFEST.json (with hash + row count), golden/CHANGELOG.md.
PR flow. Branch → edit → local check → PR → review → merge → tag.
Hash discipline. Content hash goes in MANIFEST + git tag + downstream metric rows.

Question. Walk through the PR flow for adding 20 new billing-topic rows to a 280-row golden set, ending with the new hash and tag, and show how the harness pins to the tag.

Input.

Step	Actor	Artefact
1. Sample	SME	20 real support tickets from the last week
2. Label	SME	expected_output for each
3. Peer review	Reviewer	approval or edits
4. Local check	SME	check_balance.py + hash regen
5. PR	SME	branch feature/golden-billing-batch-42
6. CI	GitHub Actions	balance check + row-count diff
7. Merge	Maintainer	squash to main
8. Tag	Maintainer	golden-v3.3
9. Harness pin	DE	eval-config.yaml → version: golden-v3.3

Code.

# 1-2. Sample + label (SME's local flow)
git checkout -b feature/golden-billing-batch-42
python harness/sample_from_prod.py --topic billing --n 20 > tmp/samples.jsonl
# SME edits tmp/samples.jsonl to write expected_output + metadata for each row

# 3. Peer review — SME opens PR, reviewer proposes edits, SME applies

# 4. Local check
cat golden/support_v3.jsonl tmp/samples.jsonl > golden/support_v3.jsonl.new
mv golden/support_v3.jsonl.new golden/support_v3.jsonl
python harness/check_balance.py golden/support_v3.jsonl   # WARN or BLOCK check
python harness/hash_manifest.py golden/support_v3.jsonl   # writes MANIFEST.json
git diff golden/MANIFEST.json                              # confirms hash bump

// golden/MANIFEST.json (after the merge)
{
  "version": "v3.3",
  "sha": "b83a11",
  "rows": 300,
  "class_balance": {
    "billing": 78,
    "account": 55,
    "product": 60,
    "shipping": 45,
    "other": 62
  },
  "generated_at": "2026-07-04T09:12:33Z",
  "generator": "harness/hash_manifest.py"
}

# eval-config.yaml — harness pins to a specific tag
golden_set:
  repo: git@github.com:acme/golden-sets.git
  path: golden/support_v3.jsonl
  tag: golden-v3.3
  expected_sha: b83a11    # harness aborts if mismatch

-- Every metric row joins by (run_id, golden_set_sha)
SELECT
  m.run_id,
  m.golden_set_sha,
  m.row_id,
  m.prediction,
  m.judge_score,
  g.topic,
  g.difficulty
FROM   eval_metric_rows m
JOIN   golden_set_rows g
  ON   m.golden_set_sha = g.sha
  AND  m.row_id         = g.row_id
WHERE  m.run_id = :run_id;

Step-by-step explanation.

The SME branches off main, samples 20 real production tickets via a helper, and labels each with an expected answer + metadata. The samples are typed into JSONL in a local scratch file, then concatenated onto the existing golden set.
Peer review happens in the PR. The reviewer proposes edits to expected answers (typos, style, missing empathy), the SME applies them, and the PR is re-pushed. Only reviewer-approved rows land in main.
The local check runs check_balance.py — in this case billing moves from 58/280 (21 percent) to 78/300 (26 percent), triggering a WARN but not a BLOCK. The SME chooses to keep the imbalance for now with a plan to add shipping rows next batch.
hash_manifest.py computes the new SHA (b83a11) and writes MANIFEST.json. The diff on MANIFEST.json is what the reviewer signs off on — a mechanical audit trail of "row count went from 280 to 300, hash went from e1f2c9 to b83a11."
The harness pins to tag: golden-v3.3 + expected_sha: b83a11. On load, it verifies both — if the tag has been re-pointed or the file has been edited without a hash bump, the harness aborts and the run does not corrupt the metric table. Every metric row also carries the SHA so downstream analysis can never confuse pre- and post-v3.3 scores.

Output.

Artefact	Before	After
Rows	280	300
SHA	e1f2c9	b83a11
Tag	golden-v3.2	golden-v3.3
CHANGELOG.md entry	—	"v3.3: +20 billing rows from prod week 27"
Harness pin	v3.2	v3.3
Downstream metric join key	(run_id, e1f2c9)	(run_id, b83a11)

Rule of thumb. Golden sets deserve the same git rigour as production migrations: branch, PR, review, tag, hash-pin. Skipping any of these is how a Monday score becomes incomparable to a Friday score without anyone noticing until an interview.

Senior interview question on golden-set design

A senior interviewer might ask: "You inherit a team where the golden set is a shared Google Sheet with 40 rows that everyone edits. Walk me through the migration to a versioned, class-balanced golden set of 300 rows in 6 weeks, without stalling the team's evaluation cadence."

Solution Using a six-week golden-set migration plan with a hybrid harness

# migration-plan.yaml — 6 weeks from Google Sheet to versioned golden set
version: v1.0

week_1:
  goal: freeze current state + baseline metric
  actions:
    - snapshot the sheet as-is into git as golden/support_v0.jsonl
    - compute content_hash + tag as golden-v0-baseline
    - run the existing harness against v0-baseline for a 4-week baseline
  deliverable: golden-v0-baseline in git; baseline metric on dashboard

week_2:
  goal: schema migration
  actions:
    - migrate 40 rows to (input, expected_output, metadata) schema
    - add topic + difficulty + language metadata (SME retroactively fills)
    - stand up check_balance.py + GitHub Actions
  deliverable: golden-v1.0 in git (40 rows, schema-conformant)

week_3:
  goal: first curation batch — sample 60 fresh production rows
  actions:
    - build sample_from_prod.py (random sample from live conversations)
    - SME labels 60 rows across 5 topic classes (12 each)
    - peer review + merge
  deliverable: golden-v2.0 (100 rows, mostly balanced)

week_4:
  goal: second curation batch — 100 more rows
  actions:
    - sample another 100 rows, focusing on under-represented classes
    - LLM-augmented labels + SME review (cheaper labelling)
  deliverable: golden-v2.1 (200 rows, balance within ±5%)

week_5:
  goal: hybrid harness — score against both v0-baseline AND v2.1
  actions:
    - harness runs against both sets on every PR
    - dashboard shows both metrics side-by-side
    - team gets used to the new numbers before cutover
  deliverable: hybrid dashboard live

week_6:
  goal: cutover
  actions:
    - final +100 rows to reach 300
    - retire v0-baseline; harness gates on v3.0 only
    - CHANGELOG + team announcement + runbook
  deliverable: golden-v3.0 (300 rows) as sole gating set

Step-by-step trace.

Week	Activity	Rows	Metric availability
1	Freeze sheet, tag v0	40	Old-sheet baseline
2	Schema migration	40	v1.0 available
3	+60 rows	100	v2.0 available
4	+100 rows	200	v2.1 available
5	Hybrid harness	200	Both v0 and v2.1
6	Cutover to v3.0	300	v3.0 is the gate

After 6 weeks, the team has a 300-row, class-balanced, git-versioned golden set with automatic balance checks on every PR, a downstream metric table joined by SHA, and a runbook for the next migration when the schema changes again. The old sheet is archived; no one edits it.

Output:

Metric	Week 1	Week 6
Golden-set rows	40	300
Storage	Google Sheet	Git JSONL
Class balance	unmeasured	±5% per class
Version control	none	git tag + SHA
PR review	none	reviewer required
Downstream metric join	none	(run_id, SHA)
Detectable regression	~2.5%	~1.1%

Why this works — concept by concept:

Freeze first, migrate second — the week-1 baseline lets the team detect regressions during the migration itself; without it, a schema change might silently mask a real quality drop. This is the same "capture golden state, then refactor" pattern you use for legacy pipeline migrations.
Schema migration before curation — moving to the (input, expected_output, metadata) shape unblocks every future capability (slice by topic, class-balance check, forbidden-pattern guard). Trying to curate before the schema is stable produces rows you have to re-edit later.
Hybrid harness in week 5 — running both metrics side-by-side is the safety net for cutover week. The team gets to compare "old sheet said 0.87, new golden set says 0.82" and understand the delta before it becomes a gate. Big-bang cutovers are how you break trust in the metric.
LLM-augmented labelling in week 4 — SMEs write 100 rows in weeks 2–3, then LLM-augmented labelling with SME review handles the next 100. The 5–10× speedup on the second batch is what makes the 6-week timeline realistic on a mid-size team.
Cost — 6 weeks of one SME's part-time attention plus roughly 40 senior-engineer hours for schema, harness, and CI plumbing. The alternative (running on a 40-row sheet forever) has an unbounded downside; the migration cost is paid once and the golden set compounds in value every week thereafter.

ETL
Topic — etl
ETL problems on versioned dataset curation

Practice →

Data analysis Topic — data-analysis Data-analysis problems on class balance and slicing

Practice →

3. Cosine similarity + embedding-based scoring

Cosine similarity is the cheap, fast, structural sanity check — and a lousy standalone truth metric

The mental model in one line: cosine similarity on embeddings of prediction and expected_output is the fastest, cheapest, most-deterministic LLM-eval metric — perfect as a sanity check and a hallucination detector, terrible as a standalone truth metric because it cannot distinguish paraphrase from contradiction. Every llm eval pipeline should compute cosine as the first metric on every row, and every serious rag evaluation stack should also compute an LLM-judge score on top for the semantic axis cosine misses.

The four axes of cosine-based scoring.

Embedding model. The cosine score is a function of the embedding — text-embedding-3-large produces different distributions than text-embedding-3-small or bge-large. Pin the model, version-stamp the score column, and never change models mid-experiment.
Threshold. A cosine of 0.85 might be "correct" on one embedding model and "irrelevant" on another. The threshold must be calibrated per embedding model per task against the golden set, not carried across from a tutorial.
Failure modes. Cosine treats paraphrase and contradiction as similar. "Yes, you can cancel anytime" and "No, you cannot cancel" embed close to each other; cosine sails through, the judge (and the customer) does not.
Aggregation. Per-row cosine is a float in -1, 1. Aggregate as mean over rows for the summary; also compute per-class means for the dashboard slice.

How embeddings become a metric.

Step 1. Embed the expected_output string with the chosen embedding model; store as a float32[dim] vector (dim=3072 for text-embedding-3-large).
Step 2. Embed the model's prediction with the same embedding model. The vectors live in the same space; cosine is defined.
Step 3. Compute cosine similarity: dot(a, b) / (norm(a) × norm(b)). For normalized embeddings this is a dot product.
Step 4. Compare to the threshold. Above threshold = "correct" (cheap-check pass); below = "flagged" (escalate to judge for the semantic axis).

Threshold calibration — the numeric discipline.

Gather. 200 golden-set rows plus 200 known-wrong predictions (paraphrased edits, contradiction injections).
Score. Compute cosine for all 400 pairs.
Choose. Pick the threshold that maximizes F1 on the "correct vs wrong" classification. Typical values land at 0.75–0.90 for text-embedding-3-large on medium-length answers.
Version-stamp. The threshold is a function of the embedding model + task; store it in the eval config with the model version. Bumping the embedding model always requires re-calibration.

Failure modes cosine cannot catch alone.

Contradiction. "Yes" and "No" embed as nearby short strings. Cosine misses the polarity flip.
Paraphrase. "Update via Settings → Billing" and "Change in the Billing settings menu" are semantically identical but embed differently on some models — cosine can under-count paraphrase correctness.
Style mismatch. A perfectly correct answer in a formal tone will score high cosine against an empathetic expected answer even though the tone-brand is off.
Long-form dilution. A 500-token answer embeds "average"-ish; a small hallucinated URL inside it barely moves the vector. Cosine is blind to needle-in-haystack failure.

When cosine actually shines.

Structural hallucination detection. Predictions with wrong entities, wrong URLs, wrong dates score dramatically lower than paraphrased-but-correct alternatives. Cosine is a great first-pass anomaly detector.
Regression detection. A prompt change that drops mean cosine by 0.10 across 300 rows is always a regression worth investigating, even if individual rows look fine.
Speed. Embeddings cost fractions of a cent; cosine is a matrix multiply. Perfect for every-PR gating where an LLM-judge would be too slow or expensive.
Determinism. Cosine is deterministic for a fixed embedding model. Unlike judges, the score is reproducible bit-for-bit — great for baseline math.

Common cosine pitfalls in the wild.

Un-normalized embeddings. Forgetting to normalize the vectors turns cosine into "dot product times garbage." Always L2-normalize before comparing.
Model version drift. OpenAI silently ships new embedding checkpoints under the same public name; pin to a specific model version and log it with every score row.
Mixed languages. Cross-lingual cosine is much noisier than same-language. Slice the metric by language before drawing conclusions.
Threshold from a Colab notebook. The threshold everyone quotes came from someone's exploration script and was never re-calibrated. Re-run calibration on every embedding-model bump.

What senior interviewers listen for.

Do you name cosine as a sanity check, not a truth metric, and pair it with a judge? — required answer.
Do you volunteer paraphrase vs contradiction as the canonical failure modes? — senior signal.
Do you name threshold calibration as a per-embedding-model discipline rather than a constant? — senior signal.
Do you distinguish normalized vs unnormalized and defend the L2 normalization step? — required answer.

Worked example — cosine scoring a 300-row golden set with text-embedding-3

Detailed explanation. Build the cosine scorer against a 300-row golden set using OpenAI's text-embedding-3-large. Batch the embedding calls for cost, L2-normalize the vectors, compute per-row cosine, aggregate by topic class, and produce the summary table with the calibrated 0.85 threshold.

Embedding model. text-embedding-3-large (dim=3072).
Batch size. 100 rows per API call.
Threshold. 0.85 (previously calibrated).
Aggregation. Per-class mean + overall mean + fraction ≥ threshold.

Question. Implement the cosine scorer end-to-end. Show the batch embedding, normalization, cosine computation, and per-class aggregation. Report the summary table.

Input.

Parameter	Value
Golden set	300 rows, 5 topic classes × 60
Embedding model	text-embedding-3-large (dim=3072)
Batch size	100
Threshold	0.85
Expected cost	300 × 2 embeddings × $0.00013 / 1K tokens × ~200 tokens = ~$0.016

Code.

# cosine_scorer.py
import json
from pathlib import Path
from typing import Iterable

import numpy as np
import pandas as pd
from openai import OpenAI

EMBED_MODEL = "text-embedding-3-large"
BATCH = 100
client = OpenAI()


def embed_batch(texts: list[str]) -> np.ndarray:
    """Return an (n, dim) float32 array of L2-normalized embeddings."""
    resp = client.embeddings.create(model=EMBED_MODEL, input=texts)
    vecs = np.asarray([d.embedding for d in resp.data], dtype=np.float32)
    norms = np.linalg.norm(vecs, axis=1, keepdims=True)
    return vecs / np.clip(norms, 1e-9, None)


def batched(items: list, size: int) -> Iterable[list]:
    for i in range(0, len(items), size):
        yield items[i : i + size]


def cosine_score(rows: list[dict], predictions: dict[str, str]) -> pd.DataFrame:
    """
    rows        : golden set (must include row_id, expected_output, metadata.topic)
    predictions : row_id -> model prediction string
    """
    expected = [r["expected_output"] for r in rows]
    predicted = [predictions[r["row_id"]] for r in rows]

    exp_vecs = np.vstack([embed_batch(b) for b in batched(expected, BATCH)])
    pred_vecs = np.vstack([embed_batch(b) for b in batched(predicted, BATCH)])

    # Dot product of L2-normalized vectors == cosine similarity
    cos = np.sum(exp_vecs * pred_vecs, axis=1)

    return pd.DataFrame({
        "row_id":    [r["row_id"] for r in rows],
        "topic":     [r["metadata"]["topic"] for r in rows],
        "cosine":    cos,
        "pass_0.85": cos >= 0.85,
    })


def summarize(df: pd.DataFrame) -> pd.DataFrame:
    by_topic = df.groupby("topic").agg(
        n=("row_id", "count"),
        mean_cosine=("cosine", "mean"),
        pass_rate=("pass_0.85", "mean"),
    ).round(4).reset_index()
    overall = pd.DataFrame([{
        "topic": "OVERALL",
        "n": len(df),
        "mean_cosine": df["cosine"].mean().round(4),
        "pass_rate": df["pass_0.85"].mean().round(4),
    }])
    return pd.concat([by_topic, overall], ignore_index=True)


if __name__ == "__main__":
    rows = [json.loads(l) for l in Path("golden/support_v3.jsonl").read_text().splitlines()]
    predictions = json.loads(Path("runs/run_2026-07-04.json").read_text())
    df = cosine_score(rows, predictions)
    print(summarize(df).to_markdown(index=False))

Step-by-step explanation.

embed_batch sends 100 strings at a time to OpenAI's embeddings endpoint, receives 100 3072-dim vectors, stacks them into a NumPy array, and L2-normalizes so cosine reduces to a plain dot product.
cosine_score embeds the 300 expected outputs and the 300 predictions in parallel batches (3 calls each), then computes cosine as an element-wise multiply-and-sum on the L2-normalized vectors. Total wall-clock is dominated by the embedding calls (~2 seconds each) — roughly 6 seconds for the full run.
Every row emits a (row_id, topic, cosine, pass_0.85) record. The pass_0.85 boolean is the threshold check; downstream aggregation treats it as a per-class pass rate.
summarize groups by topic to expose per-class means and pass rates, then appends an overall row. This is the table that goes into the PR comment — one row per topic plus overall.
Cost math: 300 × 2 embeddings × ~200 tokens/row = 120,000 tokens; at $0.00013 / 1K tokens for text-embedding-3-large, the scorer costs about $0.016 per full run. Effectively free at PR gate frequency.

Output.

topic	n	mean_cosine	pass_rate
account	60	0.9215	0.9500
billing	60	0.8842	0.8500
other	60	0.9051	0.9167
product	60	0.9127	0.9333
shipping	60	0.8776	0.8000
OVERALL	300	0.9002	0.8900

Rule of thumb. Cosine scoring is a pandas.groupby on top of a NumPy dot product. Keep it deterministic, cheap, and always paired with a judge for the semantic axis it cannot see.

Worked example — threshold calibration with a contradiction-injected set

Detailed explanation. Cosine thresholds are meaningless without calibration. Build the calibrator: take 200 golden-set pairs (correct) plus 200 contradiction-injected pairs (wrong), compute cosine on all 400, sweep the threshold to maximize F1, and produce the calibration curve.

Positive class. Correct predictions (from the golden set itself, expected as prediction).
Negative class. Contradiction-injected predictions (LLM-generated wrong-polarity answers reviewed by an SME).
Sweep. Thresholds from 0.5 to 0.99 in 0.01 steps; report precision, recall, F1.
Choose. Threshold that maximizes F1, unless business context prefers precision (fewer false-positive passes) or recall (fewer false-negative fails).

Question. Build the calibrator, sweep the threshold, and pick the operating point for a chatbot where a missed wrong answer is 3× more costly than a false-flagged correct answer.

Input.

Parameter	Value
Correct pairs	200 (from golden set)
Contradiction pairs	200 (LLM-generated, SME-reviewed)
Embedding model	text-embedding-3-large
Sweep	0.50 to 0.99 in 0.01 steps
Cost ratio (miss : false-flag)	3 : 1

Code.

# calibrate_threshold.py
import json
from pathlib import Path

import numpy as np
import pandas as pd

from cosine_scorer import embed_batch, BATCH


def cosine_pairs(a: list[str], b: list[str]) -> np.ndarray:
    va = np.vstack([embed_batch(a[i:i+BATCH]) for i in range(0, len(a), BATCH)])
    vb = np.vstack([embed_batch(b[i:i+BATCH]) for i in range(0, len(b), BATCH)])
    return np.sum(va * vb, axis=1)


def sweep(cos_pos: np.ndarray, cos_neg: np.ndarray, cost_miss: float = 3.0, cost_ff: float = 1.0) -> pd.DataFrame:
    rows = []
    for t in np.arange(0.50, 1.00, 0.01):
        tp = (cos_pos >= t).sum()
        fn = (cos_pos <  t).sum()
        fp = (cos_neg >= t).sum()
        tn = (cos_neg <  t).sum()
        precision = tp / (tp + fp) if (tp + fp) else 0.0
        recall    = tp / (tp + fn) if (tp + fn) else 0.0
        f1        = 2 * precision * recall / (precision + recall) if (precision + recall) else 0.0
        # cost-weighted score: penalize false-flags (fn on positives) and missed-wrongs (fp on negatives)
        cost = cost_ff * fn + cost_miss * fp
        rows.append({
            "threshold": round(float(t), 2),
            "tp": int(tp), "fn": int(fn), "fp": int(fp), "tn": int(tn),
            "precision": round(precision, 3),
            "recall":    round(recall, 3),
            "f1":        round(f1, 3),
            "cost":      int(cost),
        })
    return pd.DataFrame(rows)


if __name__ == "__main__":
    pos_pairs = json.loads(Path("calibration/correct_pairs.json").read_text())    # 200 pairs
    neg_pairs = json.loads(Path("calibration/contradiction_pairs.json").read_text())  # 200 pairs

    cos_pos = cosine_pairs([p["expected"] for p in pos_pairs], [p["prediction"] for p in pos_pairs])
    cos_neg = cosine_pairs([p["expected"] for p in neg_pairs], [p["prediction"] for p in neg_pairs])

    df = sweep(cos_pos, cos_neg, cost_miss=3.0, cost_ff=1.0)
    df_best_f1  = df.loc[df["f1"].idxmax()]
    df_best_cost = df.loc[df["cost"].idxmin()]
    print("Best F1 threshold:", df_best_f1.to_dict())
    print("Best cost-weighted threshold:", df_best_cost.to_dict())
    print(df.to_markdown(index=False))

Step-by-step explanation.

Positive pairs are 200 rows where prediction == expected_output (perfect answers). Negative pairs are 200 rows where a strong LLM was prompted to contradict the expected answer, then an SME reviewed to ensure each is a genuine wrong-polarity flip. Both are embedded and cosine'd.
The sweep computes confusion-matrix entries at each threshold: true positives (correct passes), false negatives (correct-but-flagged), false positives (wrong-but-passed), true negatives (wrong-and-flagged). Precision, recall, and F1 fall out of the confusion matrix.
The cost function encodes the business preference: a missed wrong answer (FP on negatives) is 3× the pain of a false-flagged correct answer (FN on positives). The cost-optimal threshold is higher than the F1-optimal threshold because higher thresholds catch more wrongs at the cost of flagging more corrects.
Empirically on text-embedding-3-large: F1 peaks around 0.82 (threshold 0.86); cost-optimal at 3:1 miss weight lands around 0.89. Both numbers are model-specific and answer-length-specific; they should be re-derived on your data.
The chosen operating point (0.89) goes into eval-config.yaml as cosine.threshold: 0.89 with a comment linking to the calibration report and the calibration set's git hash. Any future embedding-model bump requires a fresh calibration.

Output.

threshold	tp	fn	fp	tn	precision	recall	f1	cost
0.80	195	5	42	158	0.823	0.975	0.893	131
0.85	190	10	22	178	0.896	0.950	0.922	76
0.86	187	13	15	185	0.926	0.935	0.930	58
0.89	175	25	8	192	0.956	0.875	0.914	49
0.90	168	32	5	195	0.971	0.840	0.901	47
0.92	148	52	2	198	0.987	0.740	0.846	58

Rule of thumb. Calibrate the cosine threshold against a labeled positive+negative set, sweep with a cost function that reflects your business's miss/false-flag ratio, and pin the chosen threshold in config with a link back to the calibration report. Never quote a threshold "from the paper" or "from a blog post" — it will not match your data.

Worked example — the paraphrase vs contradiction blind spot

Detailed explanation. Demonstrate the canonical cosine failure mode with a synthetic 20-row set: 10 paraphrases (semantically identical, superficially different) and 10 contradictions (semantically opposite, superficially similar). Show cosine scores for each and prove why an LLM-judge is required on top.

Paraphrases. Same meaning, different words. Cosine will under-count these as false negatives.
Contradictions. Opposite meaning, similar surface form. Cosine will over-count these as false positives.
Judge job. Distinguish paraphrase from contradiction where cosine cannot.

Question. Build the demonstrator: score 10 paraphrase and 10 contradiction pairs with cosine, then with an LLM-as-judge, and compare their agreement + disagreement.

Input.

Pair type	Example expected	Example prediction	Ground truth
Paraphrase	Yes, you can cancel anytime.	You may cancel at any time.	correct
Paraphrase	The refund arrives in 3-5 business days.	Refunds land within 3 to 5 working days.	correct
Contradiction	Yes, you can cancel anytime.	No, cancellation is not permitted.	wrong
Contradiction	The refund arrives in 3-5 business days.	Refunds are not available.	wrong

Code.

# cosine_vs_judge_demo.py
import json
import numpy as np

from cosine_scorer import embed_batch
from llm_judge import judge_row       # defined in Section 4


pairs = [
    # Paraphrases (correct)
    ("Yes, you can cancel anytime.",                  "You may cancel at any time.",                          "correct"),
    ("The refund arrives in 3-5 business days.",      "Refunds land within 3 to 5 working days.",             "correct"),
    ("Please email support@company.com for help.",    "For assistance, contact support@company.com.",         "correct"),
    ("Your order will ship tomorrow.",                "The shipment is scheduled for tomorrow.",              "correct"),
    ("We accept Visa and Mastercard.",                "Visa and Mastercard payments are accepted.",           "correct"),
    ("This feature is available on Pro plans.",       "Pro plans include this feature.",                      "correct"),
    ("You'll need admin access to change settings.",  "Admin permissions are required to modify settings.",   "correct"),
    ("The trial lasts 14 days.",                      "You get a 14-day trial period.",                       "correct"),
    ("Data exports are in CSV format.",               "Exported data is provided as CSV files.",              "correct"),
    ("The API rate limit is 1000 requests/min.",      "You can make up to 1000 API calls per minute.",        "correct"),
    # Contradictions (wrong)
    ("Yes, you can cancel anytime.",                  "No, cancellation is not permitted.",                   "wrong"),
    ("The refund arrives in 3-5 business days.",      "Refunds are not available.",                           "wrong"),
    ("Please email support@company.com for help.",    "Do not email support; they don't respond.",            "wrong"),
    ("Your order will ship tomorrow.",                "Your order has been cancelled.",                       "wrong"),
    ("We accept Visa and Mastercard.",                "We do not accept credit cards.",                       "wrong"),
    ("This feature is available on Pro plans.",       "This feature is not available on any plan.",           "wrong"),
    ("You'll need admin access to change settings.",  "Anyone can change settings without any permission.",   "wrong"),
    ("The trial lasts 14 days.",                      "There is no trial period offered.",                    "wrong"),
    ("Data exports are in CSV format.",               "Data cannot be exported.",                             "wrong"),
    ("The API rate limit is 1000 requests/min.",      "There is no API access available.",                    "wrong"),
]


def demo():
    exp = embed_batch([p[0] for p in pairs])
    pred = embed_batch([p[1] for p in pairs])
    cos = np.sum(exp * pred, axis=1)

    rows = []
    for (e, p, truth), c in zip(pairs, cos):
        judge = judge_row(input_text="(hidden)", expected=e, prediction=p)   # returns 0 or 1
        rows.append({
            "type":     "paraphrase" if truth == "correct" else "contradiction",
            "cosine":   round(float(c), 3),
            "cos_pass": bool(c >= 0.85),
            "judge":    judge,
            "truth":    truth,
        })

    import pandas as pd
    df = pd.DataFrame(rows)
    print(df.to_markdown(index=False))

    cos_acc = ((df["cos_pass"] & (df["truth"] == "correct")) | (~df["cos_pass"] & (df["truth"] == "wrong"))).mean()
    jud_acc = ((df["judge"] == 1) & (df["truth"] == "correct") | (df["judge"] == 0) & (df["truth"] == "wrong")).mean()
    print(f"\ncosine  agreement with truth: {cos_acc:.2%}")
    print(f"judge   agreement with truth: {jud_acc:.2%}")


demo()

Step-by-step explanation.

The 20 pairs are hand-designed: 10 paraphrases (semantically identical, different surface form) and 10 contradictions (opposite polarity, otherwise similar surface form). This is a stress test built to break a naïve cosine gate.
Cosine scores paraphrases in the 0.75–0.85 range (below the calibrated 0.85 threshold — false negatives on the correct class) and contradictions in the 0.80–0.90 range (some above the threshold — false positives on the wrong class). The threshold cannot separate the two populations cleanly.
The LLM-judge is called with the full (expected, prediction) pair plus a rubric. For paraphrases it recognizes the semantic equivalence and returns 1 (pass); for contradictions it recognizes the polarity flip and returns 0 (fail). Judge accuracy on this stress set is typically 95–100 percent.
The comparison table exposes the exact gap: cosine agreement with ground truth is around 55–65 percent; judge agreement is 95–100 percent. On real production data the gap narrows (real predictions are less adversarial), but the direction is always the same — judge on top of cosine.
The lesson is not "throw cosine away." It is "cosine as first-pass filter, judge as truth arbiter." Cosine catches structural hallucinations cheaply; the judge catches semantic errors expensively. Use both, and know which failure mode each catches.

Output.

type	cosine	cos_pass	judge	truth
paraphrase	0.816	False	1	correct
paraphrase	0.892	True	1	correct
paraphrase	0.751	False	1	correct
contradiction	0.828	False	0	wrong
contradiction	0.874	True	0	wrong
contradiction	0.845	False	0	wrong

Cosine agreement with truth: ~60 percent. Judge agreement with truth: ~100 percent.

Rule of thumb. Any metric that cannot distinguish "yes" from "no" on similar surface forms is a false-security metric. Cosine is a great cheap gate; the judge is the truth signal that keeps the metric honest.

Senior interview question on cosine-based scoring

A senior interviewer might ask: "Walk me through your cosine similarity scorer for a RAG chatbot. How do you pick the embedding model, calibrate the threshold, handle multilingual traffic, and pair it with a judge without doubling your eval cost?"

Solution Using pinned embedding + calibrated threshold + judge-on-miss escalation

# cosine_plus_judge.py — the reference cosine + judge orchestrator
import json
from pathlib import Path
from typing import Optional

import numpy as np
import pandas as pd

from cosine_scorer import embed_batch, EMBED_MODEL, BATCH
from llm_judge import judge_row

# Pinned in config; bump forces recalibration
EMBEDDING_VERSION = "text-embedding-3-large@2024-11-01"
THRESHOLD_BY_LANGUAGE = {"en": 0.86, "es": 0.83, "fr": 0.83, "default": 0.82}


def score_row(expected_vec: np.ndarray, prediction_vec: np.ndarray) -> float:
    return float(np.dot(expected_vec, prediction_vec))


def threshold_for(language: str) -> float:
    return THRESHOLD_BY_LANGUAGE.get(language, THRESHOLD_BY_LANGUAGE["default"])


def score_batch(rows: list[dict], predictions: dict[str, str]) -> pd.DataFrame:
    exp_vecs = np.vstack([
        embed_batch([r["expected_output"] for r in rows[i:i+BATCH]])
        for i in range(0, len(rows), BATCH)
    ])
    pred_vecs = np.vstack([
        embed_batch([predictions[r["row_id"]] for r in rows[i:i+BATCH]])
        for i in range(0, len(rows), BATCH)
    ])

    out = []
    for r, ev, pv in zip(rows, exp_vecs, pred_vecs):
        lang = r["metadata"].get("language", "en")
        c = score_row(ev, pv)
        thr = threshold_for(lang)
        cos_pass = c >= thr
        # Judge-on-miss: escalate only when cosine says "wrong"
        judge = judge_row(
            input_text=r["input"],
            expected=r["expected_output"],
            prediction=predictions[r["row_id"]],
            rubric=["correctness"],
        ) if not cos_pass else None
        out.append({
            "row_id":         r["row_id"],
            "topic":          r["metadata"]["topic"],
            "language":       lang,
            "cosine":         round(c, 4),
            "cos_threshold":  thr,
            "cos_pass":       cos_pass,
            "judge_score":    judge,
            "final_pass":     cos_pass or (judge is not None and judge >= 0.7),
            "embedding":      EMBEDDING_VERSION,
        })
    return pd.DataFrame(out)


if __name__ == "__main__":
    rows = [json.loads(l) for l in Path("golden/support_v3.jsonl").read_text().splitlines()]
    predictions = json.loads(Path("runs/run_2026-07-04.json").read_text())
    df = score_batch(rows, predictions)
    print(df.groupby(["topic", "language"]).agg(
        n=("row_id", "count"),
        mean_cosine=("cosine", "mean"),
        cos_pass_rate=("cos_pass", "mean"),
        final_pass_rate=("final_pass", "mean"),
    ).round(3).to_markdown())

Step-by-step trace.

Step	Input	Output
Pin embedding	text-embedding-3-large@2024-11-01	version stamp on every row
Embed expected + prediction	300 rows	600 3072-dim vectors
Compute cosine	dot(ev, pv) after L2-norm	300 floats in [0, 1]
Threshold by language	en=0.86, es/fr=0.83	300 bool pass flags
Escalate on miss	~40 rows below threshold	40 judge calls
Aggregate final pass	cos_pass OR judge≥0.7	300 final bool

After the rollout, cosine handles ~87 percent of rows for free (below-threshold rows escalate to the judge); judge is called on only the ~13 percent that cosine flagged. Total judge cost drops by ~7× vs judging every row, while the final metric is judge-quality on the disputed subset.

Output:

Layer	Cost per full run	Detects paraphrase	Detects contradiction	Detects structural
Cosine only	$0.02	partial	no	yes
Judge only	$2.00	yes	yes	partial
Cosine + judge-on-miss	$0.30	yes (via judge)	yes (via judge)	yes (cosine)

Why this works — concept by concept:

Pinned embedding version — text-embedding-3-large@2024-11-01 is written into every score row so the metric is comparable across time. Bumping the embedding model is a first-class change that requires re-calibrating the threshold; the version stamp makes it impossible to forget.
Per-language thresholds — cross-lingual embeddings noisy differently per language pair; carrying a single "0.85" threshold across en/es/fr silently penalizes non-English rows. Per-language calibration is a one-hour exercise per language, paid once.
Judge-on-miss escalation — cosine handles the majority of rows for essentially free; the judge is called only when cosine says "wrong." This preserves judge-quality decisions on the disputed subset while cutting judge cost by roughly 7× on a well-performing model.
final_pass logic — a row passes if cosine says yes OR (cosine said no AND judge says ≥0.7). The judge acts as an appeal for cosine's false negatives on paraphrases; the OR-logic ensures paraphrases don't get punished, while contradictions still fail because the judge does not overturn cosine's passes.
Cost — per-run: ~$0.02 embeddings + ~$0.30 judge on the escalated 13 percent = ~$0.32. The all-judge alternative is ~$2.00 (6× more expensive) with only marginal quality gain on rows cosine already handled correctly. Judge-on-miss is the price-performance sweet spot.

Data analysis
Topic — data-analysis
Data-analysis problems on similarity and threshold calibration

Practice →

Optimization Topic — optimization Optimization problems on scorer cost and coverage

Practice →

4. LLM-as-judge — GPT-4-class rubric scorer

The judge is a rubric + a strong model + a structured JSON schema — anything less is vibes

The mental model in one line: llm as judge is the pattern where a strong model (GPT-4-class or better) scores the production model's prediction against the golden expected answer using an explicit rubric, returning a structured JSON with a numeric score plus a rationale — and its power comes entirely from the rubric quality, the model choice, and the output schema, not from any magic in "the LLM said so". A judge without a rubric is vibes; a rubric without a strong judge is a rubric being ignored; a judge with a rubric returning free-form prose is unusable as a metric. Engineer all three.

The four axes of a serious judge.

Rubric. An explicit list of criteria with clear pass/fail language. Typical rubrics carry 2–4 criteria (correctness, completeness, style, safety); more than 4 dilutes the judge's attention; fewer than 2 collapses the metric to a single dimension.
Model. GPT-4o, Claude Opus, or equivalent. Judge quality tracks base-model reasoning quality; using the same model as production for judgement risks correlated failure modes.
Output schema. Structured JSON with typed fields (score: float, rationale: string, optional breakdown per criterion). Function-calling or JSON-mode is mandatory — free-form text is unparseable.
Cost + sampling. Judge cost per row × golden-set size × runs per week is the finance-visible number. Sample intelligently (per-PR 20 percent sample + nightly full-set) to keep the bill sane.

The rubric — write it like an SLA.

Explicit criteria. correctness (is the factual content right?), completeness (are the required facts included?), style (is the tone appropriate?). Each criterion gets its own explicit language.
Scored on a scale. 0.0–1.0 or 1–5. Continuous scales let you aggregate as means; discrete scales are easier for judges to calibrate but coarser downstream.
Grounded in the expected answer. The judge always has access to the expected answer as ground truth; without it, the judge is just re-answering the question rather than scoring.
Failure modes named. The rubric explicitly names the failure modes to look for: "a wrong URL", "a wrong dollar amount", "an empathy mismatch". This forces the judge to check specifics rather than hand-wave.

The output schema — never take free-form text.

Use JSON mode / function calling. OpenAI's response_format={"type": "json_object"}, Anthropic's tool use, or Gemini's structured output. The judge returns typed fields; parsing is trivial.
Type every field. score: float in [0, 1], rationale: string, breakdown: {correctness: float, completeness: float, style: float}. The schema is verified with pydantic on receive.
Bounded output length. Cap rationale at 500 tokens. Longer rationales cost more without improving the score signal.
Idempotence. Set temperature=0 for reproducible judgements. Non-zero temperature turns the judge into a random noise generator.

The judge model — do not use the production model.

Same-family risk. Using GPT-4o-mini as judge for a GPT-4o-mini production model risks correlated failure modes — the judge fails to notice the same errors the production model made.
Recommendation. Judge with a stronger or different-family model. GPT-4o judges GPT-4o-mini production; Claude Opus judges GPT-4o production; Gemini Ultra judges Claude Sonnet.
Cost trade-off. A stronger judge costs more per row. Balance with sample-on-PR + full-nightly (cheap judge on the sample, stronger judge on nightly calibration).
Calibrate cross-judge agreement. Run 100 golden rows through two judges (GPT-4o and Claude Opus). Compute agreement (Cohen's κ or Pearson r on scores). Above 0.85 means either judge is fine; below 0.7 means the rubric needs tightening.

Failure modes of the judge itself.

Judge bias. Judges have systematic preferences (length, formality, structure). A verbose but wrong answer may score higher than a concise correct one. Counter with explicit anti-bias language in the rubric.
Prompt sensitivity. Small phrasing changes in the rubric shift scores. Version-hash the rubric and treat rubric edits like migrations.
Position bias. Some judges systematically favor the first option in a comparison; randomize the order in pairwise comparison setups.
Rubric drift. Over time the SME's mental rubric drifts from the written rubric. Refresh the rubric every quarter and re-calibrate.

What senior interviewers listen for.

Do you name the rubric as a first-class artefact with version control? — senior signal.
Do you insist on structured JSON output rather than free-form? — required answer.
Do you use a stronger or different-family judge than production? — senior signal.
Do you talk about judge bias and prompt sensitivity as engineering problems, not model flaws? — senior signal.

Worked example — a 3-criterion rubric + JSON schema for a support chatbot

Detailed explanation. Design the judge for the customer-support chatbot. Rubric: correctness, completeness, style. Model: GPT-4o. Output: JSON with per-criterion scores plus an overall + rationale. Show the exact prompt, the pydantic schema, and a worked scoring on a representative row.

Judge model. GPT-4o (temperature=0).
Rubric criteria. correctness (0.0-1.0), completeness (0.0-1.0), style (0.0-1.0).
Overall score. Weighted mean: 0.5*correctness + 0.3*completeness + 0.2*style.
Output. JSON via response_format={"type": "json_object"}.

Question. Write the judge prompt, the pydantic schema, and the client code. Score a representative row and interpret.

Input.

Field	Value
Input	"How do I cancel my subscription?"
Expected	"You can cancel anytime by opening Settings → Billing → Cancel. The cancellation takes effect at the end of your current billing period."
Prediction	"You may cancel any time from the billing settings; the change is effective at the end of the current cycle."

Code.

# llm_judge.py
import json
from typing import Literal, Optional

from openai import OpenAI
from pydantic import BaseModel, Field, ValidationError

client = OpenAI()

RUBRIC_HASH = "c93f11"     # bumps whenever the prompt changes


class JudgeBreakdown(BaseModel):
    correctness: float = Field(..., ge=0.0, le=1.0)
    completeness: float = Field(..., ge=0.0, le=1.0)
    style: float = Field(..., ge=0.0, le=1.0)


class JudgeResult(BaseModel):
    score: float = Field(..., ge=0.0, le=1.0)
    rationale: str
    breakdown: JudgeBreakdown


SYSTEM_PROMPT = """
You are an expert evaluator for a customer support chatbot. You score the
`prediction` against the `expected` answer on THREE criteria:

- correctness (0.0-1.0): factually accurate; no wrong dates, URLs, entities,
  or polarity flips. If the prediction contradicts the expected answer in
  any material way, correctness is 0.0.
- completeness (0.0-1.0): includes the essential facts from the expected
  answer. Missing a required step, disclaimer, or call-to-action lowers
  completeness proportionally.
- style (0.0-1.0): tone matches the expected style (empathetic where the
  expected is empathetic, formal where the expected is formal); grammar
  and phrasing are natural.

You are BLIND to answer length: a shorter correct answer scores as high as
a longer correct answer. You do NOT reward verbosity.

Return a JSON object with the schema:
{
  "score": <weighted mean: 0.5 * correctness + 0.3 * completeness + 0.2 * style>,
  "rationale": "<= 500 tokens explaining the criterion scores",
  "breakdown": {"correctness": ..., "completeness": ..., "style": ...}
}

Do not include any other fields; do not include markdown; return raw JSON.
""".strip()


def judge_row(
    input_text: str,
    expected: str,
    prediction: str,
    rubric: Optional[list[str]] = None,
    model: str = "gpt-4o",
) -> JudgeResult:
    user_prompt = (
        f"INPUT:\n{input_text}\n\n"
        f"EXPECTED:\n{expected}\n\n"
        f"PREDICTION:\n{prediction}\n"
    )
    resp = client.chat.completions.create(
        model=model,
        temperature=0.0,
        response_format={"type": "json_object"},
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user",   "content": user_prompt},
        ],
    )
    raw = resp.choices[0].message.content
    try:
        return JudgeResult.model_validate_json(raw)
    except ValidationError as e:
        raise ValueError(f"judge returned invalid JSON: {raw!r}") from e


if __name__ == "__main__":
    result = judge_row(
        input_text="How do I cancel my subscription?",
        expected=(
            "You can cancel anytime by opening Settings → Billing → Cancel. "
            "The cancellation takes effect at the end of your current billing period."
        ),
        prediction=(
            "You may cancel any time from the billing settings; the change "
            "is effective at the end of the current cycle."
        ),
    )
    print(json.dumps(result.model_dump(), indent=2))

Step-by-step explanation.

The system prompt spells out the three criteria with explicit language, calls out the specific failure modes (wrong URLs, polarity flips, missing steps), and tells the judge not to reward verbosity. This is the rubric-as-SLA discipline in one prompt.
The output schema is a pydantic model: score in [0, 1], rationale string, breakdown per-criterion. response_format={"type": "json_object"} forces JSON; temperature=0.0 makes the judgement deterministic; RUBRIC_HASH versions the prompt.
On the sample row, the prediction is a paraphrase of the expected: same steps, same effect, slightly different wording. The judge scores correctness at 1.0 (all facts match), completeness at 0.9 (mentions cancel + end-of-cycle, misses the explicit menu path), style at 0.85 (informal but appropriate). Overall = 0.5×1.0 + 0.3×0.9 + 0.2×0.85 = 0.94.
The rationale is included for auditability. When a PR author sees a per-row miss, they can read the rationale and understand why — "the prediction says 'billing settings' but the expected says 'Settings → Billing → Cancel'; the menu path is less precise" — turning the metric from a black box into a diagnostic tool.
The client caches on (input_hash, expected_hash, prediction_hash, rubric_hash, model). Deterministic judgement means identical inputs always return identical scores; cache hits are free.

Output.

{
  "score": 0.94,
  "rationale": "The prediction correctly identifies the cancellation location (billing settings) and confirms the effective date (end of current cycle), matching the factual content of the expected. It omits the specific menu path (Settings → Billing → Cancel) which lowers completeness slightly. Style is informal but appropriate for a support context; the tone is helpful.",
  "breakdown": {
    "correctness": 1.0,
    "completeness": 0.9,
    "style": 0.85
  }
}

Rule of thumb. Every judge call returns structured JSON validated by pydantic; every prompt is version-hashed; every judgement is deterministic (temperature=0). Free-form judge outputs are unusable as a metric — engineer for structure.

Worked example — cross-judge agreement calibration

Detailed explanation. Before trusting a judge, measure it against another judge. Take 100 golden-set rows plus their predictions, score with two independent judges (GPT-4o and Claude Opus), and compute agreement. If the two disagree systematically, the rubric needs work.

Judge A. GPT-4o with the 3-criterion rubric.
Judge B. Claude Opus with the same rubric.
Metrics. Pearson correlation on continuous scores; Cohen's κ on discretized pass/fail at 0.7.
Decision rule. r > 0.85 → either judge is fine; r < 0.7 → tighten the rubric or narrow the criteria.

Question. Build the cross-judge calibrator, run it on 100 rows, and interpret. What do you do if r = 0.62?

Input.

Parameter	Value
Rows	100 (from golden set)
Judge A	GPT-4o
Judge B	Claude Opus 4
Prompt hash	c93f11 (same for both)
Pass threshold	score ≥ 0.7

Code.

# cross_judge_agreement.py
import json
from pathlib import Path

import numpy as np
import pandas as pd
from scipy.stats import pearsonr, spearmanr

from llm_judge import judge_row


def score_all(rows: list[dict], predictions: dict[str, str], model: str) -> pd.Series:
    scores = []
    for r in rows:
        j = judge_row(
            input_text=r["input"],
            expected=r["expected_output"],
            prediction=predictions[r["row_id"]],
            model=model,
        )
        scores.append({"row_id": r["row_id"], "score": j.score})
    return pd.DataFrame(scores).set_index("row_id")["score"]


def cohens_kappa(a: pd.Series, b: pd.Series, threshold: float = 0.7) -> float:
    a_pass = (a >= threshold).astype(int)
    b_pass = (b >= threshold).astype(int)
    # 2x2 confusion
    n = len(a)
    tt = int(((a_pass == 1) & (b_pass == 1)).sum())
    ff = int(((a_pass == 0) & (b_pass == 0)).sum())
    tf = int(((a_pass == 1) & (b_pass == 0)).sum())
    ft = int(((a_pass == 0) & (b_pass == 1)).sum())
    p_obs = (tt + ff) / n
    p_a_pos = (tt + tf) / n
    p_b_pos = (tt + ft) / n
    p_exp = p_a_pos * p_b_pos + (1 - p_a_pos) * (1 - p_b_pos)
    return (p_obs - p_exp) / (1 - p_exp) if p_exp < 1 else 1.0


if __name__ == "__main__":
    rows = [json.loads(l) for l in Path("golden/support_v3.jsonl").read_text().splitlines()][:100]
    predictions = json.loads(Path("runs/run_2026-07-04.json").read_text())

    a = score_all(rows, predictions, model="gpt-4o")
    b = score_all(rows, predictions, model="claude-opus-4")

    r_pearson, p1 = pearsonr(a, b)
    r_spearman, p2 = spearmanr(a, b)
    kappa = cohens_kappa(a, b, threshold=0.7)

    print(f"Pearson r = {r_pearson:.3f}  (p = {p1:.4f})")
    print(f"Spearman r = {r_spearman:.3f}")
    print(f"Cohen's κ  = {kappa:.3f}")

    if r_pearson >= 0.85:
        print("PASS: either judge is fine; pick the cheaper one for PR sample.")
    elif r_pearson >= 0.70:
        print("MARGINAL: use the more expensive judge; re-examine rubric criteria.")
    else:
        print("FAIL: rubric ambiguous. Tighten criteria; re-write style axis; re-run.")

Step-by-step explanation.

Score the same 100 rows through both judges with the identical rubric. Both judges are temperature=0, so within-judge reruns are stable; the delta between judges is the "rubric ambiguity plus model bias" signal.
Compute Pearson r (linear correlation of continuous scores) and Spearman r (rank correlation, robust to scale differences). Compute Cohen's κ on the discretized pass/fail at 0.7 — this captures how well the judges agree on the decision level.
Interpret: r ≥ 0.85 means the judges agree strongly; either can be used, so pick the cheaper for high-frequency runs. r in [0.7, 0.85) is marginal — the expensive judge should stay in the gate, and the rubric deserves another look. r < 0.7 is a rubric problem: the criteria are ambiguous enough that reasonable judges disagree.
If r = 0.62, look at the per-row scores where the judges disagree. Common causes: the style axis is subjective (empathy is culture-dependent), the completeness axis is underspecified (which facts are required?), or the rubric conflates axes that need to be separated. Rewrite the ambiguous criteria and re-run.
Once the rubric is stable and r > 0.85, freeze the rubric hash and pin the cheaper judge for the PR sample. The expensive judge stays on the nightly calibration run as an anchor — if the cheap judge starts drifting from the expensive one, the calibration diff flags it.

Output.

Pearson r = 0.891  (p = 0.0000)
Spearman r = 0.867
Cohen's κ  = 0.782
PASS: either judge is fine; pick the cheaper one for PR sample.

Rule of thumb. Cross-judge calibration is the acceptance test for a rubric. A rubric that two strong judges score inconsistently is a rubric your PR author will not trust. Tighten first, then judge.

Worked example — the length-bias trap and how to instrument against it

Detailed explanation. LLM judges systematically prefer longer answers even when the shorter answer is factually complete. Instrument the judge output with per-row prediction length, plot score-vs-length, detect bias, and inject an anti-bias instruction into the rubric.

Signal. Positive correlation between len(prediction) and judge_score, controlling for correctness.
Detection. Regress judge_score on len_pred while holding expected_correctness fixed; a positive slope is length bias.
Fix. Add "you are blind to length" language + a paired-example demonstration.

Question. Build the length-bias detector against a 300-row scored set, quantify the bias slope, and update the rubric to counteract it.

Input.

Parameter	Value
Scored rows	300
Judge scores	judge_score column from prior run
Prediction length	len(prediction) in chars
Baseline (correct-answer) length	180 chars mean

Code.

# length_bias_detector.py
import json
from pathlib import Path

import numpy as np
import pandas as pd
from sklearn.linear_model import LinearRegression


def detect_length_bias(scored_df: pd.DataFrame) -> dict:
    """
    scored_df must have: judge_score (float 0-1), prediction (str), correctness (float 0-1).
    Regresses judge_score on len_pred while conditioning on correctness.
    """
    df = scored_df.copy()
    df["len_pred"] = df["prediction"].str.len()
    df["len_pred_z"] = (df["len_pred"] - df["len_pred"].mean()) / df["len_pred"].std()

    # Partial regression: judge_score ~ len_pred_z + correctness
    X = df[["len_pred_z", "correctness"]].values
    y = df["judge_score"].values
    model = LinearRegression().fit(X, y)

    slope_len = float(model.coef_[0])          # per-standard-deviation of length
    slope_correct = float(model.coef_[1])
    r_partial = float(np.corrcoef(y - model.intercept_ - slope_correct * X[:, 1], X[:, 0])[0, 1])
    return {
        "n": len(df),
        "slope_length_per_sd": round(slope_len, 4),
        "slope_correctness": round(slope_correct, 4),
        "partial_r_length": round(r_partial, 4),
        "interpretation": (
            "BIAS: judge rewards length" if slope_len > 0.02
            else "OK: no meaningful length bias"
        ),
    }


if __name__ == "__main__":
    df = pd.read_parquet("runs/run_2026-07-04_scored.parquet")
    print(detect_length_bias(df))


# --- Anti-bias rubric update (append to SYSTEM_PROMPT) ---
ANTI_BIAS_ADDENDUM = """
CRITICAL: You must be blind to answer length. Two answers that convey the same
factual content should receive the same score, regardless of length. A 40-word
correct answer scores identically to a 200-word correct answer.

Consider this pair:
- Prediction A (short): "Cancel from Settings → Billing → Cancel; effective at cycle end."
- Prediction B (long): "You can absolutely cancel your subscription at any time! To do so, please navigate to Settings, then Billing, then Cancel. The cancellation will take effect at the end of your current billing cycle. If you have any questions, feel free to reach out to our support team..."

Both should score identically (correctness=1.0, completeness=1.0), because both
contain the same factual information. Prediction B's length adds no signal.
""".strip()

Step-by-step explanation.

Regress judge_score on standardized len_pred while controlling for correctness (which is itself judge-labelled — a limitation the harness acknowledges). A positive slope on len_pred_z while correctness is fixed is length bias by construction.
On the reference set, the slope typically lands in the range 0.03–0.08 for GPT-4-class judges without an explicit anti-bias instruction — meaning a 1-standard-deviation increase in length (say, +200 chars) raises the judge score by 3–8 points on the 0-100 scale. Significant, and directionally wrong.
The interpretation triggers a rubric update: append the anti-bias language plus a paired-example demonstration. The demonstration is what actually moves the needle — instructing the judge to "consider two predictions of the same content" is far more effective than telling it to "ignore length."
After the rubric update, re-run the detector on the same 300 rows. Slope typically drops to 0.005–0.015 — small residual bias, but small enough that the metric is not systematically favoring verbose answers.
The detector runs on every nightly full-set run. If the slope creeps back up (rubric drift, model update on the judge side), the alert fires and the team re-examines the rubric before the bias corrupts weeks of PR gating.

Output.

Rubric version	Slope (length)	Slope (correctness)	Interpretation
v1.0 (no anti-bias)	0.062	0.71	BIAS: judge rewards length
v1.1 (+ anti-bias)	0.008	0.79	OK: no meaningful length bias

Rule of thumb. Judge bias detection is a nightly check, not a one-time calibration. Rubric drift is real; new judge model versions ship weekly; the harness that watches the judge is the harness the team trusts. Never assume the judge stays honest without an instrument.

Senior interview question on the judge design

A senior interviewer might ask: "Design an LLM-as-judge for a code-explanation assistant where correctness must be exact, style is 'friendly and precise,' and the judge must be robust against style drift when the production model changes. Walk me through the rubric, model choice, output schema, calibration, and the anti-bias instrumentation."

Solution Using a versioned rubric + stronger-judge + cross-model calibration + length-bias check

# code_assistant_judge.py — the reference judge for a code-explanation task
import json
from typing import Literal

from openai import OpenAI
from pydantic import BaseModel, Field

client = OpenAI()

RUBRIC_HASH = "cx-42a9"       # bump on any prompt edit
JUDGE_MODEL = "gpt-4o"         # production is Claude Sonnet; judge is different family


class CodeExplainBreakdown(BaseModel):
    factual_correctness: float = Field(..., ge=0.0, le=1.0)
    code_accuracy: float = Field(..., ge=0.0, le=1.0)  # cited APIs / syntax right
    style: float = Field(..., ge=0.0, le=1.0)


class CodeExplainResult(BaseModel):
    score: float = Field(..., ge=0.0, le=1.0)
    rationale: str
    breakdown: CodeExplainBreakdown


SYSTEM = """
You evaluate a code-explanation assistant. You compare a prediction to a
golden expected answer on THREE criteria:

1. factual_correctness (0.0-1.0): Does the explanation match the expected
   behavior? Wrong function names, wrong parameters, wrong return types, or
   wrong Big-O = 0.0. Partial matches score in between.

2. code_accuracy (0.0-1.0): Are the code snippets in the prediction runnable
   and match the expected API? Wrong imports, wrong syntax, or wrong API
   surface = 0.0. Minor style issues that do not affect runnability = 0.9-1.0.

3. style (0.0-1.0): Is the tone "friendly and precise"? Precise means using
   the exact technical term (say "generator" not "iterator" when generator
   is correct). Friendly means direct, non-condescending, no filler. Verbosity
   is NOT a positive signal.

CRITICAL: You are blind to answer length. Two explanations of the same content
score identically. A 30-word correct answer = a 200-word correct answer.

Return raw JSON:
{
  "score": <0.5*factual + 0.3*code + 0.2*style>,
  "rationale": "<= 400 tokens",
  "breakdown": {"factual_correctness": ..., "code_accuracy": ..., "style": ...}
}
""".strip()


def judge_code_row(input_text: str, expected: str, prediction: str) -> CodeExplainResult:
    user = f"INPUT:\n{input_text}\n\nEXPECTED:\n{expected}\n\nPREDICTION:\n{prediction}\n"
    resp = client.chat.completions.create(
        model=JUDGE_MODEL,
        temperature=0.0,
        response_format={"type": "json_object"},
        messages=[
            {"role": "system", "content": SYSTEM},
            {"role": "user",   "content": user},
        ],
    )
    return CodeExplainResult.model_validate_json(resp.choices[0].message.content)


# --- Nightly instrumentation harness ---
def nightly_calibration(rows, predictions):
    a = [judge_code_row(r["input"], r["expected_output"], predictions[r["row_id"]]).score for r in rows]
    b = [judge_code_row_with_alt_judge(r["input"], r["expected_output"], predictions[r["row_id"]]).score for r in rows]
    from scipy.stats import pearsonr
    return pearsonr(a, b)[0]

Step-by-step trace.

Step	Choice	Rationale
Production model	Claude Sonnet	Team's product choice
Judge model	GPT-4o	Different family; avoids correlated failures
Rubric criteria	factual_correctness, code_accuracy, style	Three axes cover this task
Weights	0.5 / 0.3 / 0.2	Factual dominates; code accuracy matters; style is tiebreaker
Output schema	pydantic-validated JSON	Structured, typed, cacheable
Rubric hash	cx-42a9	Frozen on merge; bumps invalidate cache
Length bias check	nightly	Detects rubric drift
Cross-judge calibration	weekly	Pin agreement r > 0.85

After the rollout, the judge scores every prediction on three axes with a stable weighted overall, produces per-criterion breakdowns for forensics, and is version-controlled at the rubric level. The nightly length-bias check catches drift; the weekly cross-judge calibration keeps the judge honest against a different-family alternate.

Output:

Instrument	Frequency	Alert threshold	Owner
Judge score (per PR)	on merge	<baseline - 5%	PR author
Rubric hash	on rubric edit	new hash	Reviewer
Length-bias slope	nightly	> 0.02	ML infra on-call
Cross-judge Pearson r	weekly	< 0.85	ML infra on-call
Judge cost/PR	on run	> $3	Finance ops

Why this works — concept by concept:

Different-family judge — Claude Sonnet in production, GPT-4o as judge. Correlated failure modes are the biggest hidden risk with same-family judging; switching families breaks the correlation and gives you an independent view of quality. The 5-10 percent cost premium is worth the independence.
Weighted rubric with typed schema — 0.5 factual + 0.3 code + 0.2 style is task-specific; the weights encode "what matters most for this product." Pydantic validation on receive means malformed judge output surfaces immediately as an error instead of silently poisoning the metric.
Anti-bias instruction with paired example — telling the judge "you are blind to length" is worth about 3× more with an explicit paired example than without. LLMs are pattern-matchers; showing the pattern beats declaring it.
Nightly length-bias check — the judge is an artefact under active development. Model updates, prompt drift, and rubric edits all reintroduce bias. The nightly detector is the judge's CI; without it, the harness measuring quality drifts silently.
Cost — GPT-4o judge at ~$0.006/row × 300 golden rows = $1.80 per PR. Weekly cross-judge calibration with Claude Opus adds ~$3 per week. Nightly length-bias check is a regression on existing score data — $0. Total judge cost < $100/month for a mature setup.

Data analysis
Topic — data-analysis
Data-analysis problems on scorer calibration and bias

Practice →

Optimization Topic — optimization Optimization problems on judge cost and sampling

Practice →

5. CI integration + production monitoring

Evaluation is a build gate and a production sensor — one signal, two schedulers

The mental model in one line: a mature llm ci stack runs the same eval harness in two places — on every PR against the golden set (as a build gate that blocks regressions) and on a sample of live production traffic (as a drift detector that alerts on quality decay) — and the frameworks that glue the pieces together are Ragas for RAG-specific metrics, DeepEval for pytest-native regression tests, and promptfoo for lightweight prompt A/B. The engineering discipline is exactly the pattern you already run for every other pipeline: same code path, two schedulers, two alerting surfaces.

The four axes of CI-plus-production.

PR gate. The harness runs on every PR touching prompts, retrieval, or model config. It scores the change against the golden set, compares to the previous baseline, and blocks the merge if aggregate score drops more than the configured threshold (typically 5 percent).
Nightly full-set. A cron job runs the full golden set with the stronger judge every 24 hours. Feeds the baseline the PR gate compares to; catches slow drift that PR-level sampling cannot see.
Production sampler. A sidecar samples 0.5–2 percent of live production conversations, routes them through the same harness (cheap judge + cosine), and writes scores to a rolling metrics table. This is your "did the metric decay in prod even though CI is green?" sensor.
Alerting. Regression alerts (5 percent aggregate drop) page on-call; production drift alerts (24-hour rolling mean drops 5 percent vs last-week baseline) page on-call; per-class alerts (any class drops 10 percent) page on-call. All feed the same runbook.

The PR gate — the anatomy.

Trigger. GitHub Actions on pull_request with path filter (prompts/**, retrieval/**, model_config/**).
Sample. 20 percent of the golden set (typically 300-400 rows) for a 30-second wall clock.
Score. Cosine + judge-on-miss for cost efficiency; per-row scores land in a metrics table with the PR SHA and golden-set SHA.
Compare. Load the previous main-branch baseline (last-merged run) and compute the delta. Aggregate delta ≥ -5 percent = pass; less than that = block.
Comment. Post the per-class delta table into the PR comment. The author sees exactly which class regressed.

The nightly full-set — the anatomy.

Trigger. Cron 02:00 UTC daily. Runs against main at HEAD.
Scope. Full golden set (2000 rows) with the stronger judge (GPT-4o).
Store. Metric row per (run_id, golden_set_sha, row_id, judge_score, cosine_score).
Baseline update. Aggregate mean over the run becomes the "previous baseline" the next PR run compares to.
Report. Slack post with per-class means + week-over-week deltas.

The production sampler — the anatomy.

Sample. 1 percent of live conversations, uniformly random. Higher sample for lower-traffic tenants to keep the metric statistically meaningful.
Ground truth. For sampled conversations, there is no expected answer. Score cosine against retrieved context and judge against the retrieved answer (RAG-style self-consistency), or use the LLM-judge with a "no reference" rubric ("was this answer useful given the question?").
Alert. 24-hour rolling mean drops 5 percent vs the same day-of-week last week. Rolling-window baseline handles weekly seasonality.
Cost. ~$50/week for a mid-size product; the cheapest reliable prod-quality sensor.

The frameworks — when to reach for each.

Ragas. Purpose-built for RAG evaluation. Metrics: answer_relevancy, faithfulness (does the answer contradict the retrieved context?), context_precision, context_recall. Uses LLM-as-judge under the hood. Great starter kit if you're building RAG and don't want to write the judge from scratch.
DeepEval. pytest-native LLM eval library. Write assert_test(TestCase(input=..., expected=..., metrics=[GEval(...)])) in your test files; run with pytest. Fits the "make it a pytest" instinct of most Python teams.
promptfoo. CLI + YAML for prompt A/B. promptfoo eval runs matrix of prompts × models × inputs and produces an HTML report. Best for lightweight prompt tuning, not production monitoring.
Roll your own. Once you have a mature golden set + judge + CI hook, the framework overhead is often not worth it. Ragas and DeepEval win on time-to-first-metric; a hand-rolled harness wins on flexibility once you know exactly what you want.

Common CI + production failure modes.

Baseline drift. The "previous baseline" is the last-merged score, which can be a lucky high or an unlucky low. Fix: use a 7-day moving average as the baseline, not a single run.
Flake tolerance. A one-time judge hiccup causes a false-fail PR. Fix: retry the harness once on transient judge errors; fail only on the second attempt.
Silent judge upgrade. OpenAI ships a new GPT-4o checkpoint under the same name; scores shift 2 percent overnight. Fix: pin the judge model to a dated version and log the version on every run.
Prod sampler skew. The 1 percent sample is over-representative of one tenant. Fix: stratified sample per tenant + per topic class.

What senior interviewers listen for.

Do you frame CI + prod monitoring as "same harness, two schedulers" rather than as two separate systems? — senior signal.
Do you volunteer rolling-baseline as the answer to flaky one-run baselines? — senior signal.
Do you name Ragas / DeepEval / promptfoo and know when to reach for each? — senior signal.
Do you distinguish PR gate (block regression) from production monitor (detect drift) as different problems with a shared implementation? — required answer.

Worked example — pytest-style regression harness with pass/fail gate

Detailed explanation. Build the PR-gate harness as a pytest suite. Each test case is a golden-set row; the metric is the judge score; the gate is aggregate mean vs baseline. The suite runs in the PR CI, blocks the merge on a 5 percent aggregate drop, and posts a per-class delta table into the PR comment.

Test shape. One pytest parameterization per row (or one aggregate test with a summary).
Baseline. Loaded from a JSON artefact stored in the CI cache, updated on nightly.
Gate. assert mean_score >= baseline - 0.05.
Comment. Formatted markdown table via gh pr comment.

Question. Implement the pytest harness plus the CI YAML plus the PR comment formatter. Show the exact fail behaviour for a 6 percent aggregate drop.

Input.

Parameter	Value
Framework	pytest + custom fixtures
Golden set	400 rows (20% sample of 2000)
Judge	GPT-4o (temperature=0)
Baseline	last-nightly aggregate = 0.84
Gate threshold	-0.05 (block if mean < 0.79)
PR result (simulated)	mean = 0.78 (6 percent drop)

Code.

# tests/eval/test_regression.py
import json
import statistics
from pathlib import Path

import pytest

from harness.load_golden import load_golden
from harness.predict import predict_batch
from harness.llm_judge import judge_row
from harness.cosine import cosine_batch


GATE_DELTA = 0.05
BASELINE_PATH = Path(".ci/eval_baseline.json")


@pytest.fixture(scope="session")
def golden():
    return load_golden(Path("golden/support_v3.jsonl"), expected_hash="b83a11")


@pytest.fixture(scope="session")
def predictions(golden):
    inputs = [r["input"] for r in golden]
    return dict(zip((r["row_id"] for r in golden), predict_batch(inputs)))


@pytest.fixture(scope="session")
def baseline():
    return json.loads(BASELINE_PATH.read_text()) if BASELINE_PATH.exists() else {"mean": None}


def _sample(golden, k: int = 400, seed: int = 42):
    import random
    rng = random.Random(seed)
    return rng.sample(golden, k=min(k, len(golden)))


def test_aggregate_regression(golden, predictions, baseline, record_property):
    sample = _sample(golden, k=400)
    scores = []
    per_class = {}
    for row in sample:
        j = judge_row(
            input_text=row["input"],
            expected=row["expected_output"],
            prediction=predictions[row["row_id"]],
        ).score
        scores.append(j)
        topic = row["metadata"]["topic"]
        per_class.setdefault(topic, []).append(j)

    mean_score = statistics.mean(scores)
    delta = mean_score - (baseline["mean"] or mean_score)

    record_property("mean", mean_score)
    record_property("baseline", baseline["mean"])
    record_property("delta", delta)
    record_property("per_class", {t: statistics.mean(s) for t, s in per_class.items()})

    if baseline["mean"] is not None:
        assert delta >= -GATE_DELTA, (
            f"regression: mean {mean_score:.3f} < baseline {baseline['mean']:.3f} - {GATE_DELTA}"
        )

# .github/workflows/eval-gate.yml
name: eval-gate
on:
  pull_request:
    paths:
      - 'prompts/**'
      - 'retrieval/**'
      - 'model_config/**'
      - 'golden/**'

jobs:
  eval:
    runs-on: ubuntu-latest
    timeout-minutes: 10
    env:
      OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.12'
      - run: pip install -r requirements-eval.txt
      - name: restore baseline
        uses: actions/cache@v4
        with:
          path: .ci/eval_baseline.json
          key: eval-baseline-${{ github.base_ref }}
      - name: run eval suite
        run: pytest tests/eval/ -q --junitxml=eval-report.xml
      - name: post PR comment
        if: always()
        run: python harness/pr_comment.py eval-report.xml

# harness/pr_comment.py
import os
import sys
import xml.etree.ElementTree as ET
import subprocess


def load_report(path):
    root = ET.parse(path).getroot()
    props = {p.get("name"): p.get("value") for p in root.iter("property")}
    return props


def markdown(props):
    mean = float(props["mean"])
    base = props.get("baseline") or "n/a"
    delta = float(props["delta"])
    per_class = eval(props["per_class"])
    verdict = "PASS" if delta >= -0.05 else "BLOCK"
    tbl = "\n".join(f"| {t} | {s:.3f} |" for t, s in sorted(per_class.items()))
    return f"""### LLM Eval Gate: {verdict}

| metric | value |
|---|---|
| mean | {mean:.3f} |
| baseline | {base} |
| delta | {delta:+.3f} |
| threshold | -0.050 |

**Per-class means**

| topic | mean |
|---|---|
{tbl}

_Judge: gpt-4o • Rubric: c93f11 • Golden: b83a11_
"""


if __name__ == "__main__":
    props = load_report(sys.argv[1])
    body = markdown(props)
    pr = os.environ["GITHUB_REF"].split("/")[-2]
    subprocess.run(["gh", "pr", "comment", pr, "--body", body], check=True)

Step-by-step explanation.

The pytest suite loads the golden set, samples 400 rows deterministically (seed=42), scores each with the judge, aggregates the mean, and asserts delta >= -0.05 against the baseline. The record_property calls emit per-run metadata into the JUnit XML for the PR comment step to read.
The CI workflow triggers only on PRs touching the eval-relevant paths, saving budget when someone edits the README. The actions/cache step restores the eval_baseline.json from the base branch's cache; the nightly job is responsible for writing it there.
When the PR run reports mean=0.78 against baseline 0.84 (a 6 percent drop, exceeding the 5 percent gate), the assertion fails, pytest exits non-zero, GitHub Actions marks the eval-gate check as failed, and the merge is blocked by the branch-protection rule.
The post PR comment step runs on always() so a failed run still posts the diagnostic. The comment shows aggregate + per-class means; the author sees "shipping dropped from 0.86 to 0.72, product held at 0.88" and knows immediately where to look.
Rolling the baseline: the nightly full-set run writes a fresh eval_baseline.json (aggregated over 2000 rows with the stronger judge) into the CI cache. The next day's PR runs compare against the fresh baseline; the "moving target" concern is bounded because the nightly is deterministic (fixed golden set, fixed judge prompt, temperature 0).

Output.

=========================== test session starts ===========================
tests/eval/test_regression.py::test_aggregate_regression FAILED [100%]

================================ FAILURES ================================
______________________ test_aggregate_regression ______________________
> assert delta >= -GATE_DELTA
E    AssertionError: regression: mean 0.780 < baseline 0.840 - 0.05
E    assert -0.06 >= -0.05

------- PR comment posted -------
### LLM Eval Gate: BLOCK

| metric | value |
|---|---|
| mean | 0.780 |
| baseline | 0.840 |
| delta | -0.060 |
| threshold | -0.050 |

Per-class means:
| topic | mean |
|---|---|
| account | 0.815 |
| billing | 0.782 |
| other | 0.821 |
| product | 0.883 |
| shipping | 0.723  ← centre of regression |

Rule of thumb. The gate is a pytest — the same test runner your Python devs use for unit tests. Do not build a bespoke CI system; the moment the harness feels like an isolated framework, the team stops trusting it.

Worked example — production sampler with rolling-baseline alert

Detailed explanation. Build the production sampler as an Airflow task that runs hourly, pulls a 1 percent stratified sample from the last hour of live conversations, scores each via the same cosine + judge harness, and writes rows into a metrics warehouse table. A downstream alert compares the 24-hour rolling mean against the same-time-last-week baseline and pages on-call if the drop exceeds 5 percent.

Sampler. Hourly cron; 1 percent stratified by tenant + topic.
Scoring. Cosine against retrieved context + LLM-judge with "no reference" rubric.
Storage. prod_eval_scores(conversation_id, ts, tenant, topic, cosine, judge_score, run_id).
Alert. 24-hour rolling mean vs same day-of-week last week; > 5% drop = page.

Question. Build the Airflow DAG, the scoring task, and the alert query. Show how the alert fires on a synthetic 6 percent drop.

Input.

Parameter	Value
Sample rate	1% stratified
Cadence	hourly
Judge	gpt-4o-mini (cheap; sample is large)
Storage	Snowflake `analytics.prod_eval_scores`
Alert query	24h rolling mean vs same-day last week
Alert threshold	-5%

Code.

# dags/prod_eval_sampler.py
from datetime import datetime, timedelta

from airflow import DAG
from airflow.decorators import task
from airflow.providers.snowflake.hooks.snowflake import SnowflakeHook

default_args = {"owner": "ml-infra", "retries": 2, "retry_delay": timedelta(minutes=5)}


with DAG(
    dag_id="prod_eval_sampler",
    schedule="@hourly",
    start_date=datetime(2026, 6, 1),
    catchup=False,
    default_args=default_args,
    tags=["llm-eval", "prod"],
) as dag:

    @task
    def sample_conversations(**ctx) -> list[dict]:
        window_end = ctx["logical_date"]
        window_start = window_end - timedelta(hours=1)
        hook = SnowflakeHook(snowflake_conn_id="analytics")
        return hook.get_records("""
            WITH candidates AS (
              SELECT conversation_id, tenant, topic, user_turn, model_answer,
                     retrieved_context, ts
              FROM   analytics.conversations
              WHERE  ts >= %s AND ts < %s
            ),
            stratified AS (
              SELECT *,
                     ROW_NUMBER() OVER (PARTITION BY tenant, topic ORDER BY random()) AS rn,
                     COUNT(*) OVER (PARTITION BY tenant, topic) AS n
              FROM   candidates
            )
            SELECT conversation_id, tenant, topic, user_turn, model_answer,
                   retrieved_context, ts
            FROM   stratified
            WHERE  rn <= GREATEST(1, ROUND(n * 0.01));
        """, parameters=(window_start, window_end))

    @task
    def score_and_write(rows: list[dict], **ctx):
        from harness.cosine import cosine_batch
        from harness.llm_judge import judge_no_reference

        scores = []
        for r in rows:
            cos = cosine_batch([r["retrieved_context"]], [r["model_answer"]])[0]
            j = judge_no_reference(
                question=r["user_turn"],
                context=r["retrieved_context"],
                answer=r["model_answer"],
            )
            scores.append({
                "conversation_id": r["conversation_id"],
                "ts": r["ts"],
                "tenant": r["tenant"],
                "topic": r["topic"],
                "cosine": float(cos),
                "judge_score": float(j.score),
                "run_id": ctx["run_id"],
            })

        hook = SnowflakeHook(snowflake_conn_id="analytics")
        hook.insert_rows(table="analytics.prod_eval_scores", rows=[list(s.values()) for s in scores])

    score_and_write(sample_conversations())

-- alerts/prod_drift.sql — 24h rolling vs same-day last week
WITH today AS (
  SELECT AVG(judge_score) AS mean_today
  FROM   analytics.prod_eval_scores
  WHERE  ts >= CURRENT_TIMESTAMP - INTERVAL '24 hours'
),
last_week AS (
  SELECT AVG(judge_score) AS mean_last_week
  FROM   analytics.prod_eval_scores
  WHERE  ts >= CURRENT_TIMESTAMP - INTERVAL '8 days'
    AND  ts <  CURRENT_TIMESTAMP - INTERVAL '7 days'
)
SELECT today.mean_today,
       last_week.mean_last_week,
       today.mean_today - last_week.mean_last_week AS delta,
       CASE
         WHEN today.mean_today - last_week.mean_last_week <= -0.05 THEN 'PAGE'
         WHEN today.mean_today - last_week.mean_last_week <= -0.03 THEN 'WARN'
         ELSE 'OK'
       END AS status
FROM   today, last_week;

Step-by-step explanation.

The sampler queries the last hour of live conversations, stratified 1 percent per (tenant, topic) pair to prevent one high-volume tenant from dominating the sample. The stratification uses ROW_NUMBER() + COUNT() per partition to compute the target sample size in-database.
Each sampled conversation carries the retrieved context + the model answer. Cosine is computed between context and answer (a proxy for faithfulness — is the answer grounded in what was retrieved?). Judge uses a "no reference" rubric that asks "given the question and the retrieved context, was the answer useful and faithful?"
Rows land in analytics.prod_eval_scores with run_id for lineage. Downstream dashboards slice by tenant, topic, and time; the alert query is a straight aggregation on the same table.
The alert query compares the 24-hour rolling mean against the same-day-last-week window. The last-week baseline handles weekly seasonality (Monday morning traffic ≠ Sunday evening). A -0.03 warn triggers a Slack post; a -0.05 page fires PagerDuty.
On the synthetic scenario (today mean=0.78, last week same window=0.85), delta=-0.07 → PAGE. The on-call receives a link to the drift dashboard, the per-tenant breakdown, and the runbook that walks them from "which tenant?" to "which topic?" to "which change landed in the last 7 days?"

Output.

mean_today	mean_last_week	delta	status
0.78	0.85	-0.07	PAGE

Rule of thumb. Production monitoring is the same harness as CI, just pointed at a different data source. Do not build a second scoring stack — build one, run it twice, alert on two schedules.

Worked example — using Ragas + DeepEval side by side

Detailed explanation. For teams that want a fast start without building the harness from scratch, Ragas and DeepEval are the two most-adopted 2026 frameworks. Ragas is RAG-first with pre-built metrics (faithfulness, answer_relevancy, context_precision, context_recall); DeepEval is pytest-native and general-purpose (any-metric, any-scorer). Use both: Ragas for the RAG-specific axes, DeepEval as the test-runner glue.

Ragas metrics. faithfulness (does the answer contradict the context?), answer_relevancy (does it address the question?), context_precision (are retrieved chunks relevant?), context_recall (were the necessary chunks retrieved?).
DeepEval role. Wraps Ragas metrics inside pytest test cases; runs on PR; posts pass/fail.
When to reach for each. Ragas alone for a quick RAG dashboard; DeepEval alone for non-RAG tasks; both together for pytest-native RAG gating.

Question. Wire Ragas metrics into a DeepEval test suite that runs on PR. Show the exact test file, the config, and the failure output on a faithfulness drop.

Input.

Parameter	Value
Task	RAG chatbot
Ragas metrics	faithfulness, answer_relevancy
DeepEval integration	GEval wrapper
PR gate	faithfulness >= 0.85, answer_relevancy >= 0.80

Code.

# tests/eval/test_rag_ragas.py
import json
from pathlib import Path

import pytest
from deepeval import assert_test
from deepeval.metrics import RagasMetric
from deepeval.test_case import LLMTestCase


GOLDEN = json.loads(Path("golden/rag_support_v3.jsonl").read_text().splitlines()[0])


faithfulness_metric = RagasMetric(threshold=0.85, metric="faithfulness", model="gpt-4o")
relevancy_metric    = RagasMetric(threshold=0.80, metric="answer_relevancy", model="gpt-4o")


@pytest.mark.parametrize("row", [json.loads(l) for l in Path("golden/rag_support_v3.jsonl").read_text().splitlines()])
def test_ragas(row):
    tc = LLMTestCase(
        input=row["input"],
        actual_output=row["prediction"],
        expected_output=row["expected_output"],
        retrieval_context=row["retrieved_context"],
    )
    assert_test(tc, [faithfulness_metric, relevancy_metric])

# .github/workflows/rag-eval.yml
name: rag-eval
on: pull_request

jobs:
  ragas-deepeval:
    runs-on: ubuntu-latest
    env:
      OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.12'
      - run: pip install deepeval ragas
      - run: pytest tests/eval/test_rag_ragas.py -q --html=rag-eval.html --self-contained-html
      - uses: actions/upload-artifact@v4
        with:
          name: rag-eval-report
          path: rag-eval.html

Step-by-step explanation.

LLMTestCase from DeepEval carries the four fields Ragas needs: input (the user question), actual_output (the model's answer), expected_output (the golden), retrieval_context (the chunks the RAG retrieved). Ragas metrics compute against these fields.
RagasMetric wraps Ragas's faithfulness and answer_relevancy metrics with a DeepEval-compatible threshold and a scoring model (GPT-4o). assert_test runs the metrics and raises pytest failures if any threshold is missed.
The @pytest.mark.parametrize fans out one test per row. Pytest runs them in parallel with -n auto; a 500-row set completes in ~2 minutes at 20-way parallelism.
On a faithfulness regression (say, a retrieval index change that surfaces less-relevant chunks), the faithfulness metric drops on many rows; DeepEval fails the tests; the HTML report shows exactly which rows fell below 0.85. The PR is blocked.
Ragas + DeepEval is the fastest path from "we have a RAG chatbot" to "we have a gated CI harness" — typically a day of work vs a week for hand-rolled. The trade-off is less flexibility on the metric side; once you outgrow the built-in metrics, migrate to a hand-rolled harness with the same pytest wiring.

Output.

tests/eval/test_rag_ragas.py::test_ragas[row-0] PASSED
tests/eval/test_rag_ragas.py::test_ragas[row-1] PASSED
tests/eval/test_rag_ragas.py::test_ragas[row-2] FAILED

    faithfulness = 0.72 (threshold 0.85)
    answer_relevancy = 0.88 (threshold 0.80)

    Failure reason: the answer contained a URL not present in the retrieved
    context — hallucinated citation.

Rule of thumb. Ragas + DeepEval is the "buy" decision; hand-rolled is the "build" decision. Buy when time-to-first-metric matters more than metric flexibility. Migrate to build when the built-in metrics don't cover a task-specific axis.

Senior interview question on CI + production monitoring

A senior interviewer might ask: "You join a team with a green LLM eval dashboard but customers complaining about quality. Walk me through the first two weeks — how you'd diagnose the CI-vs-prod gap, what monitoring you'd add, and how you'd close the loop."

Solution Using a "same harness, two schedulers, three alerts" hardening plan

# hardening-plan.yaml — 2 weeks from "CI green, customers unhappy" to closed-loop eval
version: v1.0

week_1:
  day_1:
    - audit golden set: does it reflect real user turns?
    - if not: sample 200 rows from real live conversations, SME-relabel
  day_2:
    - stand up production sampler (1% stratified, hourly Airflow task)
    - deploy prod scoring: cosine + cheap judge, no-reference rubric
  day_3:
    - build drift alert query: 24h rolling vs same-day last week
    - integrate to PagerDuty; test with historical data
  day_4:
    - retrofit PR gate baseline: switch from single-run to 7-day moving avg
    - reason: single-run baseline is flake-prone; moving avg is stable
  day_5:
    - retrofit per-class alerts: any class drops 10% = page (finer signal)
  weekend:
    - soak: watch the metrics; expect ~2 alerts (real signal + calibration noise)

week_2:
  day_1:
    - reconcile: for the top-3 customer-reported issues, is the golden set covering them?
    - if not: add rows in the missing class + retrain the SME labelers
  day_2:
    - cross-judge calibration: run 100 rows through GPT-4o and Claude Opus
    - if r < 0.85: tighten rubric; publish updated hash
  day_3:
    - length-bias detector: run against last 30 days of judge scores
    - if bias > 0.02: append anti-bias language; re-baseline
  day_4:
    - Ragas / DeepEval evaluation: does the built-in RAG suite catch what our custom does not?
    - integrate the specific Ragas metric that catches a real failure
  day_5:
    - runbook + on-call rotation: eval-gate + prod-drift on the same paged rotation

Step-by-step trace.

Day	Activity	Signal added
1	Golden-set audit	Coverage of real turns
2	Prod sampler	Continuous prod metric
3	Drift alert	Weekly-seasonality-aware
4	Rolling baseline	Flake-resistant PR gate
5	Per-class alerts	Localized regression detection
Week 2	Judge calibration + bias + framework fill-ins	Judge trustworthiness

After two weeks, the team has: (1) a golden set that reflects real user turns, (2) a production sampler catching drift on a 24-hour horizon, (3) a PR gate with a stable rolling baseline, (4) per-class alerts for localized regressions, and (5) a calibrated judge with instrumented length-bias detection. The CI-vs-prod gap closes; customer complaints line up with metric drops.

Output:

Signal	Before	After
Golden set reflects real turns	no	yes
Production monitoring	none	1% hourly sample
PR baseline	last-run	7-day moving avg
Per-class alerts	none	10% drop = page
Cross-judge calibration	never	weekly
Length bias monitored	no	nightly
CI-prod gap	large	closed

Why this works — concept by concept:

Same harness, two schedulers — the PR gate and the prod monitor share code paths. Debugging is easier because a bug in the harness surfaces in both places at once; the team is not maintaining two scoring stacks.
Rolling baseline — 7-day moving average absorbs individual-run noise. A single unlucky run does not lock the team out of every future PR; a real regression persists through the moving average and still triggers the gate.
Stratified prod sampler — 1 percent per (tenant, topic) prevents a single high-volume tenant from dominating. The metric is representative of the full traffic distribution, not just of the largest customer.
Cross-judge calibration + length-bias check — the judge is a moving target too. Weekly cross-model calibration keeps the judge honest; nightly length-bias check catches rubric drift before it corrupts weeks of gating.
Cost — 2 senior-engineer weeks + ~$200/month in judge + prod-sampler API cost. The alternative (unbounded customer complaints + eng escalation) is measured in engineering-days per week; the eval stack pays for itself the first time it catches a bad deploy.

ETL
Topic — etl
ETL problems on CI harnesses and drift monitoring

Practice →

Real-time analytics
Topic — real-time-analytics
Real-time analytics problems on rolling baselines and alerts

Practice →

Cheat sheet — LLM eval recipes

Golden set YAML template. path: golden/support_v3.jsonl + version: v3.2-sha-e1f2c9 + rows: 300 + class_balance: {billing: 60, account: 60, product: 60, shipping: 60, other: 60} + growth: +20 rows/week + curation: SME-labelled + LLM-augmented. Every row is (input, expected_output, metadata) where metadata carries topic, difficulty, language, source_ticket_id, sme_labeler, reviewer, retrieval_snapshot_id, forbidden_patterns, style_tag. Store as JSONL in git; content-hash on canonicalized JSON; tag per release; verify hash on load.
Cosine scorer 20-line skeleton. Pin embedding: text-embedding-3-large@2024-11-01. Batch 100 at a time; L2-normalize; dot product for cosine. Threshold per language (en=0.86, es/fr=0.83); calibrate against 200 correct + 200 contradiction pairs; sweep F1 or cost-weighted. Judge-on-miss to save cost. Every row emits (row_id, topic, cosine, cos_pass, judge_score, final_pass, embedding_version).
LLM-as-judge prompt template. System: explicit 2-4 criterion rubric with (0.0-1.0) scale + failure modes named + "you are blind to length" + paired-example demonstration. User: INPUT: ... EXPECTED: ... PREDICTION: .... Response format: {"type": "json_object"}; temperature 0; pydantic-validate on receive. Weighted overall score, e.g. 0.5*correctness + 0.3*completeness + 0.2*style. Hash the prompt; bump on every edit.
Judge model choice. Use a different-family model from production (GPT-4o judge for Claude Sonnet prod, Claude Opus judge for GPT-4o prod). Cheap sample judge (gpt-4o-mini) on PR + strong nightly judge (gpt-4o or claude-opus) for calibration. Cross-judge Pearson r ≥ 0.85 before trusting either.
pytest regression harness. assert_test(TestCase(input=..., expected=..., metrics=[GEval(...)])) with DeepEval or a hand-rolled pytest.mark.parametrize. Gate: mean >= baseline - 0.05 against 7-day moving-average baseline. Post per-class delta table into PR via gh pr comment. Cache the baseline via actions/cache keyed on base branch.
CI eval workflow. Trigger: pull_request on prompts/**, retrieval/**, model_config/**, golden/**. Sample 20 percent of the golden set (~400 rows) for the PR run; nightly full-set with the stronger judge. Timeout 5-10 minutes; parallelism 20; retry once on transient judge errors. Post pass/fail comment always.
Production sampler. Hourly Airflow task; 1 percent stratified sample per (tenant, topic) via ROW_NUMBER() OVER (PARTITION BY tenant, topic ORDER BY random()). Score with cheap judge (gpt-4o-mini) + cosine on retrieved context. Write to prod_eval_scores(conversation_id, ts, tenant, topic, cosine, judge_score, run_id).
Drift alert query. 24h_rolling_mean - same_day_last_week_mean <= -0.05 → PAGE, <= -0.03 → WARN. Use INTERVAL '8 days' to INTERVAL '7 days' for same-day-last-week baseline (handles weekly seasonality). Alert on per-class rolling means too — any class drops 10 percent = page.
Sizing the eval budget. annual_cost = golden_set_size × judge_cost_per_row × runs_per_week × 52. Levers: sample 20 percent on PR + full nightly (cuts 5×); cheap judge on sample + expensive nightly (cuts another 5×); reduce sensitive runs to weekly (cuts another 5×). Target: total eval < 5 percent of the LLM-feature's production inference bill.
Cross-judge calibration. Weekly: score 100 rows through two judges (different families), compute Pearson r + Cohen's κ at 0.7. r ≥ 0.85 → pass; r in [0.7, 0.85) → tighten rubric; r < 0.7 → rubric is broken. Log the r value; alert on deterioration week-over-week.
Length-bias detector. Nightly regression of judge_score on standardized len(prediction) while controlling for correctness. Slope > 0.02 = biased. Fix: append "you are blind to length" plus a paired example (short + long same-content predictions scored identically). Re-run detector; slope should drop below 0.01.
Ragas / DeepEval / promptfoo — when to reach for each. Ragas: RAG-specific metrics (faithfulness, answer_relevancy, context_precision, context_recall) with judge under the hood — best when you need a fast RAG dashboard. DeepEval: pytest-native, wraps any metric — best when your team already writes pytest. promptfoo: CLI matrix runner for prompt × model × input — best for lightweight prompt A/B, not production monitoring. Hand-rolled: once you outgrow built-in metrics, roll your own with the same pytest wiring.
Prompt versioning discipline. Every prompt (production, judge, RAG-retrieval rewrite) is a hashed artefact in git. Every metric row carries the prompt_hash; comparing across hashes is disallowed. Prompt edits go through PR review; the eval-gate runs against the new hash before merge.
Failure modes catalogue. Golden set drift (stagnant set diverges from prod distribution) → weekly production sampling into curation queue. Embedding version drift (silent OpenAI checkpoint change) → pin embedding to dated version. Judge bias creep (length preference returns after prompt edit) → nightly length-bias detector. Baseline flake (single unlucky run) → 7-day moving-average baseline. Prompt hash mismatch (rubric edit not propagated) → harness verifies hash on load and aborts on mismatch.

Frequently asked questions

What is LLM-as-judge?

llm as judge is the pattern where a strong LLM (typically GPT-4o, Claude Opus, or Gemini Ultra) scores your production model's prediction against a golden expected answer using an explicit rubric, returning a structured JSON with a numeric score plus a rationale. Concretely, the judge sees (input, expected_output, prediction) and a rubric like ["correctness", "completeness", "style"], and returns {"score": 0.87, "rationale": "…", "breakdown": {...}}. It is the dominant modern LLM-eval metric because it handles semantic equivalence (paraphrases score high) and semantic opposition (contradictions score low) far better than cosine similarity or BLEU. The engineering discipline that makes it work is: version-controlled rubric, structured JSON output (response_format={"type": "json_object"}), temperature=0 for reproducibility, and a stronger or different-family judge model than production to avoid correlated failure modes. Judge cost per row is the finance-visible constraint — typically $0.001-0.01 per row depending on model + rubric length, so sample-on-PR + full-nightly is the default cost-control pattern.

Cosine similarity vs LLM-as-judge — which do I use for what?

cosine similarity is the cheap, fast, deterministic first-pass metric: embed both the expected and the prediction with a pinned embedding model (text-embedding-3-large is the 2026 default), L2-normalize, dot-product, threshold. It costs a fraction of a cent per row, catches structural hallucinations (wrong entities, wrong URLs, missing named entities) beautifully, and is the right gate for "did we regress structurally by 10 percent aggregate." But it fails on paraphrase (semantically identical but different surface form) and contradiction (opposite polarity but similar surface form) — a "yes" and a "no" can embed similarly, and the metric silently misses the polarity flip. llm as judge is the truth arbiter for those cases: expensive but semantically correct. The mature pipeline runs cosine first (per-row, on every row), escalates to the judge only on cosine misses (a "judge-on-miss" pattern that cuts judge cost by ~7×), and uses both scores in the aggregate. Never use cosine alone as a truth metric; never use judge alone as a first-pass filter — the price-performance sweet spot is the cascade.

How big should my golden set be?

The floor is 200 rows for a mid-complexity task; the working default is 300-500; regression-forensics teams run at 1000-2000. The math: at N=20 rows with per-row score standard deviation of ~0.08, the two-sided 95 percent confidence half-width is 0.035 — meaning any regression under 3.5 percent is indistinguishable from noise. At N=200 the half-width drops to 0.011 — a 5 percent regression is caught with a 4× margin, and per-class slices (10-topic buckets × 20 rows each) have enough rows for statistical power. The bigger reason is class coverage: a 20-row demo set has no coverage at all, so localized regressions in unrepresented classes vanish silently; a 300-row balanced set with 5 topic classes × 60 rows each catches localized issues before they ship. Growth cadence: +20 rows/week from production sampling with SME review. Warning sign: if you cannot slice your metric by topic / difficulty / language and get a statistically meaningful number, your golden set is too small.

Do I need to re-evaluate on every prompt change?

Yes, on every PR that touches prompts, retrieval config, model config, or the golden set itself. Prompts are the highest-leverage lever on LLM behaviour; a two-word rewording of a system prompt can silently reshape entire answer classes. The mature pattern: PR gate runs a 20 percent sample (~400 rows) with the cheap judge in ~30 seconds and blocks the merge on a 5 percent aggregate drop; nightly full-set (2000 rows) with the stronger judge recalibrates the baseline. Skipping the PR gate means the first time you notice a bad prompt is when a customer opens a support ticket. Cost model: for a 400-row PR sample with a gpt-4o-mini judge at ~$0.006/row, each PR eval costs ~$2.40. At 40 PRs per week that is ~$5,000/year — a rounding error next to the cost of a single production incident. Exception: hotfix prompts that touch a well-understood, narrow behaviour can bypass the gate with an explicit override + a manual eval commitment within 24 hours. Never let the exception become the rule.

How do I detect LLM regression in production?

Stand up the same eval harness against a 1 percent stratified sample of live traffic, run hourly via Airflow / Dagster / Prefect, write per-conversation scores to a warehouse table, and alert on a 24-hour rolling mean drop vs same-day-last-week. Stratify by (tenant, topic) so a single high-volume tenant does not dominate. Score with the same cheap judge you use on the PR sample, so the production and CI metrics are directly comparable. Alert at two thresholds: -3 percent = WARN (Slack post); -5 percent = PAGE (PagerDuty). Add per-class alerts on any topic dropping more than 10 percent — localized regressions hide inside a healthy-looking aggregate. The reference alert query compares AVG(judge_score) WHERE ts >= NOW() - INTERVAL '24 hours' against AVG(judge_score) WHERE ts >= NOW() - INTERVAL '8 days' AND ts < NOW() - INTERVAL '7 days' — the last-week baseline handles weekly traffic seasonality. Cost: ~$50/week for a mid-size product; the cheapest reliable prod-quality sensor.

Should I use Ragas / DeepEval / promptfoo or roll my own harness?

Start with a framework — either Ragas (RAG-first with built-in faithfulness, answer_relevancy, context_precision, context_recall metrics) or DeepEval (pytest-native, general-purpose, wraps Ragas + any other metric). Ragas alone for a fast RAG dashboard when you have not yet decided on the judge / harness plumbing. DeepEval alone for non-RAG tasks (classification, extraction, summarization) where the built-in metrics fit. Both together for pytest-native RAG gating — DeepEval's LLMTestCase + Ragas's RagasMetric in a pytest file, gated by assert_test. promptfoo is CLI + YAML for prompt A/B — great for lightweight tuning, not production monitoring. Roll your own once you have (a) a mature golden set, (b) a task-specific rubric that no framework's built-in metric covers, and (c) a team comfortable maintaining Python infrastructure. Migration path from framework → hand-rolled is easy: keep the pytest wiring; swap the metric implementations from RagasMetric to your own scorer. The one thing not to do: build a bespoke non-pytest harness first — nobody trusts a system they cannot run locally with pytest.

Practice on PipeCode

Drill the ETL practice library → for the golden-set curation, versioned dataset, and pipeline-scheduler problems senior interviewers love.
Rehearse on the data-analysis practice library → for the class-balance, threshold-calibration, and per-slice-metric problems that motivate the LLM-eval harness in the first place.
Sharpen the tuning axis with the optimization practice library → for the eval-cost, judge-sampling, and CI-throughput problems that decide whether the harness scales.
Stack the prerequisites against PipeCode's broader 450+ data-engineering catalogue to anchor the golden-set + judge + CI intuition against real graded inputs.

Lock in LLM-evaluation muscle memory

Framework docs explain metrics. PipeCode drills explain the decision — when cosine is enough, when the judge must kick in, when the golden set is too small to trust, when the PR gate is the wrong place to catch drift. Pipecode.ai is Leetcode for Data Engineering — pattern-first practice tuned for the production trade-offs senior data engineers actually face.

Practice ETL problems →
Practice data-analysis problems →

DEV Community: Gowtham Potureddi

SQL MERGE / UPSERT / ON CONFLICT: Dialect-Aware Idempotent Writes

1. Why idempotent writes matter in 2026

The sql merge upsert mental model — idempotency as a contract, the two loading anti-patterns, and where MERGE shows up in every real DE pipeline

Worked example — the race condition in the naive SELECT-then-INSERT pattern

Worked example — idempotent replay of a Kafka batch

Worked example — a dbt incremental model that compiles to a MERGE

sql merge upsert interview question on choosing between MERGE and native UPSERT

Solution Using dual-endpoint architecture — INSERT ... ON CONFLICT for OLTP, MERGE INTO for warehouse

2. SQL Standard MERGE anatomy

sql standard merge — the ANSI SQL:2003 grammar, the three row-level branches, and how WHEN NOT MATCHED BY SOURCE extends the two-branch UPSERT into a three-way sync

Worked example — the two-branch MERGE (UPSERT) on Snowflake

Worked example — the three-branch MERGE with DELETE (full sync)

Worked example — conditional UPDATE using WHEN MATCHED AND cond

sql standard merge interview question on writing a three-branch MERGE

Solution Using three-branch MERGE with soft-delete on WHEN NOT MATCHED BY SOURCE

3. Postgres ON CONFLICT + MySQL ON DUPLICATE KEY + SQLite UPSERT

The OLTP-native sql upsert — postgres on conflict, mysql on duplicate key update, and sqlite upsert — three spellings of the same atomic conflict-resolution primitive

Worked example — the atomic counter increment on Postgres

Worked example — LWW UPSERT with timestamp guard on Postgres

Worked example — MySQL UPSERT with the row alias

postgres on conflict interview question on multi-column conflict target

Solution Using composite conflict target + atomic increment via EXCLUDED

4. Snowflake + BigQuery + Databricks MERGE

Warehouse-native snowflake merge, bigquery merge, and Databricks MERGE — the same ANSI grammar, three different storage-rewrite models, three different cost surfaces

Worked example — dbt incremental on Snowflake, from raw batch to MERGE

Worked example — BigQuery MERGE with partition prune predicate

Worked example — Databricks Delta MERGE with tombstone deletes

bigquery merge interview question on cost-optimised MERGE

Solution Using dry-run + partition prune + cluster on merge key

5. Anti-patterns and dialect matrix

The sql merge upsert failure modes — SQL Server's cardinality bug, concurrent MERGE races, non-deterministic sources — and the 8-engine dialect matrix that ties it all together

Worked example — SQL Server MERGE cardinality bug and the CTE workaround

Worked example — race-safe concurrent UPSERT under 10K QPS

Worked example — the SCD Type 2 pattern with MERGE

mysql on duplicate key update interview question on choosing among UPSERT, MERGE, and REPLACE

Solution Using ON DUPLICATE KEY UPDATE with row alias + RETURNING via ROW_COUNT()

Cheat sheet — MERGE / UPSERT recipe list

Frequently asked questions

What's the difference between MERGE and UPSERT?

When should I use INSERT ... ON CONFLICT vs MERGE in Postgres?

Why does SQL Server's MERGE have known bugs?

How do I write an idempotent MERGE in Snowflake vs BigQuery?

What's the race condition in a naive UPSERT?

How do I handle SCD Type 2 with MERGE?

Practice on PipeCode

SQL EXPLAIN ANALYZE Deep Dive: Reading Execution Plans in Postgres, Snowflake, BigQuery

1. Why execution plans matter in 2026

The sql query optimizer mental model — cost-based optimisation, statistics, cardinality estimation, and why every senior read-strategy starts with a plan

Worked example — the day-one EXPLAIN on a mystery-slow query

Worked example — cost units are not milliseconds (the day-two lesson)

Worked example — bottom-up reading discipline

SQL interview question on when to reach for EXPLAIN ANALYZE

Solution Using EXPLAIN (ANALYZE, BUFFERS) on the slow and fast versions, diffing plan trees

2. Postgres EXPLAIN ANALYZE anatomy

postgres explain analyze — the four numbers per node, the six major node types, and how BUFFERS surfaces the spill triangle

Worked example — the Seq Scan that should have been an Index Scan

Worked example — the Sort that spills and how work_mem fixes it

Worked example — EXPLAIN (ANALYZE, BUFFERS) on a Hash Join bottleneck

postgres explain analyze interview question on a two-plan diff

Solution Using bottom-up plan diffing to isolate the regressing node

3. Snowflake QUERY_PROFILE

snowflake query profile — the operator stack, micro-partition pruning percent, and how local vs remote disk spill drives warehouse sizing

Worked example — the TableScan with 0% pruning that a CLUSTER BY cures

Worked example — the remote spill that a warehouse bump cures

Worked example — the Broadcast join that broke on data growth

snowflake query profile interview question on plan reading and warehouse sizing

Solution Using the top-down profile checklist with pruning, spill, and operator time %

4. BigQuery execution graph

bigquery execution plan — the stage-based DAG, slot-milliseconds, shuffle bytes, --dry_run cost estimation, and INFORMATION_SCHEMA.JOBS

Worked example — the full-table scan on a partitioned table

Worked example — the skewed join that dominates one stage

Worked example — moving from on-demand to a reservation for cost savings

bigquery execution plan interview question on cost + performance tuning

Solution Using stage-graph diagnosis + partition pruning + materialised view

5. Dialect matrix + 5-step reading strategy

read execution plan across four engines — Postgres, Snowflake, BigQuery, SQL Server — one query, four artefacts, one senior reading strategy

Worked example — reading the same query on all four engines

Worked example — the same query optimised differently on each engine

execution plan analysis interview question on cross-engine plan reading

The `sql merge upsert` mental model — idempotency as a contract, the two loading anti-patterns, and where MERGE shows up in every real DE pipeline

`sql merge upsert` interview question on choosing between MERGE and native UPSERT

Solution Using dual-endpoint architecture — `INSERT ... ON CONFLICT` for OLTP, `MERGE INTO` for warehouse

`sql standard merge` — the ANSI SQL:2003 grammar, the three row-level branches, and how `WHEN NOT MATCHED BY SOURCE` extends the two-branch UPSERT into a three-way sync

Worked example — conditional UPDATE using `WHEN MATCHED AND cond`

`sql standard merge` interview question on writing a three-branch MERGE

The OLTP-native `sql upsert` — `postgres on conflict`, `mysql on duplicate key update`, and `sqlite upsert` — three spellings of the same atomic conflict-resolution primitive

`postgres on conflict` interview question on multi-column conflict target

Warehouse-native `snowflake merge`, `bigquery merge`, and Databricks MERGE — the same ANSI grammar, three different storage-rewrite models, three different cost surfaces

`bigquery merge` interview question on cost-optimised MERGE

The `sql merge upsert` failure modes — SQL Server's cardinality bug, concurrent MERGE races, non-deterministic sources — and the 8-engine dialect matrix that ties it all together

`mysql on duplicate key update` interview question on choosing among UPSERT, MERGE, and REPLACE

The `sql query optimizer` mental model — cost-based optimisation, statistics, cardinality estimation, and why every senior read-strategy starts with a plan

Worked example — the day-one `EXPLAIN` on a mystery-slow query

SQL interview question on when to reach for `EXPLAIN ANALYZE`

Solution Using `EXPLAIN (ANALYZE, BUFFERS)` on the slow and fast versions, diffing plan trees

`postgres explain analyze` — the four numbers per node, the six major node types, and how `BUFFERS` surfaces the spill triangle

Worked example — the `Seq Scan` that should have been an `Index Scan`

Worked example — the `Sort` that spills and how `work_mem` fixes it

Worked example — `EXPLAIN (ANALYZE, BUFFERS)` on a `Hash Join` bottleneck

`postgres explain analyze` interview question on a two-plan diff

`snowflake query profile` — the operator stack, micro-partition pruning percent, and how local vs remote disk spill drives warehouse sizing

Worked example — the TableScan with 0% pruning that a `CLUSTER BY` cures

`snowflake query profile` interview question on plan reading and warehouse sizing

`bigquery execution plan` — the stage-based DAG, slot-milliseconds, shuffle bytes, `--dry_run` cost estimation, and `INFORMATION_SCHEMA.JOBS`

`bigquery execution plan` interview question on cost + performance tuning

`read execution plan` across four engines — Postgres, Snowflake, BigQuery, SQL Server — one query, four artefacts, one senior reading strategy

`execution plan analysis` interview question on cross-engine plan reading

The three shapes of `sql pagination` — infinite scroll, page-numbered UIs, and CSV-style exports — and why the OFFSET default fails all three at scale

`sql offset fetch` and `LIMIT` across Postgres, MySQL, SQL Server, Oracle, Snowflake, and BigQuery — same primitive, six spellings, one shared perf model

Worked example — Oracle 11g's `ROWNUM` pattern vs Oracle 12c+'s `FETCH FIRST`

`sql limit offset performance` — the O(k + n) cost model, the wide-row memory story, and the "consistent read across pages" failure mode

`keyset pagination` and `seek method pagination` — WHERE (created_at, id) < (:cursor) ORDER BY DESC LIMIT n, composite index alignment, and the O(log n) deep-page path

`sql cursor pagination` and `sql infinite scroll` — opaque base64 tokens, HMAC signing, the GraphQL Relay spec, and the six-engine dialect matrix that ties it all together

`postgres lateral` and `lateral vs subquery` — CROSS JOIN LATERAL, LEFT JOIN LATERAL, and the correlation binding that makes it all work

Worked example — `LEFT JOIN LATERAL ... ON TRUE` for preserving outer rows