DEV Community: Gowtham Potureddi

PostgreSQL SQL Data Types: Practical Column-Type Guide

Gowtham Potureddi — Mon, 11 May 2026 04:08:12 +0000

Choosing the right SQL data types is one of the quiet decisions that shapes storage, correctness, and query behavior in PostgreSQL. In a tight SQL screen, interviewers often follow up on why you picked a type—not only whether the query returns rows. This guide walks through the main families, common pitfalls (rounding, time zones, type mismatches), and how to reason about casts—using PostgreSQL syntax, the same dialect PipeCode uses for practice.

If you want hands-on reps after you read, explore practice →, drill SQL problems →, browse SQL by topic →, or open Zero to FAANG SQL (full fundamentals) → for a structured path.

On this page

Why column types matter
Numeric types
Text and binary
Boolean and NULL
Date and time
Semi-structured and other types
Casting and comparison rules
Choosing types (checklist)
Frequently asked questions
Practice on PipeCode

1. Why column types matter

Storage, comparisons, indexes, and the cost of silent coercion

"Why did you pick that type?" is the single most common SQL-screen follow-up — and the cleanest answer is that a column's type controls four downstream things at once: how the value is laid out on disk, which operators compare it correctly, which indexes the planner can actually use, and when PostgreSQL has to silently coerce data behind your back. Get the type right and joins are fast, comparisons are unambiguous, and disk pages are dense. Get it wrong and you ship a schema that runs but quietly returns the wrong answer or scans 10× more pages than it should.

Pro tip: When you walk an interviewer through a CREATE TABLE, say the grain and the type in the same breath: "one row per order, order_id is BIGINT, total is NUMERIC(14,2)." That single habit signals to the interviewer that you think about column types as design decisions, not afterthoughts.

Storage footprint and on-disk layout

The storage invariant: fixed-width integer and timestamp types occupy a known number of bytes (4 or 8) and never expand; variable-width types (TEXT, NUMERIC, JSONB) carry a length prefix and grow with the value; choosing a tighter type packs more rows per 8 KB page and improves cache locality on every read. A wider type is rarely free — even when the bytes look free, the planner statistics and TOAST thresholds shift.

INTEGER — 4 bytes, range ±2.1 B; default for surrogate counts.
BIGINT — 8 bytes; required when row counts cross ~2 B or for user-facing IDs.
NUMERIC(p, s) — variable (~2 bytes overhead + 2 bytes per 4 digits); cost grows with precision.
TEXT / VARCHAR(n) — variable; no storage penalty for TEXT vs VARCHAR with the same content.

Worked example. A 100 M-row events table sized two ways:

design	per-row bytes	total
`event_id BIGINT, ts TIMESTAMPTZ, user_id BIGINT`	24	~2.4 GB
`event_id BIGINT, ts TIMESTAMPTZ, user_id TEXT (avg 18 chars)`	24 + 20 = 44	~4.4 GB

Step-by-step.

Fixed-width row (BIGINT, TIMESTAMPTZ, BIGINT) is 24 bytes on the heap regardless of values.
Replacing the integer user_id with TEXT for a UUID-shaped string adds a 4-byte length header plus the bytes of the text itself.
With ~100 M rows, the variable-width design adds ~2 GB to the table heap alone, before indexes.
The wider rows also fit fewer per 8 KB page → fewer buffer-cache hits → more I/O per query.
Net: same data, ~2× the disk and worse cache behavior.

Worked-example solution. Pick the tightest correct type:

CREATE TABLE events (
    event_id  BIGINT      PRIMARY KEY,
    ts        TIMESTAMPTZ NOT NULL,
    user_id   BIGINT      NOT NULL          -- not TEXT
);

Rule of thumb: if a value is a count or an internal identifier, it is an integer; reach for TEXT only when the value is a real human-readable string.

Equality and comparison semantics

The comparison invariant: PostgreSQL compares values within a type cleanly, but mixing types forces an implicit cast that can produce surprises — string '10' compares lexicographically ('10' < '2'), numeric 10 compares mathematically (10 > 2), and timestamps compare instant-to-instant only if both sides are TIMESTAMPTZ. The right type makes <, =, and BETWEEN behave the way humans expect.

'10' < '2' is TRUE when both are TEXT — string compare reads left-to-right.
10 < 2 is FALSE when both are INTEGER — numeric compare.
TIMESTAMP vs TIMESTAMPTZ — PostgreSQL will compare them only after coercing one side; the answer depends on the session time zone.
Collations on TEXT — 'abc' = 'ABC' is FALSE with the default C collation, possibly TRUE with a case-insensitive collation.

Worked example. A 5-row table where the comparison flips based on whether score is TEXT or INTEGER.

score (as TEXT)	order	score (as INT)	order
"10"	1	10	3
"2"	2	2	2
"100"	3	100	4
"9"	4	9	1

Step-by-step.

Stored as TEXT: ORDER BY score compares character-by-character; '1' (0x31) sorts before '9' (0x39), so '100' sorts before '2'.
Stored as INTEGER: ORDER BY score compares the numeric value; 2 < 9 < 10 < 100 — the human-expected order.
The query is identical in both cases; only the column type changed the answer.
The bug is invisible until someone audits the leaderboard and notices "100" ranked above "9".

Worked-example solution. Always store ordinal-comparable values in a numeric type:

CREATE TABLE leaderboard (
    player_id BIGINT  PRIMARY KEY,
    score     INTEGER NOT NULL CHECK (score >= 0)
);
SELECT player_id, score FROM leaderboard ORDER BY score DESC;

Rule of thumb: if you ever compare values with <, >, or BETWEEN, the type must support those operators natively — never rely on string sort for numbers or dates.

Index operator classes and planner statistics

The index invariant: a B-tree index is built against an operator class tied to a specific type; when a query compares an indexed column to a value of a different type, the planner usually has to scan instead of seek, because the function it applies to your value (the implicit cast) isn't immutable on the indexed expression. The right type matches the index; the wrong type silently disables it.

CREATE INDEX … ON t (col) — default B-tree, uses the type's default operator class.
col = $1 with matching type — index seek.
col = $1::other_type — index seek when the cast is on the literal side.
col::other_type = $1 — sequential scan; you cast the column, not the value.

Worked example. A user_id BIGINT column with a B-tree index, queried two ways.

predicate	plan	rows scanned
`WHERE user_id = 42`	Index Scan	~1
`WHERE user_id = '42'`	Index Scan (literal cast)	~1
`WHERE user_id::text = '42'`	Seq Scan	full table

Step-by-step.

WHERE user_id = 42 — both sides are BIGINT; planner uses the B-tree directly.
WHERE user_id = '42' — PostgreSQL coerces the string literal '42' to BIGINT (since BIGINT is the indexed side); index still usable.
WHERE user_id::text = '42' — the cast is on the column; PostgreSQL would have to apply the ::text function to every row to compare; the B-tree on user_id cannot help.
The third predicate triggers a full sequential scan even though an index "exists on user_id."
Diagnosis is an EXPLAIN away: Seq Scan on … Filter: ((user_id)::text = '42'::text) is the giveaway.

Worked-example solution. Keep casts on the literal side:

-- good: cast literal, index used
SELECT * FROM events WHERE user_id = '42';      -- literal '42' coerced to BIGINT

-- bad: cast column, index killed
SELECT * FROM events WHERE user_id::text = '42';

Rule of thumb: if you see a :: on a column inside a WHERE or JOIN, expect a seq scan and ask whether the underlying type should change.

Common beginner mistakes

Declaring every text column as VARCHAR(255) "just in case" — wastes nothing on storage but lies in the schema about the real constraint.
Storing numeric IDs as TEXT because the source CSV had quotes — every downstream comparison and index becomes a hazard.
Mixing TIMESTAMP and TIMESTAMPTZ in joins — comparison depends on the session time zone; you have written a query that returns different rows for different users.
Treating implicit coercion as free — the planner often hides the cost behind a seq scan and an unbroken EXPLAIN summary.
Skipping CHECK constraints because "the application handles it" — types and constraints together are the only durable schema.

SQL Interview Question on Picking Types for an Orders Schema

A junior teammate sends a CREATE TABLE orders script: order_id VARCHAR(255), total FLOAT, customer_id TEXT, placed_at TIMESTAMP. The orders application is global, has ~5 M orders per day, and is joined daily to dim_customer (customer_id BIGINT, …). Identify every type-level risk in this schema and rewrite it so reports stay correct, joins stay indexed, and storage doesn't bloat.

Solution Using Tight Native Types + `NUMERIC` + `TIMESTAMPTZ` + `CHECK` Constraints

Code solution.

CREATE TABLE orders (
    order_id    BIGSERIAL     PRIMARY KEY,
    customer_id BIGINT        NOT NULL REFERENCES dim_customer(customer_id),
    total       NUMERIC(14,2) NOT NULL CHECK (total >= 0),
    placed_at   TIMESTAMPTZ   NOT NULL DEFAULT NOW()
);
CREATE INDEX ON orders (customer_id);
CREATE INDEX ON orders (placed_at);

Step-by-step trace of the four problems:

original type	risk	fix
`order_id VARCHAR(255)`	lexicographic sort; wide rows; index mismatch	`BIGSERIAL` / `BIGINT`
`total FLOAT`	binary rounding (0.1 + 0.2 ≠ 0.3); aggregates drift	`NUMERIC(14,2)`
`customer_id TEXT`	cross-type join with `dim_customer.customer_id BIGINT`; seq scan	`BIGINT` + FK
`placed_at TIMESTAMP`	wall-clock semantics; reports differ per session TZ	`TIMESTAMPTZ`

Output: a typed, constrained schema. The daily customer-join now uses a B-tree seek on customer_id; revenue rollups are exact to the cent; "orders placed today" is unambiguous regardless of the analyst's session time zone.

Why this works — concept by concept:

BIGSERIAL PK — monotonic, 8-byte integer; supports range scans, packs tight, and matches every downstream join.
BIGINT customer_id with FK — joins are type-identical, the index is usable, and orphan rows are rejected at write time.
NUMERIC(14, 2) for money — exact decimal arithmetic; aggregates over millions of rows produce the same total a calculator would.
TIMESTAMPTZ for placed_at — every value is stored as a UTC instant; display converts to the session TZ; reports never silently shift by 24 h after a deploy.
CHECK (total >= 0) — durable invariant; even a buggy ETL run cannot insert negative revenue.
Cost — O(1) extra bytes per row vs the original; massive O(log N) per join savings vs the seq-scan caused by the type mismatch.

Inline CTA: drill the SQL practice page for type-fluency reps and the aggregation topic for grain-correct rollups.

SQL
Topic — SQL
SQL practice problems

Practice →

SQL
Topic — aggregations
SQL aggregation problems

Practice →

COURSE
Course — SQL for DE
Zero to FAANG SQL fundamentals

View course →

2. Numeric types

Integers for counts, NUMERIC for money, FLOAT for measurements

PostgreSQL splits numeric types into three families: exact integers (SMALLINT, INTEGER, BIGINT), arbitrary-precision exact decimals (NUMERIC(p, s) / DECIMAL), and binary floating point (REAL, DOUBLE PRECISION). The choice is rarely about precision in the abstract — it's about which arithmetic errors are acceptable. Integers never lose precision; NUMERIC is exact at a fixed scale; floats trade precision for speed and are the wrong default for currency.

Pro tip: When asked "what type is revenue?", say NUMERIC(p, s) and name p and s out loud — NUMERIC(14, 2) for cents up to ~$100 B, NUMERIC(18, 4) for FX rates and basis points. Knowing the scale is what separates "I know decimals exist" from "I have shipped a ledger."

`INTEGER` / `BIGINT` — surrogate keys and counts

The integer invariant: INTEGER is 4 bytes (range ±2.1 B) and BIGINT is 8 bytes (range ±9.2 quintillion); use INTEGER for small/medium counts and BIGINT for surrogate keys, monotonically increasing IDs, and anything that might ever cross 2 billion. Overflow is silent in some languages but is a hard error in PostgreSQL — once the column wraps, every insert fails.

SMALLINT — 2 bytes; rarely used outside tightly packed enum-like values.
INTEGER — 4 bytes; default for row counts, scores, age, quantities.
BIGINT — 8 bytes; default for primary keys on growing tables.
BIGSERIAL / GENERATED AS IDENTITY — 8-byte auto-incrementing PK.

Worked example. An events table grows from 1 M to 3 B rows.

year	events	INTEGER PK?	BIGINT PK?
2024	1 M	✓	✓
2025	500 M	✓	✓
2026	2.5 B	✗ overflow	✓

Step-by-step.

Start with event_id INTEGER — fits 2.1 B values.
Daily growth at 5 M / day reaches 2.1 B by mid-2026.
Next INSERT fails: ERROR: integer out of range.
Migration requires ALTER TABLE … ALTER COLUMN event_id TYPE BIGINT; — rewrites the entire table; locks scale with table size.
Doing this at 2.1 B rows means hours of downtime; doing it at table creation is free.

Worked-example solution. Use BIGINT for any growing PK:

CREATE TABLE events (
    event_id  BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
    user_id   BIGINT NOT NULL,
    ts        TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

Rule of thumb: every primary key on a table that "might be big someday" is BIGINT from day one. The 4 extra bytes per row is the cheapest insurance you can buy.

`NUMERIC(p, s)` — exact decimal for currency

The decimal invariant: NUMERIC(p, s) stores p total digits with s of them after the decimal point; arithmetic is exact at that scale; SUM(NUMERIC) over millions of rows produces the byte-identical result a careful accountant would compute by hand. The cost is performance — NUMERIC math is slower than integer or float — but for currency the trade-off is settled: exact wins.

NUMERIC(14, 2) — up to 12 digits before the decimal, 2 after; ~$1 T.
NUMERIC(18, 4) — FX rates, fractional cents (interest, allocations).
NUMERIC(38, 6) — analytics-warehouse scale; matches Snowflake / BigQuery default.
Storage — ~2 bytes overhead + 2 bytes per 4 digits; cheap up to ~$1 T.

Worked example. Summing 1,000 invoice lines of $0.10 each:

storage type	`SUM(amount)`
`DOUBLE PRECISION`	`100.00000000000007`
`NUMERIC(14, 2)`	`100.00`

Step-by-step.

0.1 cannot be represented exactly in binary floating point; the stored value is 0.1000000000000000055511….
Adding 1,000 of these in DOUBLE PRECISION accumulates 1000 * tiny_error; the result drifts.
NUMERIC(14, 2) stores 0.10 literally and adds with decimal arithmetic; 1,000 × 0.10 is exactly 100.00.
The float error is invisible until a finance lead notices a $0.00000007 discrepancy on a reconciliation report.
Once the column type is NUMERIC, the drift is impossible by construction.

Worked-example solution. Currency columns always use NUMERIC:

CREATE TABLE invoice_lines (
    line_id    BIGSERIAL      PRIMARY KEY,
    quantity   INTEGER        NOT NULL CHECK (quantity > 0),
    unit_price NUMERIC(12, 4) NOT NULL,
    line_total NUMERIC(14, 4) GENERATED ALWAYS AS (quantity * unit_price) STORED
);

Rule of thumb: anything that touches money, tax, allocations, basis points, or a regulated ledger is NUMERIC(p, s) — never FLOAT or DOUBLE PRECISION.

`REAL` / `DOUBLE PRECISION` — binary floating point and rounding

The float invariant: REAL (4 bytes, ~7 decimal digits) and DOUBLE PRECISION (8 bytes, ~15 digits) follow IEEE 754; they're fast and compact but inexact at decimal fractions; their natural home is measurements where the underlying quantity is itself approximate (sensor reading, ML feature, scientific magnitude). Floats are not "lossy currency" — they are the right type for things that were never exact to begin with.

REAL — 4 bytes; ~7 decimal digits of precision.
DOUBLE PRECISION — 8 bytes; ~15 digits; PostgreSQL's default FLOAT.
0.1 + 0.2 = 0.30000000000000004 in both.
Use cases — physical measurements, geographic coordinates, ML scores, neural-net outputs.

Worked example. Same 5 sensor readings stored two ways:

reading	`REAL`	`DOUBLE PRECISION`
23.7	23.7	23.7
0.1 + 0.2	0.3 (~0.30000001)	0.30000000000000004
3.14159265358979	3.1415927	3.141592653589793

Step-by-step.

REAL rounds aggressively after ~7 digits; fine for a temperature gauge, wrong for a price.
DOUBLE PRECISION keeps ~15 digits — enough for almost any measurement.
Neither stores 0.1 + 0.2 as exactly 0.3 because base-2 cannot represent base-10 tenths.
Equality (=) on floats is unsafe; use a tolerance (abs(a - b) < 1e-9) for "approximately equal."
For currency, both are wrong — use NUMERIC.

Worked-example solution. Use floats for genuinely approximate measurements:

CREATE TABLE sensor_readings (
    reading_id   BIGSERIAL        PRIMARY KEY,
    device_id    BIGINT           NOT NULL,
    temp_celsius DOUBLE PRECISION NOT NULL,
    ts           TIMESTAMPTZ      NOT NULL DEFAULT NOW()
);

Rule of thumb: if you would compare the value with = and care about the result, it is not a float.

Common beginner mistakes

Defaulting all PKs to SERIAL (32-bit) and discovering the overflow in production years later.
Storing money in DOUBLE PRECISION because NUMERIC "is slow" — the slowdown is invisible to humans; the rounding is not.
Using NUMERIC with no precision (NUMERIC with no (p,s)) — works but skips the documentation value of stating the scale.
Comparing floats with = instead of a tolerance window.
Using INTEGER for cents (total_cents) instead of NUMERIC(14, 2) — works but burdens every read with a /100.0.

SQL Interview Question on Reconciling a Drifting Invoice Total

The CFO reports that the monthly invoice total in the dashboard disagrees with the source-of-truth ledger by $0.0000034 on average. The dashboard sums an invoice_lines.amount column declared as DOUBLE PRECISION. Identify the cause and propose a schema fix that makes the totals byte-identical to the ledger from now on.

Solution Using `NUMERIC(14, 4)` + a Generated `line_total` Column

Code solution.

ALTER TABLE invoice_lines
    ALTER COLUMN amount TYPE NUMERIC(14, 4) USING amount::NUMERIC(14, 4);

ALTER TABLE invoice_lines
    ADD COLUMN line_total NUMERIC(14, 4)
    GENERATED ALWAYS AS (quantity * amount) STORED;

-- nightly reconciliation
SELECT SUM(line_total) AS dash_total
FROM invoice_lines
WHERE invoice_date = DATE '2026-04-13';

Step-by-step trace of the drift:

step	value	running sum (DOUBLE PRECISION)	running sum (NUMERIC)
1	0.1	0.1	0.10
2	0.1	0.2	0.20
3	0.1	0.30000000000000004	0.30
…	…	accumulating error	exact
1000	0.1	100.00000000000007	100.00

Output: dashboard total per day now matches the ledger to the cent (or to the basis point, given scale 4). No silent drift; finance closes the books without manual adjustment.

Why this works — concept by concept:

NUMERIC(14, 4) exact decimal arithmetic — every addition stays exact at four decimal places; no IEEE 754 representation error.
Generated line_total column — eliminates a class of bugs where the application computes qty * price and the database computes a slightly different number.
STORED not VIRTUAL — value is materialised once at write time; reads are plain column reads with no per-row recomputation.
Tolerance check on the ETL side — even with NUMERIC, reconciliation should compare against the source-of-truth ledger with a 0-tolerance gate.
One-time ALTER TABLE … USING — converts existing rows in place; from then on the type system makes drift impossible.
Cost — single rewrite at migration; per-row NUMERIC math is ~3× slower than DOUBLE PRECISION but invisible compared to network and disk costs.

Inline CTA: for the structured currency-and-aggregation path see SQL for Data Engineering Interviews — From Zero to FAANG.

SQL
Topic — aggregations
SQL aggregation problems

Practice →

SQL
Topic — conditional aggregation
Conditional-aggregation problems

Practice →

COURSE
Course — SQL for DE
Zero to FAANG SQL fundamentals

View course →

3. Text and binary

CHAR vs VARCHAR vs TEXT, collations, and BYTEA

PostgreSQL has three character types — CHAR(n), VARCHAR(n), and TEXT — and one binary type, BYTEA. The decision rule is short: use TEXT unless you have a hard reason to enforce a length cap, and store files outside the database with a URL or object-store key in the column. Most "text" bugs are not about storage at all — they are about collations, which control how text compares and sorts.

Pro tip: Two strings that look identical can compare unequal under a different collation. When a join "returns no rows" on string keys, your first check after EXPLAIN is SHOW lc_collate; and SELECT pg_collation_for(col1) on both columns.

`CHAR` vs `VARCHAR` vs `TEXT` — pick `TEXT` unless you need fixed-width

The text invariant: TEXT and VARCHAR(n) have the same on-disk representation in PostgreSQL — no padding, no length penalty; the only difference is the (n) constraint that throws an error on overflow. CHAR(n) pads with spaces to length, costing both storage and surprise (trailing-space equality is mostly stripped on read, but joins can still misbehave).

CHAR(n) — fixed-width; pads with spaces; storage = n bytes (plus a length header).
VARCHAR(n) — variable-width; rejects values longer than n.
TEXT — variable-width; no length limit (up to 1 GB).
citext extension — case-insensitive text via the citext type.

Worked example. Storing "abc" three ways:

type	stored bytes	trailing pad
`CHAR(5)`	`abc` (5 bytes)	yes
`VARCHAR(5)`	`abc` (3 bytes)	no
`TEXT`	`abc` (3 bytes)	no

Step-by-step.

CHAR(5) stores literally abc (5 chars), padding to length.
VARCHAR(5) stores abc; would reject abcdef with a length-violation error.
TEXT stores abc; would accept abcdef.
Equality semantics differ: CHAR(5) 'abc' = VARCHAR(5) 'abc' may be TRUE but joining a CHAR column to a VARCHAR column from another table can still fail when one side preserved trailing whitespace.
Default to TEXT — it is the simplest and never accumulates these padding surprises.

Worked-example solution. Schema for a free-form bio field:

CREATE TABLE profiles (
    user_id BIGINT PRIMARY KEY REFERENCES users(user_id),
    bio     TEXT   NOT NULL DEFAULT ''
);

Rule of thumb: use VARCHAR(n) only when you genuinely want the database to enforce a maximum length (e.g., regulator-imposed description VARCHAR(280)); otherwise reach for TEXT.

Collations and locale-aware equality

The collation invariant: a collation is a tuple of (alphabet, sort order, case-sensitivity, accent-sensitivity) that the database applies to every text comparison; the default is usually "C" (binary) or the OS locale; case-insensitive matching requires either an explicit ICU collation or the citext extension. Two databases with different locales can disagree on whether 'café' = 'cafe'.

C collation — byte-by-byte; fastest; case- and accent-sensitive.
en_US.UTF-8 — locale-aware; sorts 'a' < 'B' < 'c' (case-insensitive primary).
und-x-icu — ICU root locale; consistent across platforms.
citext — case-insensitive text type; 'ABC' = 'abc' is TRUE automatically.

Worked example. Joining users by email under different collations:

left email	right email	join match (C)	join match (citext)
`alice@x.com`	`alice@x.com`	✓	✓
`Alice@X.com`	`alice@x.com`	✗	✓
`alice@x.com`	`alice@x.com`	✗	✗ (whitespace, not case)

Step-by-step.

Default C collation does a byte compare; 'A' (0x41) is not equal to 'a' (0x61).
Same string with mixed case fails to join in C even though humans see them as the same email.
Switching the column type to citext makes the database compare case-insensitively, and the second row matches.
Whitespace differences still cause mismatches — citext does not trim; that requires BTRIM(col) in ETL.
Pick one normalization rule (lowercase + trim at write time) and apply it consistently rather than relying on collation alone.

Worked-example solution. Use citext for emails and usernames:

CREATE EXTENSION IF NOT EXISTS citext;
CREATE TABLE users (
    user_id BIGINT PRIMARY KEY,
    email   CITEXT NOT NULL UNIQUE
);

Rule of thumb: if you ever want 'Foo' = 'foo' to be TRUE, set that contract at the column type, not at every LOWER(...) call site.

`BYTEA` for binary blobs vs URL-in-SQL for files

The binary invariant: BYTEA stores raw bytes (hashes, signatures, compressed payloads, small binary tokens); large blobs (images, PDFs, ML model weights) belong in object storage (S3, GCS) with a TEXT URL or key in SQL. Databases are not file systems — every byte stored in BYTEA slows backups, replication, and query cache.

BYTEA — variable-length binary; up to 1 GB but typically used for ≤ 10 KB tokens.
SHA-256 hash — 32 bytes; perfect BYTEA use case.
Large files — store in S3; keep s3_key TEXT in SQL.
pg_largeobject — legacy API; rarely worth the complexity vs object storage.

Worked example. A documents table with two design choices:

design	per-row storage	backup time
`body BYTEA` (10 MB PDFs, 1 M rows)	10 TB in `pg_largeobject`	hours
`s3_key TEXT` (URL only, 1 M rows)	< 100 MB	seconds

Step-by-step.

Storing 10 MB PDFs in BYTEA puts all bytes in TOAST; the table grows to 10 TB.
Every pg_dump reads all 10 TB; backups become days, not minutes.
Replication lag grows; HA failover slows.
Object storage (S3) is purpose-built for large files; the database keeps only a 50-byte s3_key.
Reads still feel "one query" — the application fetches the URL from SQL, then streams the file from S3.

Worked-example solution. Store files externally; keep the key in SQL:

CREATE TABLE documents (
    document_id BIGSERIAL PRIMARY KEY,
    user_id     BIGINT    NOT NULL,
    sha256      BYTEA     NOT NULL CHECK (octet_length(sha256) = 32),
    s3_key      TEXT      NOT NULL,
    uploaded_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

Rule of thumb: the rough threshold is 100 KB — anything above that belongs in object storage; anything below is fine as BYTEA.

Common beginner mistakes

Declaring text columns as VARCHAR(255) everywhere — Java legacy from the MySQL world; meaningless in modern PostgreSQL.
Using CHAR(n) and being surprised that 'abc' = 'abc ' returns FALSE in some join contexts.
Storing emails as case-sensitive TEXT and writing LOWER(email) = LOWER($1) everywhere — set citext once at the column.
Putting megabyte payloads in BYTEA and discovering the cost only when pg_dump runs for six hours.
Forgetting to trim whitespace at ingest — ' alice@x.com' and 'alice@x.com' are different strings to the database.

SQL Interview Question on Reconciling Case-Sensitive Email Joins

A signup flow stores users.email as TEXT. The marketing dashboard joins events.email (also TEXT) to users.email to count signed-up users. Roughly 8% of events fail to match even though the user definitely signed up. Diagnose the cause and propose a column-level fix that prevents recurrence.

Solution Using `citext` + Normalised Write-Time Email

Code solution.

CREATE EXTENSION IF NOT EXISTS citext;

ALTER TABLE users   ALTER COLUMN email TYPE CITEXT USING LOWER(BTRIM(email));
ALTER TABLE events  ALTER COLUMN email TYPE CITEXT USING LOWER(BTRIM(email));

-- joins now match regardless of case; rejoin to verify
SELECT COUNT(*) FROM events e
JOIN users u ON u.email = e.email;

Step-by-step trace of the 8% miss:

event email	user email	TEXT join	CITEXT join
`Alice@x.com`	`alice@x.com`	✗	✓
`bob@x.com`	`bob@x.com`	✓	✓
`carol@x.com`	`carol@x.com`	✗	✗ (whitespace)

Output: the case-sensitivity portion of the miss disappears (≈ 7%); the remaining ≈ 1% is whitespace, fixed by BTRIM in the USING clause at migration and a BEFORE INSERT trigger going forward.

Why this works — concept by concept:

CITEXT columns — case-insensitive by construction; downstream queries never have to wrap LOWER(...) and indexes still work.
LOWER(BTRIM(email)) in the USING clause — one-shot normalisation of existing rows during the type change.
Trigger or CHECK enforcement going forward — keeps future inserts canonical.
No more LOWER(...) at every query site — every analyst joins safely without remembering the casing rule.
Existing indexes rebuild automatically — ALTER COLUMN TYPE rebuilds the index against the new operator class.
Cost — one rewrite at migration; per-row equality cost identical to TEXT.

Inline CTA: for the string-fluency syllabus see SQL for Data Engineering Interviews — From Zero to FAANG.

SQL
Topic — string manipulation
String-manipulation problems

Practice →

SQL
Topic — joins
SQL join problems

Practice →

COURSE
Course — SQL for DE
Zero to FAANG SQL fundamentals

View course →

4. Boolean and NULL

Three-valued logic and the `WHERE flag` trap

PostgreSQL has a real BOOLEAN type with three values: TRUE, FALSE, and NULL. The third value is the source of nearly every "where did my rows go?" bug — NULL is not false; it is unknown. Filters like WHERE flag silently exclude NULL rows, and WHERE NOT flag excludes them too, so a "true-or-not-true" pair of queries can together miss rows entirely.

Pro tip: Whenever you write a boolean predicate, name the third bucket out loud. "Active users are is_active = TRUE; bots are is_bot = TRUE; unknown is IS NULL and goes into the needs-investigation drawer." That habit catches the silent-exclusion bug before it ships.

`BOOLEAN` literals, `IS TRUE` / `IS FALSE` / `IS NULL`

The boolean invariant: WHERE flag returns rows where the predicate is TRUE; rows where flag is NULL (unknown) are also excluded; to include or exclude them deliberately you must use IS NULL / IS NOT NULL / IS DISTINCT FROM. Standard SQL three-valued logic treats NULL = anything as NULL, which is neither true nor false — and a WHERE clause keeps only rows that evaluate to TRUE.

TRUE / FALSE — the two non-null boolean values.
NULL — unknown; not equal to anything (including itself).
IS TRUE / IS FALSE — three-valued aware; never returns NULL.
IS DISTINCT FROM — treats two NULLs as equal; useful for join keys.

Worked example. A 5-row events table with a nullable is_bot flag:

event_id	is_bot
1	TRUE
2	FALSE
3	NULL
4	TRUE
5	NULL

predicate	rows kept
`WHERE is_bot`	1, 4 (only TRUE)
`WHERE NOT is_bot`	2 (only FALSE)
`WHERE is_bot IS NOT TRUE`	2, 3, 5
`WHERE is_bot IS NULL`	3, 5

Step-by-step.

WHERE is_bot keeps rows where the predicate is TRUE; rows 3 and 5 (NULL) are silently dropped.
WHERE NOT is_bot keeps rows where the predicate evaluates to TRUE; NOT NULL is NULL, so rows 3 and 5 are still silently dropped.
The dashboard "Bots vs non-bots" pair (is_bot true / NOT is_bot) sums to 3 rows, not 5 — two rows are missing in plain sight.
IS NOT TRUE is three-valued aware: it returns TRUE for rows 2, 3, 5 — both the false ones and the nulls.
Pick the form that matches your intent and audit any dashboard that splits a column on a boolean.

Worked-example solution. Three-valued-aware predicates:

-- bots
SELECT COUNT(*) FROM events WHERE is_bot IS TRUE;
-- non-bots, including unknown
SELECT COUNT(*) FROM events WHERE is_bot IS NOT TRUE;
-- only unknown
SELECT COUNT(*) FROM events WHERE is_bot IS NULL;

Rule of thumb: never write WHERE flag or WHERE NOT flag on a nullable boolean column without consciously deciding what NULL means.

`NOT col` vs `col = FALSE` with NULLs

The negation invariant: col = FALSE and NOT col are logically the same when col is TRUE or FALSE, but both evaluate to NULL when col IS NULL — and a WHERE clause keeps only TRUE, so both forms silently drop nulls. The fix is COALESCE(col, FALSE) or IS NOT TRUE, which collapse NULL into a definite answer.

WHERE col = FALSE — keeps rows where col is literally FALSE.
WHERE NOT col — same; both drop NULL rows.
WHERE COALESCE(col, FALSE) = FALSE — treats NULL as FALSE; keeps both.
WHERE col IS NOT TRUE — treats NULL as not-true; keeps both.

Worked example. Same events table; analyst writes "all non-bot events":

query	rows	comment
`WHERE is_bot = FALSE`	1	row 2 only — silent miss
`WHERE NOT is_bot`	1	identical; same bug
`WHERE is_bot IS NOT TRUE`	3	rows 2, 3, 5 — correct
`WHERE COALESCE(is_bot, FALSE) = FALSE`	3	also correct

Step-by-step.

Marketing asks "how many non-bot events?"; analyst writes WHERE NOT is_bot.
Result is 1; marketing thinks bots account for 4 of 5 events.
A second analyst writes WHERE is_bot IS NOT TRUE and gets 3; the difference is the NULL rows.
The dashboard's "bot vs non-bot" pie chart silently undercounts by 40%.
The fix is either a COALESCE at query time or a NOT NULL DEFAULT FALSE constraint at schema time — both make the NULL case explicit.

Worked-example solution. Default boolean columns to a known value at write time:

ALTER TABLE events
    ALTER COLUMN is_bot SET DEFAULT FALSE,
    ALTER COLUMN is_bot SET NOT NULL;
-- queries are now safe
SELECT COUNT(*) FROM events WHERE NOT is_bot;

Rule of thumb: if a boolean has no "unknown" business meaning, declare it NOT NULL DEFAULT FALSE and remove the third bucket entirely.

`COALESCE` and explicit NULL handling

The COALESCE invariant: COALESCE(a, b, c) returns the first non-NULL argument; it is the simplest way to replace NULL with a default in WHERE, ORDER BY, and aggregations — but use it deliberately, because hiding NULL is the same as throwing away information. The right pattern is to decide whether NULL means "no answer" or "definitely false," then code that intent.

COALESCE(col, default) — first non-NULL argument.
NULLIF(a, b) — returns NULL when a = b; useful for "treat empty string as NULL."
a IS DISTINCT FROM b — TRUE when values differ, treating NULL as a real value.
SUM(col) — ignores NULLs; COUNT(col) ignores NULLs; COUNT(*) includes them.

Worked example. Summing score where some rows are NULL:

row	score
1	10
2	NULL
3	20

expression	result
`SUM(score)`	30
`SUM(COALESCE(score, 0))`	30
`AVG(score)`	15 (n=2)
`AVG(COALESCE(score, 0))`	10 (n=3)

Step-by-step.

SUM ignores NULLs by SQL convention; you get the same answer with or without COALESCE.
AVG divides by COUNT(non-NULL); ignoring NULL gives 15, treating NULL as zero gives 10.
The "right" answer depends on what NULL means — missing measurement (use 15) vs zero score (use 10).
Always make the choice explicit; do not let a downstream consumer guess.
IS DISTINCT FROM is the safe way to compare keys that may be NULL: a IS DISTINCT FROM b is TRUE when one is NULL and the other is not.

Worked-example solution. Choose the aggregation rule that matches the business question:

-- "average of measurements we have"
SELECT AVG(score) FROM responses;
-- "average where missing means zero"
SELECT AVG(COALESCE(score, 0)) FROM responses;

Rule of thumb: every COALESCE should answer the question "what should the missing row contribute?" in one sentence — if you cannot answer, do not coalesce.

Common beginner mistakes

Writing WHERE flag = FALSE and assuming it includes NULL rows.
Pairing WHERE flag with WHERE NOT flag and expecting the row counts to sum to the table size.
Storing booleans as 'Y' / 'N' strings — every comparison becomes a LOWER(...) hazard; use real BOOLEAN.
Forgetting that NULL = NULL is NULL, not TRUE — join keys with NULL need IS DISTINCT FROM or pre-coalesced values.
Using AVG over a nullable column without deciding whether missing means zero or excluded.

SQL Interview Question on a Dashboard Missing 12% of Rows

A events.is_bot BOOLEAN column is nullable. The dashboard splits "bots vs humans" with WHERE is_bot and WHERE NOT is_bot. The two row counts sum to 88% of the table; nobody can explain where the missing 12% went. Identify the cause and produce a single query pair that correctly partitions every row.

Solution Using `IS TRUE` / `IS NOT TRUE` + a Schema-Level `NOT NULL` Fix

Code solution.

-- short-term query-side fix
SELECT
    COUNT(*) FILTER (WHERE is_bot IS TRUE)     AS bots,
    COUNT(*) FILTER (WHERE is_bot IS NOT TRUE) AS humans_or_unknown
FROM events;

-- long-term schema fix
UPDATE events SET is_bot = FALSE WHERE is_bot IS NULL;
ALTER TABLE events
    ALTER COLUMN is_bot SET NOT NULL,
    ALTER COLUMN is_bot SET DEFAULT FALSE;

Step-by-step trace.

step	predicate	rows
1	`WHERE is_bot` (old)	12,000
2	`WHERE NOT is_bot` (old)	76,000
3	sum	88,000 of 100,000
4	missing	12,000 rows where `is_bot IS NULL`
5	`WHERE is_bot IS NOT TRUE`	88,000 — both FALSE and NULL
6	`bots + humans_or_unknown`	100,000 ✓

Output: the two-bucket dashboard sums to 100% of rows. Schema-level NOT NULL DEFAULT FALSE makes future regression impossible.

Why this works — concept by concept:

IS TRUE / IS NOT TRUE are three-valued safe — they never return NULL; the WHERE clause keeps exactly the rows the analyst expects.
COUNT(*) FILTER (WHERE …) — single-pass two-bucket aggregation; faster than running two queries.
UPDATE … WHERE IS NULL + SET NOT NULL — one-shot remediation of historical NULLs.
DEFAULT FALSE — guarantees new rows start in a definite state.
No surprise on rerun — the dashboard's "missing 12%" cannot reappear because the column type now rules it out.
Cost — one UPDATE; the FILTER form has the same cost as two separate COUNTs combined into one scan.

Inline CTA: for the safe-NULL drill set see SQL practice page.

SQL
Topic — filtering
SQL filtering problems

Practice →

SQL
Topic — conditional aggregation
Conditional-aggregation problems

Practice →

SQL
Language — SQL
All SQL practice problems

Practice →

5. Date and time

`DATE`, `TIME`, `TIMESTAMP`, and `TIMESTAMPTZ` — instants vs wall clocks

PostgreSQL splits time into calendar dates (DATE), local wall-clock times (TIME), wall-clock timestamps (TIMESTAMP WITHOUT TIME ZONE), and absolute instants (TIMESTAMP WITH TIME ZONE, abbreviated TIMESTAMPTZ). The two-row mental model: TIMESTAMP is what a wall clock reads at a particular spot; TIMESTAMPTZ is a point on the global timeline. Every cross-region bug comes from picking the first when you wanted the second.

Pro tip: Default every event-instant column to TIMESTAMPTZ and use TIMESTAMP only when the time is intentionally local (a "9:00 AM recurring meeting" in the user's locale). Reporting that crosses regions becomes obviously correct or obviously wrong, with no middle ground.

`TIMESTAMP` without time zone — local wall-clock semantics

The wall-clock invariant: TIMESTAMP stores the literal datetime you gave it with no time-zone metadata; "2026-04-13 09:00:00" means 9:00 local wherever you happen to be; comparing two TIMESTAMP values is correct only if both came from the same time zone. It is the right type for "9:00 morning meeting in the user's local time" — and the wrong type for "the moment the user clicked Pay."

TIMESTAMP storage — 8 bytes; no time-zone info.
NOW() returns TIMESTAMPTZ — coerced to TIMESTAMP strips the zone.
Comparison — two TIMESTAMPs compare by literal value, regardless of zones.
Use case — "every Monday at 09:00 local time" recurring schedules.

Worked example. Storing a 09:00 morning meeting for two users in different zones:

user	wall-clock time	TIMESTAMP value
Alice (NYC)	9:00 AM EDT	`2026-04-13 09:00:00`
Bob (Tokyo)	9:00 AM JST	`2026-04-13 09:00:00`

Step-by-step.

Both rows look identical because the type carries no zone — the database just stores the digits the application sent.
Both meetings happen at "9:00 AM local"; they are not the same UTC instant (13 hours apart).
A query like SELECT * WHERE start_at = '2026-04-13 09:00:00' returns both rows; that is the right answer for a "9 AM morning meetings" report.
If the same column had been TIMESTAMPTZ, the two values would have been stored as different UTC instants and the report would have returned one of them or neither, depending on session settings.
Pick TIMESTAMP only when the wall-clock semantics are the actual business rule.

Worked-example solution. Recurring local-time schedule:

CREATE TABLE recurring_meetings (
    meeting_id BIGSERIAL PRIMARY KEY,
    user_id    BIGINT    NOT NULL,
    local_tz   TEXT      NOT NULL,                  -- 'America/New_York'
    start_at   TIMESTAMP NOT NULL                   -- intentional wall-clock
);

Rule of thumb: if your column answers the question "what should the clock on the wall read?", use TIMESTAMP; otherwise use TIMESTAMPTZ.

`TIMESTAMPTZ` — UTC instant, session display

The instant invariant: TIMESTAMPTZ stores every value as a UTC instant internally (8 bytes), regardless of the time-zone literal in the INSERT; output is converted to the session's TimeZone at read time; comparison is always instant-to-instant. Same data ships to every region and every report agrees on "when did this happen."

TIMESTAMPTZ storage — 8 bytes; internal representation is UTC.
INSERT … TIMESTAMPTZ '2026-04-13 09:00 EDT' — stored as 13:00 UTC.
SET TimeZone = 'Asia/Tokyo' then SELECT ts — outputs 2026-04-13 22:00:00+09.
AT TIME ZONE — converts between zones in a query.

Worked example. Same UTC instant viewed from three zones:

session TimeZone	what `SELECT ts FROM events WHERE id = 1` shows
`UTC`	`2026-04-13 13:00:00+00`
`America/New_York`	`2026-04-13 09:00:00-04`
`Asia/Tokyo`	`2026-04-13 22:00:00+09`

Step-by-step.

The instant 2026-04-13 13:00:00 UTC was inserted once into the table.
The on-disk representation is a single 8-byte number — UTC microseconds since the epoch.
Each session reads the same row, but the display function converts that instant to the session's TimeZone.
The underlying data is identical; the rendering differs.
Cross-region reports stay correct because every comparison happens on the stored UTC value, not the displayed string.

Worked-example solution. Event-instant column:

CREATE TABLE clicks (
    click_id BIGSERIAL  PRIMARY KEY,
    user_id  BIGINT     NOT NULL,
    ts       TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
SELECT user_id, COUNT(*) FROM clicks
WHERE ts >= NOW() - INTERVAL '24 hours'
GROUP BY user_id;

Rule of thumb: every "when did the event happen?" column is TIMESTAMPTZ; never TIMESTAMP.

`AT TIME ZONE` conversions and `DATE_TRUNC` pitfalls

The conversion invariant: ts AT TIME ZONE 'America/New_York' converts a TIMESTAMPTZ to a wall-clock TIMESTAMP in that zone, and the reverse (TIMESTAMP AT TIME ZONE 'America/New_York') interprets the wall-clock as a UTC instant; DATE_TRUNC('day', ts) buckets by UTC midnight unless you convert first. The pattern for "daily count in the user's local time" is DATE_TRUNC('day', ts AT TIME ZONE 'America/New_York').

TIMESTAMPTZ AT TIME ZONE 'zone' → TIMESTAMP (wall clock).
TIMESTAMP AT TIME ZONE 'zone' → TIMESTAMPTZ (instant).
DATE_TRUNC('day', ts) — uses UTC midnight; usually not what reports want.
DATE_TRUNC('day', ts AT TIME ZONE 'zone') — uses local midnight.

Worked example. Daily clicks for a US dashboard:

click	UTC `ts`	UTC day	NY day
A	`2026-04-13 03:00 UTC`	2026-04-13	2026-04-12 (still 23:00 prev day NY)
B	`2026-04-13 14:00 UTC`	2026-04-13	2026-04-13
C	`2026-04-14 02:00 UTC`	2026-04-14	2026-04-13 (still 22:00 NY)

Step-by-step.

DATE_TRUNC('day', ts) groups by UTC midnight; click A goes into UTC 2026-04-13.
But the user in NY clicked at 11 PM on April 12; the dashboard credits the wrong calendar day.
ts AT TIME ZONE 'America/New_York' converts the instant to NY wall-clock: A becomes 2026-04-12 23:00, C becomes 2026-04-13 22:00.
DATE_TRUNC('day', ts AT TIME ZONE 'America/New_York') then buckets by NY midnight; A goes into April 12, B and C into April 13.
Daily counts now match the user's perception of "yesterday."

Worked-example solution. Daily report in NY local time:

SELECT
    DATE_TRUNC('day', ts AT TIME ZONE 'America/New_York')::date AS day_ny,
    COUNT(*) AS clicks
FROM clicks
GROUP BY 1
ORDER BY 1;

Rule of thumb: if a report says "daily" or "monthly," ask whose calendar — and then AT TIME ZONE before truncating.

Common beginner mistakes

Defaulting to TIMESTAMP "because it's shorter to type" — silently breaks cross-region comparisons after the first deploy abroad.
Storing TIMESTAMP and then "adding the time zone in the app" — the database loses the original zone the moment you stored.
DATE_TRUNC('day', ts) on UTC instants for a regional dashboard — daily counts shift by hours.
Using NOW() interchangeably with CURRENT_DATE — NOW() is TIMESTAMPTZ, CURRENT_DATE is DATE in the session's zone.
Forgetting daylight saving — INTERVAL '24 hours' is not always "next day at the same wall-clock time."

SQL Interview Question on a Dashboard That Shifted 24 Hours After Deploy

The team deploys their analytics pipeline to a new region; the next morning the "orders today" dashboard shows yesterday's total. Storage is placed_at TIMESTAMP (without time zone). Diagnose the cause and propose a schema + query fix that survives any future deploy.

Solution Using `TIMESTAMPTZ` + `AT TIME ZONE` in the Reporting View

Code solution.

ALTER TABLE orders
    ALTER COLUMN placed_at TYPE TIMESTAMPTZ
    USING placed_at AT TIME ZONE 'America/New_York';

CREATE VIEW v_daily_orders AS
SELECT
    DATE_TRUNC('day', placed_at AT TIME ZONE 'America/New_York')::date AS order_day,
    COUNT(*)                                                              AS orders,
    SUM(total)                                                            AS revenue
FROM orders
GROUP BY 1;

Step-by-step trace.

step	observation
1	original `placed_at TIMESTAMP` — interpreted in the application's local zone
2	redeploy moves the app to a server in `UTC`; same `NOW()` literal now means UTC, not NY
3	rows inserted post-deploy look 4 hours older to the dashboard's NY-day buckets
4	`ALTER COLUMN … TYPE TIMESTAMPTZ USING … AT TIME ZONE 'America/New_York'` reinterprets all existing rows
5	new `TIMESTAMPTZ` column stores UTC; the view's `AT TIME ZONE` reverses to NY for display
6	dashboard buckets daily counts by NY midnight; results are stable across redeploys

Output: "orders today" matches the operations team's intuition regardless of where the application server lives. Future deploys cannot reintroduce the 24-hour shift because the column type now stores instants, not wall clocks.

Why this works — concept by concept:

TIMESTAMPTZ stores UTC — the on-disk value is the same regardless of session or server zone.
USING … AT TIME ZONE 'America/New_York' — one-shot reinterpretation of legacy rows during the type migration.
AT TIME ZONE in the view, not the table — every report stays explicit about whose calendar it uses.
DATE_TRUNC on the local wall-clock — daily buckets align to the user's perception of "today."
Stable across redeploys — server moves do not change the displayed daily count.
Cost — one rewrite per migration; per-row AT TIME ZONE is essentially free (microseconds).

Inline CTA: drill the date-functions practice topic and the filtering practice topic for time-aware predicates.

SQL
Topic — date functions
Date-function problems

Practice →

SQL
Topic — window functions
Window-function problems

Practice →

COURSE
Course — SQL for DE
Zero to FAANG SQL fundamentals

View course →

6. Semi-structured and other types

`JSONB`, `UUID`, and arrays for flexible attributes

PostgreSQL is a "relational with side quests" database — it has first-class JSONB (binary, indexable JSON), UUID (opaque distributed IDs), and array types (INTEGER[], TEXT[], JSONB[]) that make schema-flexible patterns possible without giving up SQL. The discipline is to use them deliberately: JSONB for truly sparse attributes, UUID for public/distributed identifiers, arrays for short bounded lists. Reach for them often and the schema becomes hard to query; reach for them never and you write more tables than you need.

Pro tip: Any column that becomes a frequent filter or join key belongs in a real typed column, not nested inside JSONB. Use JSONB as the "everything else" bucket for attributes that vary by row.

`JSON` vs `JSONB` — when binary indexing matters

The JSONB invariant: JSON stores the input text exactly (whitespace, key order, duplicate keys preserved) and reparses on every read; JSONB stores a binary-decoded representation that is faster to query, supports GIN indexes, and rejects duplicate keys — pay the small write-time cost for read-time speed. For event payloads, application config, and flexible user attributes, JSONB is the default.

JSON — text-faithful; preserves whitespace and duplicate keys; slow.
JSONB — binary; faster reads; canonical (no whitespace, no duplicate keys).
-> — returns JSON / JSONB.
->> — returns TEXT.
@> containment — '{"a": 1}'::jsonb @> '{"a": 1}'::jsonb is TRUE.
GIN index — CREATE INDEX … USING GIN (jsonb_col jsonb_path_ops).

Worked example. Searching event payloads for {"plan": "pro"}:

design	predicate	plan
`payload JSON`	`payload->>'plan' = 'pro'`	Seq Scan
`payload JSONB`	`payload @> '{"plan":"pro"}'::jsonb` (with `GIN`)	Index Scan

Step-by-step.

With plain JSON, every row must be parsed at query time to extract the plan key.
The planner cannot use a B-tree index because the parse step is per-row.
Switching the column to JSONB lets you create a GIN index on the document.
The containment query @> is index-eligible — PostgreSQL probes the GIN structure for documents that contain the requested subtree.
On a 50 M-row table, the difference is full table scan vs sub-second seek.

Worked-example solution. Indexed JSONB column:

CREATE TABLE events (
    event_id BIGSERIAL PRIMARY KEY,
    user_id  BIGINT    NOT NULL,
    payload  JSONB     NOT NULL,
    ts       TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
CREATE INDEX ON events USING GIN (payload jsonb_path_ops);
SELECT COUNT(*) FROM events WHERE payload @> '{"plan":"pro"}';

Rule of thumb: default to JSONB for any "flexible attributes" column; default to a real typed column for any attribute you filter on more than a few times a week.

`UUID` — opaque IDs for distributed systems

The UUID invariant: UUID is a 16-byte fixed-width identifier that does not leak ordering or count; ideal for public IDs, multi-region writes, and any context where you don't want consumers inferring growth rate from the sequence; trade-off vs BIGINT is ~2× storage and worse B-tree locality for monotonic insert patterns. Use UUIDs at the boundary (URLs, foreign systems) and BIGINTs internally if performance is critical.

UUID storage — 16 bytes; gen_random_uuid() from pgcrypto.
v4 (random) — uniform random; great privacy, bad B-tree locality.
v7 (time-ordered) — sortable by creation time; better cache behavior.
UUID vs TEXT — always declare as UUID; TEXT UUIDs lose validation and index efficiency.

Worked example. Two ways to model a public order ID:

design	bytes	URL example
`order_id BIGINT`	8	`/orders/12345678` (leaks volume)
`order_id UUID`	16	`/orders/8c3b7e2a-…` (opaque)

Step-by-step.

BIGINT is monotonic — scraping a few order URLs lets a competitor infer your daily volume.
UUID v4 is unguessable; 8c3b7e2a-… carries no information.
Storage cost: 8 extra bytes per row × millions of rows is meaningful but rarely decisive.
B-tree locality: random UUIDs spread inserts across the index; v7 (time-ordered) restores append-friendly behavior.
For most "public ID" use cases, UUID v7 is the clean middle ground.

Worked-example solution. Internal BIGINT + public UUID:

CREATE EXTENSION IF NOT EXISTS pgcrypto;
CREATE TABLE orders (
    order_id    BIGSERIAL PRIMARY KEY,
    public_id   UUID      NOT NULL UNIQUE DEFAULT gen_random_uuid(),
    customer_id BIGINT    NOT NULL
);

Rule of thumb: expose UUIDs at the API boundary; keep BIGINT joins inside the database.

Arrays — `INTEGER[]`, `TEXT[]`, and the `UNNEST` pattern

The array invariant: PostgreSQL arrays are first-class typed columns; common operations are ANY (arr) for membership, arr @> arr for containment, and UNNEST(arr) to flatten an array column into rows — useful when the list is short (≤ ~10 items) and *bounded by the row*. For unbounded or queried-often lists, a child table is the better design.

INTEGER[] — array of integers; literal '{1,2,3}'::int[].
ANY (arr) — x = ANY ('{1,2,3}'::int[]) is TRUE if x is in the array.
@> — '{1,2,3}'::int[] @> '{2}' is TRUE.
UNNEST(arr) — produces one row per array element; pivot a row of N elements into N rows.

Worked example. A users.role_ids INTEGER[] column:

user_id	role_ids
1	`{10, 20}`
2	`{20, 30}`
3	`{10}`

query	rows
`WHERE 20 = ANY (role_ids)`	1, 2
`WHERE role_ids @> '{10, 20}'`	1
`SELECT user_id, UNNEST(role_ids)`	(1,10), (1,20), (2,20), (2,30), (3,10)

Step-by-step.

Storing roles as INTEGER[] keeps the user table compact — no separate user_roles table for a small bounded set.
ANY is the array-side IN: it tests membership of one value against the column.
@> tests whether the column array contains every element of the right-hand array.
UNNEST flattens the column into rows; joining UNNEST(role_ids) to dim_role produces a per-role row.
For unbounded role sets (10 K+) the array column gets slow and a child table wins; for typical "a user has 1-5 roles" cases, arrays are clean.

Worked-example solution. A small bounded list:

CREATE TABLE users (
    user_id  BIGINT       PRIMARY KEY,
    role_ids INTEGER[]    NOT NULL DEFAULT '{}'::INTEGER[]
);
SELECT u.user_id, r.role_name
FROM users u
JOIN dim_role r ON r.role_id = ANY (u.role_ids);

Rule of thumb: arrays for short, bounded, rarely-filtered lists; child tables for everything else.

Common beginner mistakes

Storing everything as JSONB because "schemas are hard" — you trade type safety and indexability for write-time convenience.
Indexing JSON instead of JSONB — JSON cannot use GIN; the index won't help.
Picking UUID v4 PKs on a high-write table and watching B-tree fragmentation degrade write throughput.
Treating TEXT UUIDs the same as UUID columns — same data, different operator class, broken indexes.
Storing unbounded lists in arrays — once the array exceeds a few dozen entries, every read TOASTs the column and queries slow.

SQL Interview Question on Searching JSONB Payloads at 50 M-Row Scale

A 50 M-row events.payload JSONB column holds variable payloads. Marketing wants to count events where {"plan": "pro"} appears in the payload, and the query takes 60 seconds. Make it return in under 100 ms without changing the storage shape.

Solution Using a `GIN` Index with `jsonb_path_ops` + `@>` Containment

Code solution.

CREATE INDEX events_payload_gin_idx
    ON events
    USING GIN (payload jsonb_path_ops);

SELECT COUNT(*)
FROM events
WHERE payload @> '{"plan":"pro"}'::jsonb;

Step-by-step trace.

step	action	time
1	initial query `payload->>'plan' = 'pro'`	62 s, Seq Scan
2	switch predicate to `payload @> '{"plan":"pro"}'::jsonb` (no index yet)	60 s, still Seq Scan
3	`CREATE INDEX … USING GIN (payload jsonb_path_ops)` (~5 min build)	—
4	rerun containment query	85 ms, GIN Index Scan
5	`EXPLAIN ANALYZE` confirms `Bitmap Heap Scan on events`	one-pass

Output: 60 s → 85 ms — three orders of magnitude faster — with no schema change, no application change, and no data rewrite. EXPLAIN ANALYZE shows the GIN index handling the containment lookup.

Why this works — concept by concept:

@> containment is index-eligible — ->> text extraction is not; the operator choice unlocks the index.
jsonb_path_ops — specialised GIN class for containment-only queries; smaller and faster than the default jsonb_ops.
No row rewrite — CREATE INDEX builds a new index without touching the table heap; existing reads are uninterrupted.
Generalises to other keys — any future payload @> '{"key":"val"}' query benefits; no per-key index needed.
Trade-off — write throughput drops slightly (GIN updates are heavier than B-tree); usually invisible.
Cost — index build is O(N) one-time; reads become O(log N) per query.

Inline CTA: drill the filtering practice page and the SQL practice page for JSON-flavoured patterns.

SQL
Topic — filtering
SQL filtering problems

Practice →

SQL
Language — SQL
All SQL practice problems

Practice →

COURSE
Course — SQL for DE
Zero to FAANG SQL fundamentals

View course →

7. Casting and comparison rules

Implicit coercion, explicit `CAST`, and index-friendly predicates

PostgreSQL silently coerces some type mixes ('42'::text to INTEGER in an = context), refuses others, and lets you make the conversion explicit with CAST(x AS type) or its shorthand x::type. The high-leverage rule is where the cast lands: a cast on a literal is free and index-friendly; a cast on a column usually disables the index. Mixed-type joins are the canonical cause of "the query returns no rows" and "the query is suddenly 100× slower."

Pro tip: When EXPLAIN reveals Seq Scan on … on a column you indexed, scan the Filter: line for a ::type cast. The fix is usually to cast the other side or — better — to change the source column's type so no cast is needed.

Implicit coercion — when PostgreSQL guesses

The coercion invariant: PostgreSQL has a graph of allowed implicit casts (e.g., INTEGER → BIGINT, INTEGER → NUMERIC, TEXT → INTEGER in some contexts) and applies them silently when one side of a binary operator differs from the other; when no implicit cast exists, the query fails with a operator does not exist error. Implicit coercion is convenient until it produces a different answer than expected.

INTEGER = BIGINT — implicit widen; no surprise.
TEXT = INTEGER — works for literals (WHERE id = '42'); fails for columns (WHERE t.id = b.id).
DATE = TIMESTAMPTZ — implicit widen via session zone; can shift.
BOOLEAN = INTEGER — not allowed; you must cast.

Worked example. Joining a TEXT user_id to a BIGINT user_id:

left.user_id (TEXT)	right.user_id (BIGINT)	join works?
`'42'`	`42`	error / Seq Scan
`'042'`	`42`	mismatch (lexicographic ≠ numeric)
`' 42'`	`42`	mismatch (whitespace)

Step-by-step.

PostgreSQL needs both sides of = to be the same type; it tries to coerce.
Coercing TEXT → BIGINT is possible per-value ('42'::BIGINT), but the planner applies it on the column — disabling the index.
Leading zeros, whitespace, and non-digit characters cause the cast to fail mid-query.
The result is either a hard error or a slow seq scan.
The fix is upstream: align the source column types so no cross-type compare is needed.

Worked-example solution. Avoid mixed-type joins:

-- if you must cast, cast at write time, not query time
ALTER TABLE staging_users
    ALTER COLUMN user_id TYPE BIGINT USING NULLIF(user_id, '')::BIGINT;

Rule of thumb: never store an identifier as text on one side and as integer on the other side of a join. Pick one type at the warehouse contract level.

`CAST(x AS type)` vs `x::type` shorthand

The CAST invariant: CAST(x AS type) and x::type produce identical output; the longhand is SQL-standard and self-documenting; the shorthand is PostgreSQL idiomatic and shorter in expression-heavy queries. Both fail with a clear error when the conversion is illegal.

CAST(x AS type) — ANSI SQL; works in every dialect.
x::type — PostgreSQL shorthand.
Failure modes — same for both: invalid input syntax for type integer etc.
NULLIF + CAST — NULLIF(x, '')::INT collapses empty string to NULL before casting.

Worked example. Two equivalent expressions:

SELECT CAST('42' AS BIGINT), '42'::BIGINT;
SELECT CAST('not a number' AS BIGINT);  -- ERROR
SELECT NULLIF('','')::BIGINT;            -- NULL

Step-by-step.

Both CAST and :: produce the same output type and the same value.
Failing input (non-digit string) raises the same error in both forms.
NULLIF(x, '')::TYPE is the canonical "treat empty string as NULL" pattern.
In multi-expression SELECTs, :: keeps lines short; in code-review-heavy contexts, CAST is more legible.
Use whichever your team's house style prefers; do not mix unnecessarily.

Worked-example solution. Safe cast for messy ETL data:

SELECT
    NULLIF(BTRIM(user_id), '')::BIGINT AS user_id,
    raw_payload
FROM staging_events;

Rule of thumb: BTRIM + NULLIF + ::type is the three-step safe-cast pattern for noisy inputs.

Index-killing casts on indexed columns

The index-killer invariant: a WHERE predicate that wraps an indexed column in a function — including an implicit cast — usually forces a sequential scan; the planner cannot prove the function is monotonic on the index, so it falls back to scanning every row. The same query rewritten to cast the literal instead is index-eligible.

col::type = $1 — bad; column cast disables index.
col = $1::type — good; literal cast, index used.
LOWER(col) = $1 — bad unless you build a functional index on LOWER(col).
col = LOWER($1) — good; literal-side function call.

Worked example. A user_id BIGINT column indexed; two predicates:

predicate	plan
`WHERE user_id = 42`	Index Scan
`WHERE user_id::text = '42'`	Seq Scan
`WHERE user_id = '42'::bigint`	Index Scan

Step-by-step.

user_id = 42 matches the type of the indexed column directly.
user_id::text applies a function to every row; the B-tree on the original value cannot be used.
Rewriting as user_id = '42'::bigint casts the literal once and reuses the existing index.
If you genuinely need to query by the casted form, create a functional index: CREATE INDEX ON users ((user_id::text)).
The cheapest fix is almost always to change the data type so no cast is needed.

Worked-example solution. Cast the literal, never the column:

-- good
SELECT * FROM events WHERE user_id = '42';        -- literal coerced
-- bad
SELECT * FROM events WHERE user_id::text = '42';  -- column cast kills the index

Rule of thumb: every :: on the indexed side of a WHERE or JOIN is a code smell. Investigate before merging.

Common beginner mistakes

Joining a TEXT user_id to a BIGINT user_id and adding ::text on the BIGINT side — works but disables the index.
Treating '042' = 42 as TRUE everywhere — leading zeros are preserved in TEXT and lost in INTEGER.
Mixing TIMESTAMP and TIMESTAMPTZ in joins — answers depend on session TZ.
Using LIKE against a numeric column without realising it forces a ::text cast.
Forgetting to handle empty strings before casting — ''::INT is a hard error; use NULLIF.

SQL Interview Question on a Cross-Type Join Returning Zero Rows

staging_users.user_id TEXT joined to dim_users.user_id BIGINT returns 0 rows even though both tables contain user_id = 42. The planner reports a Seq Scan on dim_users. Identify every contributing cause and propose a fix that produces a sound result and keeps the dim's primary-key index usable.

Solution Using a Single-Type Schema + Explicit Literal-Side Cast

Code solution.

-- short-term: cast the staging text to BIGINT (literal-side cast on TEXT)
SELECT d.*
FROM staging_users s
JOIN dim_users d
  ON d.user_id = NULLIF(BTRIM(s.user_id), '')::BIGINT;

-- permanent fix: rewrite staging to BIGINT once
ALTER TABLE staging_users
    ALTER COLUMN user_id TYPE BIGINT
    USING NULLIF(BTRIM(user_id), '')::BIGINT;

Step-by-step trace.

step	symptom	cause
1	`WHERE d.user_id = s.user_id` errors with operator-does-not-exist	type mismatch (BIGINT vs TEXT)
2	analyst rewrites as `WHERE d.user_id::text = s.user_id`	"fixes" the error
3	query returns 0 rows	leading whitespace in `s.user_id` (`' 42'`) breaks lexicographic compare
4	`EXPLAIN` shows Seq Scan on `dim_users`	column cast on `d.user_id` killed the PK index
5	rewrite with `BTRIM` + `NULLIF` + `::BIGINT` on the staging side	index restored, whitespace tolerated
6	row count matches `dim_users.user_id` cardinality	sound result

Output: join now returns the expected rows, the dim's primary-key index is back in the plan, and the permanent ALTER COLUMN removes the per-query cast for good.

Why this works — concept by concept:

Single-type schema — after the ALTER, both sides are BIGINT; no cross-type compare ever runs.
Literal-side BTRIM + NULLIF + ::BIGINT — handles real-world dirty input without disabling the dim's index.
Index on dim_users.user_id preserved — because the cast is on the staging side, not the dim side.
Whitespace-tolerant — BTRIM eliminates the silent-zero-rows mode.
Empty-string-safe — NULLIF(x, '')::BIGINT returns NULL instead of erroring.
Cost — one rewrite at the staging layer; per-query cost drops from full table scan to O(log N) PK seek.

Inline CTA: for the join-fluency syllabus see the joins practice page and the SQL practice page.

SQL
Topic — joins
SQL join problems

Practice →

SQL
Topic — filtering
SQL filtering problems

Practice →

COURSE
Course — SQL for DE
Zero to FAANG SQL fundamentals

View course →

Choosing types (checklist)

If you are storing…	Prefer…	Watch out for…
Surrogate keys, row counts	`BIGINT` / `INTEGER`	Overflow, unnecessary `BIGSERIAL` everywhere
Money, rates, basis points	`NUMERIC(p, s)`	Float rounding in aggregates
Labels, names, free text	`TEXT` or `VARCHAR(n)`	Collation, padding with `CHAR`
Instants in distributed systems	`TIMESTAMPTZ`	Mixing with `TIMESTAMP` in joins
Nested / sparse attributes	`JSONB`	Huge documents without indexes
Public opaque IDs	`UUID`	Stringly-typed UUIDs in joins

Pro tip: When you explain a schema in a live screen, say the grain and the type together: "one row per order, order_id is BIGINT, total is NUMERIC(14,2)."

Frequently asked questions

Should I use `TEXT` or `VARCHAR(255)`?

In PostgreSQL there is no storage penalty for TEXT vs varchar with the same contents. Use VARCHAR(n) when you want the database to enforce a maximum length; otherwise TEXT is simple and common.

Is `SERIAL` still OK for primary keys?

SERIAL / BIGSERIAL are convenient; GENERATED ... AS IDENTITY is the standards-preferred spelling in modern PostgreSQL. Know both in interviews.

Why is my join returning no rows when the IDs "look the same"?

Check types and whitespace on string keys. Compare plans with EXPLAIN: mismatched types can prevent index use or change semantics of comparison. Then rehearse on SQL-tagged problems →.

When must I use `NUMERIC` instead of float?

Whenever exact decimal behavior is required—currency, tax, allocations—or when you must match a ledger or regulatory rule. Floats are for measured magnitudes where error bounds are acceptable.

Practice on PipeCode

PipeCode ships 450+ data engineering practice problems—SQL uses the PostgreSQL dialect, with editorials and topics aligned to what strong companies ask. Start from Explore practice →, open SQL practice →, filter by joins → or aggregations →, and see plans → when you want the full library.

PostgreSQL SQL Cheat Sheet — Clause Order, Joins, Aggregates, Windows

Gowtham Potureddi — Mon, 11 May 2026 03:52:46 +0000

A PostgreSQL SQL cheat sheet is only useful when every row in it maps to something you can drop straight into a query — not a wall of syntax with no operational explanation. This guide condenses real PostgreSQL fluency to four primitives: the logical clause order (FROM → WHERE → GROUP BY → HAVING → SELECT → ORDER BY → LIMIT), the six join shapes and the grain trap they create, GROUP BY with HAVING plus conditional aggregates for one-pass metrics, and window functions like ROW_NUMBER, RANK, DENSE_RANK, LAG, and LEAD for ranking and lookback. These four cover the bulk of analytical SQL — and the cheat-sheet style below is built so you can scan, copy a snippet, and tweak it for your own schema.

Every section walks through a detailed topic explanation, sub-topics with worked examples and runnable solutions, common beginner mistakes, and a worked interview-style scenario with a full traced answer. PostgreSQL syntax throughout — the dialect that drives DataLemur, CoderPad, most product-analytics live screens, and the bulk of modern data-engineering SQL corpora.

#	Topic	Why it shows up in a PostgreSQL cheat sheet
1	Logical clause order — `FROM` → `WHERE` → `GROUP BY` → `HAVING` → `SELECT` → `ORDER BY` → `LIMIT`	The single most useful PostgreSQL mental model: the order you write clauses is not the order the engine evaluates them; knowing the evaluation order eliminates 80% of parse errors and explains why `WHERE` cannot reference aggregates or column aliases.
2	Joins and grain — `INNER`, `LEFT`, `RIGHT`, `FULL`, `SELF`, `CROSS`	Joins combine rows but they also change grain; a careless `1:N` join inflates row counts silently, and the `LEFT JOIN ... IS NULL` anti-join is the canonical "find rows in A with no match in B" pattern (orphan customers, churned users).
3	`GROUP BY`, `HAVING`, and conditional aggregates	`WHERE` filters rows before grouping; `HAVING` filters groups after; `COUNT() FILTER (WHERE …)` and `SUM(CASE WHEN …)` express many metrics in one query — the universal duplicate finder `HAVING COUNT() > 1` lives here.
4	Window functions — `ROW_NUMBER`, `RANK`, `DENSE_RANK`, `LAG`, `LEAD`	Per-partition ranking without collapsing rows, top-N-per-group, second-highest salary, running totals with `SUM(...) OVER (PARTITION BY ... ORDER BY ...)`, and month-over-month deltas via `LAG`; the most-graded primitive in modern SQL screens.

1. PostgreSQL Logical Clause Order — `FROM` → `WHERE` → `GROUP BY` → `HAVING` → `SELECT` → `ORDER BY` → `LIMIT`

The seven-stage evaluation order every PostgreSQL query follows

"Why does WHERE customer_count > 5 give me a parse error when I'm clearly counting customers?" is the signature beginner question — and the answer is logical clause order. The mental model: PostgreSQL evaluates clauses in a fixed order that is different from the order you write them; FROM/JOIN builds the row set, WHERE filters rows, GROUP BY collapses rows into groups, HAVING filters groups, SELECT projects columns, ORDER BY sorts, LIMIT/OFFSET trims. WHERE cannot reference aggregate functions because aggregates do not exist until after GROUP BY; column aliases declared in SELECT cannot be referenced in WHERE for the same reason.

Pro tip: Memorize one sentence — "From-Where-Group-Having-Select-Order-Limit" — and you can decode any PostgreSQL parse error in under five seconds. The error column "customer_count" does not exist almost always means the column is a SELECT-level alias being referenced in WHERE, which runs three stages earlier; lift the predicate into HAVING (if it references an aggregate) or repeat the expression inline in WHERE.

`FROM` and `JOIN` — build the working row set

The FROM/JOIN invariant: the first stage assembles a candidate row set by listing the tables (and how they join); every subsequent stage operates on this row set. Subqueries in FROM are also evaluated here, and LATERAL joins let later subqueries reference earlier rows.

Single table — FROM orders produces one row per orders row.
Joined tables — FROM orders o JOIN customers c ON c.id = o.customer_id produces one row per matching pair.
Subquery in FROM — FROM (SELECT ...) t materializes the inner result, then treats it as a table.
LATERAL subquery — FROM orders o, LATERAL (SELECT ... WHERE x = o.id) s re-evaluates the inner subquery per outer row.

Worked example. A FROM with a LEFT JOIN that produces the right row set before any filter runs.

step	output cardinality
`FROM customers` alone	3 rows
`LEFT JOIN orders`	4 rows (Alice has 2 orders, Bob 1, Carol 0 padded with NULLs)
ready for `WHERE` filtering	4 rows

Step-by-step explanation.

The engine reads customers first, producing three rows (Alice, Bob, Carol).
For each customer, it scans orders for matching customer_id rows; Alice matches 2 orders, Bob matches 1, Carol matches 0.
Because the join is LEFT, Carol's row is preserved with the right-side columns filled with NULLs — total 4 rows.
This 4-row stream is what WHERE will see; no filtering has happened yet.
Without understanding FROM runs first, you can't reason about why a WHERE predicate on the right side of a LEFT JOIN silently converts the join into an INNER JOIN.

Worked-example solution.

SELECT c.name, o.order_id, o.amount
FROM customers c
LEFT JOIN orders o
  ON o.customer_id = c.id;

Rule of thumb: if your LEFT JOIN is producing fewer rows than expected, check whether you have a WHERE predicate that references the right-side table — that predicate runs after the join and discards the NULL-padded rows.

`WHERE` — row-level predicates before grouping

The WHERE invariant: WHERE filters individual rows from the FROM/JOIN output before GROUP BY runs; it can reference any column from the joined row set, but cannot reference aggregate functions or SELECT-level aliases. This is the cheapest place to drop rows — push predicates here whenever possible.

Row predicates — WHERE amount > 30, WHERE order_date >= '2026-01-01'.
IN / EXISTS — WHERE customer_id IN (SELECT id FROM premium), WHERE EXISTS (...).
BETWEEN — inclusive on both ends; WHERE x BETWEEN 1 AND 10 is x >= 1 AND x <= 10.
IS NULL / IS NOT NULL — the only way to check for NULL; never = NULL.

Worked example. Filter to one day of orders before grouping.

filter	rows surviving
no filter	12,847 (full day's orders)
`WHERE order_date = '2026-05-10'`	12,847 (already today's)
`WHERE order_date = '2026-05-10' AND amount > 100`	4,290 (high-value only)

Step-by-step explanation.

FROM orders returns the full row stream.
WHERE order_date = '2026-05-10' is evaluated per row; rows with other dates are dropped.
AND amount > 100 is evaluated next; this is a row predicate (not an aggregate), so it lives in WHERE correctly.
The surviving row set (4,290 rows) flows into GROUP BY if one is present, otherwise into SELECT.
Pushing the date filter into WHERE rather than HAVING is critical for index usage: a B-tree index on order_date can prune 95% of the table before any grouping happens.

Worked-example solution.

SELECT customer_id, SUM(amount) AS total
FROM orders
WHERE order_date = '2026-05-10'
  AND amount > 100
GROUP BY customer_id;

Rule of thumb: if the predicate uses only raw row columns, it belongs in WHERE; if it uses SUM, COUNT, AVG, MIN, MAX, it belongs in HAVING.

`GROUP BY` → `HAVING` → `SELECT` → `ORDER BY` → `LIMIT`

The downstream invariant: after WHERE, the engine evaluates GROUP BY (collapsing rows into one row per distinct key combination), then HAVING (filtering groups), then SELECT (projecting columns and computing expressions), then ORDER BY (sorting the final result), then LIMIT/OFFSET (trimming for pagination). SELECT-level aliases become referenceable only in ORDER BY and the outer query (in a subquery context).

GROUP BY col1, col2 — one output row per distinct (col1, col2) combination.
HAVING agg_pred — filter groups; can reference COUNT(*), SUM(col), etc.
SELECT col, agg(col2) AS x — project columns; aggregates and aliases are computed here.
ORDER BY x DESC, col — can reference SELECT aliases; deterministic with a tiebreaker.
LIMIT N OFFSET M — page slicing; always pair with ORDER BY for determinism.

Worked example. Group by customer, filter to high-spend customers, sort descending, top 5.

stage	rows
`FROM orders WHERE order_date = '2026-05-10'`	4,290
`GROUP BY customer_id`	1,720 (one row per customer)
`HAVING SUM(amount) > 500`	312 (high-spend)
`SELECT customer_id, SUM(amount) AS spend`	312 (projected)
`ORDER BY spend DESC, customer_id`	312 (sorted)
`LIMIT 5`	5

Step-by-step explanation.

WHERE produces 4,290 rows for one day with amount > 100.
GROUP BY customer_id collapses them into 1,720 buckets, one per customer.
HAVING SUM(amount) > 500 keeps only the 312 buckets whose total spend exceeds $500.
SELECT computes the alias spend = SUM(amount) and projects two columns.
ORDER BY spend DESC, customer_id sorts the 312 surviving rows by descending spend with a deterministic tiebreaker; LIMIT 5 returns just the top five.

Worked-example solution.

SELECT customer_id, SUM(amount) AS spend
FROM orders
WHERE order_date = '2026-05-10'
  AND amount > 100
GROUP BY customer_id
HAVING SUM(amount) > 500
ORDER BY spend DESC, customer_id
LIMIT 5;

Rule of thumb: every clause has a fixed slot; if you find yourself wanting WHERE to reference an aggregate, the predicate belongs in HAVING instead — and if you want ORDER BY to use a long expression, alias it in SELECT and reference the alias.

Common beginner mistakes

WHERE COUNT(*) > 1 — parse error; aggregates do not exist until after GROUP BY. Use HAVING.
Referencing a SELECT alias in WHERE — WHERE spend > 100 after SELECT SUM(amount) AS spend fails; either repeat the expression or move to HAVING.
Selecting a non-aggregated, non-GROUP BY column — strict PostgreSQL errors out with "must appear in GROUP BY"; some other dialects pick an arbitrary row silently.
LIMIT 5 without ORDER BY — non-deterministic; two runs of the same query return different rows.
Putting HAVING before GROUP BY — syntax error; the clause order is mandatory.

PostgreSQL Interview Question on Clause Order

Given orders(order_id, customer_id, amount, order_date), find every customer who placed more than 3 orders today with total spend above $500. Return customer_id and total_spend, sorted by total_spend descending.

Solution Using `WHERE` + `GROUP BY` + `HAVING` in the Right Slots

SELECT customer_id,
       SUM(amount) AS total_spend
FROM orders
WHERE order_date = CURRENT_DATE
GROUP BY customer_id
HAVING COUNT(*) > 3
   AND SUM(amount) > 500
ORDER BY total_spend DESC, customer_id;

Why this works: WHERE order_date = CURRENT_DATE filters to today's row set first (cheap, index-friendly); GROUP BY customer_id collapses to one row per customer; HAVING evaluates the two aggregate predicates together (more than 3 orders AND total > $500); SELECT projects the alias total_spend; ORDER BY total_spend DESC, customer_id produces a deterministic ordering. Single pass over today's rows with hash aggregation.

Step-by-step trace for sample data on 2026-05-10:

customer_id	orders today	sum(amount)	passes HAVING?
101	5	720	✓
102	4	410	✗ (sum ≤ 500)
103	6	1,250	✓
104	2	800	✗ (count ≤ 3)
105	4	520	✓

Three customers survive both predicates.

Output:

customer_id	total_spend
103	1250
101	720
105	520

Why this works — concept by concept:

WHERE first — order_date = CURRENT_DATE is a row predicate using a non-aggregated column; pushing it into WHERE shrinks the row set before grouping and lets the planner use a B-tree index on order_date.
GROUP BY customer_id — collapses today's rows into one bucket per customer; every subsequent aggregate is computed inside this bucket.
HAVING two-predicate AND — COUNT(*) > 3 and SUM(amount) > 500 are both aggregate predicates; combining them with AND in a single HAVING is the canonical multi-condition group filter.
SELECT projection + alias — SUM(amount) AS total_spend is computed here; the alias becomes available to ORDER BY (but not to WHERE / HAVING).
ORDER BY total_spend DESC, customer_id — descending sort on the metric with a deterministic tiebreaker via customer_id; reviewers depend on stable ordering.
O(|today's orders| + G log G) time — single hash aggregation produces G groups; final sort is G log G. With an index on (order_date, customer_id) the planner can stream rather than hash.

Inline CTA: Drill the SQL filtering practice page for WHERE patterns and the SQL aggregation practice page for GROUP BY + HAVING shapes.

SQL
Topic — filtering
SQL filtering problems

Practice →

SQL
Topic — aggregation
SQL aggregation problems

Practice →

SQL
Language — SQL
All SQL practice problems

Practice →

2. PostgreSQL Joins and Grain — `INNER`, `LEFT`, `RIGHT`, `FULL`, `SELF`, `CROSS`

Joins, anti-joins, and the grain-inflation trap in PostgreSQL

"Why is SUM(amount) returning double what I expect after I add a JOIN?" is the signature grain-inflation question — and the answer is that joins do not just combine columns; they change the row cardinality of the result. The mental model: INNER JOIN keeps only matching pairs, LEFT JOIN keeps every left row and pads the right side with NULLs, RIGHT JOIN is the mirror, FULL OUTER JOIN keeps both sides' unmatched rows, SELF JOIN joins a table to itself (for hierarchies and pair queries), CROSS JOIN produces a Cartesian product (one row per (left, right) pair). The cardinality of any join is bounded by |left| × |right|, and a 1:N relationship inflates left rows by N — the silent source of doubled metrics.

Pro tip: Before writing any join, ask "what is the grain of the result?" — orders, order lines, customer-day, or (customer, product) pair. A 1:N join (e.g., customers to orders) inflates customer rows by the number of orders; SUM(customer.lifetime_value) after that join returns lifetime value × order count, not lifetime value. Always state the grain out loud.

`INNER JOIN` — keep only matching pairs (no padding)

The INNER JOIN invariant: a left row is paired with a right row iff the join predicate is TRUE; unmatched rows on either side are discarded; the result cardinality is the count of matching pairs. This is the most common join and the fastest because the planner can short-circuit on no-match.

ON l.key = r.key — single-column equi-join; the planner hashes the right table.
Multi-column — ON l.a = r.a AND l.b = r.b for composite keys.
Non-equi — ON l.range_start <= r.point AND l.range_end >= r.point (range join).
USING (col) — shorthand when both sides share the column name; merges the column.

Worked example. Two tables, three customers, two orders; one customer has no order.

customer	order_id
Alice	101
Bob	102

Carol (no orders) does not appear — INNER JOIN dropped her.

Step-by-step explanation.

The engine reads customers (Alice, Bob, Carol) and orders (101 for Alice, 102 for Bob).
For each customers row, it scans orders for a matching customer_id.
Alice matches order_id = 101; Bob matches order_id = 102; Carol has no match.
Carol's row is silently discarded because the join is INNER — no NULL-padded row is produced.
The output has two rows because there were two matching pairs; the result cardinality is min(|customers|, |orders|) ≤ N ≤ |customers| × |orders|.

Worked-example solution.

SELECT c.name AS customer, o.order_id
FROM customers c
INNER JOIN orders o
  ON o.customer_id = c.id;

Rule of thumb: reach for INNER JOIN whenever the question is "rows where both sides exist"; it is the smallest, fastest, most common join.

`LEFT JOIN` — keep every left row, pad the right with `NULL`s (anti-join trick)

The LEFT JOIN invariant: every row from the left table appears in the output; if no right row matches, the right columns are NULL; LEFT JOIN ... WHERE right.key IS NULL keeps exactly the left rows that had no match — the anti-join idiom. RIGHT JOIN is the mirror; flip the table order and use LEFT for consistency.

LEFT JOIN — preserves every left row.
Right columns NULL when no match — the key signal for anti-joins.
LEFT JOIN ... IS NULL anti-join — "find rows in A with no match in B".
RIGHT JOIN — mirror image; rarely needed (just flip table order and use LEFT).

Worked example. Same customers + orders; Carol is preserved with NULL right-side columns.

customer	order_id
Alice	101
Bob	102
Carol	NULL

Step-by-step explanation.

For each customers row, scan orders for a matching customer_id.
Alice matches → row (Alice, 101); Bob matches → row (Bob, 102).
Carol does not match → row (Carol, NULL) is produced because the join is LEFT.
To find Carol via the anti-join: add WHERE o.order_id IS NULL after the LEFT JOIN; only Carol's row passes the filter.
Equivalent to WHERE NOT EXISTS (SELECT 1 FROM orders WHERE customer_id = c.id) and (under NOT NULL constraints) WHERE c.id NOT IN (SELECT customer_id FROM orders) — but the anti-join is immune to the NOT IN NULL-swallowing bug.

Worked-example solution.

SELECT c.name AS customer
FROM customers c
LEFT JOIN orders o
  ON o.customer_id = c.id
WHERE o.order_id IS NULL;

Rule of thumb: "find X with no Y" → LEFT JOIN ... WHERE Y.id IS NULL. Memorize this; it is the most-asked join shape in SQL interviews and the cleanest fix for the NOT IN ... NULL trap.

`FULL OUTER`, `SELF`, and `CROSS` joins — the rarer shapes

The rarer-joins invariant: FULL OUTER JOIN keeps every left row AND every right row (with NULL padding on the unmatched side); SELF JOIN joins a table to itself by aliasing it twice (employees-and-managers, parent-child, pair queries); CROSS JOIN produces every (left, right) combination — the Cartesian product. Each has a narrow but important use.

FULL OUTER JOIN — reconcile two sources; rows from either side without a match get padded.
SELF JOIN — employee/manager, hierarchical recursion (alternative to recursive CTE), pair queries.
CROSS JOIN — generate every combination (small tables only) or paired with LATERAL for top-N per row.
Implicit cross join — comma-separated tables (FROM a, b) without an ON is a CROSS JOIN — usually a bug.

Worked example. Self-join employees to itself to surface each person's manager.

name	manager_name
Alice	Carol
Bob	Carol
Carol	NULL (CEO)

Step-by-step explanation.

Alias the same employees table twice: e (for employees) and m (for managers).
LEFT JOIN on e.manager_id = m.emp_id looks each employee up against the manager rows.
Alice's manager_id points to Carol → row (Alice, Carol).
Bob's manager_id points to Carol → row (Bob, Carol).
Carol is the CEO so her manager_id IS NULL → no match → row (Carol, NULL) because the join is LEFT.

Worked-example solution.

SELECT e.name,
       m.name AS manager_name
FROM employees e
LEFT JOIN employees m
  ON m.emp_id = e.manager_id;

Rule of thumb: SELF JOIN is one-level hierarchy; for arbitrary-depth recursion (org chart traversal, BOM tree), reach for WITH RECURSIVE instead.

Common beginner mistakes

Forgetting that a 1:N JOIN inflates the left side — SUM(left.col) returns left.col × N.
Filtering the right table inside WHERE after a LEFT JOIN (e.g., WHERE o.amount > 0) — silently turns the LEFT JOIN into an INNER JOIN because NULL > 0 is NULL.
Using NOT IN (subquery) when the subquery can contain NULL — returns zero rows because x NOT IN (..., NULL, ...) is NULL, which fails the predicate.
Comma-separated FROM a, b with no ON clause — produces a Cartesian product (CROSS JOIN); usually a bug.
Joining on the wrong column (o.id = c.id instead of o.customer_id = c.id) — produces nonsense rows.

PostgreSQL Interview Question on Customers With No Orders

Given customers(id, name) and orders(order_id, customer_id, amount), return the names of customers who have never placed an order.

Solution Using `LEFT JOIN ... WHERE orders.order_id IS NULL`

SELECT c.name
FROM customers c
LEFT JOIN orders o
  ON o.customer_id = c.id
WHERE o.order_id IS NULL
ORDER BY c.name;

Why this works: the LEFT JOIN preserves every customer row regardless of whether a matching order exists; for matched customers, o.order_id carries a real value; for unmatched customers, the right-side columns are NULL and the WHERE o.order_id IS NULL predicate is TRUE; the filter keeps only the unmatched customers — the anti-join. Single pass over customers; one keyed lookup into orders per customer; no subquery materialization needed.

Step-by-step trace for the sample input:

customers.id	customers.name	LEFT JOIN orders.order_id	IS NULL?	survives?
1	Alice	101	✗	✗
2	Bob	102	✗	✗
3	Carol	NULL	✓	✓
4	Dan	NULL	✓	✓

Carol and Dan survive the filter.

Output:

name
Carol
Dan

Why this works — concept by concept:

LEFT JOIN semantics — keeps every left row; right side is NULL when there is no match. This NULL is the entire signal we filter on.
WHERE o.order_id IS NULL — o.order_id is the right-side primary key; it is NULL only when the join produced a synthetic unmatched row. A real-NULL order-id from the source table never happens because primary keys are NOT NULL.
Anti-join semantics — equivalent to NOT EXISTS (SELECT 1 FROM orders WHERE customer_id = c.id); the LEFT JOIN ... IS NULL form is typically faster on planners that materialize a hash join.
No NULL-swallowing — unlike NOT IN, the predicate is IS NULL, which is well-defined for NULL values. There is no silent zero-row failure.
ORDER BY c.name — deterministic ordering for reviewer stability.
O(|customers| + |orders|) time — hash-join build on orders.customer_id, single probe per customer. With an index on orders.customer_id this is near-linear.

Inline CTA: Drill the SQL joins practice page for INNER, LEFT, and anti-join shapes, and the SQL null-handling practice page for NULL-aware predicates.

SQL
Topic — joins
SQL join problems

Practice →

SQL
Topic — null handling
SQL null-handling problems

Practice →

SQL
Topic — filtering
SQL filtering problems

Practice →

3. PostgreSQL `GROUP BY`, `HAVING`, and Conditional Aggregates

`GROUP BY` with `HAVING`, `FILTER`, and `CASE` for one-pass metrics in PostgreSQL

"Compute total revenue, refunded revenue, and the percentage refunded — in a single query" is the signature conditional-aggregate prompt — and the cleanest PostgreSQL answer is SUM(... ) FILTER (WHERE …) clauses inside a single SELECT. The mental model: GROUP BY col collapses rows into buckets; COUNT(*), SUM(...), AVG(...), MIN(...), MAX(...) summarize each bucket; WHERE filters individual rows before grouping; HAVING filters groups after grouping; FILTER (WHERE …) and CASE WHEN … express conditional aggregates that count or sum only certain rows per group. The duplicate-finder pattern GROUP BY key HAVING COUNT(*) > 1 lives here too.

Pro tip: PostgreSQL supports the SQL standard FILTER (WHERE …) clause on every aggregate — COUNT(*) FILTER (WHERE status = 'refunded'). It produces clearer queries than SUM(CASE WHEN … THEN 1 ELSE 0 END) and is exactly what interviewers like to see. The CASE variant still works for portability across dialects.

`COUNT`, `SUM`, `AVG`, `MIN`, `MAX` — `NULL`-aware aggregates

The aggregate-NULL invariant: COUNT(*) counts every row including ones with NULL columns; COUNT(col) counts only rows where col is not NULL; SUM, AVG, MIN, MAX skip NULL values entirely; if every value in a group is NULL, the result is NULL (not 0). The distinction between COUNT(*) and COUNT(col) is the #1 source of "my counts are off by 10%" bugs.

COUNT(*) — every row in the bucket, regardless of NULLs.
COUNT(col) — non-NULL values of col only.
COUNT(DISTINCT col) — unique non-NULL values; essential after a JOIN that may have inflated rows.
SUM / AVG — numeric only; AVG is sum-of-non-null-divided-by-count-of-non-null, so NULL does not count as 0.

Worked example. Three rows in one customer's bucket: amount = 10, NULL, 30.

aggregate	result
`COUNT(*)`	3
`COUNT(amount)`	2
`SUM(amount)`	40
`AVG(amount)`	20
`MIN(amount)`	10
`MAX(amount)`	30

Step-by-step explanation.

COUNT(*) = 3 because every row in the bucket counts, regardless of amount's value.
COUNT(amount) = 2 because the NULL row is skipped; only 10 and 30 contribute.
SUM(amount) = 10 + 30 = 40; the NULL is treated as missing, not as 0.
AVG(amount) = (10 + 30) / 2 = 20; the denominator is COUNT(amount) = 2, not COUNT(*) = 3.
MIN and MAX skip the NULL and return the smallest/largest non-NULL value.

Worked-example solution.

SELECT customer_id,
       COUNT(*)        AS n_rows,
       COUNT(amount)   AS n_known,
       SUM(amount)     AS total,
       AVG(amount)     AS mean,
       MIN(amount)     AS lo,
       MAX(amount)     AS hi
FROM orders
GROUP BY customer_id;

Rule of thumb: if the metric is "people who clicked" use COUNT(DISTINCT user_id); if it is "click events" use COUNT(*); if it is "rows with a known value" use COUNT(col).

`WHERE` vs `HAVING` — row filter vs group filter

The two-clause invariant: WHERE runs before GROUP BY and references raw row columns; HAVING runs after grouping and can reference aggregate functions; trying to use WHERE COUNT(*) > 1 is a parse error because aggregates do not exist until after grouping. Both can appear in the same query.

WHERE — filter rows; uses col, col2, etc.
HAVING — filter groups; uses COUNT(*), SUM(col), etc.
Order of evaluation — FROM → WHERE → GROUP BY → HAVING → SELECT.
Performance — push predicates into WHERE whenever possible; WHERE filters before the (often expensive) sort/hash step.

Worked example. Six employees across eng and sales; find departments whose average salary exceeds 50,000 across employees earning more than 30,000.

department	salary
eng	40,000
eng	70,000
eng	25,000
sales	60,000
sales	60,000
sales	20,000

Step-by-step explanation.

WHERE salary > 30000 drops the two rows below the threshold (eng 25,000 and sales 20,000) — 4 rows remain.
GROUP BY department collapses to two buckets: eng (40,000 + 70,000) and sales (60,000 + 60,000).
The planner computes AVG(salary) per bucket: eng = 55,000; sales = 60,000.
HAVING AVG(salary) > 50000 keeps both buckets (both averages exceed 50,000).
The SELECT projects the department name and its average; final result is two rows.

Worked-example solution.

SELECT department, AVG(salary) AS avg_salary
FROM employees
WHERE salary > 30000
GROUP BY department
HAVING AVG(salary) > 50000;

Rule of thumb: aggregate predicate → HAVING; row predicate → WHERE. If the predicate uses SUM / COUNT / AVG / MIN / MAX, it must live in HAVING.

`FILTER (WHERE …)` and `CASE` — conditional aggregates

The conditional-aggregate invariant: SUM(col) FILTER (WHERE pred) and COUNT(*) FILTER (WHERE pred) apply the aggregate only to rows where the predicate is TRUE; the portable alternative is SUM(CASE WHEN pred THEN col ELSE 0 END) and COUNT(CASE WHEN pred THEN 1 END). PostgreSQL supports both; pick FILTER for clarity in PostgreSQL-only code, CASE for cross-dialect portability.

FILTER (WHERE …) — PostgreSQL/SQL-standard syntax; applies per-aggregate.
SUM(CASE WHEN … THEN col ELSE 0 END) — portable across dialects.
COUNT(CASE WHEN … THEN 1 END) — counts only matching rows; NULLs in the ELSE branch are skipped.
Multiple aggregates, one query — combine many FILTER clauses to compute several metrics in one pass.

Worked example. One pass over orders to compute total revenue, refunded revenue, and the refund rate.

customer_id	total_revenue	refunded_revenue	refund_pct
101	500	50	10.0
102	1,000	0	0.0
103	800	200	25.0

Step-by-step explanation.

SUM(amount) aggregates every row in the bucket → total_revenue.
SUM(amount) FILTER (WHERE status = 'refunded') aggregates only refunded rows → refunded_revenue.
The refund percentage is refunded_revenue / total_revenue * 100; cast one side to NUMERIC to avoid integer division.
PostgreSQL evaluates every FILTER independently per row of input; one scan computes all metrics.
The portable variant uses SUM(CASE WHEN status = 'refunded' THEN amount ELSE 0 END) — same result, slightly more verbose.

Worked-example solution.

SELECT customer_id,
       SUM(amount)                              AS total_revenue,
       SUM(amount) FILTER (WHERE status = 'refunded') AS refunded_revenue,
       ROUND(
         SUM(amount) FILTER (WHERE status = 'refunded')::NUMERIC
         / NULLIF(SUM(amount), 0) * 100, 1
       ) AS refund_pct
FROM orders
GROUP BY customer_id;

Rule of thumb: whenever you find yourself running two queries with different WHERE clauses against the same table and joining the results, refactor to a single query with two FILTER clauses — same answer, half the cost.

Common beginner mistakes

WHERE COUNT(*) > 1 — parse error; aggregates do not exist until after GROUP BY. Use HAVING.
AVG(col) and assuming NULL rows count as 0 — they are excluded from both numerator and denominator. Use AVG(COALESCE(col, 0)) only if "missing means 0" is the business rule.
COUNT(DISTINCT col) forgotten after a JOIN that inflates rows — reports inflated counts.
Integer division — 5 / 100 = 0 in PostgreSQL. Cast one operand to NUMERIC or FLOAT.
Division by zero — NULLIF(denom, 0) converts 0 to NULL, so the division returns NULL instead of erroring.

PostgreSQL Interview Question on Duplicate Emails

Given users(id, email), return every email that appears more than once, along with the number of copies.

Solution Using `GROUP BY email HAVING COUNT(*) > 1`

SELECT email,
       COUNT(*) AS n_copies
FROM users
GROUP BY email
HAVING COUNT(*) > 1
ORDER BY n_copies DESC, email;

Why this works: GROUP BY email collapses every row with the same email into a single bucket; COUNT(*) counts how many rows fell into each bucket; HAVING COUNT(*) > 1 keeps only buckets with at least two rows; ORDER BY n_copies DESC, email produces a deterministic, reviewer-friendly output. Single pass over users; sort cost dominates only when email cardinality is huge.

Step-by-step trace for the sample input:

id	email
1	alice@example.com
2	bob@example.com
3	alice@example.com
4	carol@example.com
5	bob@example.com
6	alice@example.com

FROM users — read all six rows.
No WHERE — every row passes.
GROUP BY email — three buckets: alice (3 rows), bob (2 rows), carol (1 row).
COUNT(*) — 3, 2, 1 respectively.
HAVING COUNT(*) > 1 — drops the carol bucket.
ORDER BY n_copies DESC, email — alice (3), then bob (2).

Output:

email	n_copies
alice@example.com	3
bob@example.com	2

Why this works — concept by concept:

GROUP BY email — collapses to one bucket per distinct email; the bucket is the unit of all subsequent aggregates and group-level filters.
COUNT(*) — counts every row in the bucket, perfect for "how many copies".
HAVING COUNT(*) > 1 — group-level filter; the aggregate predicate must live here, not in WHERE. This is the precise interview signal for duplicate detection.
ORDER BY n_copies DESC, email — deterministic ordering; tie-broken by email so the output is stable across runs.
O(|users| + G log G) time — single hash aggregation produces G group rows; the final sort is G log G. With an index on email, the planner may use stream aggregation and skip the hash step.

Inline CTA: Drill the SQL aggregation practice page for GROUP BY and HAVING shapes, and the SQL filtering practice page for WHERE vs HAVING distinctions.

SQL
Topic — aggregation
SQL aggregation problems

Practice →

SQL
Topic — filtering
SQL filtering problems

Practice →

SQL
Topic — null handling
SQL null-handling problems

Practice →

4. PostgreSQL Window Functions — `ROW_NUMBER`, `RANK`, `DENSE_RANK`, `LAG`, `LEAD`

Ranking, top-N-per-group, running totals, and lookback in PostgreSQL window functions

"Find the second-highest distinct salary" and "compute a running total of daily revenue" are the two signature window-function prompts — and both reduce to a window function with OVER (PARTITION BY … ORDER BY …). The mental model: a window function computes a value across a window of rows related to the current row without collapsing the rows like GROUP BY does; OVER (PARTITION BY col) defines the window boundary; OVER (ORDER BY col) defines the order within the window. ROW_NUMBER assigns unique integers; RANK skips after ties (1, 2, 2, 4); DENSE_RANK does not skip (1, 2, 2, 3); LAG looks back; LEAD looks forward; SUM/AVG/COUNT(...) OVER (...) compute running totals and moving averages.

Pro tip: Window functions cannot be referenced in WHERE of the same SELECT because they execute after WHERE. Wrap the window in a CTE or subquery, then filter on the alias. The error column "rn" does not exist after writing WHERE rn = 1 almost always means you forgot this rule.

`ROW_NUMBER` — unique sequential numbering per partition

The ROW_NUMBER invariant: ROW_NUMBER() OVER (PARTITION BY p ORDER BY o) assigns a unique integer 1, 2, 3, … to every row inside each partition p, ordered by o; ties in o are broken arbitrarily by the planner. Use it when you need a unique sequence per group regardless of tie semantics — most often for deduplication (keep rn = 1 per business key).

OVER (PARTITION BY …) — bucket the rows; without this, the window is the whole result set.
OVER (ORDER BY …) — order within the bucket; required for ROW_NUMBER/RANK/LAG/LEAD.
Ties broken arbitrarily — add a tiebreaker column to ORDER BY for determinism.
Top-N-per-group — WHERE rn <= N after ROW_NUMBER; works only when ties at rank N can be ignored.

Worked example. employees with three engineers; rank by salary descending.

department	name	salary	row_number
eng	Alice	90,000	1
eng	Bob	80,000	2
eng	Carol	80,000	3

Bob and Carol tie on salary; ROW_NUMBER still gives them unique ranks (planner-chosen unless you add a tiebreaker).

Step-by-step explanation.

PARTITION BY department defines the boundary — only eng rows are compared with each other; if there were a sales partition it would have its own 1, 2, 3 sequence.
ORDER BY salary DESC, name orders rows within the partition: Alice (90,000) first, then Bob and Carol (tied at 80,000) broken by name.
ROW_NUMBER() assigns 1, 2, 3 sequentially regardless of ties; Bob gets 2 and Carol gets 3 because name breaks the tie.
Without the , name tiebreaker, Bob/Carol order is undefined — two query runs could swap them.
To deduplicate a table that has multiple rows per (business_key, source_ts), use ROW_NUMBER() OVER (PARTITION BY business_key ORDER BY source_ts DESC) = 1 to keep the latest.

Worked-example solution.

SELECT department, name, salary,
       ROW_NUMBER() OVER (
         PARTITION BY department
         ORDER BY salary DESC, name
       ) AS rn
FROM employees;

Rule of thumb: ROW_NUMBER is the right tool for deduplication (WHERE rn = 1) and for ordered streams; reach for RANK or DENSE_RANK when ties must be honored.

`RANK` vs `DENSE_RANK` — tie semantics

The rank-vs-dense-rank invariant: both assign the same rank to tied rows; RANK then skips the next k-1 ranks (gap), while DENSE_RANK continues without a gap. For "find the Nth distinct value" questions, DENSE_RANK = N is the correct filter; for "find the Nth row in skip-aware ranking order", RANK = N is correct.

RANK — 1, 2, 2, 4 — skips after ties.
DENSE_RANK — 1, 2, 2, 3 — no skip.
ROW_NUMBER — 1, 2, 3, 4 — never ties.
Pick by semantics — "Nth highest distinct salary" → DENSE_RANK = N; "Nth-ranked row in skip ordering" → RANK = N.

Worked example. Four employees; Bob and Carol tied at second-highest salary.

name	salary	rank	dense_rank	row_number
Alice	90,000	1	1	1
Bob	80,000	2	2	2
Carol	80,000	2	2	3
Dan	70,000	4	3	4

RANK jumps 2 → 4 (skipping 3); DENSE_RANK continues 2 → 3 (no skip).

Step-by-step explanation.

All three window functions agree on Alice (rank 1) because she is alone at the top.
Bob and Carol both get rank = 2 and dense_rank = 2 because they tie on salary; row_number gives them distinct values 2 and 3.
Dan is the next-lowest salary; RANK skips ahead by the number of tied rows (2 tied → next rank is 2 + 2 = 4); DENSE_RANK continues with no gap (3).
For "second highest distinct salary", DENSE_RANK = 2 correctly returns 80,000; RANK = 2 would also work here, but RANK would not return 80,000 if three people tied for first (it would skip to 4).
For "top 3 distinct salaries", use DENSE_RANK <= 3 — it returns Alice, Bob, Carol, Dan (four rows because Bob/Carol both have dr = 2).

Worked-example solution.

SELECT name, salary,
       RANK()       OVER (ORDER BY salary DESC) AS rnk,
       DENSE_RANK() OVER (ORDER BY salary DESC) AS dr,
       ROW_NUMBER() OVER (ORDER BY salary DESC) AS rn
FROM employees;

Rule of thumb: "second highest salary" → DENSE_RANK = 2; "top 3 distinct salaries" → DENSE_RANK <= 3; never use RANK for these unless the spec explicitly says ties should consume rank slots.

`LAG`, `LEAD`, and running totals — lookback, lookahead, and `SUM(...) OVER (...)`

The lookback-and-running invariant: LAG(col, n) returns the value of col n rows back within the partition (default n=1); LEAD(col, n) is the symmetric forward; SUM(col) OVER (PARTITION BY p ORDER BY o) produces a running total within each partition. These three primitives drive month-over-month deltas, sessionization, running balances, and moving averages.

LAG(amount) OVER (ORDER BY date) — previous day's amount.
LEAD(amount) OVER (ORDER BY date) — next day's amount.
amount - LAG(amount) OVER (ORDER BY date) — day-over-day delta.
SUM(amount) OVER (ORDER BY date) — running total from start of partition through current row.

Worked example. Three days of sales; compute previous-day amount, day-over-day delta, and running total.

sales_date	amount	prev_amount	dod_delta	running_total
2026-05-09	100	NULL	NULL	100
2026-05-10	130	100	30	230
2026-05-11	120	130	-10	350

The first day has LAG = NULL because no prior row exists; consumers usually COALESCE(delta, 0) for display.

Step-by-step explanation.

LAG(amount) OVER (ORDER BY sales_date) returns the previous row's amount, ordered by date.
Day 1 (May 9): no previous row, so LAG = NULL; amount - LAG = NULL.
Day 2 (May 10): LAG = 100; delta = 130 - 100 = 30.
Day 3 (May 11): LAG = 130; delta = 120 - 130 = -10.
SUM(amount) OVER (ORDER BY sales_date) accumulates from the start of the partition through the current row: 100, 230, 350.

Worked-example solution.

SELECT sales_date,
       amount,
       LAG(amount) OVER (ORDER BY sales_date)            AS prev_amount,
       amount - LAG(amount) OVER (ORDER BY sales_date)   AS dod_delta,
       SUM(amount) OVER (ORDER BY sales_date)            AS running_total
FROM sales
ORDER BY sales_date;

Rule of thumb: LAG for "compare this row to its predecessor" (delta, retention, gap); LEAD for "what happens next" (sessionization, churn-from-here); SUM(...) OVER (...) for running totals — always PARTITION BY the entity if the table holds multiple series.

Common beginner mistakes

Using RANK when the question wants the Nth distinct value — RANK = 2 skips entirely if two rows tie for first.
Forgetting PARTITION BY for a per-group ranking — produces a global ranking instead of per-department.
Referencing the window-function alias in WHERE of the same SELECT — window functions execute after WHERE; wrap in a CTE or subquery first.
Confusing LAG (previous) with LEAD (next) — quietly produces inverted deltas.
Forgetting ORDER BY inside OVER for ROW_NUMBER/RANK/LAG/LEAD — required; the result is non-deterministic without it.

PostgreSQL Interview Question on Top 3 Salaries Per Department

Given employees(emp_id, name, department, salary), return the top 3 distinct salaries per department, with ties at rank 3 included. Output department, name, salary, and the rank.

Solution Using `DENSE_RANK() OVER (PARTITION BY department ORDER BY salary DESC)` in a CTE

WITH ranked AS (
    SELECT department,
           name,
           salary,
           DENSE_RANK() OVER (
               PARTITION BY department
               ORDER BY salary DESC
           ) AS dr
    FROM employees
)
SELECT department, name, salary, dr
FROM ranked
WHERE dr <= 3
ORDER BY department, dr, name;

Why this works: the CTE ranked materializes a per-department DENSE_RANK keyed by salary descending — dr = 1 is the highest distinct salary in that department, dr = 2 is the second-highest, and so on; the outer WHERE dr <= 3 keeps every row whose salary is in the top three distinct salaries of its department, including all ties at rank 3; the ORDER BY produces a deterministic, reviewer-friendly output. DENSE_RANK over RANK because the spec wants the top three distinct salaries; DENSE_RANK over ROW_NUMBER because ties at rank 3 must be retained.

Step-by-step trace for the sample input:

emp_id	name	department	salary
1	Alice	eng	90,000
2	Bob	eng	80,000
3	Carol	eng	80,000
4	Dan	eng	70,000
5	Eve	eng	60,000
6	Frank	sales	100,000
7	Grace	sales	90,000
8	Heidi	sales	80,000

CTE ranked — partition by department; order by salary DESC.
DENSE_RANK per partition — eng: Alice → 1, Bob → 2, Carol → 2, Dan → 3, Eve → 4. sales: Frank → 1, Grace → 2, Heidi → 3.
Outer WHERE dr <= 3 — drops Eve (dr = 4); keeps both Bob and Carol (tied at 2) and Dan (3).
ORDER BY department, dr, name — eng rows first, then sales; within department by dr, then name for tiebreak.

Output:

department	name	salary	dr
eng	Alice	90000	1
eng	Bob	80000	2
eng	Carol	80000	2
eng	Dan	70000	3
sales	Frank	100000	1
sales	Grace	90000	2
sales	Heidi	80000	3

Why this works — concept by concept:

CTE ranked — names the intermediate ranked result; the outer query then filters it like a regular table. Far cleaner than a nested subquery.
PARTITION BY department — restarts the rank at each department boundary; without this, the rank is global and the answer is wrong.
ORDER BY salary DESC — defines "highest first" inside each partition; required for any deterministic ranking.
DENSE_RANK over RANK — the spec wants the top three distinct salaries; RANK would skip after ties and miss the third distinct salary if there is a two-way tie above it.
WHERE dr <= 3 in the outer — window functions cannot be referenced in WHERE of the same SELECT; the CTE provides the materialized column the outer can filter on.
O(N log N) time — sort within each partition dominates; with an index on (department, salary DESC) the planner can stream rather than sort.

Inline CTA: More SQL window-function practice problems and SQL CTE practice problems on PipeCode.

SQL
Topic — window functions
SQL window-function problems

Practice →

SQL
Topic — CTE
SQL CTE problems

Practice →

SQL
Topic — date functions
SQL date-function problems

Practice →

Tips to use this PostgreSQL cheat sheet effectively

Hold the clause-order diagram in your head

FROM → WHERE → GROUP BY → HAVING → SELECT → ORDER BY → LIMIT. Memorize this sentence and 80% of "weird" PostgreSQL parse errors decode themselves in five seconds. The error column "x" does not exist almost always means you referenced a SELECT alias in WHERE; the error aggregate functions are not allowed in WHERE means you wanted HAVING instead.

State the grain before any `JOIN`

Before writing the JOIN, name the grain you're producing: "this is order-line grain", "this is customer-day grain", "this is (customer, product) grain". The single most common bug in analytical SQL is SUM(left.col) after a 1:N join — the metric is silently multiplied by N. If grain doubles, you'll spot it immediately.

Use `LEFT JOIN ... IS NULL` over `NOT IN` for anti-joins

NOT IN (subquery) returns zero rows when the subquery contains a single NULL because x NOT IN (..., NULL, ...) is NULL, which fails the WHERE predicate. LEFT JOIN ... WHERE right.id IS NULL and NOT EXISTS (...) are immune. Production engineers who have been burned once never write NOT IN again.

Pick `DENSE_RANK` for "Nth distinct"; pick `ROW_NUMBER` for deduplication

The single most-graded ranking distinction: DENSE_RANK = N is the Nth distinct value; RANK = N is the Nth row in skip-aware ranking order; ROW_NUMBER = N is the Nth row in arbitrary order. For "second-highest distinct salary" → DENSE_RANK = 2. For "remove duplicate rows keeping the canonical one" → ROW_NUMBER() OVER (PARTITION BY key ORDER BY tiebreaker) = 1.

Use `FILTER (WHERE …)` for one-pass conditional metrics

SUM(amount) FILTER (WHERE status = 'refunded') is cleaner than SUM(CASE WHEN status = 'refunded' THEN amount ELSE 0 END) — PostgreSQL supports both. Use FILTER in PostgreSQL-only code, CASE for cross-dialect portability. One scan, many metrics, half the cost of two queries.

Always `ORDER BY` + tiebreaker; pair `LIMIT` with `ORDER BY`

Window functions, LIMIT N, and "top result" queries all require an ORDER BY with a deterministic tiebreaker (e.g., ORDER BY salary DESC, name). Without one, two runs of the same query can return different rows in the tie band — silently wrong in production and visibly wrong in an interview if the reviewer's reference answer locks an ordering.

Use PostgreSQL-specific helpers — `EXTRACT`, `DATE_TRUNC`, `INTERVAL`, `::DATE` cast

EXTRACT(MONTH FROM ts), DATE_TRUNC('month', ts), ts - INTERVAL '1 month', ts::DATE. These four cover 95% of date arithmetic. Reach for DATE_TRUNC whenever the spec says "by month" or "by week" — it groups timestamps to the bucket boundary deterministically.

Where to practice on PipeCode

Start with the SQL practice surface for the all-language SQL corpus. Drill the four-primitive pages: SQL filtering for WHERE patterns, SQL joins for join shapes, SQL aggregation for GROUP BY + HAVING, SQL window functions for ranking and lookback. Add adjacent topics: SQL CTE, SQL subqueries, SQL null-handling, SQL date functions. The interview courses page bundles structured curricula — start with SQL for Data Engineering Interviews — From Zero to FAANG. For broader coverage, browse by topic or read the related SQL interview questions for data engineering and data lake architecture for data engineering interviews blogs.

Frequently Asked Questions

What is the logical clause order in a PostgreSQL query?

PostgreSQL evaluates clauses in the order FROM / JOIN → WHERE → GROUP BY → HAVING → SELECT → ORDER BY → LIMIT / OFFSET, regardless of the order you write them. This is why WHERE cannot reference aggregate functions (they don't exist until after GROUP BY) and why SELECT-level aliases cannot be referenced in WHERE (they're computed in stage 5). Aliases become available only in ORDER BY and the outer query in a nested context.

What is the difference between `WHERE` and `HAVING` in PostgreSQL?

WHERE filters individual rows before the GROUP BY step and can reference only raw row columns. HAVING filters whole groups after the GROUP BY step and can reference aggregate functions like COUNT(*), SUM(col), AVG(col). Trying to use an aggregate in WHERE (e.g., WHERE COUNT(*) > 1) is a parse error because the aggregate does not yet exist. Both clauses can appear in the same query.

How do I find rows in table A that have no match in table B?

The canonical PostgreSQL pattern is SELECT a.* FROM a LEFT JOIN b ON b.fk = a.pk WHERE b.pk IS NULL — the LEFT JOIN preserves every left row, and the WHERE b.pk IS NULL filter keeps only the ones where no right-side match was found. This is the anti-join pattern. Equivalent to WHERE NOT EXISTS (SELECT 1 FROM b WHERE b.fk = a.pk). Both are safer than NOT IN (subquery), which returns zero rows if the subquery contains a single NULL.

What is the difference between `RANK`, `DENSE_RANK`, and `ROW_NUMBER`?

All three assign integers within a window. ROW_NUMBER gives every row a unique sequential integer (1, 2, 3, 4), even on ties. RANK gives tied rows the same rank but skips after them (1, 2, 2, 4). DENSE_RANK gives tied rows the same rank with no skip (1, 2, 2, 3). For "Nth distinct value" use DENSE_RANK = N; for "Nth row in skip-aware ranking order" use RANK = N; for "Nth row in arbitrary order" or "deduplicate keeping one canonical row" use ROW_NUMBER = 1.

What does `FILTER (WHERE …)` do in PostgreSQL aggregates?

SUM(col) FILTER (WHERE pred) and COUNT(*) FILTER (WHERE pred) apply the aggregate only to rows where the predicate is TRUE; rows where the predicate is FALSE or NULL are skipped for that aggregate, while other aggregates in the same SELECT still see them. The portable cross-dialect equivalent is SUM(CASE WHEN pred THEN col ELSE 0 END) and COUNT(CASE WHEN pred THEN 1 END). Use FILTER for clarity in PostgreSQL-only code.

How do I compute a running total in PostgreSQL?

Use SUM(col) OVER (PARTITION BY p ORDER BY o) — the window aggregate accumulates from the start of each partition through the current row in the order defined by ORDER BY. Example: SUM(amount) OVER (PARTITION BY customer_id ORDER BY order_date) gives a per-customer running total of order amounts ordered by date. Drop PARTITION BY for a single global running total.

Why is `LIMIT 5` returning different rows on different runs?

LIMIT without ORDER BY is non-deterministic — PostgreSQL returns whatever rows it sees first, which depends on the query plan, parallelism, and table physical layout. Always pair LIMIT with ORDER BY <col> DESC, <tiebreaker> so two runs return the same rows. Reviewers depend on stable ordering, and dashboards break silently when row order drifts.

Start practicing PostgreSQL SQL problems

Data Lake Architecture for Data Engineering Interviews

Gowtham Potureddi — Mon, 11 May 2026 03:20:43 +0000

Data lake architecture questions in data-engineering interviews almost always reduce to four primitives: medallion zones (bronze → silver → gold) for progressive refinement, an ingestion → metadata catalog → compute flow on object storage, the lake vs cloud warehouse vs lakehouse decision driven by open table formats (Iceberg, Delta, Hudi), and a disciplined answer shape that covers grain, idempotency, lineage, and aggregate reconciliation. Whether the prompt is "design our analytics lake from scratch", "how would you land CDC from Postgres into the lake", "when would you pick a lakehouse over a warehouse", or "why do counts drift between the lake and the source app", interviewers grade the same handful of mental models — and candidates who skip straight to vendor names without naming the primitives lose the round.

This guide walks four topic clusters end-to-end, each with a detailed topic explanation, per-sub-topic explanation with a worked example and its solution, common beginner mistakes, and an interview-style scenario with a full answer that traces the design step by step. Every section ends with a concept-by-concept breakdown that explains why the design works, what it costs, and where beginners typically slip. Storage examples assume an S3-style object store on the cloud, but every primitive transfers to GCS, Azure Blob / ADLS, or any other modern object backend.

1. Bronze / Silver / Gold Medallion Zones for Data Lake Architecture

Progressive refinement through landing/bronze, refined/silver, and curated/gold zones

"Walk me through how you would lay out an analytics lake from scratch" is the signature opening prompt — and the cleanest answer is medallion architecture with three numbered zones. The mental model: landing/bronze is an append-only mirror of the source payloads with minimal transformation; refined/silver applies dedup, type coercion, and conformed business keys; curated/gold publishes subject-area tables and star-schema facts/dims that downstream applications and BI tools consume. Each zone has a different SLA, different read/write permissions, and different retention. The names vary across vendors — Databricks coined "bronze/silver/gold", AWS uses "raw/curated/consumption", Microsoft uses "landing/refined/analytics" — but the three-tier shape is universal.

Pro tip: When you whiteboard the medallion zones, label each box with who writes, who reads, and what breaks if the job reruns. Idempotent writes and clear grain matter as much in a lake as they do in a warehouse — interviewers grade the candidate who naturally adds these annotations without prompting.

Landing / bronze — append-only mirror of source payloads

The landing-zone invariant: bronze is an append-only, immutable copy of the source payload with minimal transformation; the schema is captured but not enforced; partitioning is by **ingest_date (or ingest_hour for high-frequency sources); replays are safe because writes never overwrite**. The zone optimizes for fidelity and replay, not query performance.

Append-only writes — every batch produces a new file under a date-partitioned prefix; MERGE and UPDATE are forbidden.
Source-payload fidelity — store the raw shape (JSON, Avro, CSV, Parquet snapshot) plus an ingest_id and source_ts per row.
Partition by ingest_date — makes back-fill, replay, and audit trivially scoped.
Retention — keep at least 30-90 days; audits and reconciliations need historical bronze.

Worked example. A Postgres CDC pipeline lands daily JSON snapshots into s3://analytics-lake/bronze/orders/.

prefix	files	purpose
`bronze/orders/ingest_date=2026-04-11/`	`part-00000.json`	Apr 11 snapshot
`bronze/orders/ingest_date=2026-04-12/`	`part-00000.json`	Apr 12 snapshot
`bronze/orders/ingest_date=2026-04-13/`	`part-00000.json`	Apr 13 snapshot

Step-by-step explanation.

The source app emits one JSON snapshot per day at 02:00 UTC.
The ingestion job lands each snapshot under a calendar-keyed prefix bronze/orders/ingest_date=YYYY-MM-DD/ so partition pruning works for any date filter downstream.
Each batch is also stamped with a unique ingest_id (timestamp + UUID) sub-prefix so retries write fresh files instead of overwriting a previous attempt.
Files inside a partition are append-only part-NNNNN.json; bronze never edits a written file — corrected payloads land as new files under a new ingest_id.
After three days you have three day-partitions; each is independently re-readable with WHERE ingest_date = 'YYYY-MM-DD' and any single day can be replayed without touching the others.

Worked-example solution. A landing-zone object layout:

s3://analytics-lake/bronze/orders/
  ingest_date=2026-04-13/
    ingest_id=20260413T0200Z/
      part-00000.json
      part-00001.json

Rule of thumb: never edit a bronze file. If a payload is wrong, drop a corrected file under a new ingest_id and let the silver-layer dedup logic resolve it; never overwrite history.

Refined / silver — deduped, typed, conformed business keys

The refined-zone invariant: silver applies dedup against natural or business keys, coerces types to a canonical schema, conforms key columns across sources, and may emit slowly-changing-dimension (SCD) history; the zone is the single source of truth for downstream application code and most analyst SQL. Idempotency at the silver layer is non-negotiable — re-running a daily job must produce byte-identical output.

Dedup on (business_key, source_ts) — ROW_NUMBER() OVER (PARTITION BY business_key ORDER BY source_ts DESC) = 1.
Type coercion — JSON strings → typed columns; epoch ms → TIMESTAMP; cents → DECIMAL(18,2).
Conformed dimensions — customer_id, product_id, geo_id mapped to one canonical form across every source.
SCD type 2 — emit (valid_from, valid_to, is_current) columns when downstream consumers need point-in-time joins.

Worked example. Bronze orders rows arrive twice for order_id=448 due to a CDC retry; silver dedup keeps the latest.

order_id	source_ts	bronze_rn
448	2026-04-12 09:30:00	2
448	2026-04-12 09:30:15	1 (kept)
449	2026-04-12 10:00:00	1 (kept)

Step-by-step explanation.

Bronze contains three rows for ingest_date = 2026-04-12: two for order_id = 448 (a CDC retry produced two payloads at 09:30:00 and 09:30:15) and one for order_id = 449.
ROW_NUMBER() OVER (PARTITION BY order_id ORDER BY source_ts DESC) numbers rows independently inside each order_id group, with the latest source_ts getting rn = 1.
For order_id = 448: the row at 09:30:15 is later, so it gets rn = 1; the 09:30:00 row gets rn = 2.
For order_id = 449: only one row, so it gets rn = 1 automatically.
The outer WHERE rn = 1 keeps two rows — the latest order_id = 448 and the only order_id = 449 — and silently drops the duplicate, producing a deterministic single-row-per-business-key silver table.

Worked-example solution.

WITH ranked AS (
    SELECT *,
           ROW_NUMBER() OVER (
               PARTITION BY order_id
               ORDER BY source_ts DESC
           ) AS rn
    FROM bronze.orders
    WHERE ingest_date = DATE '2026-04-12'
)
SELECT order_id,
       customer_id,
       amount::DECIMAL(18,2)            AS amount,
       order_status,
       source_ts                         AS as_of_ts,
       CURRENT_TIMESTAMP                 AS silver_loaded_at
FROM ranked
WHERE rn = 1;

Rule of thumb: the silver zone is where ETL bugs hide — invest in unit-tested dedup logic, schema-evolution tests, and aggregate reconciliation against bronze totals before promoting to gold.

Curated / gold — subject-area tables and star schemas

The curated-zone invariant: gold publishes tables shaped for downstream consumption: dimensional models (fact tables + conformed dimensions), subject-area marts, or one-big-table (OBT) flattenings; SLAs are stricter, freshness is tracked, and consumer contracts are explicit. Each gold table maps to exactly one consumer class — analysts, dashboards, ML feature pipelines, or reverse-ETL into operational systems.

Star schema — fact_orders joined to dim_customer, dim_product, dim_date; one row per business event.
Subject-area marts — domain-scoped denormalized tables (e.g., mart_marketing_attribution).
OBT flattening — when consumers prefer one wide table over a join (Looker, Power BI dashboards).
Consumer contracts — column types, refresh cadence, breakage policy declared in dbt-style metadata.

Worked example. A gold star schema for the orders subject area.

table	grain	example columns
`fact_orders`	one row per order line	`order_id`, `line_id`, `customer_key`, `product_key`, `date_key`, `qty`, `revenue`
`dim_customer`	one row per customer (SCD2)	`customer_key`, `customer_id`, `name`, `valid_from`, `valid_to`
`dim_product`	one row per product	`product_key`, `product_id`, `category`
`dim_date`	one row per calendar date	`date_key`, `date`, `iso_week`, `is_weekend`

Step-by-step explanation.

fact_orders is the central transactional table at order-line grain — one row per line item, with numeric measures (qty, revenue) and foreign-key columns to every dimension.
dim_customer is an SCD2 dimension: a single real-world customer can appear in multiple rows over time, each with valid_from / valid_to / is_current columns to capture historical attribute changes.
dim_product is a simpler Type-1 dimension: one row per product, current state only — overwrites on update with no history.
dim_date is the conformed date dimension: one row per calendar date with pre-computed week, month, quarter, year, and is_weekend columns so dashboards never have to compute date math at query time.
Joins from fact_orders to each dimension use the surrogate keys (customer_key, product_key, date_key) — never the natural business IDs — so SCD2 history is preserved when the same customer's row evolves over time.

Worked-example solution.

CREATE TABLE gold.fact_orders AS
SELECT
    s.order_id,
    s.line_id,
    dc.customer_key,
    dp.product_key,
    dd.date_key,
    s.qty,
    s.qty * s.unit_price AS revenue
FROM silver.order_lines s
JOIN gold.dim_customer dc
  ON dc.customer_id = s.customer_id
 AND s.order_ts BETWEEN dc.valid_from AND dc.valid_to
JOIN gold.dim_product  dp ON dp.product_id = s.product_id
JOIN gold.dim_date     dd ON dd.date      = s.order_ts::DATE;

Rule of thumb: gold tables are the only zone customers should reference by name; if a dashboard reads silver directly, your contract is leaking. Use views or feature-flagged exposures rather than letting consumers couple to interim grains.

Common beginner mistakes

Treating bronze as a junk drawer with no ingest_date partitioning — replay and audit become impossible.
Doing dedup at gold instead of silver — every downstream job has to repeat the work and answers diverge.
Letting consumers query silver directly — silver schemas can change without notice; gold contracts are explicit.
Skipping ingest_id and source_ts lineage columns — when counts drift, you have no way to reconstruct what landed when.
Mixing batch and streaming writes into the same bronze prefix without a partition key for write-mode — late arrivals overwrite eager batches.

Data Lake Interview Question on Designing Layered Zones

A team dumps daily JSON exports of orders into a single S3 prefix. Analysts complain that order counts drift versus the source application by 0.5–2% on most days. Design a three-zone medallion layout that fixes the drift, makes the discrepancy investigable, and supports daily reruns without producing duplicates.

Solution Using Bronze (append-only) + Silver (dedup) + Gold (star schema)

1. Move existing daily dumps into:
     s3://analytics-lake/bronze/orders/ingest_date=YYYY-MM-DD/ingest_id=<batch>/
   Append-only; never overwrite a date partition.

2. Build silver/orders as a daily MERGE that:
     - Dedups bronze rows by ROW_NUMBER() OVER (PARTITION BY order_id ORDER BY source_ts DESC) = 1
     - Coerces JSON fields to a typed schema
     - Joins against dim_customer / dim_product on conformed keys
     - Carries ingest_id + source_ts as lineage columns

3. Promote to gold/fact_orders only after a silver-vs-source aggregate-reconciliation job
   passes a tolerance threshold (e.g., |silver_count - source_count| / source_count < 0.001).

4. Surface a row-count + revenue-sum dashboard sourced from BOTH bronze and the source app's
   replica, so any future drift surfaces within one ingest cycle.

Why this works: the append-only bronze layer makes the discrepancy investigable — every historical payload is preserved with ingest_id and ingest_date, so analysts can replay any day's source state; the silver dedup converts CDC retries and late-arriving rows into a deterministic single row per order_id; the gold layer is gated by an aggregate-reconciliation step that catches drift before it reaches dashboards; and the dual-source row-count dashboard surfaces residual drift immediately. The combination addresses both the prevention (idempotent dedup) and detection (reconciliation + dashboard) sides of the failure mode.

Step-by-step trace for the drift scenario on 2026-04-12:

step	action	observation
1	bronze ingests `ingest_date=2026-04-12`	12,847 raw rows including 12 CDC retries
2	silver dedup keeps `rn = 1`	12,835 unique `order_id`s
3	source-app replica reports	12,835 orders for 2026-04-12
4	reconciliation passes	drift = 0 / 12,835 = 0.0%
5	promote to gold/fact_orders	12,835 fact rows; counts match dashboard

Output: the fixed-state contract per ingest day:

metric	bronze	silver	source	gold	drift
row count	12,847	12,835	12,835	12,835	0.0%
total revenue	$4,128,931	$4,128,931	$4,128,931	$4,128,931	0.0%

Why this works — concept by concept:

Append-only bronze with ingest_date partitioning — every payload is preserved and addressable; replay is a WHERE ingest_date = ... filter rather than a re-ingest.
Silver dedup via ROW_NUMBER over (order_id ORDER BY source_ts DESC) — collapses CDC retries to a deterministic single row per business key; idempotent on rerun.
Lineage columns ingest_id + source_ts — every silver row points back to a specific bronze file and source moment; forensic debugging is one join away.
Aggregate reconciliation gate before gold — drift cannot reach dashboards because gold is gated on |silver - source| / source < threshold; failures page the on-call rather than silently corrupt the BI tool.
Dual-source dashboard — surface drift instantly even when reconciliation isn't perfect; the early-warning loop pays for itself the first incident.
O(|bronze|) time per day — single linear scan + window for dedup; reconciliation adds one aggregate per zone, negligible compared to ingest cost.

Inline CTA: Drill the ETL practice page for medallion-zone problems and the dimensional modeling practice page for star-schema patterns at the gold layer.

ETL
Topic — ETL pipelines
ETL practice problems

Practice →

SQL
Topic — dimensional modeling
Dimensional modeling problems

Practice →

SQL
Language — SQL
All SQL practice problems

Practice →

2. Ingestion → Catalog → Compute Flow on Object Storage

Sources to query engines through metadata catalogs in data lake architecture

"How does data physically get from a Postgres source into a query engine like Spark or Trino on the lake?" is the signature design follow-up — and the cleanest answer is the ingest → register → query flow with three distinct components. The mental model: sources (databases, APIs, streaming platforms, file feeds) ingest into object storage as files; a metadata catalog (Hive Metastore, AWS Glue, Unity Catalog, Polaris, Iceberg REST catalog) maps logical tables to physical file paths and column schemas; compute engines (Spark, Trino, Presto, DuckDB-in-the-cloud, Snowflake external tables) read the catalog to discover tables and read the object store to fetch data. The decoupling is the entire value proposition — many engines can read the same footprint, and storage scales independently from compute.

Pro tip: When the interviewer asks "where does Spark get the schema from?", the answer is the catalog, not the file. Files (Parquet, ORC, Avro) carry their own schema in the footer, but the catalog is what makes a logical table addressable across sessions and engines. State this distinction explicitly — it separates candidates who learned data lake architecture by reading docs from those who learned by debugging production.

Object storage as the storage layer — S3, GCS, ADLS

The object-store invariant: modern lakes use cloud object storage (Amazon S3, Google Cloud Storage, Azure Data Lake Storage / ADLS Gen2) rather than HDFS; storage is **infinitely scalable, durable, and decoupled from compute, with eventual-consistency semantics that the table format is responsible for masking**. Files are typically Parquet (columnar, compressed) or ORC; Avro shows up in streaming pipelines.

Hive-style partitioning — s3://bucket/table/col=value/file.parquet for partition pruning at query time.
File sizes — target 128MB-1GB per file; smaller files trigger the small-file problem (excessive metadata, slow planning).
Compaction — periodic batch jobs that rewrite many small files into fewer large ones.
Eventual consistency — S3 was eventually consistent for many years; the table format handles the retry / commit semantics that mask this from queries.

Worked example. A Hive-style partition layout for a daily-loaded orders table in silver.

prefix	role
`s3://analytics-lake/silver/orders/`	table root
`…/ingest_date=2026-04-13/`	partition value
`…/ingest_date=2026-04-13/part-00000.parquet`	data file
`…/_delta_log/` or `…/metadata/`	table-format metadata (if Delta/Iceberg)

Step-by-step explanation.

The table root s3://analytics-lake/silver/orders/ is the registered location in the catalog; everything under it belongs to one logical table.
Each child prefix ingest_date=YYYY-MM-DD/ is one Hive partition value; the key=value syntax is the convention every engine (Spark, Trino, Athena, Snowflake) recognizes.
Inside each partition, multiple Parquet files (~180MB each) split the data so a Spark reader can fetch them in parallel; the file count is bounded by your micro-batch size.
The _delta_log/ (Delta) or metadata/ (Iceberg) prefix holds the table-format commit log — a sequence of JSON files describing every transaction, which is what gives you ACID and time travel on top of plain object storage.
A query with WHERE ingest_date = '2026-04-13' triggers partition pruning: the planner reads only files under that one prefix, skipping every other day's files entirely — the difference between 200ms and 60s.

Worked-example solution. Object layout for a partitioned silver table:

s3://analytics-lake/silver/orders/
  ingest_date=2026-04-13/
    part-00000.parquet  (180MB, 1.2M rows)
    part-00001.parquet  (165MB, 1.1M rows)
  ingest_date=2026-04-12/
    part-00000.parquet  (175MB)
  _delta_log/                              # Delta Lake commit log
    00000000000000000001.json
    00000000000000000002.json

Rule of thumb: if your average file size is below 50MB, schedule a daily compaction job; if it's above 1GB, your partitions are too coarse. Both extremes hurt query latency.

Metadata catalog — Hive Metastore, AWS Glue, Unity Catalog

The catalog invariant: a metadata catalog maps logical names (silver.orders) to physical locations (s3://analytics-lake/silver/orders), column schemas, partition definitions, and table properties; it is the single source of truth for "what tables exist" across every compute engine that reads the lake. The catalog can be a long-running service (Hive Metastore, AWS Glue Data Catalog, Databricks Unity Catalog) or a REST API on top of files (Iceberg REST catalog, Polaris).

Logical → physical mapping — silver.orders → s3://...; column names, types, partition keys.
Engine-agnostic — Spark, Trino, Presto, Snowflake external tables, Athena, DuckDB all read the same catalog.
Schema evolution — add column, widen type, rename (with caveats); the catalog records the evolution history.
Permissions — many catalogs (Unity, Glue with Lake Formation) carry table/column-level access policies.

Worked example. Registering a partitioned silver.orders table in Glue.

field	value
logical name	`silver.orders`
location	`s3://analytics-lake/silver/orders/`
input format	`parquet`
partition keys	`ingest_date STRING`
schema	`order_id BIGINT, customer_id BIGINT, amount DECIMAL(18,2), source_ts TIMESTAMP`

Step-by-step explanation.

CREATE EXTERNAL TABLE silver.orders declares a logical name in the catalog without copying or moving any data files.
The column list (order_id BIGINT, …) declares the schema the engine should expect; Parquet files store their own schema in the footer, but the catalog is the canonical answer the planner trusts.
PARTITIONED BY (ingest_date STRING) declares the partition column; this column is derived from the prefix path, not stored in the data files, which keeps each partition lean.
LOCATION 's3://analytics-lake/silver/orders/' is the prefix the engine scans when reading; data files must already exist at this location.
MSCK REPAIR TABLE silver.orders walks the S3 prefix, discovers any partition values it doesn't yet know about, and registers them; without this command after a backfill, the planner returns zero rows for the new dates.

Worked-example solution.

CREATE EXTERNAL TABLE silver.orders (
    order_id     BIGINT,
    customer_id  BIGINT,
    amount       DECIMAL(18,2),
    source_ts    TIMESTAMP,
    ingest_id    STRING
)
PARTITIONED BY (ingest_date STRING)
STORED AS PARQUET
LOCATION 's3://analytics-lake/silver/orders/';

MSCK REPAIR TABLE silver.orders;

Rule of thumb: always run MSCK REPAIR TABLE (or the engine equivalent) after a backfill that adds new partition prefixes; otherwise the catalog won't know about them and the partition predicate will return zero rows.

Compute engines — Spark, Trino, Presto, DuckDB

The compute invariant: compute engines read the catalog to discover tables, plan queries with partition pruning and predicate pushdown, then read the relevant Parquet/ORC files from object storage; storage and compute scale independently and the same data can be queried by multiple engines simultaneously. Spark dominates for batch + streaming pipelines; Trino/Presto dominate for interactive SQL; DuckDB is rising for single-node analytics.

Spark — JVM, batch + streaming, rich ecosystem (Iceberg/Delta connectors, Spark SQL, MLlib, Structured Streaming).
Trino / Presto — interactive SQL across many catalogs; great for federated queries across lake + warehouse.
DuckDB — single-node, embeddable, blazing fast for sub-TB analytics; popular for ad-hoc + notebooks.
Snowflake / BigQuery / Redshift external tables — read lake data from inside a managed warehouse.

Worked example. A Spark SQL query against silver.orders with partition pruning.

layer	action	data scanned
catalog	resolve `silver.orders` → `s3://...`	metadata only
planner	prune partitions for `ingest_date = '2026-04-13'`	one partition
Spark workers	read Parquet column-block for `amount`	~50MB
executor	aggregate `SUM(amount)`	local

Step-by-step explanation.

Spark resolves silver.orders against the catalog — pure metadata fetch, zero data scanned, returns the location plus the partition schema.
The planner sees WHERE ingest_date = '2026-04-13' and prunes the partition list to a single value, so workers only need to list files under one S3 prefix instead of all of them.
Workers issue an S3 LIST for that single partition, fetching a list of ~one to ten Parquet file paths.
Each Parquet reader uses footer metadata to skip every column except amount, then streams just that column block — typically 50MB instead of the full 1GB Parquet file.
Each task computes a partial SUM(amount) locally; a final shuffle sums the partial values to one number — the entire query is O(rows in one partition) and runs in sub-second time.

Worked-example solution.

SELECT SUM(amount) AS daily_revenue
FROM silver.orders
WHERE ingest_date = '2026-04-13';

Rule of thumb: always include the partition key in your WHERE clause to enable partition pruning; without it, the planner reads every partition (terabytes), and your query goes from 500ms to 50 seconds.

Common beginner mistakes

Skipping the catalog and reading raw S3 paths in every job — schemas drift, no central source of truth, no permissions.
Ignoring file-size budgets — millions of 5KB files (the small-file problem) make Spark planning slower than the actual scan.
Not declaring partition keys — full-table scans on every query, costs balloon by 100x.
Mixing file formats inside one logical table (some Parquet, some JSON) — the planner can't push predicates and queries error out.
Forgetting to refresh the catalog after a backfill — MSCK REPAIR TABLE or REFRESH TABLE is the single most-forgotten command.

Data Lake Interview Question on CDC Ingestion from Postgres

Design a near-real-time ingestion pipeline that lands changes from a 10TB Postgres database into the lake, registers them in a catalog, and exposes them to Spark and Trino with sub-five-minute freshness.

Solution Using Debezium → Kafka → Iceberg with Hive Metastore

Postgres (with logical replication enabled)
      │
      ▼
Debezium connector (CDC reader, emits change events)
      │
      ▼
Kafka topic per table (key = primary key; value = before/after JSON or Avro)
      │
      ▼
Spark Structured Streaming job (1-minute trigger):
      - Reads Kafka topic
      - Writes to bronze.orders_cdc as append-only Iceberg files (partitioned by event_date)
      │
      ▼
Hive Metastore / Glue catalog:
      - bronze.orders_cdc registered with Iceberg metadata
      - silver.orders_current registered as a Spark MERGE-on-read view
      │
      ▼
Compute consumers:
      - Trino: SELECT * FROM silver.orders_current WHERE event_date = today
      - Spark batch: nightly compaction + table-maintenance

Why this works: Debezium reads the Postgres write-ahead log (WAL) directly via logical replication, so it captures every insert/update/delete with no impact on the source; Kafka decouples the producer from the consumer and absorbs traffic spikes; the Spark Structured Streaming job runs with a one-minute trigger, so the lake is at most one minute behind; Iceberg's ACID transactions make concurrent micro-batch writes safe; the Hive Metastore registers the table once, and both Trino and Spark see the same schema; partitioning by event_date enables prune-friendly time-window queries; nightly compaction keeps file sizes in the 128MB-1GB sweet spot.

Step-by-step trace for an order update at 09:30:00.000:

time	component	action
09:30:00.000	Postgres	UPDATE orders SET status='shipped' WHERE order_id=448
09:30:00.150	Debezium	reads WAL, emits change event to Kafka
09:30:00.300	Kafka	persists change event to topic `orders.cdc`
09:30:30.000	Spark Streaming	next 1-min trigger; reads change events
09:30:35.000	Spark Streaming	writes Parquet to `bronze.orders_cdc/event_date=2026-04-13/`
09:30:35.500	Iceberg	commits new snapshot; catalog updated
09:30:40.000	Trino	`SELECT … FROM silver.orders_current WHERE order_id=448` returns updated row

End-to-end latency: ~40 seconds. Well within the five-minute SLA.

Output: the consumer-visible contract per minute:

metric	target	actual
freshness (P50)	< 5 min	~40 sec
freshness (P99)	< 5 min	~2 min
dropped events	0	0
schema-drift incidents	< 1/quarter	0 last quarter

Why this works — concept by concept:

Postgres logical replication + Debezium — captures every row change at the WAL layer; no impact on source query performance; no missed events.
Kafka as the decoupler — handles backpressure, replays, and multiple downstream consumers; lake outages don't lose source events.
Spark Structured Streaming with 1-minute trigger — micro-batch sweet spot; latency vs throughput trade-off favors throughput here.
Iceberg table format — ACID commits make concurrent micro-batch writes safe; time travel makes "what did the table look like at 09:30?" a one-line query.
Hive Metastore as the unified catalog — Spark and Trino see the same schema; no per-engine duplication.
event_date partitioning + nightly compaction — bounds query scan size and keeps file count manageable; both maintenance jobs are idempotent.
End-to-end latency ~40s — well inside the 5-min SLA; the 4.5-min headroom absorbs Kafka rebalances and Spark micro-batch jitter without alerting.

Inline CTA: Drill the streaming practice page for Kafka + micro-batch problems and the Python practice page for PySpark Structured Streaming patterns. Course: PySpark Fundamentals.

ETL
Topic — streaming
Streaming practice problems

Practice →

PYTHON
Language — Python
Python practice for data pipelines

Practice →

ETL
Topic — ETL pipelines
ETL practice problems

Practice →

3. Lake vs Cloud Warehouse vs Lakehouse — Iceberg, Delta, Hudi

Pattern selection and open table formats in data lake architecture

"When would you pick a lakehouse over a warehouse?" and "what is the difference between Iceberg, Delta Lake, and Hudi?" are the two signature pattern-selection prompts — and they share one mental model: a data lake is files + a catalog; a cloud warehouse is a managed ACID SQL system with proprietary storage; a lakehouse is a lake plus an open table format that adds ACID, time travel, partition evolution, and concurrent writers — bringing warehouse-like semantics to object storage. Iceberg, Delta Lake, and Hudi are the three dominant open table formats, each with slightly different trade-offs.

Pro tip: Most large organizations run a blend: lake for flexible high-volume ingestion and ML feature stores, warehouse or lakehouse SQL for curated analytics. Don't propose a single-pattern solution to a system-design question — describe the boundary between the two and the contracts that flow across it. That's the senior signal.

Data lake — files on object storage with a catalog

The lake invariant: a data lake is object storage (S3/GCS/ADLS) plus a metadata catalog plus open file formats (Parquet, ORC, Avro) plus convention-based partitioning; reads are cheap and parallel, writes are eventual-consistent unless wrapped in a table format, and the cost model is storage + compute-at-query-time. Lakes shine when data shapes are diverse and high-volume.

Strengths — accepts any data format; massive scale; cheap storage; many engines can read.
Watch-outs — no ACID without a table format; no time travel; concurrent writes can corrupt the table; small-file problem.
Best fit — ML feature stores, log archives, raw event data, ingestion landing zones.
Cost — storage ~$0.023/GB/month (S3 Standard); compute pay-per-query.

Worked example. A 50TB clickstream feature store in S3 + Glue.

attribute	value
storage	S3 Standard, ~$1,150/month for 50TB
catalog	AWS Glue (free for first million objects)
compute	Athena, ~$5/TB scanned
typical query	scan 100GB → ~$0.50

Step-by-step explanation.

Storage line: 50TB × 1024GB × $0.023/GB/month (S3 Standard pricing) ≈ $1,150/month — this is the floor regardless of query activity.
Catalog line: AWS Glue is free for the first million metadata objects; a 50TB clickstream table partitioned by year/month/day fits comfortably under that limit.
Compute line: Athena charges per TB scanned, not per query — write efficient SQL (use the partition predicate, project only needed columns) and you pay only for what you actually read.
Typical query: a partition-pruned + column-projected scan touches ~100GB → 0.1 TB × $5/TB ≈ $0.50; an unpruned full-table scan would touch 50TB → $250 per query.
Net at this scale: storage dominates the monthly bill (~$1,150) and compute scales linearly with query discipline — bad queries cost real money, good queries are nearly free.

Worked-example solution. A lake-first deployment for clickstream:

s3://feature-lake/raw_events/year=2026/month=04/day=13/
  part-00000.parquet
  part-00001.parquet
   …
Glue catalog: feature_lake.raw_events
Athena query: SELECT user_id, COUNT(*) FROM feature_lake.raw_events WHERE day = '2026-04-13' GROUP BY user_id;

Rule of thumb: a pure lake is the right answer when data is high-volume, schema-flexible, and primarily consumed by ML or batch analytics; reach for a lakehouse the moment you need ACID or concurrent writers.

Cloud warehouse — managed ACID SQL on proprietary storage

The warehouse invariant: a cloud warehouse (Snowflake, BigQuery, Redshift, Synapse) is a managed system that owns both storage and compute, exposes SQL as the primary interface, provides ACID transactions out of the box, and handles indexing, statistics, and query optimization automatically. Warehouses shine when data is structured and the primary consumer is analyst SQL.

Strengths — mature SQL; ACID; managed governance products (RBAC, masking); workload management.
Watch-outs — proprietary storage = vendor lock; cost at huge semi-structured scale; less flexible for non-tabular data.
Best fit — curated analytics, BI dashboards, financial reporting, dimensional models.
Cost — ~$2-5 per credit-hour or per-TB-scanned; storage ~$0.02-0.04/GB/month (compressed).

Worked example. A 5TB curated finance mart in Snowflake.

attribute	value
storage	Snowflake, ~$200/month for 5TB compressed
compute	Small warehouse, ~$2/credit-hour
typical query	dashboard refresh in ~30 seconds
ACID	full transactions across multi-table updates

Step-by-step explanation.

Storage line: Snowflake compresses raw data 3-5x, so 5TB raw becomes ~1-1.5TB stored at ~$23-46/TB/month, landing around $200/month total.
Compute line: a Small warehouse runs at ~$2/credit-hour; nightly ELT jobs plus business-hours dashboards consume ~50-200 credits/month for a finance mart of this size.
Typical query: dashboard refresh hits a sub-30-second target because data is co-located with compute and the planner has full statistics.
ACID guarantee: multi-table updates within a BEGIN ... COMMIT block are atomic — the finance close cannot land half-updated, which is the whole reason finance reports run on a warehouse rather than a raw lake.
Net at 5TB scale: the warehouse premium (~$200 storage) is small versus a lake's ~$115 equivalent; ergonomics, SQL-first BI integration, and ACID tilt the choice clearly toward warehouse.

Worked-example solution. A curated star schema in Snowflake:

CREATE OR REPLACE TABLE finance.fact_revenue AS
SELECT date_key, region_key, product_key, SUM(amount) AS revenue
FROM silver.order_lines
GROUP BY 1, 2, 3;

Rule of thumb: warehouses are the right answer when SQL ergonomics and ACID matter more than format flexibility; reach for a lakehouse when you need both and the ability to query the same data from outside the warehouse.

Lakehouse with Iceberg / Delta / Hudi — ACID on object storage

The lakehouse invariant: a lakehouse is an open-table-format layer (Apache Iceberg, Delta Lake, Apache Hudi) on top of object storage that adds ACID transactions, schema evolution, partition evolution, time travel, and safe concurrent writers; the data sits in standard Parquet files but is governed by a JSON/Avro commit log that any engine can read. Lakehouse architectures combine lake scale with warehouse-like semantics.

Apache Iceberg — table format invented at Netflix; broad engine support (Spark, Trino, Snowflake, BigQuery, Dremio); REST catalog spec.
Delta Lake — invented at Databricks; strong Spark integration; commit log in _delta_log/; OSS Delta works across engines.
Apache Hudi — invented at Uber; optimized for upsert-heavy CDC workloads; merge-on-read and copy-on-write modes.
All three — provide ACID, time travel, schema evolution, and partition pruning; pick by ecosystem and team skill.

Worked example. A 50TB lakehouse on S3 + Iceberg + Spark/Trino.

dimension	data lake	warehouse	lakehouse (Iceberg)
storage cost	✓ cheap	✗ expensive	✓ cheap
ACID transactions	✗	✓	✓
concurrent writers	✗	✓	✓
time travel	✗	depends	✓
schema evolution	manual	managed	managed
vendor lock	none	high	low (open standard)
ML / Python access	direct	via connector	direct

Step-by-step explanation.

Storage cost row: lake and lakehouse both win because data sits in cheap object storage; warehouse loses at scale because storage is bundled with managed compute.
ACID + concurrent writers rows: warehouse and lakehouse both provide them out of the box; pure lake does not — concurrent writers can corrupt a lake table without an open table format on top.
Time travel row: only the lakehouse exposes it natively via the Iceberg/Delta snapshot log; some warehouses offer it as a managed feature; pure lake has no concept.
Schema evolution row: lakehouse and warehouse both manage adding/widening columns as a metadata commit; pure-lake users do it manually with file rewrites.
Vendor lock + ML/Python rows: pure lake is open standard; lakehouse is open standard with a richer feature set; warehouse is proprietary and ML access usually requires connectors that copy data back out — which is why ML teams gravitate to lake/lakehouse for feature stores.

Worked-example solution. Creating an Iceberg table via Spark SQL:

CREATE TABLE lakehouse.orders (
    order_id     BIGINT,
    customer_id  BIGINT,
    amount       DECIMAL(18,2),
    order_date   DATE
)
USING iceberg
PARTITIONED BY (days(order_date))
LOCATION 's3://lakehouse-bucket/orders/';

MERGE INTO lakehouse.orders t
USING staging.orders_delta s
ON t.order_id = s.order_id
WHEN MATCHED THEN UPDATE SET *
WHEN NOT MATCHED THEN INSERT *;

Rule of thumb: the lakehouse pattern is the right answer when you need ACID + time travel + concurrent writers + the ability to query from multiple engines; pick Iceberg for the broadest engine support, Delta for tightest Databricks/Spark integration, Hudi for upsert-heavy CDC.

Common beginner mistakes

Conflating "lake" with "Hadoop / HDFS" — modern lakes are object storage; HDFS is the legacy on-prem variant.
Picking a lakehouse "because it's modern" without matching it to the workload — for pure curated SQL analytics, a warehouse is often simpler and cheaper.
Treating Iceberg / Delta / Hudi as interchangeable — Hudi is upsert-tuned; Delta is Spark-tightest; Iceberg is most engine-agnostic. The choice has long-term implications.
Forgetting that lakehouses still need governance — IAM, lineage, quality tests, contracts; the "open" part is the storage, not the operational discipline.
Underestimating the operational cost of running an open lakehouse vs a managed warehouse — engineering time matters.

Data Lake Interview Question on Pattern Selection

A retail company stores 200TB of clickstream events plus a 5TB curated finance mart and a 1TB ML feature store. Should they run on a pure data lake, a cloud warehouse, a lakehouse, or a hybrid? Walk through your decision.

Solution Using a Hybrid — Lakehouse for Clickstream + Features, Warehouse for Finance

Workload                  Volume    Pattern recommended    Why
────────────────────────  ────────  ─────────────────────  ─────────────────────────────────────────
Clickstream events        200 TB    Lakehouse (Iceberg)    Volume + schema flexibility + ML access
ML feature store           1 TB     Lakehouse (Iceberg)    Same engine, same catalog as clickstream
Curated finance mart       5 TB     Cloud warehouse        SQL ergonomics, ACID across many tables, BI tools
                                                            Snowflake / BigQuery / Redshift

Boundary contract:
  - Clickstream + features stay in S3 + Iceberg
  - Finance mart loads nightly from Iceberg via Snowflake external tables
  - Reverse-ETL syncs finance summaries back into the lakehouse for ML feature joins

Why this works: clickstream at 200TB is the workload that justifies the cheaper object-storage cost model; the lakehouse table format adds ACID and time travel that the team will need for replays and audits; the ML feature store sits on the same engine + catalog so feature engineers can JOIN against clickstream without a cross-system data hop; the finance mart at only 5TB is small enough that warehouse storage cost is negligible, and the team's BI tools and analyst SQL ergonomics dominate the decision; the boundary contract (Snowflake external tables) lets finance read curated lake tables without copying them, and reverse-ETL closes the loop for ML.

Step-by-step trace of the decision:

step	question	answer
1	Is volume > 50TB?	yes (clickstream) → lake or lakehouse
2	Need ACID + concurrent writers + time travel?	yes (CDC + ML feature recomputation) → lakehouse, not pure lake
3	Pick a table format	Iceberg (broadest engine support across Spark, Trino, Snowflake)
4	Is the curated SQL workload < 10TB?	yes (finance, 5TB) → warehouse is fine
5	Pick a warehouse	Snowflake (ergonomics + multi-cloud + Iceberg external table support)
6	Boundary contract	Snowflake external tables on Iceberg; reverse-ETL nightly job

Output: the recommended architecture summary:

zone	technology	volume	primary consumer
Clickstream lakehouse	S3 + Iceberg + Spark/Trino	200 TB	ML pipelines, analyst SQL via Trino
ML feature store	S3 + Iceberg + Spark	1 TB	ML training + serving
Finance warehouse	Snowflake (managed)	5 TB	Finance analysts, BI dashboards
Boundary	Snowflake external tables on Iceberg	—	finance reads curated lake data zero-copy

Why this works — concept by concept:

Volume-driven storage choice — 200TB at warehouse storage cost ($0.02-0.04/GB/month) = ~$5K/month; same data on S3 = ~$4.6K/month and available to ML directly. The cost gap widens with growth.
Lakehouse for ACID + time travel — clickstream replays and ML feature recomputation need transactional snapshots; a pure lake without Iceberg cannot give you that.
Warehouse for curated SQL — finance analysts live in BI tools; warehouse SQL ergonomics + ACID across multi-table updates dominates the cost-per-query argument at 5TB scale.
Iceberg as the open boundary — Snowflake reads Iceberg tables natively via external tables; no nightly copy job, no schema drift between systems.
Reverse-ETL closes the loop — finance summaries flow back to the lakehouse so ML features can JOIN against revenue without leaving the lake stack.
Operational cost trade-off — running both a lakehouse and a warehouse is more engineering than a single managed warehouse; the cost is justified at this volume mix but not at 5TB total.

Inline CTA: More SQL practice problems for warehouse-style queries and data modeling practice for star-schema and OBT patterns. Course: Data Modeling for Data Engineering Interviews.

SQL
Topic — dimensional modeling
Dimensional modeling problems

Practice →

SQL
Language — data modeling
Data modeling problems

Practice →

SQL
Language — SQL
All SQL practice problems

Practice →

4. Interview Answer Shape — Grain, Idempotency, Lineage, Reconciliation

A five-step template for data lake design rounds

"Design our company's analytics data lake" is the canonical open-ended system-design prompt — and the cleanest answer is a five-step template that walks the interviewer through the load-bearing decisions in a fixed order. The mental model: clarify grain → separate landing from conformed → make loads idempotent → attach lineage keys → reconcile aggregates against source. Following this template demonstrates that you have shipped data pipelines before, and it gives the interviewer five concrete spots to drill deeper. Candidates who jump straight to vendor names or who skip the grain question lose the round, regardless of how many tools they can name.

Pro tip: State the template out loud at the start: "I'd answer this in five steps — first clarify grain, then separate landing from conformed, then make loads idempotent, then attach lineage keys, then explain how I'd reconcile aggregates against the source." This gives the interviewer a road map and makes it easy for them to interrupt at any step with "tell me more about X" — which is exactly the signal you want.

Step 1 — Clarify the grain and the metric definition

The grain invariant: the grain of a fact table is the business event one row represents — orders, order lines, shipments, page views, user-day, user-session — and ambiguous grain is the single most common bug in data engineering. Ask the interviewer "are we counting orders or order lines?" before drawing a box. The answer changes joins, group-bys, and reconciliation totals.

Order grain — one row per order; COUNT(*) = number of orders.
Order-line grain — one row per line item; COUNT(DISTINCT order_id) = number of orders.
User-day grain — one row per user per day; SUM(events) = events per user per day.
Session grain — one row per session; rolling LAG over events to define session boundaries.

Worked example. "How many orders did we ship last week?" against fact_shipments.

grain candidate	implied metric
order grain	`COUNT(*) WHERE shipped_date BETWEEN ...`
order-line grain	`COUNT(DISTINCT order_id) WHERE shipped_date BETWEEN ...`
shipment grain	`COUNT(DISTINCT order_id) WHERE shipment_event = 'shipped'`

Step-by-step explanation.

If the table is at order grain (one row per order), COUNT(*) WHERE shipped_date BETWEEN ... directly counts orders shipped — clean and simple.
If the table is at order-line grain (one row per item per order), COUNT(*) over-counts every multi-item order; the right answer becomes COUNT(DISTINCT order_id).
If the table is at shipment grain (one row per shipment event per line, including partial shipments and cancellations), filter by event_type = 'shipped' first and then COUNT(DISTINCT order_id).
Without naming the grain, the same SQL can produce three different "right" numbers — and the analyst, dashboard, and source-of-truth Slack thread will each pick a different one.
Stating the grain in the first sentence of every interview answer prevents this entire class of bug — and the same rule applies in production: every fact table should have its grain documented in the catalog comment.

Worked-example solution. Always state grain explicitly:

"This fact_shipments table has shipment grain — one row per shipment_event per order_line.
For 'orders shipped last week' I'll do COUNT(DISTINCT order_id) where event_type = 'shipped'
and shipped_date BETWEEN start_of_week AND end_of_week."

Rule of thumb: the first sentence of every interview answer should name the grain. Even if the interviewer doesn't ask, declaring grain demonstrates senior intent.

Step 2 — Separate landing from conformed (bronze vs silver)

The separation invariant: landing is what the source sent; conformed is what the business agrees to call truth; never let analysts query landing directly because schemas change without notice. The bronze/silver split is the architectural manifestation of this rule.

Landing / bronze — append-only, source-fidelity, partitioned by ingest_date.
Conformed / silver — deduplicated, typed, with conformed business keys.
Curated / gold — subject-area marts and dimensional models for downstream consumption.
Boundary — only the silver and gold layers carry consumer contracts; bronze is for re-processors only.

Worked example. A pipeline lands daily JSON snapshots; without a separation layer, analysts join directly against bronze.orders and break every time the source adds a column.

layer	who reads	breakage tolerance
bronze.orders	re-processors only	high (re-process on demand)
silver.orders	analyst ad-hoc, ML	low (contract change ≥ 30 days notice)
gold.fact_orders	dashboards, BI	zero (versioned column contracts)

Step-by-step explanation.

Bronze is owned by the re-processors only — no SLA, no consumer contract; analysts who query it get whatever the source app emitted today, including freshly-renamed columns and broken types.
Silver is the contract layer — analyst ad-hoc SQL, ML feature pipelines, and reverse-ETL all read it; breakage requires ≥30-day notice so consumers can adapt.
Gold has zero breakage tolerance — dashboards and BI tools couple to specific column names + types; any change requires explicit version bumping (gold.fact_orders_v2) so old dashboards keep working.
Without these boundaries, a source app's column rename cascades immediately into a broken executive dashboard, and the data team learns about it from a Slack screenshot.
With these boundaries, the silver-layer owner absorbs the upstream change inside the dedup logic, gold contracts stay intact, and the dashboard never breaks — the architecture has done its job.

Worked-example solution. State the layer boundaries explicitly:

"I'd split the platform into three layers — bronze for raw landing, silver for conformed,
gold for analytics-ready. Bronze is for re-processors only; analysts and dashboards read
silver and gold. The boundary contract is documented and breakage requires 30-day notice
for silver and explicit version bumping for gold."

Rule of thumb: any answer that allows analysts to query the landing zone has a hidden bug-factory; the bronze/silver split is what prevents source-schema chaos from cascading into BI.

Step 3 — Idempotent loads — same input → same output, every time

The idempotency invariant: a daily load is idempotent if re-running it (after any failure, manual intervention, or backfill) produces byte-identical output; without idempotency, retries cause duplicates and counts drift silently. Idempotency is achieved through MERGE instead of INSERT, partition-overwrite semantics, or table-format ACID transactions.

MERGE on a business key — WHEN MATCHED UPDATE SET * + WHEN NOT MATCHED INSERT *.
Partition overwrite — INSERT OVERWRITE INTO silver.orders PARTITION (ingest_date='2026-04-13').
Iceberg / Delta MERGE — ACID transaction; safe for concurrent writers.
Functional idempotency — pure transformations whose output depends only on inputs, never on NOW() or random.

Worked example. A retry on a half-completed daily load should produce the same final state as the original successful run.

run	rows in silver before	rows after
original	0	12,835
retry (after partial failure)	12,401	12,835 (no duplicates)
backfill 2026-04-12 a week later	12,820	12,820 (overwritten cleanly)

Step-by-step explanation.

Original run: silver starts at 0 rows; the MERGE writes 12,835 unique rows after dedup; final count = 12,835.
Retry after a partial failure: silver already has 12,401 rows (the partial write that crashed); the MERGE updates the existing rows and inserts only the missing 434; final count = 12,835 — no duplicates.
Backfill 2026-04-12 a week later: partition-overwrite semantics drop the existing 12,820 rows for that date and replace them with the freshly recomputed 12,820; final count = 12,820 — clean.
The key invariant: every rerun produces the same final state regardless of the starting state — that's what idempotency means.
Without idempotency, the retry would have inserted 434 duplicate rows (12,835 - 12,401), and the backfill would have either errored on the unique constraint or silently created shadow data that broke the next dashboard refresh.

Worked-example solution.

MERGE INTO silver.orders t
USING (
    SELECT * FROM bronze.orders
    WHERE ingest_date = '2026-04-13'
    QUALIFY ROW_NUMBER() OVER (PARTITION BY order_id ORDER BY source_ts DESC) = 1
) s
ON t.order_id = s.order_id
WHEN MATCHED THEN UPDATE SET *
WHEN NOT MATCHED THEN INSERT *;

Rule of thumb: if your interviewer asks "what happens if this job runs twice", and your answer involves any kind of cleanup script, you don't have idempotency — restructure.

Step 4 — Attach row-level lineage keys — `ingest_id`, `source_ts`, `pipeline_version`

The lineage invariant: every silver and gold row carries the columns that let you reconstruct which source payload produced it and which pipeline version transformed it; without lineage, debugging "why does this row look wrong" is forensic archaeology.

ingest_id — unique identifier of the bronze batch (e.g., timestamp + UUID).
source_ts — timestamp from the source system (CDC) for ordering.
pipeline_version — git SHA or version tag of the transformation code.
silver_loaded_at — when the row entered silver; useful for SLA metrics.

Worked example. Analysts notice that revenue for order_id=448 is wrong; with lineage, they can trace it back to the exact bronze file and pipeline version that produced it.

field	value
order_id	448
revenue	$99.00 (wrong; should be $999.00)
ingest_id	`20260412T0200Z_a3f2`
source_ts	`2026-04-12 09:30:15`
pipeline_version	`v2.1.7` (commit `b3a4d72`)
silver_loaded_at	`2026-04-12 02:15:32`

Step-by-step explanation.

An analyst notices order_id = 448 shows revenue $99 instead of the expected $999 in the BI dashboard.
They look the row up in silver: SELECT ingest_id, source_ts, pipeline_version FROM silver.orders WHERE order_id = 448.
The result tells them exactly which bronze batch produced this row (ingest_id = '20260412T0200Z_a3f2'), the source moment (source_ts = 2026-04-12 09:30:15), and the pipeline version that ran (v2.1.7).
They open the bronze file at that ingest_id. If the source payload already shows $99, it's a source bug — file a ticket with the upstream team and replay from a known-good source_ts.
If the bronze payload shows $999 but silver shows $99, the bug is in the pipeline. Run git log v2.1.7 to find the exact commit, fix the transformation, deploy v2.1.8, and backfill the affected ingest_date partition — total recovery time ~30 minutes instead of multi-day forensic SQL.

Worked-example solution. Carry lineage in every silver row:

SELECT order_id,
       customer_id,
       amount::DECIMAL(18,2)            AS amount,
       source_ts,
       ingest_id,
       'v2.1.7'                          AS pipeline_version,
       CURRENT_TIMESTAMP                 AS silver_loaded_at
FROM (deduped bronze rows);

Rule of thumb: if the dashboard shows a wrong number and you can't answer "which source file produced this row?" in under five minutes, your lineage isn't strong enough.

Step 5 — Aggregate reconciliation against the source

The reconciliation invariant: a daily job compares aggregate metrics (row counts, sums, distinct counts) between the lake and the source system, alerts on drift above a tolerance, and blocks promotion to gold until the drift is investigated. Reconciliation is the difference between "we trust the lake" and "we hope the lake is right".

Row count — COUNT(*) in lake vs source for the same time window.
Sum reconciliation — SUM(amount) in lake vs source.
Distinct count — COUNT(DISTINCT user_id) to catch dedup bugs.
Tolerance threshold — typically 0.1% for high-volume facts, 0.01% for finance.

Worked example. Daily reconciliation between silver and source-app replica.

metric	silver	source	drift	passes?
row count	12,835	12,835	0.0%	✓
sum(amount)	$4,128,931	$4,128,931	0.0%	✓
count(distinct user_id)	8,712	8,712	0.0%	✓

Step-by-step explanation.

The daily reconciliation job runs after the silver load completes for the prior day.
It computes three metrics over silver.orders: COUNT(*), SUM(amount), and COUNT(DISTINCT user_id) for the same date.
It computes the same three metrics over source_replica.orders (a read-only replica of the source-app database) for the same date.
For each metric, drift is calculated as ABS(silver - source) / source; the gate passes only if every metric is below the tolerance (0.001 = 0.1% for facts; 0.0001 for finance).
If all three pass: silver promotes to gold and the dashboard refresh proceeds. If any fail: the gate blocks promotion, pages the on-call engineer, and emits the failing metric to a drift dashboard for investigation — the BI team never sees stale or wrong numbers because they see no refresh.

Worked-example solution. A reconciliation gate before gold promotion:

WITH lake AS (
    SELECT COUNT(*) AS n, SUM(amount) AS s
    FROM silver.orders
    WHERE source_ts::DATE = '2026-04-13'
),
src AS (
    SELECT COUNT(*) AS n, SUM(amount) AS s
    FROM source_replica.orders
    WHERE order_ts::DATE = '2026-04-13'
)
SELECT
    ABS(lake.n - src.n) * 1.0 / src.n AS row_drift,
    ABS(lake.s - src.s) * 1.0 / src.s AS sum_drift,
    CASE
        WHEN ABS(lake.n - src.n) * 1.0 / src.n < 0.001
         AND ABS(lake.s - src.s) * 1.0 / src.s < 0.001
        THEN 'PASS'
        ELSE 'FAIL'
    END AS gate
FROM lake, src;

Rule of thumb: never promote to gold without a reconciliation gate; the BI team will discover any drift the hard way otherwise, and trust takes years to rebuild.

Common beginner mistakes

Skipping Step 1 (grain) and going straight to architecture — every downstream answer is wrong if grain is wrong.
Letting analysts query the bronze zone directly — schema drift cascades into BI dashboards.
"Idempotent" loads that depend on NOW() — re-runs produce different rows; not actually idempotent.
Lineage limited to the pipeline level (not the row level) — debugging "this row is wrong" is a multi-day forensic effort.
Reconciliation that only checks row counts but not sums — COUNT matches when the dedup deletes the wrong row but the count happens to be right.

Data Lake Interview Question on a Full System-Design Walkthrough

Walk through your end-to-end answer to "design our company's analytics data lake" using the five-step template.

Solution Using the Five-Step Template

1. CLARIFY GRAIN
   "Before I draw any boxes — what's the canonical fact event? Orders, order lines, shipments?
    What's the metric we ultimately care about? Revenue, user counts, latency?"
   → assume: order grain; canonical metric = daily revenue per region.

2. SEPARATE LANDING FROM CONFORMED
   bronze.orders   ← S3 append-only daily JSON, partitioned by ingest_date
   silver.orders   ← deduped + typed + conformed customer_key/region_key
   gold.fact_orders ← star schema with dim_customer, dim_region, dim_date

3. IDEMPOTENT LOADS
   - bronze: append-only writes by ingest_id (never overwrite)
   - silver: MERGE on order_id with QUALIFY ROW_NUMBER() = 1 dedup
   - gold: INSERT OVERWRITE PARTITION (date_key) for the affected day(s)
   Re-runs produce byte-identical output.

4. ROW-LEVEL LINEAGE
   Carry ingest_id, source_ts, pipeline_version, silver_loaded_at on every silver row.
   Carry silver_loaded_at and pipeline_version on every gold row.
   Forensic queries: "show me every silver.orders row where pipeline_version='v2.1.6'."

5. AGGREGATE RECONCILIATION
   Daily SQL job: compare COUNT(*), SUM(amount), COUNT(DISTINCT user_id) between
   silver.orders and the source-app replica for the prior day. Drift > 0.1% blocks
   gold promotion and pages on-call. Drift dashboard surfaces history at a glance.

Why this works: the template gives the interviewer a clear road map (so they know where to drill) while demonstrating that the candidate has shipped this kind of pipeline before; each step addresses a specific failure mode (grain ambiguity, schema drift, retry duplicates, debugging dead-ends, silent data corruption); the order is non-arbitrary — Step N depends on Step N-1, and skipping any step weakens the foundation; every step has a concrete artifact (a layer, a SQL pattern, a column, a job) so the interviewer can ask "show me what that looks like" and get a specific answer.

Step-by-step trace through a sample interview round:

time (min)	step	candidate output
0-2	grain	"Are we counting orders or order lines? Confirmed: orders."
2-7	landing vs conformed	drew bronze/silver/gold split with ownership boxes
7-12	idempotency	walked through silver MERGE; named QUALIFY ROW_NUMBER dedup
12-15	lineage	listed `ingest_id`, `source_ts`, `pipeline_version` columns
15-20	reconciliation	sketched daily-reconciliation SQL job + drift dashboard
20-25	open questions	streaming variant, schema evolution, multi-region replication

Output: the recommended interview-round shape:

step	minutes	failure mode addressed
1 — grain	0-2	ambiguous metric → wrong joins
2 — landing vs conformed	2-7	source-schema drift → BI breakage
3 — idempotency	7-12	retries → duplicates → drift
4 — lineage	12-15	"why is this row wrong" → forensic dead-end
5 — reconciliation	15-20	silent corruption → trust loss

Why this works — concept by concept:

Step 1 anchors the conversation in business semantics — grain is the foundation; getting it right makes Steps 2-5 simpler, getting it wrong makes them all moot.
Step 2 turns architecture into ownership — naming the layer boundary makes it easy to talk about who reads what, who's allowed to break what, and what notice consumers get.
Step 3 prevents the most common production incident — non-idempotent loads are the #1 source of "duplicate row" bug reports; demonstrating the MERGE + QUALIFY ROW_NUMBER pattern signals senior fluency.
Step 4 turns debugging from hours to minutes — lineage columns are the difference between "I can fix this in 10 min" and "I'll get back to you tomorrow."
Step 5 is the operational backstop — even with steps 1-4 done well, you need reconciliation to catch the failures you didn't anticipate; the gate-before-promotion pattern blocks drift before consumers see it.
The template's value compounds — each step makes the next one easier, and skipping any step weakens the foundation that the later steps build on.

Inline CTA: More ETL practice problems for end-to-end pipeline design and the data modeling practice page for grain and dimensional patterns. Course: ETL System Design for Data Engineering Interviews.

ETL
Topic — ETL pipelines
ETL practice problems

Practice →

SQL
Language — data modeling
Data modeling problems

Practice →

SQL
Language — SQL
All SQL practice problems

Practice →

Tips to crack data lake architecture interviews

Master the four primitives — zones, ingest flow, pattern selection, answer template

If you can draw the bronze/silver/gold zones with ownership labels, walk the ingest → catalog → compute flow without skipping the catalog, articulate when a lakehouse beats a warehouse and when it doesn't, and structure your answer using the five-step grain → idempotency → lineage → reconciliation template — you can clear most data-engineering system-design rounds. The remaining 20% is dialect-specific (Spark vs Snowflake idioms, Iceberg vs Delta semantics) and behavioral.

Always state grain in the first sentence

Before drawing any boxes, name the grain: "this is order-line grain" or "this is user-day grain". Every wrong answer in a system-design round can be traced back to a grain ambiguity that nobody named. Stating grain explicitly costs five seconds and saves the entire round.

Pick Iceberg unless you have a reason not to

Iceberg has the broadest engine support (Spark, Trino, Snowflake, BigQuery, Dremio, Athena) and is the most engine-agnostic of the three open table formats. Pick Delta if your stack is Databricks-centric and Spark-only. Pick Hudi if your workload is upsert-heavy CDC. State the choice and the reason out loud — "I'd pick Iceberg for engine portability" — interviewers grade the reasoning more than the choice.

Treat idempotency as table stakes, not advanced

MERGE instead of INSERT, partition-overwrite for backfill, and pure transformations whose output depends only on inputs (never NOW() or random) — these are baseline expectations, not advanced techniques. If you forget to mention idempotency in a system-design round, the interviewer will assume you have not shipped a production pipeline.

Use Spark for batch, Trino for interactive, DuckDB for ad-hoc

Spark dominates batch + streaming with the richest connector ecosystem; Trino dominates federated interactive SQL across many catalogs; DuckDB is rising fast for single-node ad-hoc analytics under 1TB. Naming the right tool for the workload (without over-explaining) signals breadth.

Reconciliation is what separates "we trust the lake" from "we hope the lake is right"

Always include a reconciliation step that compares aggregate metrics between the lake and the source system, alerts on drift above a tolerance, and blocks gold promotion until drift is investigated. The five seconds it takes to mention reconciliation is the difference between a senior signal and a mid-level signal.

Where to practice on PipeCode

Start with the ETL practice page for medallion-zone and end-to-end pipeline problems. Drill the related topic pages: streaming, dimensional modeling, SQL practice, Python practice, data modeling practice. The interview-first courses page bundles structured curricula — start with ETL System Design for Data Engineering Interviews, Data Modeling for Data Engineering Interviews, or PySpark Fundamentals. For broader coverage, browse by topic or read the SQL interview questions for data engineering and top data engineering interview questions 2026 blogs.

Frequently Asked Questions

What is data lake architecture?

Data lake architecture is the set of conventions — layered zones (bronze/silver/gold), an ingest → catalog → compute flow on object storage, an open table format for ACID semantics, and disciplined ownership and quality contracts — that turn raw object storage into a trustworthy analytics platform. Without these conventions, a "data lake" devolves into a data swamp where nobody can trust the numbers.

What is the difference between a data lake, a data warehouse, and a lakehouse?

A data lake is cheap, flexible object storage with file-based reads and no built-in ACID; a cloud warehouse (Snowflake, BigQuery, Redshift) is a managed system with proprietary storage, full ACID, and SQL-first ergonomics; a lakehouse is a lake plus an open table format (Iceberg, Delta Lake, Hudi) that adds ACID, time travel, schema evolution, and concurrent writers — bringing warehouse-like semantics to object storage. Most organizations run a hybrid: lake/lakehouse for high-volume + ML workloads, warehouse for curated SQL.

What are bronze, silver, and gold layers?

Bronze (or landing/raw) is an append-only mirror of source payloads with minimal transformation. Silver (or refined/conformed) applies dedup, type coercion, and conformed business keys; this is the source of truth for downstream applications. Gold (or curated/consumption) publishes subject-area marts and star-schema fact + dim tables for analyst SQL and BI dashboards. The names vary across vendors but the three-tier shape is universal.

Do I need Iceberg, Delta Lake, or Hudi for every project?

No. Small teams can start with well-partitioned Parquet and strict naming conventions. Reach for an open table format when you need ACID transactions, concurrent writers, partition evolution, time travel, or simpler upserts and deletes. Pick Iceberg for the broadest engine support, Delta for tightest Databricks/Spark integration, Hudi for upsert-heavy CDC workloads.

What is the small-file problem?

When a lake table accumulates millions of small files (e.g., 5KB each from frequent micro-batch writes), query planning spends more time listing files in the catalog and metastore than actually scanning data — a Spark or Trino query that should take 500ms can take 50 seconds. The fix is scheduled compaction jobs that rewrite many small files into fewer 128MB-1GB files, plus targeting larger micro-batch sizes upstream.

How do I handle schema evolution in a data lake?

Open table formats handle schema evolution gracefully — adding a column or widening a type is a single metadata commit. Without a table format, schema evolution requires rewriting partitions or carrying a column-version field on every row. Either way, the silver layer should be the schema-stability boundary: bronze accepts whatever the source sends, silver enforces a canonical schema, and changes to silver require explicit consumer notice (typically 30 days).

How does this connect to data engineering interviews on PipeCode?

System-design questions still reduce to SQL queries, Python data transforms, and dimensional modeling decisions. PipeCode focuses on those signals with 450+ problems — drill SQL aggregations and joins, Python pipeline patterns, and dimensional models, then layer on system-design depth via the courses. Use Practice once you can draw the medallion zones and the ingest → catalog → compute flow confidently.

Start practicing data lake architecture problems

Data Engineering Roadmap for Freshers (2026): A 13-Step Beginner's Guide from SQL to Your First Data Engineering Job

Gowtham Potureddi — Mon, 11 May 2026 03:17:57 +0000

Data engineering is one of the fastest-growing tech careers in 2026. Companies collect huge amounts of data every day, and data engineers build the systems that collect, clean, transform, store, and deliver that data so analysts, scientists, and product teams can use it. If you're a fresher and confused about where to start, this data engineering roadmap for freshers lays out a clear, ordered 13-step path — what to learn first, what to learn next, what to build, and how to prove the work to a recruiter.

This guide is a beginner-first walkthrough for how to become a data engineer in 2026 without a CS degree, three certificates, or a Spark cluster on day one. The 13 steps are grouped into five learning blocks below, each with a tiny worked example you can run on your laptop. Most freshers fail because they jump to Spark too early, ignore SQL depth, avoid projects, or watch tutorials without practising — the roadmap below fixes all four. Examples use PostgreSQL SQL (the dialect every coding-environment interview defaults to) and standard-library Python so you can run everything on a laptop without setup overhead. Default plan: about 6–9 months at 10–15 hours per week to be job-ready, 9–12 months at 6–8 hours per week for working learners.

1. Step 1 — Master SQL: The Most Important Skill for a Data Engineer

SQL fundamentals, joins, aggregations, window functions, and the queries you'll write every day

SQL is the foundation of data engineering — you'll write it daily for querying, cleaning, transforming, joining datasets, building reports, and writing ETL logic. Master SQL first; everything else becomes easier.

The five SQL skill clusters every fresher needs:

Basics — SELECT, WHERE, ORDER BY, LIMIT, DISTINCT.
Aggregations — COUNT, SUM, AVG, MIN, MAX, GROUP BY, HAVING.
Joins — INNER, LEFT, RIGHT, FULL, SELF.
Window functions — ROW_NUMBER, RANK, DENSE_RANK, LAG, LEAD.
Advanced — CTEs, subqueries, CASE, NULL handling, date functions, indexes, query optimisation.

Pro tip: SQL is non-negotiable. Drill it daily on a free coding environment (DataLemur, LeetCode SQL, StrataScratch, HackerRank SQL). Most fresher rejections at the SQL screen are not from missing syntax — they are from joining at the wrong grain or putting an aggregate in the wrong clause.

SQL basics — `SELECT`, `WHERE`, `ORDER BY`, `LIMIT`, `DISTINCT`

The bedrock SQL shape: SELECT cols FROM table WHERE row_filter ORDER BY col DESC LIMIT N. That one query covers most "show me the top X by Y" prompts you'll ever write.

SELECT cols FROM table — pick the columns you actually need; never SELECT * in production.
WHERE filter — row-level predicate; runs before grouping.
ORDER BY col DESC — sort the result; ASC is default, DESC is biggest-first.
LIMIT N — keep only the top N rows.
DISTINCT col — collapse duplicates to a single value per group.

Example input. A 4-row employees table with name and salary.

name	salary
Alice	70000
Bob	45000
Carol	90000
Dan	55000

Question. Return the names and salaries of employees who earn more than 50,000, sorted from highest to lowest salary.

Code solution.

SELECT name, salary
FROM employees
WHERE salary > 50000
ORDER BY salary DESC;

Explanation of code. WHERE salary > 50000 runs first and drops Bob (45000). The remaining three rows are then sorted by salary in descending order, so Carol (highest) comes first, Alice second, Dan third. No LIMIT, so all three qualifying rows are returned.

Step-by-step.

step	clause	result
1	`FROM employees`	scan all 4 rows
2	`WHERE salary > 50000`	drop Bob (45000); 3 rows left
3	`ORDER BY salary DESC`	sort: Carol (90000) → Alice (70000) → Dan (55000)
4	`SELECT name, salary`	project the two named columns

Output.

name	salary
Carol	90000
Alice	70000
Dan	55000

Rule of thumb: always name the columns in the SELECT; SELECT * outside an exploratory REPL is a code smell.

Aggregations — `GROUP BY` + `HAVING`

The aggregation shape: SELECT dim, AGG(col) FROM table GROUP BY dim HAVING AGG_filter. GROUP BY collapses many rows to one row per group; HAVING filters the resulting groups (you cannot put an aggregate in WHERE).

COUNT(*) — number of rows per group.
SUM(col) / AVG(col) / MIN(col) / MAX(col) — collapse a numeric column.
GROUP BY dim — one output row per distinct value of dim.
HAVING AGG > N — keep only groups whose aggregate exceeds N.

Example input. A 6-row employees table with department and salary.

name	department	salary
Alice	Engineering	90000
Bob	Engineering	80000
Carol	Sales	50000
Dan	Sales	55000
Eve	Marketing	65000
Frank	Marketing	60000

Question. Return the average salary per department, but only show departments whose average exceeds 60,000.

Code solution.

SELECT department, AVG(salary) AS avg_salary
FROM employees
GROUP BY department
HAVING AVG(salary) > 60000;

Explanation of code. GROUP BY department collapses the six rows into three groups — Engineering, Sales, Marketing. AVG(salary) computes the per-group average: Engineering 85000, Sales 52500, Marketing 62500. HAVING AVG(salary) > 60000 then drops Sales (52500 fails the threshold) and keeps the other two.

Step-by-step.

step	clause	result
1	`FROM employees`	scan all 6 rows
2	`GROUP BY department`	3 groups — Engineering (2 rows), Sales (2 rows), Marketing (2 rows)
3	`AVG(salary)` per group	Engineering 85000, Sales 52500, Marketing 62500
4	`HAVING AVG(salary) > 60000`	drop Sales (52500 fails); keep Engineering + Marketing
5	`SELECT department, AVG(salary) AS avg_salary`	project the 2 surviving rows

Output.

department	avg_salary
Engineering	85000
Marketing	62500

Rule of thumb: row predicates → WHERE; aggregate predicates → HAVING. Putting AVG > X in WHERE is a parse error.

Joins — connecting tables on a common key

Joins combine columns from two tables on a matching key. The four every fresher needs: INNER (only matched rows survive), LEFT (all rows from the left table, even unmatched), RIGHT (mirror of LEFT, rarely used), FULL (all rows from both sides). SELF JOIN joins a table to itself for hierarchies (manager / employee, parent / child).

INNER JOIN — strict match on both sides.
LEFT JOIN — keep every left row; NULL on the right when no match.
RIGHT JOIN — same as LEFT with sides swapped; usually rewrite as LEFT.
FULL JOIN — keep every row from both sides; useful for reconciliation.
SELF JOIN — alias the same table twice (employees a JOIN employees b ON a.manager_id = b.id).

Example input. An orders table and a customers table.

orders:

order_id	customer_id
101	C1
102	C2
103	C1

customers:

customer_id	customer_name
C1	Alice
C2	Bob

Question. Return one row per order showing order_id and the matching customer_name.

Code solution.

SELECT o.order_id, c.customer_name
FROM orders o
JOIN customers c
    ON o.customer_id = c.customer_id;

Explanation of code. The INNER JOIN (the default form when you just write JOIN) matches each order to its customer using customer_id. Order 101 → Alice, order 102 → Bob, order 103 → Alice. All three orders have a matching customer, so every order survives.

Step-by-step.

step	action	result
1	scan `orders` (left side)	3 rows: 101→C1, 102→C2, 103→C1
2	for each row, look up `customer_id` in `customers`	C1→Alice (twice), C2→Bob
3	`INNER JOIN` keeps only matched pairs	all 3 orders matched, 0 dropped
4	`SELECT o.order_id, c.customer_name`	project the 2 named columns

Output.

order_id	customer_name
101	Alice
102	Bob
103	Alice

Rule of thumb: always give every table a short alias (o, c) and prefix every column (o.order_id, c.customer_name) — the SQL becomes self-documenting.

Common beginner mistakes

Using SELECT * everywhere — production queries always name the columns.
Putting an aggregate in WHERE instead of HAVING — parse error in PostgreSQL.
Joining at the wrong grain (one-to-many without thinking) — the #1 source of "the number is suddenly 3× too high" bugs.
Memorising syntax without internalising which side keeps its rows in a LEFT JOIN — the part that breaks numbers.
Skipping window functions because they "look hard" — interviewers love them; they take a week to learn.

Worked Problem on Ranking Top Earners per Department with Window Functions

Example input. A 6-row employees table mixing departments and salaries.

name	department	salary
Alice	Engineering	90000
Bob	Engineering	80000
Carol	Sales	50000
Dan	Sales	55000
Eve	Marketing	65000
Frank	Marketing	60000

Question. Rank each employee by salary within their department (highest = rank 1) and return only the top earner per department. Use a window function — pure GROUP BY cannot keep both the rank and the row's other columns.

Solution Using `ROW_NUMBER() OVER (PARTITION BY department ORDER BY salary DESC)`

Code solution.

SELECT department, name, salary
FROM (
    SELECT
        department,
        name,
        salary,
        ROW_NUMBER() OVER (
            PARTITION BY department
            ORDER BY salary DESC
        ) AS rank
    FROM employees
) t
WHERE rank = 1;

Explanation of code. ROW_NUMBER() OVER (PARTITION BY department ORDER BY salary DESC) assigns a strict 1, 2, 3 sequence within each department, ordered by salary from highest to lowest. The outer WHERE rank = 1 keeps only the top-paid row per department. The wrapping subquery is needed because PostgreSQL evaluates window functions after WHERE, so we cannot filter rank = 1 in the same level where we compute it.

Output.

department	name	salary
Engineering	Alice	90000
Marketing	Eve	65000
Sales	Dan	55000

Step-by-step trace for the input rows above:

department	name	salary	rank
Engineering	Alice	90000	1
Engineering	Bob	80000	2
Marketing	Eve	65000	1
Marketing	Frank	60000	2
Sales	Dan	55000	1
Sales	Carol	50000	2

After WHERE rank = 1: three rows — one per department, the top earner.

Why this works — concept by concept:

PARTITION BY department — defines the group inside which the ranking happens; without it, the rank would be global across all employees.
ORDER BY salary DESC — descending so rank 1 is the highest-paid; ascending would give the lowest.
ROW_NUMBER not RANK — strict 1, 2, 3; ties produce one rank-1 row per partition, which is what "top earner" demands.
Outer WHERE rank = 1 filter — Postgres cannot filter window-function output in the same query level; the wrap is required.
one row per department guaranteed — ROW_NUMBER (not RANK or DENSE_RANK) ensures no ties, so the result has exactly one row per group.
Cost — O(N log N) from the partitioned sort; with an index on (department, salary DESC) this becomes O(N).

Inline CTA: drill the SQL practice page for short curated reps; the structured path for fresher SQL is SQL for Data Engineering Interviews — From Zero to FAANG.

SQL
Topic — window functions
SQL window-function problems

Practice →

SQL
Topic — joins
SQL join problems

Practice →

SQL
Topic — aggregation
SQL aggregation problems

Practice →

2. Step 2 — Learn Python for Data Engineering

Core Python, file handling, Pandas, and the API requests every DE writes

Python is the glue language for everything outside the database — ETL scripts, automation, data pipelines, API integrations, transformations. You don't need to be a Python wizard; you need to be fluent at reading CSVs, calling APIs, transforming data with Pandas, and writing small testable functions.

Three Python skill clusters every fresher needs:

Core Python — variables, loops, functions, lists / dicts / sets, classes, exception handling.
File handling — read and write CSV, JSON, and Excel files using the standard library and Pandas.
Libraries — Pandas for data transformation; Requests for API calls; PySpark later (Step 6) for big-data processing.

Pro tip: the 10% of Python you actually use day-to-day is csv, json, pathlib, collections, dataclasses, typing, and pandas. Skip metaclasses, descriptors, and async event loops on day one — they're irrelevant to fresher DE work.

Core Python — loops, lists, and small functions

The fresher Python invariant: write small testable functions over loops over lists / dicts. Type hints (def f(x: int) -> int:) make a 2-month-old script readable when you come back to it.

Variables and types — int, float, str, bool, None.
Lists, dicts, sets — ordered, key-value, unique-only.
Loops — for x in xs: over iterables.
Functions — single-responsibility; takes inputs, returns outputs.
Exception handling — try / except FileNotFoundError for fragile I/O.

Example input. A Python list of three integers.

data = [1, 2, 3]

Question. Multiply each number by 2 and print the result. Show the canonical for loop pattern that every other Python data-engineering script will mirror.

Code solution.

data = [1, 2, 3]

for num in data:
    print(num * 2)

Explanation of code. for num in data: walks the list one element at a time, binding the current value to num. Inside the loop body, num * 2 doubles the value and print(...) writes it to stdout. The pattern generalises directly to "for every row in this CSV, do something" — replace data with csv.DictReader(f) and you have an ETL skeleton.

Step-by-step.

iteration	`num`	`num * 2`	stdout
1	1	2	`2`
2	2	4	`4`
3	3	6	`6`
end	—	—	loop exits when list is exhausted

Output.

2
4
6

Rule of thumb: if your Python script grows past 100 lines and has zero functions, it's a notebook draft, not a script — refactor before sharing it.

File handling — reading CSV and JSON

Most data-engineering Python is reading a file, transforming the contents, and writing the result somewhere. The standard library has csv and json modules that cover 90% of fresher needs; for anything richer reach for Pandas.

open(path, encoding='utf-8') — open a text file safely.
csv.DictReader(f) — iterate CSV rows as dictionaries (column-name access).
json.load(f) — parse a JSON file into a Python dict / list.
pathlib.Path('file.csv') — modern path object; works on Windows, macOS, Linux.

Example input. A data.json file containing one JSON object.

{"name": "Alice", "salary": 70000}

Question. Open data.json, parse it into a Python dict, and print the parsed result.

Code solution.

import json

with open("data.json") as f:
    data = json.load(f)

print(data)

Explanation of code. with open("data.json") as f: opens the file safely (the with block guarantees the file is closed when the block exits, even on error). json.load(f) parses the file's contents into a Python object — a dict here because the JSON started with {. Printing the dict shows the parsed data.

Step-by-step.

step	action	result
1	`with open("data.json") as f`	file handle `f` opens in text mode
2	`json.load(f)` reads bytes	parses JSON object → Python `dict`
3	bind result to `data`	`data = {"name": "Alice", "salary": 70000}`
4	exit `with` block	file auto-closed (even on error)
5	`print(data)`	dict printed to stdout

Output.

{'name': 'Alice', 'salary': 70000}

Rule of thumb: always use with open(...) rather than the bare open() call — it auto-closes the file and handles exceptions cleanly.

Pandas for tabular data — `read_csv`, `groupby`, `sum`

Pandas is the Python library every DE uses for transforming tabular data. The three operations you'll do hundreds of times: read a CSV into a DataFrame, group by one or more columns, aggregate with sum / mean / count. Requests is the API-call counterpart.

pd.read_csv('file.csv') — read a CSV into a DataFrame.
df.groupby('col') — group rows by a column.
.sum() / .mean() / .count() — aggregate the groups.
requests.get(url).json() — fetch a URL and parse the JSON response.

Example input. A sales.csv file with 5 rows across two regions.

order_id	region	amount
1	North	100
2	North	150
3	South	80
4	South	120
5	North	70

Question. Read sales.csv into a Pandas DataFrame, group by region, and print the sum of amount per region.

Code solution.

import pandas as pd

df = pd.read_csv("sales.csv")
print(df.groupby("region").sum())

Explanation of code. pd.read_csv("sales.csv") loads the entire CSV into a DataFrame, with the first row treated as column headers. df.groupby("region") produces a grouped object that buckets rows by region. .sum() aggregates every numeric column within each bucket — here that's order_id (sum of IDs, usually meaningless) and amount (the metric we care about).

Step-by-step.

step	action	result
1	`pd.read_csv("sales.csv")`	DataFrame with 5 rows × 3 columns
2	`df.groupby("region")`	bucket rows: North = {1, 2, 5}; South = {3, 4}
3	`.sum()` per bucket	North: order_id sum = 8, amount = 320 (100+150+70); South: order_id sum = 7, amount = 200 (80+120)
4	`print(...)`	the two-row grouped frame prints to stdout

Output.

        order_id  amount
region
North          8     320
South          7     200

Rule of thumb: when the data fits in memory and you don't need a database, Pandas is faster than writing the SQL — but for anything past a few million rows, push the work back into SQL or PySpark.

Common beginner mistakes

Skipping type hints — code becomes unreadable in 2 months.
Reading huge CSVs into Pandas without chunksize — your laptop runs out of RAM.
Using requests without a timeout — a hung API call freezes your script forever (requests.get(url, timeout=10)).
Not handling None / missing values — int(None) crashes with a TypeError.
Writing 200-line scripts as one big block — break into def-defined functions.

Worked Problem on Building a CSV-to-Summary Python ETL Script

Example input. A sales.csv file with 5 rows.

order_id	region	amount
1	North	100
2	North	150
3	South	80
4	South	120
5	North	70

Question. Build a small Python ETL script that reads sales.csv, sums amount per region, writes the result to summary.csv, and prints the count of rows processed. This is the canonical Phase-1 portfolio script every fresher should ship to GitHub.

Solution Using Pandas + a writeable summary path

Code solution.

import pandas as pd
from pathlib import Path

def summarise_sales(input_path: Path, output_path: Path) -> int:
    df = pd.read_csv(input_path)
    summary = df.groupby("region", as_index=False)["amount"].sum()
    summary.to_csv(output_path, index=False)
    return len(df)

if __name__ == "__main__":
    rows = summarise_sales(Path("sales.csv"), Path("summary.csv"))
    print(f"processed {rows} rows")

Explanation of code. The function takes two Path objects so it's testable — you can call it from a test with mock paths instead of hardcoding filenames. pd.read_csv(input_path) loads the CSV, groupby("region", as_index=False)["amount"].sum() produces a clean two-column summary (as_index=False keeps region as a column rather than becoming the index), and to_csv(output_path, index=False) writes the summary back out without Pandas' default integer index column. The function returns the row count so the caller can log a clean status line.

Output.

summary.csv:

region	amount
North	320
South	200

stdout: processed 5 rows

Step-by-step trace for the input rows above:

step	action	result
1	`pd.read_csv("sales.csv")`	DataFrame with 5 rows × 3 columns
2	`df.groupby("region", as_index=False)["amount"].sum()`	2-row summary DataFrame
3	`summary.to_csv("summary.csv", index=False)`	file written to disk
4	`return len(df)`	returns `5`
5	`print(f"processed {rows} rows")`	stdout: `processed 5 rows`

Why this works — concept by concept:

Path objects for testable I/O — paths are inputs, not hardcoded constants, so the function works with any source / destination.
groupby(..., as_index=False) — keeps region as a regular column instead of the DataFrame index; the resulting CSV reads naturally.
["amount"].sum() — selects the metric column before aggregation; otherwise Pandas would also sum order_id, which is meaningless.
to_csv(..., index=False) — suppresses Pandas' default integer index column; the CSV has only the two columns you actually want.
return + print separation — the function returns a value (good for tests); the caller decides whether to print it (good for scripts vs imports).
Cost — O(N) where N is the input row count; fits in memory up to a few million rows.

Inline CTA: for fresher Python reps see Python practice page; the structured path is Python for Data Engineering Interviews — Complete Fundamentals.

PYTHON
Language — Python
Python practice problems

Practice →

COURSE
Course — Python for DE
Python for Data Engineering Interviews

View course →

SQL
Topic — CSV processing
SQL CSV-processing problems

Practice →

3. Steps 3-5 — Databases, Data Warehousing, and ETL/ELT

How data is stored, modeled, and moved through pipelines

Three closely-related steps in one section because they answer the same question: where does the data live, and how does it get there?

Step 3 — Databases. Relational (PostgreSQL, MySQL) for transactional workloads; NoSQL (MongoDB, Cassandra, Redis) for specialised cases. Learn keys, normalisation, transactions, indexing, ACID.
Step 4 — Data Warehousing. Snowflake, BigQuery, Redshift store analytics-ready data in fact tables + dimension tables, organised as a star schema (fact in the middle, dimensions hanging off). Heavily asked in interviews.
Step 5 — ETL / ELT. ETL = Extract → Transform → Load (transform before loading). ELT = Extract → Load → Transform (load raw, then transform inside the warehouse). Plus batch vs streaming pipelines, incremental loads, and CDC (change data capture).

Pro tip: the three databases worth installing for practice: PostgreSQL (covers 90% of relational SQL you'll see at work), SQLite (zero-setup local dev), and one NoSQL (MongoDB is the friendliest). Skip Redis until you genuinely need a cache; skip Cassandra until you genuinely have wide-column data.

Relational databases — tables, keys, normalisation, ACID

Relational databases store data in tables with primary keys (one column uniquely identifies each row) and foreign keys (a column in one table references the primary key of another). Normalisation splits data so each fact lives in exactly one place — no duplication, no inconsistency. ACID properties (Atomicity, Consistency, Isolation, Durability) guarantee that transactions either fully succeed or fully roll back.

Primary key — uniquely identifies a row (customer_id in customers).
Foreign key — points to another table's primary key (orders.customer_id → customers.customer_id).
Normalisation — 1NF / 2NF / 3NF — split tables until each fact lives once.
Indexing — speeds up lookups; trade-off is slower writes and extra storage.
ACID transactions — BEGIN; … COMMIT; (or ROLLBACK; on failure).

Example input. A two-table relational design — customers references orders via the foreign key customer_id.

customers:

customer_id	customer_name
C1	Alice
C2	Bob

orders:

order_id	customer_id	amount
101	C1	100
102	C2	200
103	C1	50

Question. Write the CREATE TABLE statements for customers and orders with proper primary keys and a foreign key from orders to customers. Then write a transactional INSERT that adds a new customer plus their first order atomically — both rows commit or neither does.

Code solution.

CREATE TABLE customers (
    customer_id   TEXT PRIMARY KEY,
    customer_name TEXT NOT NULL
);

CREATE TABLE orders (
    order_id    INT PRIMARY KEY,
    customer_id TEXT NOT NULL REFERENCES customers(customer_id),
    amount      NUMERIC(10,2) NOT NULL
);

BEGIN;
INSERT INTO customers (customer_id, customer_name) VALUES ('C3', 'Carol');
INSERT INTO orders (order_id, customer_id, amount) VALUES (104, 'C3', 75);
COMMIT;

Explanation of code. The customers table declares customer_id as PRIMARY KEY (uniqueness + index automatically created). The orders table's customer_id is REFERENCES customers(customer_id) — a foreign key that prevents you from inserting an order for a non-existent customer. The BEGIN; … COMMIT; block makes both inserts a single atomic transaction: if the second insert fails for any reason, the first is rolled back automatically — the database never ends up with a customer who has no order or an order pointing to a missing customer.

Step-by-step.

step	statement	result
1	`CREATE TABLE customers`	empty table, `customer_id` enforced unique
2	`CREATE TABLE orders`	empty table; FK rejects orphan `customer_id`
3	`BEGIN`	open a transaction — changes are invisible until commit
4	`INSERT INTO customers ('C3', 'Carol')`	row staged; FK in `orders` will accept C3 later
5	`INSERT INTO orders (104, 'C3', 75)`	row staged; FK satisfied because C3 exists in-tx
6	`COMMIT`	both rows persisted atomically; on error, both rolled back

Output.

After the transaction commits:

customer_id	customer_name
C1	Alice
C2	Bob
C3	Carol

order_id	customer_id	amount
101	C1	100
102	C2	200
103	C1	50
104	C3	75

Rule of thumb: every multi-row write that has to be "all or nothing" goes inside a BEGIN; … COMMIT; block — that's the entire point of a relational database.

Data warehousing — fact tables, dimension tables, star schema

A data warehouse stores analytics-ready data optimised for fast SELECT queries (not for high-volume INSERT / UPDATE). The canonical model is the star schema — one fact table in the middle that records events (sales, clicks, logins) surrounded by dimension tables that describe context (customers, products, dates). Heavily tested at interviews.

Fact table — measures events; mostly numeric columns + foreign keys to dimensions.
Dimension table — descriptive context; mostly text columns (customer name, product category).
Star schema — one fact in the centre, dimensions hanging off as star points.
Snowflake schema — dimensions further normalised into sub-dimensions.
Partitioning / clustering — physical layout choices that speed up filtered queries.

Example input. A star-schema design for an e-commerce sales fact with three dimensions.

fact_sales:

sale_id	date_id	customer_id	product_id	amount
S1	20260501	C1	P1	100
S2	20260501	C2	P2	200

dim_customer:

customer_id	customer_name
C1	Alice
C2	Bob

dim_product:

product_id	product_name
P1	Book
P2	Headphones

dim_date:

date_id	day	month	year
20260501	1	5	2026

Question. Write a query that joins the fact to all three dimensions and returns customer_name, product_name, month, and amount for every sale. This is the canonical "fact + dim rollup" report every BI dashboard runs.

Code solution.

SELECT
    c.customer_name,
    p.product_name,
    d.month,
    f.amount
FROM fact_sales f
JOIN dim_customer c ON c.customer_id = f.customer_id
JOIN dim_product  p ON p.product_id  = f.product_id
JOIN dim_date     d ON d.date_id     = f.date_id;

Explanation of code. The fact table sits in the middle and is joined once to each dimension on the matching surrogate key. Because each dimension has exactly one row per dimension key, the joins do not multiply rows — the output has the same number of rows as fact_sales. The SELECT then pulls the descriptive columns from the dimensions plus the amount from the fact.

Step-by-step.

step	action	result
1	scan `fact_sales`	2 rows (S1, S2)
2	join `dim_customer` on `customer_id`	S1 → Alice, S2 → Bob; row count unchanged (1:1)
3	join `dim_product` on `product_id`	S1 → Book, S2 → Headphones; row count unchanged
4	join `dim_date` on `date_id`	both rows pick up month=5; row count unchanged
5	`SELECT` 4 projected columns	final 2-row report

Output.

customer_name	product_name	month	amount
Alice	Book	5	100
Bob	Headphones	5	200

Rule of thumb: fact tables hold the measure; dimensions hold the context. If you can't tell whether a column belongs in the fact or the dim, ask "is this a number we'll aggregate, or text we'll group by?"

ETL vs ELT, batch vs streaming, and CDC in plain English

ETL = extract, transform, load — read source data, transform it in a separate engine (Spark, Python), then load the clean result into the warehouse. ELT = extract, load, transform — load the raw source straight into the warehouse, then transform with SQL. Modern cloud warehouses are powerful enough that ELT has become the default. Batch processes data on a schedule (every hour / day); streaming processes data as it arrives (sub-second). CDC (change data capture) tracks INSERT / UPDATE / DELETE events on a source so the warehouse stays in sync without re-loading the whole table.

ETL — transform outside the warehouse (older pattern; Spark, Python, custom).
ELT — transform inside the warehouse with SQL (newer; dbt, Snowflake, BigQuery).
Batch — scheduled jobs (hourly, daily); cheaper, simpler, slightly stale data.
Streaming — event-by-event processing (Kafka, Flink); fresher, more expensive.
CDC — incremental change tracking; loads only what changed since last run.

Example input. A daily-batch ETL skeleton in Python that loads yesterday's orders, transforms them, and writes a curated table.

source: raw orders dropped to S3 daily under s3://orders/2026-05-08/orders.csv
target: warehouse table fact_orders, partitioned by order_date

Question. Sketch the three-stage ETL pipeline shape — extract reads the CSV, transform cleans / dedupes / casts types, load writes to the warehouse. Use plain Python pseudocode; the goal is the shape not a runnable example.

Code solution.

import pandas as pd

def extract(date: str) -> pd.DataFrame:
    return pd.read_csv(f"s3://orders/{date}/orders.csv")

def transform(df: pd.DataFrame) -> pd.DataFrame:
    df = df.drop_duplicates(subset=["order_id"])
    df["order_date"] = pd.to_datetime(df["order_date"]).dt.date
    df["amount"]     = df["amount"].astype(float)
    return df

def load(df: pd.DataFrame, partition_date: str) -> int:
    # warehouse-specific COPY or INSERT INTO fact_orders WHERE order_date = partition_date
    return len(df)

if __name__ == "__main__":
    date = "2026-05-08"
    rows = load(transform(extract(date)), date)
    print(f"loaded {rows} rows for {date}")

Explanation of code. extract is the only function that knows where the source is; transform is pure (no I/O) and easy to unit-test; load is the only function that writes to the warehouse. Splitting the pipeline into three named functions makes the script readable, testable, and easy to swap (you can replace extract with a Postgres reader without touching transform). The dedupe + type-cast inside transform is the canonical "raw → curated" cleaning step.

Step-by-step.

step	function	what it does	result
1	`extract("2026-05-08")`	read S3 path for that day	raw DataFrame from CSV
2	`transform(df)` step a	`drop_duplicates(subset=["order_id"])`	duplicate orders removed
3	`transform(df)` step b	`pd.to_datetime(...).dt.date`	`order_date` cast to date type
4	`transform(df)` step c	`astype(float)`	`amount` cast to float
5	`load(df, date)`	warehouse `COPY` / `INSERT`	row count returned
6	`print(...)`	stdout summary	`loaded 5 rows for 2026-05-08`

Output.

loaded 5 rows for 2026-05-08

Rule of thumb: always separate extract, transform, load into three named functions — even when the pipeline is small. The shape is what reviewers look for.

Common beginner mistakes

Treating data warehouses like OLTP databases — running thousands of UPDATEs per minute (warehouses optimise for SELECT, not UPDATE).
Modelling everything in one wide table — kills performance and makes joins impossible later.
Confusing batch and streaming — batch is the default; pick streaming only when you genuinely need sub-second freshness.
Forgetting CDC — re-loading the whole customers table every night when only 100 rows changed wastes hours.
Skipping the staging step — going source → curated directly means you can't reproduce yesterday's run.

Worked Problem on Building an Idempotent Daily ETL with Quality Checks

Example input. A daily CSV orders_2026-05-08.csv that lands in S3. The warehouse has a fact_orders table partitioned by order_date. The pipeline must be idempotent — running it twice with the same input produces the same output.

order_id	order_date	amount
1	2026-05-08	100
2	2026-05-08	200
3	2026-05-08	50

Question. Write a Python script that loads the daily CSV, replaces today's partition (so a rerun does not double-count), and runs three data-quality checks (row count > 0, no NULL order_ids, no duplicate order_ids). Fail loudly with a non-zero exit code if any check fails.

Solution Using `TRUNCATE` of today's partition + three quality checks

Code solution.

import sys
import pandas as pd
import psycopg2

LOAD_DATE = "2026-05-08"

def run(conn, csv_path: str) -> int:
    df = pd.read_csv(csv_path)
    with conn.cursor() as cur:
        cur.execute("DELETE FROM fact_orders WHERE order_date = %s;", (LOAD_DATE,))
        for _, row in df.iterrows():
            cur.execute(
                "INSERT INTO fact_orders (order_id, order_date, amount) VALUES (%s, %s, %s);",
                (int(row["order_id"]), row["order_date"], float(row["amount"])),
            )
        cur.execute("SELECT COUNT(*) FROM fact_orders WHERE order_date = %s;", (LOAD_DATE,))
        if cur.fetchone()[0] == 0:
            return 1
        cur.execute("SELECT COUNT(*) FROM fact_orders WHERE order_id IS NULL;")
        if cur.fetchone()[0] > 0:
            return 1
        cur.execute("""
            SELECT COUNT(*) FROM (
              SELECT order_id, COUNT(*) c FROM fact_orders GROUP BY 1 HAVING COUNT(*) > 1
            ) d;
        """)
        if cur.fetchone()[0] > 0:
            return 1
    conn.commit()
    return 0

if __name__ == "__main__":
    conn = psycopg2.connect(dbname="warehouse")
    sys.exit(run(conn, f"orders_{LOAD_DATE}.csv"))

Explanation of code. DELETE FROM fact_orders WHERE order_date = LOAD_DATE wipes today's partition before re-inserting — that's what makes the pipeline idempotent (a rerun overwrites today's slice instead of appending). The INSERT loop loads every CSV row with explicit type casts so dates land as dates and amounts land as numbers. Three quality checks then verify the load worked — non-zero row count, no null primary keys, no duplicates. Any failure returns exit code 1 so the orchestrator (Airflow, cron) notices automatically and the developer is paged.

Output.

After a healthy run:

order_id	order_date	amount
1	2026-05-08	100
2	2026-05-08	200
3	2026-05-08	50

Exit code: 0. A second run of the same script produces an identical fact_orders (idempotent).

Step-by-step trace for a clean 3-row CSV:

step	action	result
1	`DELETE WHERE order_date = '2026-05-08'`	today's partition wiped
2	`INSERT` 3 CSV rows	3 rows in today's partition
3	row-count check	3 > 0 → pass
4	null-PK check	0 nulls → pass
5	duplicate-PK check	0 dupes → pass
6	`commit`	exit `0`

Why this works — concept by concept:

DELETE of today's partition before insert — makes the pipeline idempotent; rerun overwrites instead of appending.
explicit type casts in the INSERT — int(), float(), ISO date strings make the warehouse see clean types.
three quality checks inside the same job — checks live next to the load, not in a "we'll add monitoring later" backlog.
non-zero exit code on failure — Airflow / cron / GitHub Actions detect the failure automatically.
conn.commit() only on success — bad runs roll back; the warehouse is never left half-loaded.
Cost — O(rows in today's CSV); the historical fact_orders is only scanned for the duplicate check.

Inline CTA: for the structured ETL learning path see ETL System Design for Data Engineering Interviews and the Data Modeling course.

ETL
Topic — ETL
ETL practice problems

Practice →

COURSE
Course — Data Modeling
Data Modeling for Data Engineering Interviews

View course →

COURSE
Course — ETL System Design
ETL System Design for DE Interviews

View course →

4. Steps 6-9 — Apache Spark, Airflow, Cloud, and Data Modeling

From single-machine SQL and Pandas to production-scale pipelines

After SQL, Python, databases, and ETL fundamentals are solid, four scaling skills turn you from a script-writer into a production data engineer:

Step 6 — Apache Spark. The industry standard for large-scale processing; PySpark is its Python API. Learn DataFrames, transformations, actions, Spark SQL.
Step 7 — Workflow orchestration. Apache Airflow runs your pipelines on a schedule. Learn DAGs (directed acyclic graphs), tasks, operators, dependencies.
Step 8 — Cloud platforms. Modern data engineering lives on AWS, Azure, or GCP. Pick AWS first — it's the most asked. Learn S3, EC2, Lambda, Glue, Redshift, IAM.
Step 9 — Data modeling. OLTP vs OLAP, normalisation vs denormalisation, slowly changing dimensions (SCDs), fact-vs-dim design. Read the Kimball Data Warehouse Toolkit.

Pro tip: these are scale-and-production skills. Don't open them until SQL and Python are second nature. The most common fresher failure mode is "I learned Spark but I can't write a LEFT JOIN correctly under pressure." Master the foundations first.

Apache Spark + PySpark — the big-data engine

Apache Spark processes data that doesn't fit on a single machine by splitting work across a cluster. PySpark is its Python API — almost everything you do in Pandas has a PySpark equivalent, just distributed. The simplest entry point is SparkSession.builder.getOrCreate() followed by spark.read.csv(...) to load data.

SparkSession — the entry point; creates the cluster connection.
DataFrame — the main abstraction; like a Pandas DataFrame but distributed.
Transformations — select, filter, groupBy — lazy, build a plan.
Actions — show, count, write — trigger actual execution.
Spark SQL — register a DataFrame as a table and run SQL against it.

Example input. A sales.csv file similar to the Pandas example, but big enough that we want Spark to process it on a cluster.

order_id	region	amount
1	North	100
2	South	200
3	North	150

Question. Write a minimal PySpark script that reads sales.csv and shows the first few rows. Show the canonical SparkSession setup that every PySpark script begins with.

Code solution.

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("demo").getOrCreate()

df = spark.read.csv("sales.csv", header=True, inferSchema=True)
df.show()

Explanation of code. SparkSession.builder.appName("demo").getOrCreate() either creates a new Spark session or attaches to an existing one — either way, you end up with a spark object that knows how to talk to the cluster. spark.read.csv("sales.csv", header=True, inferSchema=True) loads the file as a DataFrame, treating the first row as headers and inferring column types. df.show() is an action that triggers execution and prints the first 20 rows to stdout.

Step-by-step.

step	call	kind	result
1	`SparkSession.builder.appName(...).getOrCreate()`	setup	`spark` session attached to a (local or cluster) executor
2	`spark.read.csv(..., header=True, inferSchema=True)`	transformation (lazy)	DataFrame plan registered — no rows scanned yet
3	`df.show()`	action	plan executes: read CSV → infer types → render first 20 rows
4	stdout	—	grid-formatted table printed

Output.

+--------+------+------+
|order_id|region|amount|
+--------+------+------+
|       1| North|   100|
|       2| South|   200|
|       3| North|   150|
+--------+------+------+

Rule of thumb: in PySpark, transformations (filter, select, groupBy) are lazy — nothing runs until you call an action like .show(), .count(), or .write(). That's why the same PySpark code can be reused for 1 GB and 1 TB datasets.

Apache Airflow — DAGs, tasks, scheduling

Airflow is the workflow orchestrator most data teams use. You write a DAG (directed acyclic graph) of tasks; Airflow runs them on a schedule, respects dependencies, retries failures, and surfaces alerts. The minimum viable DAG is two tasks chained with >>.

DAG — the workflow; a Python file in dags/ directory.
Task — a single unit of work (run a SQL query, call an API, run a PySpark job).
Operator — a reusable task type (BashOperator, PythonOperator, SQLExecuteQueryOperator).
Dependencies — task1 >> task2 means "run task1 then task2."
Schedule — schedule_interval='@daily', '0 3 * * *' (cron), or None for manual.

Example input. A simple two-stage daily ETL — extract data from an API, load it into a warehouse table.

task1: extract — call the API, write raw JSON to S3
task2: load — read S3 JSON, INSERT into fact_events
schedule: daily at 03:00 UTC

Question. Write a minimal Airflow DAG that defines two PythonOperator tasks extract_task and load_task, and chains them so load_task only runs after extract_task succeeds.

Code solution.

from datetime import datetime
from airflow import DAG
from airflow.operators.python import PythonOperator

def extract():
    pass  # call the API, write raw JSON to S3

def load():
    pass  # read S3 JSON, INSERT into fact_events

with DAG(
    dag_id="etl_pipeline",
    start_date=datetime(2026, 5, 1),
    schedule_interval="@daily",
    catchup=False,
) as dag:
    extract_task = PythonOperator(task_id="extract", python_callable=extract)
    load_task    = PythonOperator(task_id="load",    python_callable=load)
    extract_task >> load_task

Explanation of code. The with DAG(...) as dag: block defines the DAG metadata — its name, when it starts, how often it runs (@daily is shorthand for "every day at midnight"), and whether to backfill missed runs (catchup=False means "no, just run from now on"). Two PythonOperator tasks wrap the actual Python functions. extract_task >> load_task declares the dependency — Airflow will only run load_task if extract_task succeeds.

Step-by-step.

step	when	action	result
1	parse time	`DAG(...)` instantiates	DAG `etl_pipeline` registered in Airflow metadata
2	parse time	`PythonOperator(...)` ×2	two tasks attached to the DAG
3	parse time	`extract_task >> load_task`	dependency edge added (extract → load)
4	every day at midnight	scheduler triggers a DAG run	`extract` task starts
5	extract succeeds	scheduler sees green upstream	`load` task starts
6	load succeeds	DAG run marked success	green tick in calendar grid
6′	extract fails	downstream skipped	red tick; alert fires

Output.

In the Airflow UI, this DAG appears as two boxes connected by an arrow:

[extract] → [load]

Each daily run produces a tick in the calendar grid; failures are red, successes are green.

Rule of thumb: one DAG = one logical workflow. If you find yourself writing 50 tasks in a single DAG, you probably want 5 DAGs of 10 tasks each — easier to debug, easier to retry.

Cloud platforms — AWS first, then expand

Modern data engineering is cloud-based. Pick one platform first and learn its data services before branching out — most teams use AWS, so it's the highest-leverage starting point. Azure and GCP are equally valid second choices once you have one cloud under your belt.

S3 — object storage; where raw data lands.
EC2 — virtual machines; rarely touched directly anymore.
Lambda — serverless functions; great for small ETL triggers.
Glue — managed ETL service; runs Spark jobs without you managing the cluster.
Redshift — AWS data warehouse; SQL-compatible.
IAM — identity and access; non-optional — every cloud bug eventually traces back to permissions.

Example input. You have a daily CSV that lands in S3 at s3://my-bucket/orders/{date}/orders.csv and a Redshift table fact_orders to load it into.

Question. Write the AWS CLI / SQL pseudocode that copies the CSV from S3 into Redshift on a schedule. (Don't worry about IAM details; the goal is the shape.)

Code solution.

-- Inside Redshift, run on a schedule from Airflow / cron
COPY fact_orders
FROM 's3://my-bucket/orders/2026-05-08/orders.csv'
IAM_ROLE 'arn:aws:iam::ACCOUNT:role/RedshiftS3ReadRole'
CSV
IGNOREHEADER 1;

Explanation of code. COPY ... FROM 's3://...' is the Redshift-specific bulk-load command — it pulls a file directly from S3 into a table without needing an intermediate machine. IAM_ROLE references an AWS IAM role that grants Redshift permission to read that S3 bucket — without this, the copy fails with a permission error. CSV tells Redshift the file format; IGNOREHEADER 1 skips the column-header row.

Step-by-step.

step	actor	action	result
1	Airflow / cron	submits `COPY` to Redshift	command queued
2	Redshift leader	assumes `RedshiftS3ReadRole`	temporary AWS credentials obtained
3	Redshift compute nodes	parallel-fetch the S3 object	bytes streamed direct to slices
4	parser	apply `CSV, IGNOREHEADER 1`	header row skipped; data rows parsed
5	loader	bulk-insert into `fact_orders`	rows committed
6	system catalogue	log to `STL_LOAD_COMMITS`	row count + reject count recorded

Output.

The CSV's rows land in fact_orders. A small status row is logged in STL_LOAD_COMMITS showing how many rows were copied and whether any were rejected.

Rule of thumb: for AWS, S3 + IAM are the two services you actually need to be fluent in. Everything else (Lambda, Glue, Redshift) layers on top.

Common beginner mistakes

Learning Spark before SQL is solid — Spark is just bigger SQL with more failure modes.
Writing 1,000-line Airflow DAGs — split into smaller DAGs that each do one thing.
Storing AWS credentials in code — always use IAM roles or environment variables, never hardcode.
Ignoring data modeling because it "feels theoretical" — interviewers test it heavily.
Trying to learn all three clouds at once — pick AWS first; the others are easier once you know one.

Worked Problem on Designing a Slowly Changing Dimension (Type 2) for Customer Addresses

Example input. Customer C1 lives at "12 Old St" until 2026-03-15, then moves to "88 New Ave". The fact tables need to know which address was current at the time of each historical sale.

customer_id	address	valid_from	valid_to	is_current
C1	12 Old St	2025-01-01	2026-03-14	FALSE
C1	88 New Ave	2026-03-15	(NULL)	TRUE

Question. Write the SQL to handle the address change as an SCD Type 2 update — close the old row by setting valid_to and is_current = FALSE, then insert a new row with the new address and is_current = TRUE. This pattern preserves historical correctness without losing the past.

Solution Using `UPDATE` to close the old row + `INSERT` for the new one

Code solution.

-- Step 1: close the existing current row
UPDATE dim_customer
SET valid_to = DATE '2026-03-14',
    is_current = FALSE
WHERE customer_id = 'C1' AND is_current = TRUE;

-- Step 2: insert the new current row
INSERT INTO dim_customer (customer_id, address, valid_from, valid_to, is_current)
VALUES ('C1', '88 New Ave', DATE '2026-03-15', NULL, TRUE);

Explanation of code. SCD Type 2 keeps full history by adding new rows rather than overwriting old ones. The UPDATE finds the row where is_current = TRUE for the customer and closes it — sets valid_to to the day before the change and is_current to FALSE. The INSERT then adds the new row with valid_from set to the change date, valid_to left NULL (still current), and is_current = TRUE. Historical fact tables can join to this dim with a date predicate to find the address that was current at the time of each sale.

Output.

After the two statements:

customer_id	address	valid_from	valid_to	is_current
C1	12 Old St	2025-01-01	2026-03-14	FALSE
C1	88 New Ave	2026-03-15	(NULL)	TRUE

A query like WHERE is_current = TRUE returns only the current address. A historical join uses WHERE sale_date BETWEEN valid_from AND COALESCE(valid_to, DATE '9999-12-31') to pick the right address per sale.

Step-by-step trace for the change on 2026-03-15:

step	action	result
1	`UPDATE` closes row 1	`valid_to = 2026-03-14`, `is_current = FALSE`
2	`INSERT` adds row 2	new row with `valid_from = 2026-03-15`, `is_current = TRUE`
3	dimension now has 2 rows for C1	one historical, one current

Why this works — concept by concept:

SCD Type 2 keeps history — old rows are not overwritten; both versions of the customer's address coexist with date ranges.
valid_from / valid_to define the row's lifetime — the date range during which this row was the truth.
is_current = TRUE flag — shortcut for dashboards that always want the latest; saves an ORDER BY ... LIMIT 1 lookup.
historical joins use BETWEEN — pick the dim row whose date range contains the fact row's date.
COALESCE(valid_to, '9999-12-31') — handles the open-ended current row whose valid_to is NULL.
Cost — two row-level operations; constant time per dimension change.

Inline CTA: for the deeper modeling syllabus see Data Modeling for Data Engineering Interviews; when you do start Spark, the gentle entry point is PySpark Fundamentals.

COURSE
Course — PySpark
PySpark Fundamentals

View course →

COURSE
Course — Spark internals
Apache Spark Internals

View course →

SQL
Topic — slowly changing data
SCD practice problems

Practice →

5. Steps 10-13 — Streaming, Portfolio Projects, Git, and Interview Prep

From skills to a job offer — proving the work and clearing the loop

The last four steps turn your skills into a job offer. Streaming systems handle real-time data, portfolio projects prove you can ship, Git makes your code visible, and interview prep closes the deal.

Step 10 — Streaming systems. Kafka, event-driven architectures, message queues, real-time processing. Required for advanced roles; optional for first jobs.
Step 11 — Build five portfolio projects. SQL analytics, Python ETL, Airflow pipeline, PySpark large-data, cloud deployment. Put all on GitHub.
Step 12 — Master Git. clone, add, commit, push, branch, merge — every company uses Git from day one.
Step 13 — Interview prep. SQL questions (joins, windows, aggregations, ranking), Python questions (dicts, strings, lists, hashmaps), system-design basics (ETL architecture, lake vs warehouse, batch vs streaming, scalability).

Pro tip: projects beat certificates. A GitHub repo with a clean README and a runnable pipeline outperforms a stack of certifications. Your top-of-funnel signal to recruiters is "here's the URL to my orders-batch-etl project" — not your transcript.

Streaming systems — Kafka in plain English

Kafka is a distributed message queue that lets producers publish events to a "topic" and consumers read them in order. Event-driven architectures use Kafka as the spine — payment events flow in, multiple downstream consumers (fraud detection, analytics, notifications) read the same stream independently. Required for advanced / senior DE roles; optional for fresher first jobs.

Producer — writes events to a Kafka topic.
Topic — a named append-only log; events stay in order.
Consumer — reads events from a topic; multiple consumers per topic are fine.
Partition — topics are split into partitions for parallelism.
Use case — live payment events flowing into a fraud-detection model.

Example input. A payment event payload that a producer wants to publish to the payments topic.

event = {
    "payment_id": "PAY-1001",
    "amount": 250.00,
    "currency": "USD",
    "user_id": "U42",
    "ts": "2026-05-08T10:15:00Z",
}

Question. Sketch the producer-side Python code that publishes this event to a Kafka topic called payments. Use kafka-python (the most popular client). Include just the producer setup + send call.

Code solution.

import json
from kafka import KafkaProducer

producer = KafkaProducer(
    bootstrap_servers=["localhost:9092"],
    value_serializer=lambda v: json.dumps(v).encode("utf-8"),
)

event = {
    "payment_id": "PAY-1001",
    "amount": 250.00,
    "currency": "USD",
    "user_id": "U42",
    "ts": "2026-05-08T10:15:00Z",
}

producer.send("payments", event)
producer.flush()

Explanation of code. KafkaProducer(bootstrap_servers=[...]) connects to one or more Kafka brokers. The value_serializer lambda turns the Python dict into JSON bytes (Kafka stores raw bytes, not Python objects). producer.send("payments", event) queues the event for delivery to the payments topic; producer.flush() blocks until the queued messages are actually sent. Downstream consumers (fraud detection, analytics) can read this event independently and in order.

Step-by-step.

step	action	result
1	`KafkaProducer(bootstrap_servers=[...])`	TCP connection to broker established; metadata fetched
2	`value_serializer = json.dumps(...).encode("utf-8")`	every send will convert dict → JSON bytes
3	`producer.send("payments", event)`	record buffered in the producer's in-memory queue
4	broker picks a partition (hashed key or round-robin)	record assigned to a partition log
5	`producer.flush()`	blocks until all buffered records are acknowledged
6	subscribed consumers	next `poll()` returns the new event

Output.

The event is appended to the payments topic. Any consumer subscribed to payments will receive it on its next poll() call:

{"payment_id": "PAY-1001", "amount": 250.0, "currency": "USD", "user_id": "U42", "ts": "2026-05-08T10:15:00Z"}

Rule of thumb: fresher first jobs rarely need Kafka. Master batch (Step 5) before opening Step 10. Mention Kafka in interviews only if you've actually shipped a project that uses it.

Five portfolio projects — what to build, in order

Projects matter more than certificates. Build all five and put them on GitHub with a clean README.md for each. The five build on each other — by the end you have a production-grade portfolio.

Project 1 — SQL analytics. E-commerce sales dashboard built entirely in SQL.
Project 2 — Python ETL. Extract API data → clean → store in PostgreSQL.
Project 3 — Airflow pipeline. Schedule the Python ETL as a daily DAG.
Project 4 — PySpark large-data pipeline. Process millions of rows.
Project 5 — Cloud project. Deploy the ETL pipeline on AWS.

Example input. Project 1 — an e-commerce dataset (orders, customers, products) for which you'll write the SQL behind a sales dashboard.

orders (sample):

order_id	customer_id	product_id	order_date	amount
1	C1	P1	2026-04-01	100
2	C2	P2	2026-04-15	200
3	C1	P1	2026-05-01	150

Question. For Project 1, write the canonical "monthly revenue per product" SQL. This is the single query the entire dashboard hangs off — get this right and the rest of the dashboard is just WHERE and ORDER BY variations.

Code solution.

SELECT
    p.product_name,
    DATE_TRUNC('month', o.order_date) AS month,
    SUM(o.amount) AS revenue
FROM orders o
JOIN products p ON p.product_id = o.product_id
WHERE o.order_date >= DATE '2026-01-01'
GROUP BY p.product_name, DATE_TRUNC('month', o.order_date)
ORDER BY month, p.product_name;

Explanation of code. DATE_TRUNC('month', o.order_date) collapses every order date to the first day of its month, so all April orders aggregate together. The JOIN brings in product_name from products so the dashboard can label rows. GROUP BY collapses to one row per (product, month). ORDER BY produces a chronologically-readable result. Wrap this in a saved view or a dbt model and the dashboard renders automatically.

Step-by-step.

step	clause	result
1	`FROM orders o`	scan all order rows
2	`JOIN products p ON p.product_id = o.product_id`	each order picks up its `product_name`
3	`WHERE o.order_date >= DATE '2026-01-01'`	drop pre-2026 rows
4	`DATE_TRUNC('month', o.order_date)`	every date snapped to month-start (e.g. 2026-04-15 → 2026-04-01)
5	`GROUP BY product_name, month`	bucket by (product, month)
6	`SUM(o.amount)` per bucket	revenue total per group
7	`ORDER BY month, product_name`	chronological, then alphabetical

Output.

product_name	month	revenue
Book	2026-04-01	100
Headphones	2026-04-15 (truncated to 2026-04-01)	200
Book	2026-05-01	150

(Real output would have one row per (product, month) combination; the sample is too small for a strong rollup.)

Rule of thumb: every Project 1 SQL should be runnable on a free PostgreSQL sandbox with a 100-row sample dataset. Put both the SQL and the sample data in your GitHub repo so a recruiter can clone and run it in 60 seconds.

Git, GitHub, and the resume bullet

Git is non-optional infrastructure. Every team's workflow assumes you can clone a repo, branch off, commit, and push. The bare minimum command set fits on a single screen.

git clone <url> — copy a remote repo locally.
git checkout -b feature/x — create + switch to a new branch.
git add file — stage a change.
git commit -m "..." — record the staged changes.
git push origin <branch> — push the branch to GitHub; open a pull request.
git merge / git rebase — combine branches.

Example input. You've finished Project 1 (the SQL analytics dashboard) on your laptop and want to push it to GitHub under your account.

Question. Show the canonical six-command workflow: clone an empty template repo, branch off, add the files you've written, commit with a descriptive message, push the branch, and open a pull request.

Code solution.

git clone https://github.com/<you>/sql-sales-dashboard.git
cd sql-sales-dashboard
git checkout -b feature/initial-dashboard
# (write README.md, schema.sql, queries.sql, sample-data/)
git add README.md schema.sql queries.sql sample-data/
git commit -m "Add Project 1: SQL sales dashboard with sample data"
git push origin feature/initial-dashboard
# open a pull request on github.com

Explanation of code. clone brings the empty repo to your laptop. checkout -b creates a feature branch (never push to main directly on a team repo, even your own — make the habit). After writing the project files, add stages them, commit records the change with a one-line message that future-you can scan, and push sends the branch to GitHub. The pull request is the artifact a recruiter or interviewer will actually look at.

Step-by-step.

step	command	result
1	`git clone <url>`	empty repo copied to laptop
2	`cd sql-sales-dashboard`	move into the working tree
3	`git checkout -b feature/initial-dashboard`	new branch created and checked out
4	write `README.md`, `schema.sql`, `queries.sql`, `sample-data/`	working tree now has 4 untracked items
5	`git add ...`	files staged for commit
6	`git commit -m "..."`	snapshot recorded with descriptive message
7	`git push origin feature/initial-dashboard`	branch published to GitHub
8	open PR on github.com	reviewable artifact link a recruiter can click

Output.

A GitHub repo URL with a feature branch and a pull request — both visible to anyone you share the link with. The README renders directly on the repo home page, becoming your portfolio artifact.

Rule of thumb: if you can't clone, branch, commit, and push within 60 seconds without looking commands up, Git is still on your to-do list. Practice it daily until it's muscle memory.

Common beginner mistakes

Trying to learn Kafka before mastering batch ETL — Kafka adds complexity without removing any.
Building one giant project instead of five small ones — recruiters skim; five clear repos beat one tangled one.
Pushing to main directly — every commit becomes part of history with no review trail.
No README.md per project — repos without READMEs are invisible.
Skipping interview prep — solid skills + zero practice = solid skills wasted at the screen.

Worked Problem on Picking Project 2 and Writing the Resume Bullet

Example input. You've shipped Project 1 (SQL dashboard). Project 2 is the Python ETL — extract from an API, clean, store in PostgreSQL. The repo will be python-api-etl. The recruiter call is in two weeks.

Question. Sketch the four-file layout for the Project 2 repo plus the one-line resume bullet you'll lead with on the recruiter call. The goal: a stranger should be able to read the repo, run it locally, and understand the work in 5 minutes.

Solution Using a four-file repo layout + a metric-led resume bullet

Code solution.

python-api-etl/
├── README.md           # 60-second pitch + how to run
├── etl.py              # extract / transform / load functions
├── tests/
│   └── test_etl.py     # one test per function
└── requirements.txt    # pinned dependencies

Resume bullet (lead with the metric):

Built a Python ETL pipeline that ingests 10K daily API records into a PostgreSQL warehouse with row-level validation and CI-friendly exit codes. (github.com/<you>/python-api-etl)

Explanation of code. Four files is the floor — one for documentation, one for code, one for tests, one for dependencies. The README is what a recruiter sees first; lead with the what and how to run, then explain the why. The resume bullet leads with a quantitative metric (10K daily records) and ends with the GitHub URL — recruiters scan for both, in that order.

Output.

Your GitHub now has a runnable, documented ETL repo. The recruiter receives the link, clicks through, sees the README, and forwards your resume to the hiring manager. The bullet on the resume becomes the first sentence of the recruiter's pitch to the hiring manager.

Step-by-step trace of how a recruiter reads it:

step	recruiter action	what they see
1	clicks the GitHub link in the resume	repo home page with the README rendered
2	scans the first paragraph of README	"Daily API → PostgreSQL with validation"
3	scrolls to "How to run"	three commands (`git clone`, `pip install`, `python etl.py`)
4	clicks `etl.py`	sees three named functions; reads in 30 seconds
5	clicks `tests/`	tests exist; quality signal confirmed

Why this works — concept by concept:

one repo per project — recruiters skim; five clean repos beat one tangled monorepo.
README-first design — the home page is the pitch; lead with what + how to run.
tests in tests/ — even one test per function is a quality signal.
pinned requirements.txt — anyone can clone and run; no "works on my machine" surprises.
metric-led resume bullet — 10K daily records is concrete; "ETL pipeline" alone is generic.
Cost — about a weekend of focused work for the project; 30 minutes for the bullet.

Inline CTA: for fresher interview reps see SQL practice page, Python practice page, and the canonical course path SQL for Data Engineering Interviews — From Zero to FAANG.

SQL
Hub — all practice
Browse all data-engineering practice

Practice →

COURSE
Course — SQL for DE
SQL for Data Engineering Interviews

View course →

COURSE
Hub — all courses
Browse all DE courses

View courses →

Tips to master the data engineering roadmap (best learning order + timeline)

Follow the order — and the calendar

The 13 steps above have a best learning order that works for most freshers — skip ahead at your own risk. The order plus a realistic timeline:

Order: SQL → Python → Databases → Pandas → ETL concepts → Data Warehousing → PySpark → Airflow → Cloud (AWS) → Kafka → Projects → Interview prep.
2-3 months — SQL + Python basics solid.
4-6 months — intermediate DE (warehousing, ETL, modeling).
6-9 months — job-ready (Airflow, cloud, projects shipped).
9-12 months — strong fresher profile (Spark, streaming basics, polished portfolio).

Most freshers fail for the same four reasons — avoid them

The failure modes are predictable. Watch for these in your own routine:

Jumping to Spark too early. Spark is just bigger SQL with more failure modes; without solid SQL it's noise.
Ignoring SQL depth. Beyond SELECT and JOIN, the bar at the screen is window functions + grain reasoning. Drill them.
Avoiding projects. Tutorials and certifications are signals; shipped code on GitHub is proof.
Watching tutorials without practice. Watch the video → close it → rebuild the example without it. If you can't, you didn't learn it.

The winning formula

Every successful fresher career follows the same five-step loop: learn → practice → build → publish → interview. Pick a topic, drill it on a coding-environment, build a small artifact, push to GitHub, then interview for jobs that touch that topic. Repeat for each step in the roadmap.

Books worth buying

Designing Data-Intensive Applications (Martin Kleppmann) — the modern systems book; read once a quarter.
The Kimball Data Warehouse Toolkit (Ralph Kimball) — the canonical modeling reference.

Where to practice on PipeCode

Start with the SQL practice page and the Python practice page; the structured paths are SQL for Data Engineering Interviews — From Zero to FAANG and Python for Data Engineering Interviews — Complete Fundamentals. After SQL and Python land, drill ETL practice, window functions, joins, and the deeper Data Modeling course and ETL System Design course. Pivot to peer guides — the Airbnb DE interview guide, the top DE interview questions 2026, and the SQL data types Postgres guide.

Frequently Asked Questions

How long does it really take to become a data engineer in 2026?

If consistent, 6-9 months at 10-15 hours per week is enough to be job-ready for junior / fresher data-engineering roles; 9-12 months produces a strong fresher profile with Spark, streaming basics, and a polished portfolio. The 2-3 months mark is where SQL and Python basics click; 4-6 months gets you through warehousing, ETL, and modeling. The single biggest predictor of speed is consistency — 10 hours a week for 6 months beats 40 hours a week for 6 weeks.

Do I need to learn all 13 steps before applying for jobs?

No — start applying as soon as Steps 1-5 are solid (SQL, Python, databases, warehousing, ETL/ELT). Roles you can target with the first five steps done: junior data engineer, junior analytics engineer, data engineer intern, ETL developer trainee. Steps 6-9 (Spark, Airflow, Cloud, Modeling) turn "hireable" into "competitive." Steps 10-13 (Streaming, Projects, Git, Interview prep) close the deal. Apply earlier than you think you should — interviewing is itself a skill that needs reps.

Should I master one cloud or learn all three (AWS, Azure, GCP)?

Pick one first and master its core data services before touching the others. AWS is the most asked at fresher interviews and the most widely deployed in industry — start there. The core AWS services for fresher DE work: S3 (object storage), IAM (access control), Lambda (serverless functions), Glue (managed ETL), Redshift (warehouse). Once you have one cloud under your belt, the other two are easy because the concepts (object storage, IAM, serverless, managed ETL, warehouse) are the same — only the names change.

Is Apache Spark required for fresher data-engineering jobs?

For most fresher first jobs, no — but knowing what Spark is and when it appears is required. The honest fresher posture: "I've shipped batch ETL with Python and SQL; I know Spark is the next step when data outgrows a single machine; I've done the PySpark Fundamentals tutorial and would learn the rest on the job." That's enough for 80% of fresher screens. Roles at Spark-heavy shops (Databricks customers, ad-tech, large e-commerce) will test deeper — for those, ship a PySpark project as part of your Step 11 portfolio.

What does a data engineer actually do day-to-day?

Day-to-day, a data engineer writes SQL queries, builds and maintains batch pipelines, models new tables, fixes data quality issues, and reviews other engineers' pipelines. A typical week: Monday — investigate a Slack message about a wrong dashboard number (usually a grain or null-handling bug); Tuesday-Wednesday — model a new dimension table for a product launch; Thursday — code review on a teammate's Airflow DAG; Friday — add a quality check that would have caught Monday's bug. Spark, Kafka, and lakehouse architecture appear at scale-heavy companies; the day-to-day at most companies is SQL + modeling + pipelines.

What's the difference between a data engineer, data analyst, and data scientist?

Data engineers build the pipelines and tables; analysts query them for business questions; scientists run experiments and ML models on top. In a typical e-commerce team: a DE owns the daily ETL that loads cur_orders; an analyst writes the SQL behind the daily revenue dashboard; a scientist runs the A/B test that decides whether the new checkout flow ships. The roles overlap on SQL — every analytics person writes it — but only DEs own the infrastructure that produces the tables everyone else queries. Salaries also follow this stack — DEs are typically paid more than analysts and on par with scientists at most companies.

Start practicing data engineering interview problems

SQL Interview Questions for Data Engineering

Gowtham Potureddi — Sun, 10 May 2026 15:45:00 +0000

SQL interview questions for data engineering circle around the same four primitives in every loop, regardless of the company name on the JD: JOIN semantics with LEFT JOIN ... IS NULL for anti-joins, GROUP BY with HAVING for aggregate filters and duplicate detection, window functions like ROW_NUMBER, RANK, DENSE_RANK, LAG, and LEAD for ranking and lookback, and CTEs (including recursive CTEs) plus correlated subqueries for multi-step logic and top-N-per-group queries. Whether the prompt is "find customers who never placed an order", "count duplicate emails", "second-highest salary", or "top 3 salaries per department", the same handful of mental models keeps showing up — interviewers grade fluency with these primitives over memorized syntax.

This guide walks four topic clusters end-to-end, each with a detailed topic explanation, per-sub-topic explanation with a worked example and its solution, common beginner mistakes, and an interview-style problem with a full solution that traces the query step by step. Every example uses PostgreSQL-flavored syntax — the dialect that drives DataLemur, the live-coding environments at most product-analytics companies, and the bulk of public SQL interview corpora — and every solution ends with a concept-by-concept breakdown that explains why the query is correct, what the cost is, and where beginners typically slip.

1. SQL Joins for Data Engineering — `INNER`, `LEFT`, and Anti-Joins

`INNER JOIN`, `LEFT JOIN`, and the `LEFT JOIN ... IS NULL` anti-join in SQL for data engineering

"Find customers who never placed an order" is the signature SQL join interview question — and the cleanest answer is not a NOT IN subquery or a NOT EXISTS correlated query but a LEFT JOIN with WHERE right.key IS NULL. The mental model: INNER JOIN keeps only matching pairs, LEFT JOIN keeps every left row and pads the right side with NULLs when there is no match, and filtering for right.id IS NULL after the LEFT JOIN isolates exactly the left rows that had no match — the anti-join. The same primitive surfaces churned-user, never-bought, never-clicked, and never-converted queries across product-analytics interviews.

Pro tip: LEFT JOIN ... WHERE right.id IS NULL is generally as fast as or faster than NOT IN (subquery) because NOT IN returns NULL (not FALSE) for any NULL in the subquery and silently drops every outer row. State this gotcha out loud — interviewers grade the candidate who knows why NOT IN can return zero rows when the data has a single NULL.

`INNER JOIN`: keep only matching rows

The INNER JOIN invariant: a row from the left table is paired with a row from the right table iff the join predicate evaluates to TRUE; rows on either side that have no match are discarded. The cardinality of the result is at most |left| × |right| but typically far less — bounded by the multiplicity of matching keys.

Predicate — ON l.key = r.key is the common shape; multi-column predicates are fine.
No padding — unmatched rows on either side are silently dropped.
Result schema — every column from both tables (use aliases to disambiguate).
Cardinality risk — 1:N on the right inflates left rows; N:M is a Cartesian-by-key explosion.

Worked example. Two tables, customers(id, name) with rows (1, Alice), (2, Bob), (3, Carol) and orders(order_id, customer_id, amount) with rows (101, 1, 50), (102, 1, 30), (103, 2, 80).

customer	order_id	amount
Alice	101	50
Alice	102	30
Bob	103	80

Carol has no row — INNER JOIN drops her.

Worked-example solution.

SELECT c.name AS customer, o.order_id, o.amount
FROM customers c
INNER JOIN orders o
  ON o.customer_id = c.id;

Rule of thumb: reach for INNER JOIN when the question is "rows where both sides exist"; it is the smallest, fastest, most common join.

`LEFT JOIN`: keep every left row, pad the right with `NULL`

The LEFT JOIN invariant: every row from the left table appears in the output; if the join predicate matches at least one right row, the right columns are filled in; otherwise the right columns are NULL. The result has at least |left| rows and at most |left| × max_right_match rows.

All left rows preserved — even unmatched ones.
Right columns are NULL when there is no match — the key signal for the anti-join trick.
RIGHT JOIN — the mirror image; rarely needed, since you can flip the table order.
FULL OUTER JOIN — keeps unmatched rows from both sides; less common in interviews.

Worked example. Same customers and orders. Carol stays in the output.

customer	order_id	amount
Alice	101	50
Alice	102	30
Bob	103	80
Carol	NULL	NULL

Worked-example solution.

SELECT c.name AS customer, o.order_id, o.amount
FROM customers c
LEFT JOIN orders o
  ON o.customer_id = c.id;

Rule of thumb: LEFT JOIN is the right answer whenever the question asks for "every X, with Y when it exists" — a churn report, a coverage report, a left-padded join for downstream pipelines.

`LEFT JOIN ... IS NULL` anti-join: rows with no match

The anti-join invariant: a LEFT JOIN followed by WHERE right.key IS NULL keeps exactly the left rows for which no right row matched. Equivalent in result to NOT EXISTS and (under NOT NULL constraints) to NOT IN, but typically faster than the latter and immune to the NULL-swallowing bug.

LEFT JOIN — preserves every left row.
WHERE right.<pk> IS NULL — strips out every left row that did match, leaving only the unmatched ones.
NOT EXISTS equivalent — WHERE NOT EXISTS (SELECT 1 FROM right WHERE right.fk = left.pk).
NOT IN pitfall — returns zero rows if the subquery contains a single NULL; avoid in production data.

Worked example. Same customers and orders; Carol has no order.

customer	matched_order_id	passes filter?
Alice	101	✗ (matched)
Bob	103	✗ (matched)
Carol	NULL	✓ (anti-match)

Worked-example solution.

SELECT c.name AS customer
FROM customers c
LEFT JOIN orders o
  ON o.customer_id = c.id
WHERE o.order_id IS NULL;

Rule of thumb: "find X with no Y" → LEFT JOIN ... WHERE Y.id IS NULL. Memorize this pattern; it is the single most-asked SQL join shape in data-engineering interviews.

Common beginner mistakes

Filtering the right table inside WHERE after a LEFT JOIN (e.g., WHERE o.amount > 0) — silently turns the LEFT JOIN back into an INNER JOIN because NULL > 0 is NULL, which fails the filter.
Using NOT IN (subquery) when the subquery can return NULL — drops every outer row.
Forgetting to alias both sides of the join — id is ambiguous when both tables have it.
Joining on the wrong column (o.id = c.id instead of o.customer_id = c.id) — produces a Cartesian-flavored mess.
Using LEFT JOIN when INNER JOIN is correct — leaves spurious NULL rows in the output.

SQL Interview Question on Customers With No Orders

Given customers(id, name) and orders(order_id, customer_id, amount), return the names of customers who have never placed an order.

Solution Using `LEFT JOIN ... WHERE orders.order_id IS NULL`

SELECT c.name
FROM customers c
LEFT JOIN orders o
  ON o.customer_id = c.id
WHERE o.order_id IS NULL;

Why this works: the LEFT JOIN preserves every customer row regardless of whether a matching order exists; for matched customers, o.order_id carries a real value; for unmatched customers, the right-side columns are NULL and o.order_id IS NULL is TRUE; filtering on that predicate keeps only the unmatched customers — the anti-join. Single pass over customers; one keyed lookup into orders per customer; no subquery materialization.

Step-by-step trace for the sample input:

customers.id	customers.name	LEFT JOIN orders.order_id	IS NULL?	survives?
1	Alice	101	✗	✗
1	Alice	102	✗	✗
2	Bob	103	✗	✗
3	Carol	NULL	✓	✓

Only Carol's row survives the WHERE filter.

Output:

name
Carol

Why this works — concept by concept:

LEFT JOIN semantics — keeps every left row; right side is NULL when there is no match. This NULL is the entire signal we filter on.
WHERE o.order_id IS NULL — o.order_id is the right-side primary key; it is NULL only when the join produced a synthetic unmatched row. A real-NULL order-id from the source table never happens because primary keys are NOT NULL.
Anti-join semantics — equivalent to NOT EXISTS (SELECT 1 FROM orders WHERE customer_id = c.id); the LEFT JOIN ... IS NULL form is typically faster on planners that materialize a hash join.
No NULL-swallowing — unlike NOT IN, the predicate is IS NULL, which is well-defined for NULL values. There is no silent zero-row failure.
O(|customers| + |orders|) time — hash-join build on orders.customer_id, single probe per customer. Index on orders.customer_id makes this near-linear.

Inline CTA: Drill the SQL joins practice page for INNER, LEFT, and anti-join shapes, and the SQL filtering practice page for the WHERE vs HAVING distinction.

SQL
Topic — joins
SQL join problems

Practice →

SQL
Topic — filtering
SQL filtering problems

Practice →

SQL
Language — SQL
All SQL practice problems

Practice →

2. SQL Aggregations and `GROUP BY` for Data Engineering

`GROUP BY`, `HAVING`, and aggregate functions in SQL for data engineering

"Find duplicate emails in the users table" and "find the department with the highest average salary" are the two signature aggregation prompts — and both reduce to GROUP BY + an aggregate function + HAVING filter. The mental model: GROUP BY col collapses rows that share the same col value into a single output row; COUNT(*), SUM(...), AVG(...), MIN(...), MAX(...) summarize each bucket; WHERE filters individual rows before grouping; HAVING filters whole groups after grouping. Aggregate predicates belong in HAVING; row predicates belong in WHERE. Mixing the two up is the most common parse-error in live coding rounds.

Pro tip: When the question is "find duplicates", the canonical shape is SELECT key, COUNT(*) FROM t GROUP BY key HAVING COUNT(*) > 1. State the HAVING COUNT(*) > 1 half out loud while writing — it signals that you understand WHERE cannot reference aggregates and that you can compose row-level and group-level filters in the right order.

`COUNT`, `SUM`, `AVG`, `MIN`, `MAX` — `NULL`-aware aggregates

The aggregate-NULL invariant: COUNT(*) counts every row including ones with NULL columns; COUNT(col) counts only rows where col is not NULL; SUM, AVG, MIN, MAX skip NULL values entirely; if every value in a group is NULL, the result is NULL (not 0). Beginners conflate COUNT(*) and COUNT(col) and silently report wrong totals.

COUNT(*) — every row in the bucket, regardless of NULLs.
COUNT(col) — non-NULL values of col only.
COUNT(DISTINCT col) — unique non-NULL values; essential after a JOIN that may have inflated rows.
SUM / AVG — numeric only; AVG is sum-of-non-null-divided-by-count-of-non-null, so NULL does not count as 0.

Worked example. Three rows in one group: amount = 10, NULL, 30.

aggregate	result
`COUNT(*)`	3
`COUNT(amount)`	2
`SUM(amount)`	40
`AVG(amount)`	20
`MIN(amount)`	10
`MAX(amount)`	30

Worked-example solution.

SELECT user_id,
       COUNT(*)        AS n_rows,
       COUNT(amount)   AS n_known,
       SUM(amount)     AS total,
       AVG(amount)     AS mean,
       MIN(amount)     AS lo,
       MAX(amount)     AS hi
FROM orders
GROUP BY user_id;

Rule of thumb: if the metric is "people who clicked" use COUNT(DISTINCT user_id); if it is "click events" use COUNT(*); never confuse the two.

`WHERE` vs `HAVING` — row filter vs group filter

The two-clause invariant: WHERE runs before GROUP BY and references raw row columns only; HAVING runs after grouping and can reference aggregate functions; trying to use WHERE COUNT(*) > 1 is a parse error because COUNT(*) does not exist until after grouping. They are not interchangeable — both can appear in the same query.

WHERE — filter rows; uses col, col2, etc.
HAVING — filter groups; uses COUNT(*), SUM(col), etc.
Order of evaluation — FROM → WHERE → GROUP BY → HAVING → SELECT.
Performance — push predicates into WHERE whenever possible; WHERE filters before the (often expensive) sort/hash step.

Worked example. employees(department, salary) with six rows; ask for departments whose average salary exceeds 50,000 across employees earning more than 30,000.

department	salary
eng	40,000
eng	70,000
eng	25,000
sales	60,000
sales	60,000
sales	20,000

WHERE salary > 30000 drops the 25k and 20k rows; GROUP BY department then computes AVG; HAVING AVG(salary) > 50000 keeps only departments whose surviving rows average above 50k.

Worked-example solution.

SELECT department, AVG(salary) AS avg_salary
FROM employees
WHERE salary > 30000
GROUP BY department
HAVING AVG(salary) > 50000;

Rule of thumb: aggregate predicate → HAVING; row predicate → WHERE. If the predicate uses SUM / COUNT / AVG / MIN / MAX, it must live in HAVING.

`HAVING COUNT(*) > 1` — the universal duplicate finder

The duplicate-detection invariant: SELECT key, COUNT(*) FROM t GROUP BY key HAVING COUNT(*) > 1 returns every distinct key value that appears more than once in t, along with its multiplicity. Replace key with the column you want to dedupe on (email, user_id, order_id), and the same query surfaces every duplicate group.

GROUP BY key — one bucket per distinct value.
COUNT(*) > 1 — keeps only buckets with at least two rows.
COUNT(*) = N — variant for "exactly N copies"; rare but cleanly expressible.
Composite keys — GROUP BY a, b, c HAVING COUNT(*) > 1 finds duplicate (a, b, c) triples.

Worked example. users(id, email) with two alice@example.com rows.

id	email
1	alice@example.com
2	bob@example.com
3	alice@example.com

Group by email; the alice@example.com group has COUNT(*) = 2; the bob@example.com group has COUNT(*) = 1 and is filtered out.

Worked-example solution.

SELECT email, COUNT(*) AS n_copies
FROM users
GROUP BY email
HAVING COUNT(*) > 1;

Rule of thumb: every "find duplicates" question reduces to GROUP BY key HAVING COUNT(*) > 1; reach for ROW_NUMBER() OVER (PARTITION BY key) only when you need to delete the duplicates and keep one canonical row.

Common beginner mistakes

Writing WHERE COUNT(*) > 1 — parse error; WHERE cannot reference aggregates.
Selecting a non-aggregated, non-GROUP BY column — strict SQL rejects this; lax dialects pick an arbitrary value silently.
Forgetting COUNT(DISTINCT user_id) after a JOIN that inflates rows — reports inflated user counts.
Using AVG(col) and forgetting that NULL rows are excluded from the denominator — wrong for "treat missing as 0" metrics; use AVG(COALESCE(col, 0)) only when the spec says so.
Putting HAVING before GROUP BY — syntax error; the order is WHERE → GROUP BY → HAVING.

SQL Interview Question on Duplicate Emails

Given users(id, email), return every email that appears more than once in the table, along with the number of copies. Output email and n_copies.

Solution Using `GROUP BY email HAVING COUNT(*) > 1`

SELECT email,
       COUNT(*) AS n_copies
FROM users
GROUP BY email
HAVING COUNT(*) > 1
ORDER BY n_copies DESC, email;

Step-by-step trace for the sample input:

id	email
1	alice@example.com
2	bob@example.com
3	alice@example.com
4	carol@example.com
5	bob@example.com
6	alice@example.com

FROM users — read all six rows.
No WHERE — every row passes.
GROUP BY email — three buckets: alice (3 rows), bob (2 rows), carol (1 row).
COUNT(*) — 3, 2, 1 respectively.
HAVING COUNT(*) > 1 — drops the carol bucket.
ORDER BY n_copies DESC, email — alice (3), then bob (2).

Output:

email	n_copies
alice@example.com	3
bob@example.com	2

Why this works — concept by concept:

GROUP BY email — collapses to one bucket per distinct email; the bucket is the unit of all subsequent aggregates and group-level filters.
COUNT(*) — counts every row in the bucket, including ones with NULL non-key columns; perfect for "how many copies".
HAVING COUNT(*) > 1 — group-level filter; the aggregate predicate must live here, not in WHERE. This is the precise interview signal for duplicate detection.
ORDER BY n_copies DESC, email — deterministic ordering; tie-broken by email so the output is stable across runs and reviewers.
O(|users| + G log G) time — single hash-aggregation over users produces G group rows; the final sort is G log G. With an index on email, the planner may use stream aggregation and skip the hash step.

Inline CTA: Drill the SQL aggregation practice page for GROUP BY and HAVING shapes, and the SQL null-handling practice page for NULL-aware aggregates.

SQL
Topic — aggregation
SQL aggregation problems

Practice →

SQL
Topic — null handling
SQL null-handling problems

Practice →

SQL
Topic — filtering
SQL filtering problems

Practice →

3. SQL Window Functions for Data Engineering — `ROW_NUMBER`, `RANK`, `DENSE_RANK`, `LAG`, `LEAD`

Window functions for ranking and lookback in SQL for data engineering

"Find the second-highest salary" and "find the top 3 salaries per department" are the two signature window-function prompts — and both reduce to a DENSE_RANK() OVER (PARTITION BY ... ORDER BY ...) filter. The mental model: a window function computes a value across a set of rows ("the window") that are related to the current row, without collapsing the rows like GROUP BY does; OVER (PARTITION BY col) defines the window boundary; OVER (ORDER BY col) defines the order within the window. ROW_NUMBER assigns unique sequential numbers; RANK skips after ties (1, 2, 2, 4); DENSE_RANK does not skip (1, 2, 2, 3). LAG looks back; LEAD looks forward. These five primitives drive almost every "ranking" or "lookback" SQL interview question.

Pro tip: When the question is "second-highest salary", reach for DENSE_RANK over RANK — DENSE_RANK = 2 reliably means "second distinct salary" even when ties exist at the top, while RANK = 2 skips entirely if there are two rows tied for first. State this distinction; interviewers grade it.

`ROW_NUMBER` — unique sequential numbering per partition

The ROW_NUMBER invariant: ROW_NUMBER() OVER (PARTITION BY p ORDER BY o) assigns a unique integer 1, 2, 3, ... to every row inside each partition p, ordered by o; ties in o are broken arbitrarily by the planner. Use it when you need a unique sequence per group regardless of tie semantics — most often for deduplication.

OVER (PARTITION BY ...) — bucket the rows.
OVER (ORDER BY ...) — order within the bucket.
Ties broken arbitrarily — add a tiebreaker column to ORDER BY for determinism.
Top-N-per-group — WHERE rn <= N after ROW_NUMBER; works only when ties at rank N are not desired.

Worked example. employees(department, name, salary) with three engineers; rank by salary desc.

department	name	salary	row_number
eng	Alice	90,000	1
eng	Bob	80,000	2
eng	Carol	80,000	3

Bob and Carol tie on salary; ROW_NUMBER still gives them unique ranks (planner-chosen).

Worked-example solution.

SELECT department, name, salary,
       ROW_NUMBER() OVER (PARTITION BY department ORDER BY salary DESC, name) AS rn
FROM employees;

Rule of thumb: ROW_NUMBER is the right tool for deduplication (WHERE rn = 1) and for ordered streams; reach for RANK or DENSE_RANK when ties must be honored.

`RANK` vs `DENSE_RANK` — tie semantics

RANK — 1, 2, 2, 4 — skips after ties.
DENSE_RANK — 1, 2, 2, 3 — no skip.
ROW_NUMBER — 1, 2, 3, 4 — never ties.
Pick by question semantics — "Nth highest distinct salary" → DENSE_RANK = N; "Nth-highest-salaried row in ranking order with skips" → RANK = N.

Worked example. Same employees, plus a third tied row.

name	salary	rank	dense_rank	row_number
Alice	90,000	1	1	1
Bob	80,000	2	2	2
Carol	80,000	2	2	3
Dan	70,000	4	3	4

RANK jumps 2 → 4; DENSE_RANK continues 2 → 3.

Worked-example solution.

SELECT name, salary,
       RANK()       OVER (ORDER BY salary DESC) AS rnk,
       DENSE_RANK() OVER (ORDER BY salary DESC) AS dr,
       ROW_NUMBER() OVER (ORDER BY salary DESC) AS rn
FROM employees;

`LAG` and `LEAD` — lookback and lookahead

The lookback invariant: LAG(col) OVER (PARTITION BY p ORDER BY o) returns the value of col in the previous row within the partition (or NULL for the first row); LEAD returns the next row's value. Both take an optional offset (default 1) and an optional default value to substitute for NULL. They power running deltas, month-over-month growth, sessionization gap detection, and previous-event-aware analytics.

LAG(col, n, dflt) — value n rows back; dflt (default NULL) when out of range.
LEAD(col, n, dflt) — symmetric forward.
PARTITION BY user_id — restart the lookback at each user.
col - LAG(col) — the delta-from-previous-row idiom.

Worked example. sales(sales_date, amount) with a contiguous month series; compute month-over-month delta.

sales_date	amount	lag_amount	delta
2026-01-01	100	NULL	NULL
2026-02-01	130	100	30
2026-03-01	120	130	-10

The first row's LAG is NULL; subtraction yields NULL; consumers usually COALESCE(delta, 0) for display.

Worked-example solution.

SELECT sales_date,
       amount,
       LAG(amount) OVER (ORDER BY sales_date) AS prev_amount,
       amount - LAG(amount) OVER (ORDER BY sales_date) AS mom_delta
FROM sales;

Rule of thumb: LAG for "compare this row to its predecessor" (delta, retention, gap); LEAD for "what happens next" (sessionization, churn-from-here). Always PARTITION BY the entity if the table holds multiple series.

Common beginner mistakes

Using RANK when the question wants the Nth distinct value — RANK = 2 skips entirely if two rows tie for first.
Forgetting PARTITION BY for a per-group ranking — produces a global ranking instead of per-department.
Using WHERE rn = 2 directly without wrapping in a subquery or CTE — window functions cannot be referenced in WHERE of the same SELECT (they run after WHERE).
Confusing LAG (previous) with LEAD (next) — quietly produces inverted deltas.
Forgetting ORDER BY inside OVER — required for ROW_NUMBER, RANK, LAG, LEAD; the result is non-deterministic without it.

SQL Interview Question on Second-Highest Salary

Given employees(emp_id, name, salary), return the second-highest distinct salary. If there is no second-highest distinct salary (e.g., all employees earn the same), return NULL.

Solution Using `DENSE_RANK() OVER (ORDER BY salary DESC)`

SELECT MAX(salary) AS second_highest_salary
FROM (
    SELECT salary,
           DENSE_RANK() OVER (ORDER BY salary DESC) AS dr
    FROM employees
) ranked
WHERE dr = 2;

Why this works: DENSE_RANK() OVER (ORDER BY salary DESC) numbers each row by its salary rank, with no gaps after ties — dr = 1 is the highest distinct salary, dr = 2 is the second-highest distinct salary; the outer WHERE dr = 2 filters to that group; MAX(salary) collapses to a single row and (critically) returns NULL if no rows match — handling the "no second-highest" edge case gracefully without a LIMIT 1 OFFSET 1 that would error or return zero rows.

Step-by-step trace for the sample input:

emp_id	name	salary
1	Alice	90,000
2	Bob	80,000
3	Carol	80,000
4	Dan	70,000

Inner SELECT reads all four rows.
DENSE_RANK() OVER (ORDER BY salary DESC) assigns: Alice → 1, Bob → 2, Carol → 2, Dan → 3.
Outer WHERE dr = 2 keeps Bob and Carol.
MAX(salary) collapses to a single row: 80,000.
No-match path — if every employee earned 90,000, the inner query has only dr = 1 rows; outer WHERE dr = 2 filters to zero rows; MAX(salary) returns NULL (the spec).

Output:

second_highest_salary
80000

Why this works — concept by concept:

DENSE_RANK() over RANK() — DENSE_RANK = 2 is the second-distinct-salary, regardless of how many people tie for first; RANK = 2 would skip if two people tied for first.
OVER (ORDER BY salary DESC) — single global window, ordered by salary descending; no PARTITION BY because the question is over the whole table.
Subquery wrapper — required because window functions cannot be referenced in WHERE of the same SELECT; the outer query reads the materialized rank.
MAX(salary) collapses ties — when dr = 2 matches multiple rows (a tie), MAX returns one value; when dr = 2 matches zero rows, MAX returns NULL — both edge cases handled.
O(N log N) time — sort by salary dominates; ranking is O(N) over the sorted stream.

Inline CTA: Drill the SQL window-functions practice page for ROW_NUMBER, RANK, DENSE_RANK, LAG, and LEAD shapes, and the SQL date-functions practice page for time-series window patterns.

SQL
Topic — window functions
SQL window-function problems

Practice →

SQL
Topic — date functions
SQL date-function problems

Practice →

SQL
Topic — aggregation
SQL aggregation problems

Practice →

4. SQL CTEs and Subqueries for Data Engineering — `WITH`, Recursive CTEs, and Correlated Subqueries

CTE composition, recursive CTEs, and correlated subqueries in SQL for data engineering

"Find the top 3 salaries per department" and "find employees earning above their department average" are the two signature CTE-and-subquery prompts — and they showcase the two complementary patterns. The mental model: a CTE (WITH name AS (SELECT ...)) names an intermediate result you reference like a table; a recursive CTE (WITH RECURSIVE) repeatedly evaluates a base case plus a recursive case, terminating when no new rows are added; a correlated subquery is a subquery whose WHERE clause references the outer query's alias, re-evaluating per outer row. CTEs win on readability for multi-step logic; correlated subqueries win on per-row predicates against the same table.

Pro tip: When the prompt is "top N per group", reach for the CTE-plus-DENSE_RANK pattern, not LIMIT N — LIMIT does not respect partitions and only works on the global stream. The CTE makes the per-group ranking explicit and trivial to reason about; the alternative correlated-subquery WHERE col >= (SELECT col FROM t2 ... LIMIT 1 OFFSET N - 1) is an interview red flag.

`WITH name AS (SELECT ...)` — non-recursive CTEs for readability

The CTE invariant: WITH name AS (SELECT ...) SELECT ... FROM name defines an intermediate result (a "common table expression") that subsequent SELECTs reference like a table; the engine may inline or materialize it depending on cost; multiple CTEs can be chained in a single WITH clause separated by commas. CTEs let you build up complex queries step by step without nesting subqueries five layers deep.

Single CTE — WITH a AS (...) SELECT * FROM a.
Chained CTEs — WITH a AS (...), b AS (SELECT * FROM a WHERE ...) SELECT * FROM b.
Re-reference — a CTE can be referenced multiple times in the main query.
No mandatory materialization — modern planners often inline; PostgreSQL 12+ removed the implicit fence.

Worked example. Find departments where the average salary is above 50,000, then list every employee in those departments.

step	output rows
`high_paying` CTE	departments with `AVG(salary) > 50000`
outer `SELECT`	every employee whose department is in `high_paying`

Worked-example solution.

WITH high_paying AS (
    SELECT department
    FROM employees
    GROUP BY department
    HAVING AVG(salary) > 50000
)
SELECT e.name, e.salary, e.department
FROM employees e
JOIN high_paying h ON h.department = e.department;

Rule of thumb: if you find yourself nesting a subquery three levels deep, refactor to a CTE — the engine produces the same plan but the human review takes a tenth of the time.

`WITH RECURSIVE` — recursive CTEs for sequences and hierarchies

The recursive-CTE invariant: a WITH RECURSIVE CTE has two parts joined by UNION ALL: an anchor query (base case, evaluated once) and a recursive query (refers to the CTE itself, evaluated repeatedly until it returns no new rows); the planner accumulates results across iterations until termination. Use it for sequence generation (1..N, dates), hierarchy traversal (org charts, BOM trees, parent-child relationships), and graph-style reachability queries.

Anchor (base) — SELECT 1 or SELECT root_id FROM table WHERE parent IS NULL.
UNION ALL — combines anchor and recursive output.
Recursive query — references the CTE name; must converge (return zero rows eventually).
Termination — the engine stops when the recursive step produces no new rows.

Worked example. Generate the integers 1 through 5.

iteration	new row
anchor	1
step 1	2
step 2	3
step 3	4
step 4	5
step 5	(none — `n = 5` fails the predicate)

Worked-example solution.

WITH RECURSIVE nums(n) AS (
    SELECT 1
    UNION ALL
    SELECT n + 1
    FROM nums
    WHERE n < 5
)
SELECT n FROM nums;

Rule of thumb: always include a termination predicate (WHERE n < N); a missing or wrong predicate produces an infinite loop that most planners eventually kill with an out-of-memory error.

Correlated subqueries — per-row predicates against the same table

The correlated-subquery invariant: a subquery whose WHERE clause references a column of the outer query is re-evaluated for every outer row; this enables predicates like "salary above this row's department average" without a JOIN. Powerful but expensive; the planner sometimes rewrites them into joins or hash-aggregations automatically.

Outer alias reference — WHERE inner.dept = outer.dept.
EXISTS — short-circuit-friendly; stop on first match.
NOT EXISTS — anti-join equivalent; immune to NULL like LEFT JOIN ... IS NULL.
Performance — index the inner referenced column; otherwise the planner runs an inner scan per outer row.

Worked example. Find employees whose salary is above their department's average.

name	dept	salary	dept_avg	survives?
Alice	eng	90,000	76,667	✓
Bob	eng	80,000	76,667	✓
Carol	eng	60,000	76,667	✗
Dan	sales	60,000	60,000	✗

Worked-example solution.

SELECT name, department, salary
FROM employees e
WHERE salary > (
    SELECT AVG(salary)
    FROM employees
    WHERE department = e.department
);

Rule of thumb: prefer a CTE + window function (AVG(salary) OVER (PARTITION BY department)) for performance on large tables; the correlated subquery is the clearer but slower form. Interviewers grade you on knowing both.

Common beginner mistakes

Forgetting WITH RECURSIVE for a recursive CTE — non-RECURSIVE CTEs cannot self-reference; parse error.
Missing the termination predicate in WITH RECURSIVE — infinite loop.
Using LIMIT N for top-N-per-group — LIMIT is global; it does not respect partitions.
Writing a correlated subquery without an index on the inner column — quadratic blow-up on large tables.
Re-materializing the same CTE under different names — the planner may run it twice; inline a single shared CTE instead.

SQL Interview Question on Top 3 Salaries Per Department

Given employees(emp_id, name, department, salary), return the top 3 distinct salaries per department, with ties at rank 3 included. Output department, name, salary, and the rank.

Solution Using `DENSE_RANK() OVER (PARTITION BY department ORDER BY salary DESC)` in a CTE

WITH ranked AS (
    SELECT department,
           name,
           salary,
           DENSE_RANK() OVER (
               PARTITION BY department
               ORDER BY salary DESC
           ) AS dr
    FROM employees
)
SELECT department, name, salary, dr
FROM ranked
WHERE dr <= 3
ORDER BY department, dr, name;

Why this works: the CTE ranked materializes a per-department DENSE_RANK keyed by salary descending — dr = 1 is the highest distinct salary in that department, dr = 2 is the second-highest, and so on; the outer WHERE dr <= 3 keeps every row whose salary is in the top three distinct salaries of its department, including all ties at rank 3; the ORDER BY produces a deterministic, reviewer-friendly output. DENSE_RANK over RANK because we want the top three distinct salaries; DENSE_RANK over ROW_NUMBER because ties at rank 3 must be retained (the spec).

Step-by-step trace for the sample input:

emp_id	name	department	salary
1	Alice	eng	90,000
2	Bob	eng	80,000
3	Carol	eng	80,000
4	Dan	eng	70,000
5	Eve	eng	60,000
6	Frank	sales	100,000
7	Grace	sales	90,000
8	Heidi	sales	80,000

CTE ranked — partition by department; order by salary DESC.
DENSE_RANK per partition — eng: Alice → 1, Bob → 2, Carol → 2, Dan → 3, Eve → 4. sales: Frank → 1, Grace → 2, Heidi → 3.
Outer WHERE dr <= 3 — drops Eve (dr = 4); keeps both Bob and Carol (tied at 2) and Dan (3).
ORDER BY department, dr, name — eng rows first, then sales; within department by dr, then name for tiebreak.

Output:

department	name	salary	dr
eng	Alice	90000	1
eng	Bob	80000	2
eng	Carol	80000	2
eng	Dan	70000	3
sales	Frank	100000	1
sales	Grace	90000	2
sales	Heidi	80000	3

Why this works — concept by concept:

CTE ranked — names the intermediate ranked result; the outer query then filters it like a regular table. Far cleaner than a nested subquery.
PARTITION BY department — restarts the rank at each department boundary; without this, the rank is global and the answer is wrong.
ORDER BY salary DESC — defines "highest first" inside each partition; required for any deterministic ranking.
DENSE_RANK over RANK — the spec wants the top three distinct salaries; RANK would skip after ties and miss the third distinct salary if there is a two-way tie above it.
WHERE dr <= 3 in the outer — window functions cannot be referenced in WHERE of the same SELECT; the CTE provides the materialized column the outer can filter on.
O(N log N) time — sort within each partition dominates; with an index on (department, salary DESC) the planner can stream rather than sort.

Inline CTA: More SQL CTE practice problems and SQL subquery practice problems on PipeCode.

SQL
Topic — CTE
SQL CTE problems

Practice →

SQL
Topic — subqueries
SQL subquery problems

Practice →

SQL
Topic — window functions
SQL window-function problems

Practice →

Tips to crack SQL interviews for data engineering roles

Master the four primitives — joins, aggregates, windows, CTEs

If you can write LEFT JOIN ... IS NULL for orphans, GROUP BY ... HAVING COUNT(*) > 1 for duplicates, DENSE_RANK() OVER (PARTITION BY ...) for top-N-per-group, and a WITH RECURSIVE CTE for sequence generation without thinking — you can pass most fresher and mid-level data-engineering SQL rounds. These four primitives compose into 80% of the questions you will see; the remaining 20% is LAG/LEAD lookback, COALESCE null-safety, and date arithmetic.

Know the difference between `WHERE`, `HAVING`, and the `OVER` clause

WHERE filters individual rows before grouping; HAVING filters groups after aggregation; OVER (...) defines a window for a window function and runs after WHERE but before SELECT. Window functions cannot be referenced in WHERE of the same SELECT — wrap them in a CTE or subquery first. State the order of evaluation (FROM → WHERE → GROUP BY → HAVING → window → SELECT → ORDER BY → LIMIT) when an interviewer asks; it is graded as fundamental literacy.

Pick `DENSE_RANK` for "Nth distinct"; pick `ROW_NUMBER` for deduplication

Use `LEFT JOIN ... IS NULL` over `NOT IN` for anti-joins

NOT IN (subquery) returns zero rows if the subquery contains a single NULL because x NOT IN (..., NULL, ...) evaluates to NULL, which fails the WHERE predicate. LEFT JOIN ... WHERE right.id IS NULL and NOT EXISTS (...) are both immune to this. Production data engineers who have been bitten by this once never write NOT IN again. State the gotcha out loud.

Practice on PostgreSQL — it is the default dialect of most live coders

DataLemur, CoderPad, HackerRank's SQL practice, most product-analytics live screens, and most public SQL interview corpora use PostgreSQL syntax. Drill EXTRACT(...), INTERVAL '1 month', DATE_TRUNC, ::DATE casting, and COALESCE until they are reflexive. MySQL-only quirks (back-tick identifiers, LIMIT N OFFSET M) and SQL-Server-only quirks (TOP N, OFFSET ... FETCH ...) are second priority — only drill them if you know the company uses that dialect.

Always `ORDER BY` and add a tiebreaker for determinism

Read the data shape before writing the query

Before typing, ask: what is the grain of each table (one row per...)? What are the keys? Are there NULLs in the join columns? Is email indexed? The most common "almost passed" failure mode is correct happy-path SQL that breaks on the actual schema — duplicate primary keys, NULL foreign keys, off-by-one cardinality. A 30-second schema sweep prevents it.

Where to practice on PipeCode

Start with the SQL practice surface for the all-language SQL corpus. Drill the four-primitive pages: SQL joins, SQL aggregation, SQL window functions, SQL CTE, SQL subqueries. Add adjacent topics: SQL filtering, SQL null-handling, SQL date functions. The interview courses page bundles structured curricula. For broader coverage, browse by topic or read the top data engineering interview questions 2026 blog and the SQL data types in PostgreSQL guide.

Frequently Asked Questions

What SQL topics are most asked in data engineering interviews?

Four primitives carry the loop: joins (especially LEFT JOIN ... IS NULL for anti-joins), aggregations (GROUP BY plus HAVING, with COUNT(*) > 1 as the universal duplicate finder), window functions (ROW_NUMBER, RANK, DENSE_RANK, LAG, LEAD for ranking and lookback), and CTEs (WITH and WITH RECURSIVE for multi-step logic and sequence generation). Adjacent shapes — WHERE vs HAVING, NULL handling with COALESCE, dedup via ROW_NUMBER, and indexes — round out the typical fresher and mid-level loop.

What is the difference between `RANK`, `DENSE_RANK`, and `ROW_NUMBER`?

What is the difference between `WHERE` and `HAVING`?

What is a CTE and when should I use one over a subquery?

A CTE (Common Table Expression, written as WITH name AS (SELECT ...)) is a named intermediate result you reference like a table in subsequent SELECTs. Use a CTE when the same intermediate is referenced multiple times, when the multi-step logic is deeply nested as a subquery, or when you need recursion (WITH RECURSIVE). Use a subquery when the intermediate is referenced exactly once and inlined cleanly. Modern PostgreSQL no longer materializes CTEs by default, so the performance gap has narrowed — pick by readability.

How do I find duplicate rows in SQL?

The canonical pattern is SELECT key, COUNT(*) FROM table GROUP BY key HAVING COUNT(*) > 1, which returns every distinct key value that appears more than once along with its multiplicity. To find duplicate (a, b, c) triples, use GROUP BY a, b, c. To delete duplicates while keeping one canonical row per key, use ROW_NUMBER() OVER (PARTITION BY key ORDER BY tiebreaker) inside a CTE and delete every row where ROW_NUMBER > 1.

What is the best way to find the second-highest salary?

The cleanest answer is SELECT MAX(salary) FROM (SELECT salary, DENSE_RANK() OVER (ORDER BY salary DESC) AS dr FROM employees) t WHERE dr = 2. DENSE_RANK = 2 reliably gives the second-distinct-salary even when ties exist at the top; the outer MAX(salary) collapses ties at rank 2 into a single value and returns NULL when no second-distinct-salary exists — both edge cases handled. Avoid LIMIT 1 OFFSET 1 — it errors on empty result sets in some dialects and does not handle ties correctly.

Should I learn PostgreSQL or MySQL for SQL interviews?

PostgreSQL is the default dialect of most modern data-engineering interview platforms (DataLemur, CoderPad, most product-analytics live screens) because of its strong window-function support, EXTRACT / INTERVAL date arithmetic, and ::TYPE casting syntax. Drill PostgreSQL first; learn MySQL-specific quirks (back-tick identifiers, slightly different LIMIT syntax, no FULL OUTER JOIN until 8.0.31) only if the company explicitly uses MySQL. Either way, the four primitives — joins, aggregates, windows, CTEs — are dialect-agnostic.

Start practicing SQL data engineering problems

COALESCE in SQL — First Non-NULL, LEFT JOIN Defaults, and Interview Patterns

Gowtham Potureddi — Sun, 03 May 2026 15:52:58 +0000

COALESCE in SQL is the single most-asked NULL-handling primitive at every data-engineering interview that touches production analytics. The mental model: COALESCE(expr1, expr2, ..., exprN) returns the first argument that is not NULL, evaluated left-to-right; if every argument is NULL, the result is NULL. That one rule covers a vast surface — fallback chains for "best available column," default values to neutralize LEFT JOIN misses, sentinel labels like 'NONE' for missing dimensions, and zero-substitution before downstream math. Four sub-primitives carry the loop: left-to-right evaluation with short-circuit semantics, the COALESCE(left_join_col, default) pattern that turns outer-join NULLs into reportable defaults, the dialect-portability matrix versus CASE / ISNULL / NVL / IFNULL, and the pitfall set around NULL ≠ 0 semantics, type coercion, empty strings, and the COALESCE(NULLIF(col, ''), 'default') composition.

This guide walks four concept clusters end-to-end, each with a detailed topic explanation, per-sub-topic explanation with a worked example and its solution, and an interview-style worked problem with a full solution that explains why it works. The mix matches the actual surface area COALESCE covers in production SQL — analytics dashboards, ETL transforms, BI reports, and whiteboard interview questions. Examples use PostgreSQL-friendly syntax; engine differences are called out where they bite (ISNULL is SQL Server, NVL is Oracle, IFNULL is MySQL — all of them are two-argument cousins of the standard-SQL COALESCE).

Top COALESCE concepts and SQL patterns

The four numbered sections below follow this concept map (one row per H2):

#	Topic (sections 1–4)	Why it matters in SQL data engineering
1	`COALESCE` evaluation order and basic fallback patterns	`COALESCE(a, b, c)` is left-to-right, short-circuits on first non-`NULL`, and follows engine-specific type-precedence rules; the bedrock primitive every other pattern composes on.
2	`COALESCE` with `LEFT JOIN` for default values in analytics and BI	Outer joins to lookup tables (FX rates, dimensions, slowly-changing-dim surrogates) produce `NULL` on misses; `COALESCE(fx.rate, 1)` and `COALESCE(p.promo_code, 'NONE')` turn that `NULL` into a default that downstream metrics can consume.
3	`COALESCE` vs `CASE`, `ISNULL`, and `NVL` — portability and when to pick which	`COALESCE` is standard SQL and portable across PostgreSQL / MySQL / SQL Server / Oracle / Snowflake / BigQuery; `ISNULL` (SQL Server, 2-arg), `NVL` (Oracle, 2-arg), `IFNULL` (MySQL, 2-arg) are dialect cousins; `CASE` is the right tool when the logic is not "first non-`NULL`."
4	`COALESCE` pitfalls — NULL semantics, type coercion, empty strings, and `NULLIF`	`NULL` ≠ `0`; replacing unknown with zero changes the metric meaning. Empty string `''` is not `NULL` in PostgreSQL; combine with `NULLIF` for "empty-or-null." Mixing types without `CAST` raises errors or silently coerces.

Concept-based framing rule: COALESCE is the primitive; the four sections are the operating contexts where it shows up. Master the evaluation rule first (§1), then the LEFT JOIN default pattern (§2), then the dialect comparison (§3), then the pitfalls (§4). State the left-to-right-first-non-NULL rule out loud in any interview answer that touches null handling — interviewers grade the explicit verbalization, not just the correct query.

1. `COALESCE` Evaluation Order and Basic Fallback Patterns

Left-to-right first-non-NULL evaluation in SQL for data engineering

"Return the first non-NULL value from a list of expressions" is the canonical COALESCE semantics. The mental model: the engine evaluates arguments left-to-right; the first one whose value is not NULL becomes the result; remaining arguments may not be evaluated at all (most engines short-circuit); if every argument is NULL, the result itself is NULL. Same primitive powers any "priority-ordered fallback" pattern — pick the user's work_email, else personal_email, else a literal 'no-email@example.com'; pick the trade's settled_price, else mark_price, else last_close; pick the column you trust, else the column you mostly trust, else a sentinel.

Pro tip: Even when the engine documents short-circuit evaluation, never put side-effecting expressions in later COALESCE arguments — volatile functions, RAISE statements, or sequence calls. State this explicitly in interviews; it signals production fluency. The reader-friendly contract is "every argument is a pure expression; the engine picks the first non-NULL."

Syntax: `COALESCE(expr1, expr2, ..., exprN)`

The syntax invariant: COALESCE takes 1 to N arguments (most engines require ≥ 2) and returns a single value of the unified result type. The unified type is determined by precedence rules across all argument types — mixing INT and BIGINT is fine and returns BIGINT; mixing INT and TEXT without an explicit CAST raises a type error in PostgreSQL.

COALESCE(a, b) — minimum form; first argument a if non-NULL, else b.
COALESCE(a, b, c, d, e) — N-ary form; ordered fallback chain.
COALESCE(NULL) — single-argument form; rejected by PostgreSQL parser.
Type unification — all arguments must coerce to a common type; explicit CAST(... AS ...) keeps intent obvious.

Worked example. Mixed literal arguments; the engine returns 'third' because the first two are NULL.

step	value
arg 1	`NULL`
arg 2	`NULL`
arg 3	`'third'` (first non-`NULL`)
result	`'third'`

Worked-example solution.

SELECT COALESCE(NULL, NULL, 'third', 'fourth') AS first_non_null;
-- first_non_null = 'third'

Rule of thumb: COALESCE reads as "try this, else try this, else try this" — the order of arguments encodes business priority directly in SQL.

Short-circuit evaluation and side-effect safety

The short-circuit invariant: most engines stop evaluating arguments once they find a non-NULL; PostgreSQL, SQL Server, Oracle, and MySQL all document this behavior, but the SQL standard does not require it. Treat short-circuit as a performance optimization, not a contract — never rely on it for correctness when later arguments have side effects.

COALESCE(safe_col, expensive_lookup()) — expensive_lookup() only runs when safe_col is NULL.
COALESCE(col, RAISE_NOTICE('...')) — never rely on the side-effect ordering; the optimizer may rewrite.
Volatile functions — random(), now(), sequence calls can be evaluated even when later in the list.
Pure functions only — write the contract as "every argument is a pure expression."

Worked example. Cheap column first, expensive subquery second.

input	expensive_lookup called?
`cached_value = 100`	no
`cached_value = NULL`	yes

Worked-example solution.

SELECT user_id,
       COALESCE(
         cached_email,                           -- cheap, indexed
         (SELECT email FROM email_lookup
          WHERE user_id = u.user_id LIMIT 1)    -- expensive, only runs on NULL cached
       ) AS effective_email
FROM users u;

Rule of thumb: put the cheapest reliably-non-NULL column first; let short-circuit save the expensive lookups for rows that need them.

Type precedence and `CAST` for unambiguous types

The type-precedence invariant: COALESCE returns a single unified type computed from the precedence rules of all argument types; mixing incompatible types (e.g. INTEGER with TEXT) is a parse error in PostgreSQL and a silent coercion in MySQL. Use CAST(... AS ...) to make intent explicit.

COALESCE(int_col, 0) — both INTEGER; result INTEGER. ✓
COALESCE(numeric_col, 0) — NUMERIC and INTEGER unify to NUMERIC. ✓
COALESCE(int_col, 'unknown') — type error in PostgreSQL.
COALESCE(CAST(int_col AS TEXT), 'unknown') — explicit cast resolves the conflict.

Worked example. Avoid the type error by casting INTEGER to TEXT before mixing with a string default.

expression	result type	safe?
`COALESCE(qty, 'N/A')`	error	✗
`COALESCE(CAST(qty AS TEXT), 'N/A')`	TEXT	✓

Worked-example solution.

SELECT order_id,
       COALESCE(CAST(qty AS TEXT), 'N/A') AS qty_label
FROM orders;

Rule of thumb: when you need a string sentinel for a numeric column, cast the numeric first; never trust implicit coercion across families.

Common beginner mistakes

Calling COALESCE with one argument — most engines reject; minimum is 2.
Putting volatile / side-effecting expressions in later positions — non-deterministic results.
Mixing types without CAST — silent coercion in MySQL, parse error in PostgreSQL.
Assuming all engines short-circuit — relying on it is a portability bug.
Returning NULL and surprising downstream — if every argument is NULL, so is the result; add a guaranteed-non-NULL literal at the end if the metric needs a default.

Worked Problem on COALESCE Evaluation Order

Given a contacts table with three nullable email columns — work_email, personal_email, legacy_email — return one column effective_email per row that is the first non-NULL of the three, falling back to the literal 'no-email@example.com' when every column is NULL.

Solution Using `COALESCE` with a literal final fallback

SELECT
    contact_id,
    COALESCE(
        work_email,
        personal_email,
        legacy_email,
        'no-email@example.com'
    ) AS effective_email
FROM contacts;

Why this works: the four-argument COALESCE evaluates each column left-to-right; on the first non-NULL it short-circuits and returns that value; the trailing literal 'no-email@example.com' guarantees a non-NULL result for rows where all three columns are NULL. One pass, no CASE, no joins, fully PostgreSQL-portable.

Step-by-step trace for four sample rows:

contact_id	work_email	personal_email	legacy_email	effective_email
1	`alice@work.com`	`alice@home.com`	`NULL`	`alice@work.com`
2	`NULL`	`bob@home.com`	`bob@old.net`	`bob@home.com`
3	`NULL`	`NULL`	`carol@old.net`	`carol@old.net`
4	`NULL`	`NULL`	`NULL`	`no-email@example.com`

Row 1 — work_email is non-NULL → return immediately.
Row 2 — work_email is NULL; personal_email is non-NULL → return.
Row 3 — first two are NULL; legacy_email is non-NULL → return.
Row 4 — all three are NULL; literal 'no-email@example.com' returned.

Output:

contact_id	effective_email
1	alice@work.com
2	bob@home.com
3	carol@old.net
4	no-email@example.com

Why this works — concept by concept:

Left-to-right evaluation — COALESCE walks arguments in source order; the priority encoded in the SQL exactly matches the business rule "prefer work, then home, then legacy."
Short-circuit on first non-NULL — the engine stops as soon as it has a value; rows with a populated work_email never touch personal_email or legacy_email.
Literal final fallback — the trailing string literal guarantees effective_email is never NULL; downstream consumers (reports, joins, filters) can treat the column as NOT NULL.
Single-pass aggregation — no CASE, no JOIN, no subquery; the SELECT scans contacts once and emits one output row per input row.
O(N) time / O(1) space — N rows scanned, no auxiliary structures; the per-row work is constant 4-argument evaluation.

Inline CTA: Drill the SQL null-handling practice page for the full curated set of COALESCE-style problems.

SQL
Topic — null handling
SQL null-handling problems

Practice →

SQL
Topic — conditional logic
SQL conditional-logic problems

Practice →

SQL
Topic — filtering
SQL filtering problems

Practice →

2. `COALESCE` with `LEFT JOIN` for Default Values in Analytics and BI

COALESCE for outer-join NULL handling in SQL for data engineering

"Replace NULL from a LEFT JOIN miss with a sensible default" is the most common production use of COALESCE. The mental model: LEFT JOIN keeps every row from the left table; rows that have no match in the right table get NULL for every right-table column; COALESCE(right_col, default) neutralizes those NULLs into business-meaningful defaults — 1 for missing FX rates (treat as USD), 'NONE' for missing promo codes, 0 for missing aggregate counts. Same primitive powers any "fact + lookup" pipeline — sales fact joined to currency dim, click event joined to user dim, transaction joined to merchant category.

Pro tip: Choose the default carefully. COALESCE(fx.rate, 1) is correct only when the source amount is already in USD; if amounts are in unknown local currencies, defaulting to 1 silently fabricates dollars. State the assumption in the SQL comment or the surrounding documentation.

`LEFT JOIN` + `COALESCE(right_col, default)` for fact-dim joins

The fact-dim invariant: a fact table (transactions, events, orders) is LEFT JOIN-ed to a dim table (currency rates, customer reference, product catalog); rows with no dim match return NULL for every dim column; COALESCE substitutes a default that downstream math can consume.

COALESCE(dim_col, default) — the canonical wrapping pattern.
LEFT JOIN — keep all fact rows; never drop unmatched.
Right-side filtering — putting WHERE dim.col = X after a LEFT JOIN silently turns it into INNER JOIN (the WHERE filters out the NULL rows); use AND dim.col = X inside the ON clause if you need to keep all fact rows.
Default semantics — pick a value that means "no match" without poisoning the metric.

Worked example. Three transactions in two currencies; FX table has rates only for EUR.

transaction	currency	fx.rate (after LEFT JOIN)	coalesced rate
t1	USD	`NULL`	1
t2	EUR	`1.08`	1.08
t3	USD	`NULL`	1

Worked-example solution.

SELECT t.transaction_id,
       t.amount,
       t.currency,
       COALESCE(fx.rate, 1) * t.amount AS amount_usd
FROM transactions t
LEFT JOIN fx_rates fx
    ON fx.currency = t.currency;

Rule of thumb: LEFT JOIN ... COALESCE(right_col, default) is the canonical "fact + dim with safe defaults" pattern; it appears in nearly every BI dashboard SQL.

"First non-blank across sources" via chained `COALESCE`

The chained-coalesce invariant: multiple LEFT JOINs to supplemental tables (primary, secondary, fallback) return per-table columns that may all be NULL; COALESCE(a.col, b.col, c.col, default) picks the first source that actually has data.

Primary source — most reliable column (e.g., production CRM).
Secondary source — supplemental (e.g., marketing CRM).
Tertiary source — historical (e.g., legacy import).
Default literal — guaranteed non-NULL final fallback.

Worked example. User name from CRM, fall back to billing system, fall back to login email.

crm.name	billing.name	login.email	display_name
`Alice Lee`	`Alice L.`	`alice@x.com`	`Alice Lee`
`NULL`	`Bob B.`	`bob@x.com`	`Bob B.`
`NULL`	`NULL`	`carol@x.com`	`carol@x.com`
`NULL`	`NULL`	`NULL`	`Unknown`

Worked-example solution.

SELECT u.user_id,
       COALESCE(crm.full_name, billing.display_name, login.email, 'Unknown') AS display_name
FROM users u
LEFT JOIN crm     ON crm.user_id     = u.user_id
LEFT JOIN billing ON billing.user_id = u.user_id
LEFT JOIN login   ON login.user_id   = u.user_id;

Rule of thumb: chain LEFT JOINs for multiple sources, then COALESCE across them in priority order; the SQL reads top-to-bottom exactly like the business rule.

`COALESCE(SUM(x), 0)` for grouped aggregates with empty groups

The aggregate-coalesce invariant: SUM, COUNT, AVG over an empty group returns NULL; wrapping the aggregate in COALESCE(SUM(x), 0) returns a numeric default so the report row still appears with a zero metric instead of disappearing.

SUM(x) over empty group — returns NULL.
COUNT(*) — returns 0, not NULL (the exception).
COALESCE(SUM(x), 0) — 0 for empty groups; report row remains.
COALESCE(AVG(x), 0) — 0 for empty groups; semantic consideration — may mislead.

Worked example. Three regions; only two have orders; the third should show 0 not be omitted.

region	SUM(amount)	COALESCE
North	1500	1500
South	800	800
West	NULL	0

Worked-example solution.

SELECT r.region,
       COALESCE(SUM(o.amount), 0) AS total_amount
FROM regions r
LEFT JOIN orders o ON o.region = r.region
GROUP BY r.region;

Rule of thumb: always wrap SUM / AVG in COALESCE(..., 0) when joining a dimension that should produce a row even with zero matching facts; without it, empty groups silently disappear.

Common beginner mistakes

Filtering WHERE dim.col = X after a LEFT JOIN — silently downgrades to INNER JOIN; move to AND dim.col = X in the ON clause.
Defaulting FX rate to 1 when the source amount is in unknown currency — fabricates dollars.
Forgetting COALESCE(SUM(x), 0) for aggregate metrics — empty groups disappear from reports.
Using COALESCE to skip rows — that's WHERE's job, not COALESCE's.
Mismatched join keys — COALESCE cannot fix a missing or wrong ON predicate.

Worked Problem on LEFT JOIN with COALESCE Default

Given an orders table and an optional promos table, return a single column promo_label per order that shows the promo code when matched and the literal 'NONE' when there is no matching promo.

Solution Using `LEFT JOIN` + `COALESCE(p.promo_code, 'NONE')`

SELECT
    o.order_id,
    COALESCE(p.promo_code, 'NONE') AS promo_label
FROM orders o
LEFT JOIN promos p ON p.promo_id = o.promo_id;

Why this works: LEFT JOIN keeps every order, even those with no promo_id or whose promo_id doesn't match any row in promos; the unmatched rows get NULL for p.promo_code; COALESCE substitutes the literal 'NONE' so the output column is non-NULL and dashboard-safe.

Step-by-step trace for sample data:

order_id	promo_id	matched promo_code	COALESCE result
1001	P1	`SUMMER20`	`SUMMER20`
1002	NULL	(no row)	`NONE`
1003	P9	(no row)	`NONE`
1004	P2	`WELCOME10`	`WELCOME10`

LEFT JOIN — every order row survives.
Match — order 1001 → P1 → SUMMER20; order 1004 → P2 → WELCOME10.
No-match — orders 1002 (NULL key) and 1003 (key not in promos) get NULL for p.promo_code.
COALESCE — NULL rows become 'NONE'; matched rows pass through.

Output:

order_id	promo_label
1001	SUMMER20
1002	NONE
1003	NONE
1004	WELCOME10

Why this works — concept by concept:

LEFT JOIN preserves left-side rows — every order stays in the result; no order silently disappears because of a missing or invalid promo_id.
Right-side NULL on miss — orders with no matching promo see p.promo_code = NULL; this is the LEFT JOIN contract, not a bug.
COALESCE(p.promo_code, 'NONE') — sentinel substitution; the output column becomes non-NULL and downstream code can treat it as a finite enum (SUMMER20, WELCOME10, ..., NONE).
Sentinel, not silent zero — 'NONE' is a string sentinel that flags "no promo applied"; it is never confused with a real code.
O(|orders| + |promos|) time — hash-join cost; O(|orders|) output rows.

Inline CTA: Practice SQL join problems to drill the LEFT JOIN + COALESCE muscle.

SQL
Topic — joins
SQL join problems

Practice →

SQL
Topic — null handling
SQL null-handling problems

Practice →

SQL
Topic — aggregation
SQL aggregation problems

Practice →

3. `COALESCE` vs `CASE`, `ISNULL`, and `NVL` — Portability and When to Pick Which

Standard SQL portability versus dialect-specific NULL helpers in SQL for data engineering

"Which NULL-handling construct should I use in this engine?" is a question every SQL data engineer answers daily. The mental model: COALESCE is the standard-SQL N-argument primitive supported on every major engine; ISNULL, NVL, and IFNULL are dialect-specific 2-argument cousins that exist for historical reasons; CASE WHEN ... IS NOT NULL THEN ... ELSE ... END is the general-purpose conditional that handles any logic, not just NULL fallback. Reach for COALESCE by default; reach for CASE when the branches are not "first non-NULL."

Pro tip: In multi-engine analytics codebases (e.g., dbt models targeting both Snowflake and BigQuery, or migrations from legacy SQL Server to PostgreSQL), COALESCE is the only portable choice. Hard-coding ISNULL or NVL in shared code is a portability bug waiting for the migration ticket.

`COALESCE` (standard SQL) — the portable default

The portability invariant: COALESCE is part of the SQL:1992 standard and is supported by PostgreSQL, MySQL, SQL Server, Oracle, Snowflake, BigQuery, Redshift, Databricks SQL, DuckDB, SQLite, and every other major engine you'll encounter. It accepts 2 to N arguments.

N-argument support — COALESCE(a, b, c, d, ...) works everywhere.
Standard-SQL identifier — no version-specific syntax.
Recommended default — choose COALESCE unless you have a dialect-specific reason.
Most engines short-circuit — performance-equivalent to dialect cousins.

Worked example. Same query runs unmodified on PostgreSQL, MySQL, SQL Server, Oracle, BigQuery.

dialect	query	works?
PostgreSQL	`SELECT COALESCE(a, b, c) FROM t;`	✓
MySQL	same	✓
SQL Server	same	✓
Oracle	same	✓
BigQuery	same	✓

Worked-example solution.

SELECT user_id,
       COALESCE(work_email, personal_email, 'no-email@example.com') AS contact
FROM users;
-- runs identically across PostgreSQL / MySQL / SQL Server / Oracle / BigQuery / Snowflake

Rule of thumb: if portability matters at all, COALESCE is the only correct choice; it never gets you locked into a single dialect.

`ISNULL` (SQL Server), `NVL` (Oracle), `IFNULL` (MySQL) — dialect 2-arg cousins

The dialect-cousin invariant: all three are 2-argument NULL-replacement functions; they exist for historical reasons predating standard COALESCE adoption; they have minor type-coercion quirks vs COALESCE that occasionally bite (e.g., SQL Server ISNULL truncates to the type of the first argument, while COALESCE uses precedence).

function	engine	arity	type rule
`ISNULL(a, b)`	SQL Server	2	result type = type of `a` (truncation risk)
`NVL(a, b)`	Oracle	2	result type = type of `a`
`IFNULL(a, b)`	MySQL, BigQuery	2	result type = standard precedence

Worked example. SQL Server ISNULL truncation gotcha — ISNULL(varchar(3), 'longer string') truncates the second argument to 3 characters.

query	result	gotcha
`ISNULL(CAST('hi' AS VARCHAR(3)), 'goodbye')`	`'goo'`	truncated to len-3
`COALESCE(CAST('hi' AS VARCHAR(3)), 'goodbye')`	`'goodbye'`	no truncation

Worked-example solution.

-- SQL Server-only (avoid in portable code)
SELECT ISNULL(work_email, personal_email) FROM users;

-- Oracle-only
SELECT NVL(work_email, personal_email) FROM users;

-- MySQL / BigQuery
SELECT IFNULL(work_email, personal_email) FROM users;

-- Standard SQL — runs everywhere
SELECT COALESCE(work_email, personal_email) FROM users;

Rule of thumb: unless you are maintaining an existing single-dialect codebase that already uses the cousin, prefer COALESCE; the cousins offer no benefit and lock you into one engine.

`CASE WHEN ... IS NOT NULL THEN ... ELSE ... END` — when logic is not "first non-NULL"

The CASE invariant: CASE is the general-purpose conditional in SQL; it handles any boolean predicate, not just IS NOT NULL; reach for CASE when the branches involve ranges, flags, transformations, or any logic richer than "pick the first non-NULL".

Two-argument CASE equivalent to COALESCE(a, b) — verbose but explicit.
Range branches — CASE WHEN amount > 1000 THEN 'large' WHEN amount > 100 THEN 'medium' ELSE 'small' END.
Multiple-condition logic — combine IS NULL, comparisons, and string predicates.
Per-branch transformations — different SQL functions per branch.

Worked example. Categorize order amounts; COALESCE doesn't apply because the logic is range-based.

amount	category
1500	large
250	medium
50	small
NULL	unknown

Worked-example solution.

SELECT order_id,
       CASE
         WHEN amount IS NULL THEN 'unknown'
         WHEN amount > 1000 THEN 'large'
         WHEN amount > 100  THEN 'medium'
         ELSE 'small'
       END AS amount_category
FROM orders;
-- COALESCE cannot express ranges; CASE is the right tool

Rule of thumb: if your logic is "first non-NULL," use COALESCE; if your logic is anything else, use CASE; if you find yourself nesting COALESCE with CASE, the CASE alone is usually clearer.

Common beginner mistakes

Using ISNULL in PostgreSQL — ISNULL(a, b) is not a function in PostgreSQL; it's a keyword for IS NULL. Parse error.
Using NVL outside Oracle — non-portable; COALESCE instead.
Hitting the SQL Server ISNULL truncation gotcha — use COALESCE for safe type unification.
Reaching for CASE when COALESCE is shorter — verbose and harder to review.
Reaching for COALESCE when the logic is not first-non-NULL — pick CASE.

Worked Problem on Picking the Right NULL Construct

Given a users table with three nullable columns — work_email, personal_email, and phone — return a column effective_contact that is the first non-NULL of the three, prefixed by 'email: ' for emails and 'phone: ' for phone numbers, falling back to 'no contact'.

Solution Using `COALESCE` inside a `CASE`

SELECT
    user_id,
    CASE
        WHEN work_email IS NOT NULL OR personal_email IS NOT NULL
            THEN 'email: ' || COALESCE(work_email, personal_email)
        WHEN phone IS NOT NULL
            THEN 'phone: ' || phone
        ELSE 'no contact'
    END AS effective_contact
FROM users;

Why this works: the outer CASE handles the per-channel transformation (different prefix for email vs phone), which COALESCE alone cannot express; the inner COALESCE(work_email, personal_email) collapses the two email candidates with first-non-NULL priority, then the prefix concatenation runs only when at least one email is present; the WHEN phone IS NOT NULL branch handles the secondary channel; the ELSE 'no contact' is the guaranteed fallback. This is the canonical "use COALESCE for null pickers, CASE for everything else" idiom.

Step-by-step trace for four sample rows:

user_id	work_email	personal_email	phone	effective_contact
1	`a@w.com`	`a@h.com`	`5551111`	`email: a@w.com`
2	`NULL`	`b@h.com`	`5552222`	`email: b@h.com`
3	`NULL`	`NULL`	`5553333`	`phone: 5553333`
4	`NULL`	`NULL`	`NULL`	`no contact`

Row 1 — work_email non-null → first WHEN matches → email: a@w.com.
Row 2 — work_email null but personal_email non-null → first WHEN matches → COALESCE returns b@h.com → email: b@h.com.
Row 3 — both emails null → second WHEN matches → phone: 5553333.
Row 4 — all three null → ELSE → no contact.

Output:

user_id	effective_contact
1	email: a@w.com
2	email: b@h.com
3	phone: 5553333
4	no contact

Why this works — concept by concept:

COALESCE for null-picker — collapses work_email/personal_email into one value with first-non-NULL priority; clean, standard SQL.
CASE for per-branch transformation — different string prefixes per channel; COALESCE cannot express this because every branch needs a different formatting rule.
Composition — COALESCE nested inside CASE is idiomatic; the outer CASE switches behavior, the inner COALESCE resolves null preference.
Guaranteed-non-NULL ELSE — the ELSE 'no contact' ensures the output column is never NULL regardless of input.
O(N) time / O(1) space — single linear pass; per-row constant work for both the CASE and the COALESCE.

Inline CTA: More SQL CASE-when problems and conditional-logic problems for the dialect-portability muscle.

SQL
Topic — case when
SQL CASE-when problems

Practice →

SQL
Topic — conditional logic
SQL conditional-logic problems

Practice →

SQL
Topic — null handling
SQL null-handling problems

Practice →

4. `COALESCE` Pitfalls — NULL Semantics, Type Coercion, Empty Strings, and `NULLIF`

Avoiding silent semantic bugs in COALESCE-heavy SQL for data engineering

"Where does COALESCE quietly produce wrong answers?" is the question senior interviewers probe to separate fluent candidates from rote memorizers. The mental model: COALESCE is purely mechanical — first non-NULL — but the meaning of NULL is not; defaulting NULL to 0 for a financial KPI changes the metric semantics; empty string '' is not NULL so COALESCE skips it; type mixing without CAST is a coercion landmine; combining COALESCE with NULLIF produces the "empty-or-null → default" idiom. Drill the four pitfalls below and you'll never ship a COALESCE bug to production.

Pro tip: State the NULL semantics out loud before writing the SQL. If NULL means "unknown," replacing it with 0 may turn a missing data point into a real-looking zero. The interviewer wants to hear "I'm coalescing to 0 because the metric definition says missing rentals count as zero rentals" — not "I added COALESCE to make the dashboard not show NULL."

`NULL` ≠ `0` — semantic awareness for KPIs and metrics

The semantic invariant: NULL typically means "unknown"; 0 is a concrete numeric fact; replacing NULL with 0 in display code is usually fine but in calculation code can fabricate facts. Treat the substitution as a documented business decision, not a code-style preference.

Display — COALESCE(amount, 0) is fine in dashboards (the user sees 0, knows it's a zero).
Aggregate — AVG(COALESCE(amount, 0)) changes the average semantically; AVG ignores NULL natively, so coalescing first lowers the average.
Math — COALESCE(qty, 0) * price may be wrong if qty IS NULL means "we don't know how much was ordered."
Documented sentinel — for metrics, prefer carrying the NULL and handling it at the report layer.

Worked example. Average of [100, 200, NULL].

approach	computation	average
`AVG(amount)`	`(100+200)/2`	150
`AVG(COALESCE(amount, 0))`	`(100+200+0)/3`	100

Worked-example solution.

-- Choose intentionally; both are valid for different metric definitions
SELECT AVG(amount)                       AS avg_known_only,
       AVG(COALESCE(amount, 0))          AS avg_with_nulls_as_zero,
       COUNT(*)                          AS total_rows,
       COUNT(amount)                     AS non_null_rows
FROM orders;

Rule of thumb: the NULL-vs-0 decision is a metric-definition decision, not a coding style; document it inline.

Empty strings are not `NULL` — combine with `NULLIF`

The empty-string invariant: in PostgreSQL and most engines, '' and NULL are different; COALESCE(col, 'default') returns '' when col = '' (because '' is non-NULL); to treat empty-as-null, wrap with NULLIF(col, '') first.

NULLIF(a, b) — returns NULL when a = b, else returns a.
COALESCE(NULLIF(col, ''), 'default') — the canonical "empty-or-null → default" idiom.
Oracle quirk — Oracle's varchar treats '' AS NULL; not portable.
MySQL behavior — '' is not NULL; same as PostgreSQL.

Worked example. Three rows: real value, empty string, real NULL.

input	`COALESCE(col, 'X')`	`COALESCE(NULLIF(col, ''), 'X')`
`'hello'`	`'hello'`	`'hello'`
`''`	`''`	`'X'`
`NULL`	`'X'`	`'X'`

Worked-example solution.

SELECT raw_value,
       COALESCE(raw_value, 'X')                AS naive_default,
       COALESCE(NULLIF(raw_value, ''), 'X')    AS empty_or_null_default
FROM samples;

Rule of thumb: whenever a string column might be empty AND COALESCE is the right tool, wrap with NULLIF(col, '') to handle both cases.

Type-coercion landmines and explicit `CAST`

The type-coercion invariant: COALESCE returns a single unified type; mixing incompatible types is a parse error in strict engines (PostgreSQL) and a silent coercion in lenient ones (MySQL); CAST keeps intent explicit and review-friendly.

COALESCE(int_col, 'unknown') — error in PostgreSQL.
COALESCE(CAST(int_col AS TEXT), 'unknown') — explicit, portable.
COALESCE(numeric_col, 0) — NUMERIC and INTEGER unify cleanly.
COALESCE(date_col, '1900-01-01'::DATE) — explicit ::DATE cast on the literal.

Worked example. Mixing INTEGER and TEXT cleanly via CAST.

input	naive	with CAST
`qty = 5`	error	`'5'`
`qty = NULL`	error	`'unknown'`

Worked-example solution.

SELECT order_id,
       COALESCE(CAST(qty AS TEXT), 'unknown') AS qty_label
FROM orders;

Rule of thumb: whenever COALESCE mixes families (numeric + string, date + string), insert an explicit CAST on whichever argument needs it; never trust implicit coercion to do the right thing.

Common beginner mistakes

Defaulting NULL to 0 without considering the metric semantics — fabricates zeros into "unknown" rows.
Using COALESCE(col, 'default') for empty strings — '' is not NULL; wrap with NULLIF first.
Mixing INTEGER and TEXT without CAST — parse error in PostgreSQL.
Using COALESCE for string concatenation — that's STRING_AGG / LISTAGG's job.
Forgetting that AVG ignores NULL natively — coalescing first changes the average.

Worked Problem on Empty-or-NULL Default with NULLIF

Given a customers table with a display_name column that is sometimes NULL, sometimes '' (empty string), and sometimes a real value, return a column effective_name that is the real value when present, the literal 'Anonymous' when the input is NULL or empty.

Solution Using `COALESCE(NULLIF(display_name, ''), 'Anonymous')`

SELECT
    customer_id,
    COALESCE(NULLIF(display_name, ''), 'Anonymous') AS effective_name
FROM customers;

Why this works: NULLIF(display_name, '') returns NULL when display_name equals '', else returns display_name unchanged; COALESCE(..., 'Anonymous') then substitutes the literal 'Anonymous' for NULL (which now covers both the original NULL and the converted ''). The composition handles both edge cases in one expression.

Step-by-step trace for four sample rows:

customer_id	display_name	NULLIF(display_name, '')	COALESCE	effective_name
1	`'Alice Lee'`	`'Alice Lee'`	`'Alice Lee'`	`Alice Lee`
2	`''`	`NULL`	`'Anonymous'`	`Anonymous`
3	`NULL`	`NULL`	`'Anonymous'`	`Anonymous`
4	`'Bob B.'`	`'Bob B.'`	`'Bob B.'`	`Bob B.`

Row 1 — real value passes through NULLIF unchanged → COALESCE short-circuits.
Row 2 — empty string converted to NULL by NULLIF → COALESCE substitutes 'Anonymous'.
Row 3 — original NULL passes through NULLIF (NULL = '' evaluates to NULL, not true) → COALESCE substitutes.
Row 4 — real value passes through unchanged.

Output:

customer_id	effective_name
1	Alice Lee
2	Anonymous
3	Anonymous
4	Bob B.

Why this works — concept by concept:

NULLIF(a, b) semantics — returns NULL when a = b, else returns a; converts '' to NULL so COALESCE can treat both as "missing."
Composition with COALESCE — the inner NULLIF normalizes the input, the outer COALESCE applies the default; reads exactly like the business rule "empty-or-null → 'Anonymous'."
No CASE needed — CASE WHEN display_name IS NULL OR display_name = '' THEN 'Anonymous' ELSE display_name END is verbose; the two-function composition is cleaner and equivalent.
Single-column output — effective_name is non-NULL and non-empty by construction; downstream code can treat it as a clean string.
O(N) time / O(1) space — one linear scan; per-row constant work for the NULLIF + COALESCE composition.

Inline CTA: Drill more SQL null-handling problems on PipeCode.

SQL
Topic — null handling
SQL null-handling problems

Practice →

SQL
Topic — conditional logic
SQL conditional-logic problems

Practice →

SQL
Topic — filtering
SQL filtering problems

Practice →

Tips to master COALESCE in SQL

State the rule out loud before writing the query

Every interview answer that touches NULL should open with "left-to-right, first non-NULL, returns NULL only if every argument is NULL." That single sentence demonstrates fluency. Candidates who write COALESCE in the query without naming the rule are graded as memorizers; candidates who name the rule and then write the query are graded as fluent. The rule takes 7 seconds to say; it's worth a level on the interviewer's rubric.

Drill the four primitives

The four primitives in this guide map directly to the surface area COALESCE covers in production: left-to-right evaluation with short-circuit semantics (the rule itself plus the side-effect-safety contract), LEFT JOIN + COALESCE(right_col, default) (the canonical "fact + dim with safe defaults" pattern that drives every BI dashboard), COALESCE vs CASE / ISNULL / NVL / IFNULL (the dialect portability matrix), and the pitfall set (NULL ≠ 0, empty string ≠ NULL, type coercion, NULLIF composition).

Pick PostgreSQL-flavored answers in interviews

Most coding-environment interviews — DataLemur, StrataScratch, LeetCode SQL, CoderPad live coding — default to PostgreSQL. Drill PostgreSQL syntax: COALESCE (not ISNULL), EXTRACT(MONTH FROM ts) (not MONTH(ts)), INTERVAL '30 days' (not DATEADD), || for string concatenation (not +), ::TYPE for casts (not CONVERT(TYPE, ...)).

Combine `COALESCE` with `NULLIF` for empty-or-null

The COALESCE(NULLIF(col, ''), 'default') idiom is the canonical "treat empty string as null and fall back to a default" pattern. Memorize it. PostgreSQL keeps '' and NULL distinct; without the inner NULLIF, COALESCE returns the empty string unchanged. Naming this composition unprompted in interviews is a strong product-fluency signal.

Choose `CASE` when the logic is not "first non-NULL"

COALESCE is for ordered fallbacks. CASE is for everything else — ranges, flags, transformations, branch-specific formatting. If you find yourself writing nested COALESCE calls or COALESCE inside CASE to express conditional logic, the pure CASE is usually clearer. The decision tree: "first non-NULL?" → COALESCE; "anything richer?" → CASE.

Default thoughtfully — `NULL` carries information

A NULL in a financial KPI is information ("we don't know"); replacing it with 0 is a documented business decision. State the decision in the SQL comment or in the surrounding pull-request description. A senior interviewer wants to hear "I'm coalescing to 0 because the metric definition says missing rentals count as zero" — not "I added COALESCE so the dashboard wouldn't show NULL."

Where to practice on PipeCode

Start with the SQL null-handling practice page for the curated set of COALESCE-style problems. After that, drill the matching topic pages: conditional logic, CASE when, joins, aggregation, filtering, window functions, CTE. The SQL for data engineering interviews course bundles structured curricula. For broader coverage, browse all practice problems or pivot to peer guides — the Airbnb DE interview guide, the top DE interview questions 2026 blog, and the SQL data types Postgres guide.

Communication and approach under time pressure

Talk through the rule first ("left-to-right, first non-NULL"), the obvious application second ("here that means prefer work_email then personal_email then a literal default"), and the edge cases third ("if every column is NULL, the literal kicks in; if NULL could mean 0 semantically, I'd think twice about defaulting to 0"). Interviewers grade process as much as the final query. Leave 30 seconds for a sweep: empty string vs NULL, type mixing, side-effect safety on later arguments, the LEFT JOIN-becomes-INNER trap if a WHERE filters the right table.

Frequently Asked Questions

Is `COALESCE` the same as `IFNULL`?

Not exactly. IFNULL (MySQL, BigQuery) and ISNULL (SQL Server) are usually two-argument forms. COALESCE accepts many arguments and is part of the SQL:1992 standard, which is why teams prefer it in portable analytics code. PipeCode's SQL practice is PostgreSQL-oriented, so canonical solutions use COALESCE everywhere. If you're working in a MySQL-only codebase that already uses IFNULL, you can keep it; if you're writing portable code or migrating between engines, switch to COALESCE.

Does `COALESCE` skip evaluating later arguments?

Most major databases short-circuit once they find a non-NULL value — PostgreSQL, MySQL, SQL Server, Oracle, and Snowflake all document this behavior. That's useful for performance, but the SQL standard does not require short-circuit, and you should not rely on side effects in later arguments (volatile functions, RAISE statements, sequence calls). In interviews, saying "typically short-circuits, but I would not depend on side effects" is a strong answer.

Can I nest `COALESCE`?

Yes — COALESCE(a, b, COALESCE(c, d)) works — but a flat list is easier to read and review. Prefer COALESCE(a, b, c, d) when all fallbacks are simple columns or literals. The only legitimate reason to nest is when one fallback itself depends on another expression that needs its own fallback chain.

When should I use `CASE` instead of `COALESCE`?

Use CASE when the logic is not "first non-NULL" — for example, ranges (amount > 1000 → 'large'), flags (is_premium → premium pricing), or different transformations per branch. COALESCE stays the right default when you only need ordered fallbacks. For more conditional patterns, browse conditional-logic SQL on PipeCode and CASE-when problems.

How do I handle empty strings vs `NULL` together?

Use the COALESCE(NULLIF(col, ''), 'default') composition. NULLIF(col, '') returns NULL when col is empty (and returns col unchanged otherwise); the outer COALESCE then treats both the original NULL and the converted-empty as missing and applies the default. This is the canonical PostgreSQL idiom for "missing or blank → default" — drill it; it shows up at every interview that touches dirty user-input data.

What's the right default value for `COALESCE`?

The right default depends on the metric definition, not on coding style. For displays and dashboards, 0 for numerics and 'NONE' / 'Unknown' for strings are common. For aggregates, COALESCE(SUM(x), 0) keeps empty groups visible in reports. For financial or scientific metrics where NULL means "unknown," carry the NULL and handle it at the report layer instead of replacing it. Document the decision inline so the next reviewer knows it was intentional.

Start practicing SQL COALESCE problems

Facebook Data Engineering Interview Questions & Prep Guide

Gowtham Potureddi — Sun, 03 May 2026 05:43:11 +0000

Facebook data engineering interview questions are bilingual, product-analytics-flavored, and PostgreSQL-grounded on the SQL side. Facebook Inc. rebranded to Meta Platforms Inc. in October 2021, but the data-engineering interview shape — and the question patterns that show up in the live phone screen and onsite — has not moved. The standard technical phone screen is 5 minutes intro + 30 minutes SQL + 30 minutes Python + 5 minutes Q&A, with the candidate choosing whether to start with SQL or Python. Four primitives carry the loop: n*(n+1)/2 - sum(arr) arithmetic-series sum-formula (and its XOR self-cancellation alternative) for the missing-number array problem, character-by-character tokenization plus two-pass evaluation for arithmetic formula parsing, correlated EXISTS subqueries with EXTRACT(MONTH FROM ts - INTERVAL '1 month') for month-over-month MAU retention, and CTE composition plus self-joins for post-hiatus aggregation and friend-recommendation queries.

This guide walks four topic clusters end-to-end, each with a detailed topic explanation, per-sub-topic explanation with a worked example and its solution, and an interview-style problem with a full solution that explains why it works. The mix matches a curated 2-problem Facebook set (1 EASY Python array + 1 MEDIUM Python array+math+bit+string parser) plus two adjacent SQL primitives — EXISTS month-over-month retention and CTE + self-join aggregation — that show up on every Meta SQL question list and at every product-analytics onsite. The interview is bilingual SQL + Python; candidates who prep only one language stutter on the half they avoided.

1. Python Array Missing Number — Sum Formula and XOR

Sum-formula and XOR self-cancellation in Python for data engineering

"Given an array of n distinct integers in the range [0, n] with exactly one missing, return the missing number" is Facebook's signature EASY Python prompt (Missing Number). The mental model: the arithmetic-series identity 0 + 1 + … + n = n*(n+1)/2 gives the expected sum; subtracting the actual sum reveals the missing number in O(n) time and O(1) space. The mirror primitive is XOR self-cancellation — a ^ a == 0 and a ^ 0 == a — which gives the same answer using bit operations and never overflows.

Pro tip: State both approaches out loud before writing code. Interviewers grade the candidate's awareness that the sum-formula can overflow on huge n (Python ints are arbitrary precision so it's fine in practice, but the answer in Java or C++ requires the XOR variant). Naming both demonstrates breadth.

Arithmetic-series sum formula: `n*(n+1)/2`

The sum-formula invariant: the sum of integers from 0 to n inclusive is n*(n+1)/2. Subtracting sum(arr) from this expected total yields the missing element when exactly one integer is absent from the contiguous range.

n*(n+1)//2 — integer division in Python; produces the expected total.
sum(nums) — O(n) linear scan; produces the actual total.
expected - actual — the difference is the missing value.
Constant space — no auxiliary structure required.

Worked example. nums = [0, 1, 3, 4], n = 4.

step	value
`n`	4
`n*(n+1)//2`	10
`sum(nums)`	8
`missing`	2

Worked-example solution.

def missing_number(nums: list[int]) -> int:
    n = len(nums)
    return n * (n + 1) // 2 - sum(nums)

Rule of thumb: the sum-formula is the cleanest one-liner in Python; reach for it whenever the prompt guarantees a contiguous integer range.

XOR self-cancellation: `a ^ a == 0`

The XOR invariant: a ^ a == 0 and a ^ 0 == a; XOR is commutative and associative so the order does not matter. XORing every element of nums with every element of range(n + 1) cancels every value that appears in both, leaving only the one missing from nums.

a ^ a == 0 — self-cancellation property.
a ^ 0 == a — identity property.
Commutative + associative — order-independent.
No overflow — bit operations never overflow in any language.

Worked example. Same nums = [0, 1, 3, 4], n = 4.

step	XOR chain
nums	`0 ^ 1 ^ 3 ^ 4` = 6
range	`0 ^ 1 ^ 2 ^ 3 ^ 4` = 4
combined	`6 ^ 4` = 2

Worked-example solution.

from functools import reduce
from operator import xor

def missing_number_xor(nums: list[int]) -> int:
    n = len(nums)
    return reduce(xor, nums + list(range(n + 1)))

Rule of thumb: the XOR variant is the right answer when the prompt asks about overflow safety or when you want to demonstrate bit-manipulation fluency.

Set-difference fallback for non-contiguous ranges

The set-difference invariant: when the input range is not contiguous (e.g., "find the missing element from [10, 20, 30, 40, 60] knowing the full set should be [10, 20, 30, 40, 50, 60]"), the sum and XOR shortcuts no longer apply; reach for set(expected) - set(actual).

set(expected) - set(actual) — O(n) time, O(n) space.
(set(expected) - set(actual)).pop() — return the single missing element.
Multiple missing elements — the same set-difference returns all of them.
Slower than sum / XOR — uses O(n) extra space; pick this only when the range is non-contiguous.

Worked example. actual = [10, 20, 30, 40, 60], expected = [10, 20, 30, 40, 50, 60].

step	output
`set(expected) - set(actual)`	`{50}`

Worked-example solution.

def missing_from_set(actual: list[int], expected: list[int]) -> int:
    return (set(expected) - set(actual)).pop()

Rule of thumb: contiguous range → sum-formula or XOR (O(1) space); non-contiguous → set-difference (O(n) space).

Common beginner mistakes

Using n * (n + 1) / 2 (float division) instead of n * (n + 1) // 2 (integer division) — produces a float result that fails int-typed tests.
Computing n as max(nums) instead of len(nums) — wrong by one when the missing element is the max itself.
Forgetting that range(n) excludes n — use range(n + 1) for inclusive [0, n].
Sorting nums first — O(n log n) instead of O(n); signals algorithmic weakness.
Returning the difference set instead of a single value — read the contract; the missing-number problem returns one int.

Python Interview Question on Missing Number

Given an array nums containing n distinct integers in the range [0, n], return the single missing number in O(n) time and O(1) space.

def missing_number(nums: list[int]) -> int:
    pass  # your code here

Solution Using `n*(n+1)//2 - sum(nums)`

def missing_number(nums: list[int]) -> int:
    n = len(nums)
    return n * (n + 1) // 2 - sum(nums)

Why this works: len(nums) gives n because the array contains exactly one fewer element than the full range [0, n]; the arithmetic-series identity 0 + 1 + … + n = n*(n+1)/2 produces the expected total; subtracting sum(nums) (the actual total) yields the missing value; the integer-division // keeps the result an int. O(n) time for the single linear sum, O(1) extra space — no auxiliary structure.

Step-by-step trace for nums = [3, 0, 1]:

step	value
`n = len(nums)`	3
expected = `3 * 4 // 2`	6
actual = `sum([3, 0, 1])`	4
missing = `6 - 4`	2

Output:

input	expected	actual	missing
`[3, 0, 1]`	6	4	2

Why this works — concept by concept:

Arithmetic-series identity — 0 + 1 + ... + n = n*(n+1)/2 is a closed-form expression; computing it is O(1) regardless of n.
Linear sum(nums) — O(n) scan of the array; the only data-touching cost in the algorithm.
Integer division // — keeps the expected total as a Python int; n*(n+1) is always even so the division has no remainder.
Subtraction reveals the gap — expected - actual cancels every shared value and leaves the missing one.
O(n) time / O(1) space — single pass over nums, constant extra storage, no recursion or auxiliary collection.

Inline CTA: Drill the Facebook Python practice page for the curated EASY array problem and the Facebook array practice page for the company_topic surface.

PYTHON
Company — Facebook (Python)
Facebook Python practice problems

Practice →

PYTHON
Company — Facebook / array
Facebook array problems

Practice →

PYTHON
Topic — array
Python array problems

Practice →

2. Python Arithmetic Formula Evaluator — Array, Math, Bit, String Parsing

Tokenization and two-pass evaluation in Python for data engineering

"Given a string like '3 + 5 * 2', parse and evaluate it with standard precedence" is Facebook's signature MEDIUM Python prompt (Arithmetic Formula Evaluator). The mental model: a length-1 character-by-character scan tokenizes the string into a flat list of integers and operator characters; a first pass collapses every * and / left-to-right; a second pass collapses the remaining + and - left-to-right; the final list has one element — the result. Same primitive powers any "expression evaluator" pipeline — calculator apps, formula columns in spreadsheets, simple DSL interpreters.

Pro tip: Avoid eval(s) even when the test prompt allows it. Production code never trusts arbitrary strings, and interviewers grade the candidate who writes a real parser. State the no-eval rule out loud before coding; the senior signal is unmistakable.

Tokenization: split a string into numbers and operators

The tokenization invariant: scan the string character-by-character; accumulate digits into an integer buffer, flush the buffer when an operator is hit, append the operator as its own token; whitespace is skipped. The output is a flat list alternating between integers and operator chars.

Digit accumulation — if c.isdigit(): num = num * 10 + int(c).
Operator flush — elif c in '+-*/': tokens.append(num); tokens.append(c); num = 0.
Final flush — append the trailing num after the loop.
Whitespace — skip with if c.isspace(): continue.

Worked example. s = '3 + 5 * 2'.

step	tokens
start	`[]`
`3`	`[3]`
`+`	`[3, '+']`
`5`	`[3, '+', 5]`
`*`	`[3, '+', 5, '*']`
`2`	`[3, '+', 5, '*', 2]`

Worked-example solution.

def tokenize(s: str) -> list:
    tokens, num, in_num = [], 0, False
    for c in s:
        if c.isspace():
            continue
        if c.isdigit():
            num = num * 10 + int(c)
            in_num = True
        else:
            if in_num:
                tokens.append(num)
                num, in_num = 0, False
            tokens.append(c)
    if in_num:
        tokens.append(num)
    return tokens

Rule of thumb: always handle multi-digit numbers explicitly — '12 + 3' is two operands, not three; the digit accumulator is what makes that work.

Two-pass evaluation: handle `*` and `/` first, then `+` and `-`

The precedence invariant: the first pass walks the token list, finds every * and /, and replaces the triple (left, op, right) with (left op right); the second pass does the same for + and -. After the first pass, the token list contains only + and - operators between integers — left-to-right evaluation produces the result.

First pass — handle * and / (higher precedence).
Second pass — handle + and - (lower precedence).
Left-to-right within precedence — preserves the standard expression semantics.
Integer division — // for / if the contract is integer arithmetic; int(a / b) for truncate-toward-zero.

Worked example. Continuing tokens = [3, '+', 5, '*', 2].

pass	tokens
input	`[3, '+', 5, '*', 2]`
pass 1 (`*`)	`[3, '+', 10]`
pass 2 (`+`)	`[13]`
result	`13`

Worked-example solution.

def collapse(tokens: list, ops: set) -> list:
    result = [tokens[0]]
    i = 1
    while i < len(tokens):
        op, right = tokens[i], tokens[i + 1]
        if op in ops:
            left = result.pop()
            if op == '*':
                result.append(left * right)
            elif op == '/':
                result.append(left // right)
            elif op == '+':
                result.append(left + right)
            else:  # '-'
                result.append(left - right)
        else:
            result.append(op)
            result.append(right)
        i += 2
    return result

Rule of thumb: two passes is the simplest correct approach; reach for the stack-based shunting-yard variant only when parens or unary minus enter the spec.

Stack-based evaluator for general expressions

The stack invariant: maintain a stack of partial results; when scanning a token, the operator decides whether to update the top of the stack (* or /) or to push a new term (+ or -). The final answer is sum(stack).

+ — push the next operand onto the stack.
- — push the negated next operand.
* — multiply the top of stack by the next operand in place.
/ — integer-divide the top of stack by the next operand in place.

Worked example. s = '3+5*2', single-pass with stack.

token	stack
`3`	`[3]`
`+ 5`	`[3, 5]`
`* 2`	`[3, 10]`
end	sum = 13

Worked-example solution.

def calculate(s: str) -> int:
    s = s.replace(' ', '')
    stack, num, op = [], 0, '+'
    for i, c in enumerate(s):
        if c.isdigit():
            num = num * 10 + int(c)
        if c in '+-*/' or i == len(s) - 1:
            if op == '+': stack.append(num)
            elif op == '-': stack.append(-num)
            elif op == '*': stack.append(stack.pop() * num)
            else: stack.append(int(stack.pop() / num))
            op, num = c, 0
    return sum(stack)

Rule of thumb: the stack-based single-pass evaluator is the production-grade answer; the two-pass variant is the easier explanation but allocates more.

Common beginner mistakes

Using eval(s) — works on the test cases but is graded as a fail; production code never evals untrusted input.
Forgetting multi-digit numbers — '12 + 3' becomes [1, 2, '+', 3] instead of [12, '+', 3].
Doing left-to-right without precedence — 3 + 5 * 2 becomes (3 + 5) * 2 = 16 instead of 3 + (5 * 2) = 13.
Using int(a / b) when the contract says // — produces wrong rounding for negative integers.
Skipping the trailing-number flush — the last operand never enters the token list.

Python Interview Question on Arithmetic Formula Evaluator

Given a string s containing integer operands and the operators + - * / (no parentheses, integer arithmetic, standard precedence), return the integer result.

def calculate(s: str) -> int:
    pass  # your code here

Solution Using single-pass stack-based evaluator

def calculate(s: str) -> int:
    s = s.replace(' ', '')
    stack, num, op = [], 0, '+'
    for i, c in enumerate(s):
        if c.isdigit():
            num = num * 10 + int(c)
        if c in '+-*/' or i == len(s) - 1:
            if op == '+':
                stack.append(num)
            elif op == '-':
                stack.append(-num)
            elif op == '*':
                stack.append(stack.pop() * num)
            else:  # '/'
                stack.append(int(stack.pop() / num))
            op, num = c, 0
    return sum(stack)

Why this works: the single pass tokenizes and evaluates simultaneously; op tracks the operator that applies to the just-finished number; on +/- we push a new term onto the stack; on */// we update the top of stack in place — which is exactly the precedence-respecting behavior we need; the trailing flush (i == len(s) - 1) handles the last number; sum(stack) collapses the additive sub-results.

Step-by-step trace for s = '3+5*2':

i	c	num	op_pending	stack
0	`3`	3	`+`	`[]`
1	`+`	0	`+`	`[3]`
2	`5`	5	`+`	`[3]`
3	`*`	0	`*`	`[3, 5]`
4	`2`	2	`*`	`[3, 5]`
end	—	0	—	`[3, 10]`

sum(stack) = 3 + 10 = 13.

Output:

input	result
`'3+5*2'`	13
`'12-3*2'`	6
`'10/3'`	3

Why this works — concept by concept:

Single-pass tokenize-and-evaluate — one scan of the string drives both tokenization and stack updates; no separate token list materialized.
op tracks the previous operator — applies to the number that just finished accumulating, which is the key trick for the stack-update timing.
Stack push for + and - — additive operations push new terms; the final sum collapses them.
In-place stack update for * and / — multiplicative operations modify the top of stack so they bind tighter than additive ones; this is what gives precedence without a second pass.
Trailing flush via i == len(s) - 1 — the last operand has no following operator to trigger a flush; the index check forces the final stack update.
O(n) time / O(n) space — one pass over the string; the stack holds at most one element per + or - token, which is O(n) worst case.

Inline CTA: More SQL string problems and array problems for breadth.

PYTHON
Company — Facebook (Python)
Facebook Python practice problems

Practice →

PYTHON
Topic — string
Python string problems

Practice →

PYTHON
Topic — array
Python array problems

Practice →

3. SQL Window Functions and `EXISTS` for Monthly Active User Retention

Correlated EXISTS subquery for month-over-month MAU retention in SQL for data engineering

"Return the number of monthly active users in July 2022 — users who were active in both July AND June" is Meta's signature SQL retention prompt (DataLemur Q4). The mental model: a user is a July-MAU iff they have at least one event in July AND at least one event in June; a correlated EXISTS subquery checks the second condition row-by-row against the same user_actions table; EXTRACT(MONTH FROM curr_month.event_date - INTERVAL '1 month') shifts the comparison window. Same primitive powers any "active in current period AND in previous period" retention metric — week-over-week active users, day-over-day session retention, cohort-N-day return.

Pro tip: Correlated EXISTS subqueries are graded as the right answer over self-joins for retention queries. Self-joins explode the row count when a user has many events; EXISTS short-circuits on the first match per outer row. State this performance distinction to the interviewer.

`EXTRACT(MONTH FROM ts)` and `INTERVAL '1 month'` arithmetic

The date-arithmetic invariant: EXTRACT(MONTH FROM ts) returns the month component of ts as an integer 1-12; ts - INTERVAL '1 month' shifts ts back exactly one calendar month respecting end-of-month edges. Combining both produces "month of one month ago" — the comparison key for retention.

EXTRACT(MONTH FROM ts) — month number 1-12.
EXTRACT(YEAR FROM ts) — year number; combine with month for unique periods.
ts - INTERVAL '1 month' — calendar shift, handles 31-day → 30-day automatically.
DATE_TRUNC('month', ts) — alternative; returns first day of the month as a DATE.

Worked example. event_date = '2022-07-15'.

expression	output
`EXTRACT(MONTH FROM event_date)`	7
`event_date - INTERVAL '1 month'`	2022-06-15
`EXTRACT(MONTH FROM event_date - INTERVAL '1 month')`	6

Worked-example solution.

SELECT EXTRACT(MONTH FROM event_date)                            AS curr_month,
       EXTRACT(MONTH FROM event_date - INTERVAL '1 month')       AS prev_month
FROM user_actions
WHERE event_date = '2022-07-15';

Rule of thumb: combine EXTRACT(MONTH FROM ...) and EXTRACT(YEAR FROM ...) when retention spans calendar years; month-only is wrong for Dec→Jan transitions.

Correlated `EXISTS` subquery for previous-month presence

The correlated-subquery invariant: WHERE EXISTS (SELECT 1 FROM ... WHERE inner.col = outer.col) returns TRUE for outer rows whose col value has at least one matching row in the inner query; the inner query references the outer alias and re-evaluates per outer row. Short-circuits on the first match.

EXISTS (SELECT 1 FROM ... WHERE ...) — the canonical pattern.
Correlation — inner WHERE clause references outer alias.
Short-circuit — stops scanning inner rows on first match.
NOT EXISTS — mirror image; "no match found in inner."

Worked example. Two users; user 445 has June + July rows, user 742 has only July.

user_id	event_date	EXISTS June row?
445	2022-06-30	(no — this IS June)
445	2022-07-05	✓ (matches user 445's June row)
742	2022-07-03	✗

Worked-example solution.

SELECT user_id
FROM user_actions AS curr_month
WHERE EXTRACT(MONTH FROM event_date) = 7
  AND EXTRACT(YEAR  FROM event_date) = 2022
  AND EXISTS (
      SELECT 1
      FROM user_actions AS last_month
      WHERE last_month.user_id = curr_month.user_id
        AND EXTRACT(MONTH FROM last_month.event_date) = 6
        AND EXTRACT(YEAR  FROM last_month.event_date) = 2022
  );

Rule of thumb: EXISTS beats self-join for retention because it short-circuits per outer row and never explodes the cardinality.

Alternative: window function with self-join over month buckets

The window-alternative invariant: a self-join on user_id between two pre-bucketed tables (or LAG/LEAD over month-truncated dates inside a partition) achieves the same retention answer with different performance trade-offs. Useful when the prompt asks for a continuous N-month-streak metric.

DATE_TRUNC('month', event_date) — bucket per month.
LAG(month_bucket) OVER (PARTITION BY user_id ORDER BY month_bucket) — previous month bucket per user.
Self-join — JOIN user_actions u2 ON u2.user_id = u1.user_id AND u2.month_bucket = u1.month_bucket - INTERVAL '1 month'.
Window approach scales better — single sort, no nested scan.

Worked example. Same data; window approach.

user_id	curr_month	prev_month_bucket	retained?
445	2022-07	2022-06	✓
742	2022-07	NULL	✗

Worked-example solution.

WITH per_user_months AS (
  SELECT DISTINCT user_id, DATE_TRUNC('month', event_date) AS month_bucket
  FROM user_actions
)
SELECT user_id
FROM per_user_months curr
JOIN per_user_months prev
  ON prev.user_id    = curr.user_id
 AND prev.month_bucket = curr.month_bucket - INTERVAL '1 month'
WHERE curr.month_bucket = '2022-07-01';

Rule of thumb: one-month retention → EXISTS; multi-month streak → window functions over DISTINCT user_id, month_bucket rows.

Common beginner mistakes

Comparing EXTRACT(MONTH FROM ts) = 6 without checking the year — June 2021 silently counts as June 2022.
Using a self-join when EXISTS is cleaner — explodes the cardinality.
Forgetting WHERE EXTRACT(MONTH FROM event_date) = 7 on the outer query — counts all-time MAU, not July-specific.
Comparing event_date::date directly without INTERVAL '1 month' — fails on calendar boundaries (30 vs 31 day months, leap years).
Returning all rows instead of distinct user counts — COUNT(DISTINCT user_id) is the metric.

SQL Interview Question on Active User Retention

Given user_actions(user_id, event_id, event_type, event_date), return the count of monthly active users for July 2022 — users who had at least one event in July AND at least one event in June. Output month (numeric) and monthly_active_users.

Solution Using correlated `EXISTS` subquery

SELECT
    EXTRACT(MONTH FROM curr_month.event_date) AS mth,
    COUNT(DISTINCT curr_month.user_id)        AS monthly_active_users
FROM user_actions AS curr_month
WHERE EXTRACT(MONTH FROM curr_month.event_date) = 7
  AND EXTRACT(YEAR  FROM curr_month.event_date) = 2022
  AND EXISTS (
      SELECT 1
      FROM user_actions AS last_month
      WHERE last_month.user_id = curr_month.user_id
        AND EXTRACT(MONTH FROM last_month.event_date)
            = EXTRACT(MONTH FROM curr_month.event_date - INTERVAL '1 month')
        AND EXTRACT(YEAR FROM last_month.event_date)
            = EXTRACT(YEAR FROM curr_month.event_date - INTERVAL '1 month')
  )
GROUP BY EXTRACT(MONTH FROM curr_month.event_date);

Why this works: the outer query restricts to July 2022 events; the correlated EXISTS short-circuits on the first June-2022 event for the same user_id; INTERVAL '1 month' makes the comparison work across year boundaries (Jan retention against December of prior year); COUNT(DISTINCT user_id) collapses multiple July events per user to a single MAU contribution; the EXTRACT(MONTH ...) GROUP BY is mechanically required by the SELECT alias.

Step-by-step trace for the DataLemur sample:

user_id	event_date	event_type	curr_month?	EXISTS prev?	counted?
445	2022-06-30	sign-in	✗ (June)	—	—
742	2022-07-03	sign-in	✓	✗	—
445	2022-07-05	like	✓	✓	✓
742	2022-07-05	comment	✓	✗	—
648	2022-07-18	like	✓	✗	—

Only user 445 satisfies both conditions → MAU = 1.

Output:

mth	monthly_active_users
7	1

Why this works — concept by concept:

EXTRACT(MONTH FROM ts) + EXTRACT(YEAR FROM ts) — joint period-key avoids cross-year false positives.
Correlated EXISTS — short-circuits on first match per outer row; O(N · M) worst case but typically far less in practice.
INTERVAL '1 month' shift — calendar-aware month subtraction; handles end-of-month edges and year transitions automatically.
COUNT(DISTINCT user_id) — collapses multiple July events per user; the contract demands "users", not "events."
GROUP BY EXTRACT(MONTH FROM ...) — required because the SELECT references a non-aggregate; produces one output row per month.
O(|user_actions| × log|user_actions|) time — index lookup per outer row inside the EXISTS; with a (user_id, event_date) index this is near-linear.

Inline CTA: More SQL window-function problems and aggregation problems on PipeCode.

SQL
Company — Facebook
Facebook SQL practice problems

Practice →

SQL
Topic — window functions
SQL window-function problems

Practice →

SQL
Topic — aggregation
SQL aggregation problems

Practice →

4. SQL CTE and Self-Join for Post Hiatus and Friend Recommendations

CTE composition with MIN/MAX aggregates and self-joins in SQL for data engineering

"For each user who posted at least twice in 2024, return the days between their first and last post" is Meta's signature post-hiatus prompt (DataLemur Q1). The mental model: MAX(post_date) - MIN(post_date) per user gives the hiatus; WHERE EXTRACT(YEAR FROM post_date) = 2024 filters to the year; HAVING COUNT(post_id) > 1 ensures the user actually posted multiple times. The same CTE-and-self-join skeleton scales up to the friend-recommendation pattern (Q6) — a CTE captures private_events and a self-join produces non-friend pairs who attended the same events.

Pro tip: Always alias the result of date subtraction (AS days_between for post hiatus, AS user_pair for friend-rec). PostgreSQL returns an INTERVAL from date subtraction by default — explicit casting (::DATE - ::DATE) returns a plain int. State the cast in the SELECT.

`MIN` / `MAX` aggregates per user with date subtraction

The aggregate-date invariant: MAX(post_date) - MIN(post_date) over a GROUP BY user_id window returns the per-user span of activity; casting to ::DATE first ensures the subtraction returns a plain integer-day count, not an INTERVAL. Filter to the year first via WHERE, group second.

MIN(post_date::DATE) — earliest post date per group.
MAX(post_date::DATE) — latest post date per group.
MAX - MIN — INTERVAL if both are timestamps, int if both are DATE.
EXTRACT(DAY FROM ...) cast — alternative if working with timestamps.

Worked example. Two users, six posts in 2024.

user_id	post_dates	min	max	days_between
151652	07/10, 07/12	07/10	07/12	2
661093	07/08, 07/29	07/08	07/29	21

Worked-example solution.

SELECT user_id,
       MAX(post_date::DATE) - MIN(post_date::DATE) AS days_between
FROM posts
WHERE EXTRACT(YEAR FROM post_date) = 2024
GROUP BY user_id;

Rule of thumb: when the prompt says "days between first and last", MAX(::DATE) - MIN(::DATE) is one line; never compute it via window functions when GROUP BY suffices.

`HAVING COUNT() > 1` for multi-event users

The filter-on-aggregate invariant: HAVING COUNT(post_id) > 1 filters group rows after the GROUP BY to keep only users with at least 2 posts; WHERE cannot reference aggregates and would parse-error. The two clauses are not interchangeable.

WHERE — row-level filter (year predicate).
HAVING — group-level filter (count predicate).
HAVING COUNT(post_id) > 1 — strictly more than 1.
HAVING COUNT(*) >= 2 — equivalent here since post_id is non-null.

Worked example. Three users; one posted only once in 2024.

user_id	post_count_2024	survives?
151652	2	✓
661093	2	✓
004239	1	✗ (filtered)

Worked-example solution.

SELECT user_id,
       MAX(post_date::DATE) - MIN(post_date::DATE) AS days_between
FROM posts
WHERE EXTRACT(YEAR FROM post_date) = 2024
GROUP BY user_id
HAVING COUNT(post_id) > 1;

Rule of thumb: a single-row hiatus is 0 days, which is misleading; the HAVING COUNT > 1 filter is non-negotiable for the post-hiatus prompt.

CTE composition for multi-step logic + self-join for friend-rec / pair queries

The CTE-and-self-join invariant: WITH cte AS (SELECT ...) names an intermediate result; subsequent SELECTs reference it like a table; for pair queries (e.g., friend recommendations), self-join the same CTE on a not-equal-id condition to generate every ordered pair. Combine with friendship-status filtering to surface non-friend pairs.

CTE definition — WITH private_events AS (SELECT user_id, event_id FROM event_rsvp WHERE attendance_status IN ('going', 'maybe') AND event_type = 'private').
Self-join — JOIN private_events e2 ON e1.event_id = e2.event_id AND e1.user_id != e2.user_id.
Friendship-status filter — JOIN friendship_status fs ... WHERE fs.status = 'not_friends'.
Pair count — HAVING COUNT(*) >= 2 for the "two-or-more shared events" requirement.

Worked example. Three users (111, 222, 333) attended event 234; only (222, 333) are not friends.

pair	shared_events	friend?	recommend?
(111, 222)	1	friends	✗
(111, 333)	1	not_friends	✗ (only 1 shared)
(222, 333)	2	not_friends	✓

Worked-example solution.

WITH private_events AS (
  SELECT user_id, event_id
  FROM event_rsvp
  WHERE attendance_status IN ('going', 'maybe')
    AND event_type = 'private'
)
SELECT fs.user_a_id, fs.user_b_id
FROM private_events e1
JOIN private_events e2
  ON e1.event_id = e2.event_id
 AND e1.user_id != e2.user_id
JOIN friendship_status fs
  ON fs.user_a_id = e1.user_id
 AND fs.user_b_id = e2.user_id
WHERE fs.status = 'not_friends'
GROUP BY fs.user_a_id, fs.user_b_id
HAVING COUNT(*) >= 2;

Rule of thumb: CTEs make multi-step logic readable; self-joins on the same CTE are the canonical shape for pair / recommendation queries.

Common beginner mistakes

Computing MAX - MIN without ::DATE cast — returns an INTERVAL, not days.
Using WHERE COUNT(post_id) > 1 — parse error; WHERE cannot reference aggregates.
Forgetting the year filter — counts all-time hiatus instead of 2024-only.
Self-joining without e1.user_id != e2.user_id — generates (user, user) pairs that pollute the answer.
Returning a single direction (user_a, user_b) when the prompt says recommendations are bidirectional — add the mirror by UNION or by including both orderings in the GROUP BY.

SQL Interview Question on Average Post Hiatus

Given posts(user_id, post_id, post_date), for each user who posted at least twice in 2024, return the number of days between the user's first and last post in 2024. Output user_id and days_between.

Solution Using `MAX - MIN` per user with `HAVING COUNT > 1`

SELECT
    user_id,
    MAX(post_date::DATE) - MIN(post_date::DATE) AS days_between
FROM posts
WHERE EXTRACT(YEAR FROM post_date) = 2024
GROUP BY user_id
HAVING COUNT(post_id) > 1;

Why this works: the WHERE clause restricts the row stream to 2024 posts before grouping; GROUP BY user_id collapses to one row per user; MAX(post_date::DATE) - MIN(post_date::DATE) produces the hiatus as an integer day count thanks to the ::DATE cast; HAVING COUNT(post_id) > 1 strips out users who posted only once. Single-pass aggregation; no self-join needed.

Step-by-step trace for the DataLemur sample:

user_id	post_id	post_date
151652	599415	2024-07-10
661093	624356	2024-07-29
004239	784254	2024-07-04
661093	442560	2024-07-08
151652	111766	2024-07-12

WHERE filter — all 5 rows are in 2024; nothing dropped.
GROUP BY user_id — three groups: 151652 (2 posts), 661093 (2 posts), 004239 (1 post).
MIN / MAX per group — 151652 → (07-10, 07-12); 661093 → (07-08, 07-29); 004239 → (07-04, 07-04).
MAX - MIN — 2 days, 21 days, 0 days.
HAVING COUNT > 1 — strips 004239 (only 1 post).

Output:

user_id	days_between
151652	2
661093	21

Why this works — concept by concept:

WHERE EXTRACT(YEAR FROM post_date) = 2024 — row-level filter; runs before grouping for correctness and performance.
GROUP BY user_id — collapses to one row per user.
MAX(post_date::DATE) - MIN(post_date::DATE) — date subtraction returns an integer day count after the ::DATE cast; INTERVAL would be the wrong shape.
HAVING COUNT(post_id) > 1 — group-level filter; aggregate predicate must live in HAVING, not WHERE.
O(|posts| + G log G) time — single scan + sort of G groups; no self-join.

Inline CTA: More SQL CTE problems and SQL join problems on PipeCode.

SQL
Company — Facebook
Facebook SQL practice problems

Practice →

SQL
Topic — CTE
SQL CTE problems

Practice →

SQL
Topic — joins
SQL join problems

Practice →

Tips to crack Facebook data engineering interviews

Facebook = Meta — the rebrand survived but the SQL bar didn't move

Facebook Inc. rebranded to Meta Platforms Inc. in October 2021. The legal entity is Meta; the consumer products (Facebook, Instagram, WhatsApp, Messenger, Threads) keep their original brands. The data-engineering interview loop, the SQL bar, and the question shapes did not change with the rebrand. Search for both "Facebook data engineer interview" and "Meta data engineer interview" — every external article you find under one name applies under the other.

30+30 phone-screen format — pick your opener

The standard Meta data-engineering technical phone screen is 5 minutes intro + 30 minutes SQL + 30 minutes Python + 5 minutes Q&A, with the candidate choosing whether to open with SQL or Python. Pick whichever you'd rather attack first when you're freshest; the second half always feels harder under fatigue, so save your stronger language for the closer if you can. State your preferred opener at the start so the interviewer can plan.

Two role variants — pick the one that matches your loop

Meta has two distinct DE loops: the standard "Data Engineer" role (algo-heavy Python plus SQL) and the "Data Engineer — Product Analytics" role (5 SQL questions plus 5 algo coding questions, more product-sense flavor). Confirm which loop you're in during the recruiter call; the prep mix shifts — Product Analytics candidates drill more SQL retention / cohort patterns; standard DE candidates drill more Python algorithms.

Drill the four primitives

The four primitives in this guide map directly to the curated 2 PipeCode Python problems plus the two adjacent SQL primitives every Meta SQL list rotates through: n*(n+1)/2 - sum(arr) and XOR self-cancellation for the missing-number array problem (Python EASY, #83), two-pass tokenize-and-evaluate or single-pass stack-based evaluator for arithmetic formula parsing (Python MEDIUM, #273), correlated EXISTS with INTERVAL '1 month' for month-over-month MAU retention (DataLemur Q4), and CTE composition + self-joins with MIN/MAX aggregates and HAVING COUNT > 1 for post hiatus and friend recommendations (DataLemur Q1 / Q6).

Product-analytics SQL emphasis

Meta's data-engineering SQL questions are heavily product-analytics-flavored — MAU retention, post hiatus, power users, click-through rates, friend recommendations. Drill the Meta-style SQL practice surface with a focus on EXISTS subqueries, CTE composition, MIN/MAX per-user aggregates, and CASE WHEN + ROUND for percentage metrics. Avoid generic "joins and group-by" prep; Meta's bar is higher.

IC4-IC6 leveling and behavioral expectations

Meta levels DE roles from IC4 (Senior, ~5+ YoE) through IC6 (Staff, ~10+ YoE) and IC7 (Senior Staff). Behavioral rounds at IC5+ probe ownership ("tell me about a time you owned a critical pipeline migration"), navigating ambiguity, and cross-functional collaboration with product / DS / ML teams. Have STAR-format stories ready for each. Compensation per Levels.fyi: IC4 ~$340K-$430K total, IC5 ~$430K-$580K, IC6 $600K+.

Where to practice on PipeCode

Start with the Facebook practice page and the language-scoped Facebook Python practice page for the curated 2-problem set. Hit the company_topic Facebook — array page for the only Facebook-tagged topic surface available. After that, drill the matching topic pages: array, string, window functions, aggregation, CTE, joins, date functions, filtering. The interview courses page bundles structured curricula. For broader coverage, browse by topic, or pivot to peer guides — the Airbnb DE interview guide, the top DE interview questions 2026 blog, and the SQL data types Postgres guide.

Communication and approach under time pressure

Talk through the invariant first ("this is a missing-number problem with sum-formula and XOR alternatives"), the brute force second ("a sort-then-scan would also work but is O(n log n)"), and the optimal third ("but the sum-formula gives O(n) time and O(1) space"). Interviewers grade process as much as the final answer. Leave 5 minutes for an edge-case sweep: empty input, single-element array, zero-only array, year-boundary date arithmetic, NULL in a user_id partition. The most common "almost passed" failure mode is correct happy-path code that crashes on edge cases — a 30-second sweep prevents it.

Frequently Asked Questions

What is the Facebook (Meta) data engineering interview process?

The Meta data engineering interview opens with a recruiter screen (15-30 min), then a technical phone screen with the 5 minute intro + 30 minute SQL + 30 minute Python + 5 minute Q&A format (candidate-choice on opener), then a 4-5 round virtual onsite covering 2 SQL rounds, 1 Python algorithm round, 1 system design round, and 1 behavioral round. Senior roles (IC5+) add a data-architecture round. End-to-end the loop runs three to four weeks.

Is Facebook the same as Meta? Which name should I search for?

Yes — Facebook Inc. rebranded to Meta Platforms Inc. in October 2021. The legal entity is Meta; the consumer brands (Facebook, Instagram, WhatsApp, Messenger, Threads) keep their original names. Search for both "Facebook data engineer interview" and "Meta data engineer interview" — every external article you find under one name applies under the other. The DataLemur SQL guide is titled "Facebook/Meta SQL Interview Questions" because both names index the same content.

What programming languages does Facebook test in data engineering interviews?

Meta tests Python and SQL in the technical phone screen and onsite — bilingual by design. The phone screen is exactly 30 minutes of each, with the candidate choosing the opener. Python emphasizes algorithms (array missing-number, string parsing, hash-table aggregation, sliding-window patterns). SQL emphasizes product-analytics queries (MAU retention, post hiatus, power users, CTR, friend recommendations) on PostgreSQL syntax. Python questions for the Meta data engineer interview lean medium-difficulty algorithm style, not data-pipeline scripting.

What is the difference between the standard DE role and the Data Engineer — Product Analytics variant?

Standard Data Engineer at Meta = algo-heavier Python loops with bilingual SQL coverage (the curated PipeCode set's algorithm focus). Data Engineer — Product Analytics = a product-sense-flavored variant where the technical phone screen is 5 SQL questions + 5 algo coding questions in one session, more weighted toward retention / cohort / funnel analytics SQL than pure algorithm fluency. The recruiter call confirms which loop you're in; ask explicitly if it isn't stated.

What SQL questions does Meta ask data engineers?

Meta SQL interview questions concentrate on six product-analytics shapes: (1) post hiatus / first-and-last-event aggregations via MIN/MAX(::DATE) with HAVING COUNT > 1; (2) power-user identification via JOIN + 2-condition HAVING; (3) MAU retention via correlated EXISTS subquery with INTERVAL '1 month' shifts; (4) friend recommendations via CTE + self-join over event_rsvp with friendship_status filtering; (5) average-shares-per-post via LEFT JOIN + COALESCE; (6) ad click-through rate via CASE WHEN + ROUND and 2022-year filters. Drill all six against the PostgreSQL dialect; CoderPad is the live coding environment.

What is the Meta data engineer salary range?

Meta data engineer total compensation per Levels.fyi: IC4 (Senior, ~5+ YoE) $340K-$430K total comp ($180K-$220K base + RSUs + bonus); IC5 (Staff, ~10+ YoE) $430K-$580K total ($210K-$260K base); IC6 (Senior Staff, ~12+ YoE) $600K+ total. RSU refreshers are annual; equity vests on a 4-year schedule with a typical 25% cliff. Negotiation success rates run 10-20% with competing offers. The "meta data engineer salary" search keyword (vol 260) reflects strong candidate interest in this number.

Start practicing Facebook data engineering problems

Square Data Engineering Interview Questions & Prep Guide

Gowtham Potureddi — Sun, 03 May 2026 05:21:28 +0000

Square data engineering interview questions are SQL-heavy, fintech-flavored, and PostgreSQL-grounded. Square rebranded to Block Inc. in December 2021, but the SQL bar — and the question shapes that show up in the live CoderPad pair-programming round — has not moved. Four primitives carry the loop: GROUP BY sender_id + COUNT(*) + ORDER BY DESC LIMIT 10 for top-N invoice senders, DATEDIFF / INTERVAL '30 days' cohort math for 30-day-post-signup activity windows, AVG(stars) OVER (PARTITION BY product_id, month) window aggregates for monthly product analytics, and COUNT(DISTINCT col) over status-filtered payment rows for fintech-grade transaction analysis. The framings are everyday Block / Square / CashApp data engineering — invoice ranking, cohort retention, monthly review averages, payment success counts.

This guide walks four SQL topic clusters end-to-end, each with a detailed topic explanation, per-sub-topic explanation with a worked example and its solution, and an interview-style problem with a full solution that explains why it works. The mix matches a curated 2-problem set (1 EASY ranking+sorting+aggregation, 1 MEDIUM aggregation+date-functions+cohort-analysis) plus the two adjacent SQL primitives — window functions and payment-flow COUNT DISTINCT — that show up on every Block / Square / CashApp SQL question list. The interview is PostgreSQL on CoderPad; candidates who prep in MySQL / Snowflake / BigQuery dialect stutter on INTERVAL, DATE_PART, and EXTRACT syntax. Drill PostgreSQL-flavored answers from the start.

1. SQL Ranking, Sorting, and Aggregation for Top-N Invoice Senders

Top-N entity ranking via GROUP BY + ORDER BY DESC + LIMIT in SQL for data engineering

"Return the top 10 senders by completed invoice count" is Square's signature EASY SQL prompt (Top 10 Invoice Senders). The mental model: GROUP BY sender_id collapses one row per sender; COUNT(*) produces the per-sender invoice count; ORDER BY count DESC ranks senders descending; LIMIT 10 returns only the top 10. Same primitive powers any "top-N entities by event volume" pipeline — top-N customers by order count, top-N products by review volume, top-N regions by daily active users.

Pro tip: Always alias the count column (AS invoice_count) and sort by the alias in ORDER BY. Mixing literal COUNT(*) and the alias is style noise that interviewers grade. Add a deterministic tiebreaker (, sender_id ASC) when ties at the cut matter — Square's interviewers probe ties at the boundary frequently.

GROUP BY + COUNT(*) for per-sender aggregation

The aggregation invariant: GROUP BY sender_id collapses all invoice rows that share the same sender_id into one output row; COUNT(*) counts the number of rows in each group. Every non-aggregate column in the SELECT must appear in GROUP BY (or be functionally dependent on it).

GROUP BY sender_id — collapses to one row per sender.
COUNT(*) — total rows per group; equivalent to COUNT(invoice_id) when invoice_id is non-null.
COUNT(DISTINCT col) — unique values per group; useful when senders can repeat invoices.
SELECT sender_id, COUNT(*) — only sender_id is referenced ungrouped; the GROUP BY makes this legal.

Worked example. Five invoices, three distinct senders.

step	output
input	`[(101, s1), (102, s2), (103, s1), (104, s3), (105, s1)]`
`GROUP BY sender_id`	`[s1: 3 rows, s2: 1 row, s3: 1 row]`
`COUNT(*) AS cnt`	`[(s1, 3), (s2, 1), (s3, 1)]`

Worked-example solution.

SELECT sender_id, COUNT(*) AS invoice_count
FROM invoices
GROUP BY sender_id;

Rule of thumb: GROUP BY on the dimension you want one-row-per-value, COUNT(*) for the metric, alias the count immediately so subsequent clauses can reference it.

ORDER BY count DESC + LIMIT N for top-N rankings

The top-N invariant: ORDER BY <metric> DESC sorts groups by the metric in descending order; LIMIT N returns only the first N. For ties at the cut, dialects differ — PostgreSQL (Square's CoderPad default) returns whichever order the planner picks unless you add an explicit tiebreaker.

ORDER BY invoice_count DESC — sorts groups by the aliased column.
LIMIT 10 — first 10 groups in sort order; PostgreSQL syntax.
OFFSET 10 LIMIT 10 — pagination; ranks 11-20.
Deterministic tiebreak — , sender_id ASC makes ties stable across runs.

Worked example. Sort 4 senders by invoice_count and take top 3.

sender_id	invoice_count	rank
s1	4	1
s2	2	2
s3	1	3
s4	1	(cut)

Worked-example solution.

SELECT sender_id, COUNT(*) AS invoice_count
FROM invoices
GROUP BY sender_id
ORDER BY invoice_count DESC, sender_id ASC
LIMIT 3;

Rule of thumb: always include a deterministic tiebreaker — fintech audits demand that ranked lists be reproducible across runs.

Tiebreaks via `RANK()`, `DENSE_RANK()`, and `ROW_NUMBER()`

The tiebreak invariant: ROW_NUMBER always gives a unique sequence (1, 2, 3, 4); RANK gives the same number for ties and skips the next (1, 1, 3, 4); DENSE_RANK gives the same number for ties without skipping (1, 1, 2, 3). Choose based on whether the prompt wants strict top-N row count, "all senders tied at rank N", or compact ranks without gaps.

ROW_NUMBER() OVER (ORDER BY count DESC) — strict 1..N sequence; ties resolved by planner.
RANK() OVER (ORDER BY count DESC) — ties share rank, next rank skips.
DENSE_RANK() OVER (ORDER BY count DESC) — ties share rank, next rank does not skip.
LIMIT 10 — chops to top 10 rows after sort; doesn't address ties unless paired with RANK filter.

Worked example. Three senders tied at 5 invoices, one at 3, one at 2.

sender_id	invoice_count	ROW_NUMBER	RANK	DENSE_RANK
s1	5	1	1	1
s2	5	2	1	1
s3	5	3	1	1
s4	3	4	4	2
s5	2	5	5	3

Worked-example solution.

SELECT sender_id, invoice_count,
       ROW_NUMBER() OVER (ORDER BY invoice_count DESC) AS rn,
       RANK()       OVER (ORDER BY invoice_count DESC) AS rk,
       DENSE_RANK() OVER (ORDER BY invoice_count DESC) AS drk
FROM (SELECT sender_id, COUNT(*) AS invoice_count
      FROM invoices GROUP BY sender_id) t;

Rule of thumb: "top 10" with strict cardinality → LIMIT 10; "all senders tied at top 10" → wrap with WHERE rk <= 10; "compact ranks 1..K with no gaps" → DENSE_RANK.

Common beginner mistakes

Forgetting GROUP BY — SELECT sender_id, COUNT(*) without grouping is a parse error in PostgreSQL.
Forgetting ORDER BY ... DESC — top 10 silently becomes "first 10 by planner whim."
Returning more than N rows by skipping LIMIT — graded as a wrong answer even when the top N are correct.
Hardcoding ties to a single row when the prompt says "all senders tied at rank 10" — use RANK filter, not LIMIT.
Mixing COUNT(*) and COUNT(DISTINCT invoice_id) — non-equivalent when invoice_id can repeat (it usually can't, but state the assumption).

SQL Interview Question on Top 10 Invoice Senders

Given an invoices(invoice_id, sender_id, recipient_id, sent_at, amount) table, write a query that returns the top 10 senders by completed invoice count, ordered by count descending and breaking ties by sender_id ascending. Output two columns: sender_id and invoice_count.

Solution Using `GROUP BY` + `COUNT(*)` + `ORDER BY DESC` + `LIMIT 10`

SELECT
    sender_id,
    COUNT(*) AS invoice_count
FROM invoices
GROUP BY sender_id
ORDER BY invoice_count DESC, sender_id ASC
LIMIT 10;

Why this works: GROUP BY sender_id collapses the row stream to one row per sender; COUNT(*) produces the metric for each group; ORDER BY invoice_count DESC ranks groups descending; the , sender_id ASC tiebreak ensures stable output across runs (fintech-audit-grade); LIMIT 10 returns the first 10 ranked rows.

Step-by-step trace for an 8-row sample:

invoice_id	sender_id
101	s1
102	s2
103	s1
104	s3
105	s1
106	s2
107	s4
108	s1

Group by sender_id — four groups: s1 (4 rows), s2 (2 rows), s3 (1 row), s4 (1 row).
COUNT(*) — produces (s1, 4), (s2, 2), (s3, 1), (s4, 1).
Order by invoice_count DESC, sender_id ASC — (s1, 4), (s2, 2), (s3, 1), (s4, 1).
LIMIT 10 — only 4 senders exist; all four returned.

Output:

sender_id	invoice_count
s1	4
s2	2
s3	1
s4	1

Why this works — concept by concept:

GROUP BY sender_id — collapses rows that share sender_id; produces one output row per sender.
COUNT(*) aggregate — counts rows in each group; equivalent to COUNT(invoice_id) when invoice_id is non-null (always true since it's the primary key).
ORDER BY invoice_count DESC ranking — sorts groups descending by the metric; the DESC keyword is the entire difference between top-N and bottom-N.
Deterministic tiebreak — , sender_id ASC ensures the output is reproducible across runs; fintech audits require this.
LIMIT 10 cut — chops to the top 10 ranked rows; PostgreSQL syntax (CoderPad default).
O((|invoices|) + G log G) time — |invoices| rows scanned for the GROUP BY, then O(G log G) to sort the group output where G is the number of senders.

Inline CTA: Drill the Square SQL practice page for the curated EASY problem and the Square aggregation practice page for the only company-tagged topic surface available.

SQL
Company — Square (SQL)
Square SQL practice problems

Practice →

SQL
Company — Square / aggregation
Square aggregation problems

Practice →

SQL
Topic — ranking
SQL ranking problems

Practice →

2. SQL Aggregation, Date Functions, and Cohort Analysis for Post-Signup Activity

30-day cohort retention via DATEDIFF + GROUP BY + HAVING in SQL for data engineering

"Return every user who has at least 5 events in the 30 days after signup" is Square's signature MEDIUM SQL prompt (Users with High Activity After 30 Days Signup). The mental model: the cohort is defined by signup_date; the activity window is signup_date + INTERVAL '30 days'; events filtered to that window are aggregated GROUP BY user_id; HAVING COUNT(*) >= 5 returns the active users. Same primitive powers any retention or cohort analysis — N-day-post-signup activity, post-purchase repeat behavior, post-event engagement windows.

Pro tip: PostgreSQL has multiple ways to express the 30-day boundary: signup_date + INTERVAL '30 days', signup_date + INTERVAL '30 day', (event_at - signup_date) <= INTERVAL '30 days', (event_at::date - signup_date::date) <= 30. Pick one and stick with it. Mixing forms within the same query is a syntax-fluency red flag.

Date arithmetic: `DATEDIFF`, `DATE_TRUNC`, and `INTERVAL '30 days'`

The date-arithmetic invariant: PostgreSQL date math returns either a DATE (when adding INTERVAL) or an INTERVAL (when subtracting two dates); both are comparable to literal intervals. DATEDIFF is not native PostgreSQL — use (d1 - d2)::int or EXTRACT(DAY FROM (d1 - d2)).

signup_date + INTERVAL '30 days' — returns a date 30 days after signup.
(event_at - signup_date) <= INTERVAL '30 days' — interval comparison; works for both DATE and TIMESTAMP.
DATE_TRUNC('week', event_at) — snaps to the week boundary; useful for weekly cohorts.
EXTRACT(DAY FROM (event_at - signup_date)) — pulls the day-count out as an integer.

Worked example. Signup on 2025-01-01; events at 2025-01-10 and 2025-02-15.

event_at	signup_date	diff	within 30 days?
2025-01-10	2025-01-01	9 days	✓
2025-02-15	2025-01-01	45 days	✗

Worked-example solution.

SELECT event_at,
       (event_at::date - signup_date::date) AS days_since_signup,
       (event_at <= signup_date + INTERVAL '30 days') AS within_window
FROM users JOIN events USING (user_id);

Rule of thumb: prefer event_at <= signup_date + INTERVAL '30 days' over (event_at - signup_date) <= INTERVAL '30 days' — the additive form reads as "the event happened within the window" without a subtraction step.

Cohort definition via `signup_date + INTERVAL '30 days'`

The cohort invariant: a cohort is a set of users defined by a shared signup date or signup-date-bucket; the cohort's activity window is that signup-date plus a fixed interval. The cohort filter goes in WHERE, the per-user aggregation happens in GROUP BY, and the threshold check goes in HAVING.

Cohort by exact signup date — WHERE u.signup_date = '2025-01-01'.
Cohort by week — WHERE DATE_TRUNC('week', u.signup_date) = '2025-01-06'.
Cohort by all users — no extra filter; aggregate per user.
Activity window — WHERE e.event_at <= u.signup_date + INTERVAL '30 days'.

Worked example. Two users signed up on different dates; activity-window check per user.

user_id	signup_date	window_end
u1	2025-01-01	2025-01-31
u2	2025-01-15	2025-02-14

Worked-example solution.

SELECT u.user_id, u.signup_date,
       u.signup_date + INTERVAL '30 days' AS window_end
FROM users u;

Rule of thumb: compute the window boundary in SELECT (or a CTE) when you need to reference it multiple times; recomputing signup_date + INTERVAL '30 days' in three different clauses signals copy-paste.

`COUNT(*) per user` with `HAVING` threshold

The threshold invariant: GROUP BY user_id collapses to one row per user; COUNT(*) produces the per-user activity count; HAVING COUNT(*) >= 5 filters groups whose count meets the threshold. HAVING is "WHERE on aggregates" — it filters group rows, not source rows.

HAVING COUNT(*) >= 5 — group-level filter; aggregate predicate.
HAVING COUNT(*) >= 5 AND MAX(event_at) > signup_date + INTERVAL '14 days' — compound threshold.
Avoid WHERE COUNT(*) >= 5 — WHERE cannot reference aggregates; PostgreSQL parser rejects it.
Avoid WHERE to enforce the post-signup window when you want to count zero-event users — outer join + HAVING instead.

Worked example. Three users; threshold 5 events.

user_id	events_in_window	passes?
u1	6	✓
u2	3	✗
u3	5	✓

Worked-example solution.

SELECT u.user_id, COUNT(*) AS events_in_window
FROM users u
JOIN events e ON e.user_id = u.user_id
WHERE e.event_at <= u.signup_date + INTERVAL '30 days'
GROUP BY u.user_id
HAVING COUNT(*) >= 5;

Rule of thumb: row-level predicates → WHERE; aggregate predicates → HAVING. Crossing them is a graded conceptual error.

Common beginner mistakes

Using DATEDIFF(event_at, signup_date) in PostgreSQL — that's MySQL syntax; PostgreSQL uses (d1 - d2)::int or EXTRACT(DAY FROM ...).
Putting the aggregate threshold in WHERE — parser rejects; use HAVING.
Filtering signup_date in HAVING — that's a row predicate; should be in WHERE for performance.
Forgetting the JOIN — querying events alone misses the per-user signup_date anchor.
Missing the INTERVAL '30 days' boundary — counting all-time activity instead of the 30-day window inflates results.
Timezone bugs — event_at and signup_date in different timezones produces off-by-one days; cast both to a single timezone when in doubt.

SQL Interview Question on 30-Day Post-Signup Activity

Given users(user_id, signup_date) and events(event_id, user_id, event_at), return every user_id whose events in the 30 days after signup are at least 5. Output user_id and events_in_window, ordered by events_in_window descending.

Solution Using `JOIN` + `WHERE INTERVAL` + `GROUP BY` + `HAVING COUNT(*) >= 5`

SELECT
    u.user_id,
    COUNT(*) AS events_in_window
FROM users u
JOIN events e
    ON e.user_id = u.user_id
WHERE e.event_at <= u.signup_date + INTERVAL '30 days'
  AND e.event_at >= u.signup_date
GROUP BY u.user_id
HAVING COUNT(*) >= 5
ORDER BY events_in_window DESC;

Why this works: the JOIN pairs each event with its user's signup_date; the WHERE clause restricts events to the 30-day window post-signup (>= signup_date AND <= signup_date + INTERVAL '30 days'); GROUP BY u.user_id collapses to one row per user; COUNT(*) produces the per-user in-window event count; HAVING COUNT(*) >= 5 filters users whose count crosses the threshold; ORDER BY events_in_window DESC ranks active users by engagement.

Step-by-step trace for three users:

user_id	signup_date	events_in_first_30d
u1	2025-01-01	6
u2	2025-01-15	3
u3	2025-02-10	5

JOIN — pairs each event row with its signup_date from users.
WHERE window filter — keeps only events where event_at is in [signup_date, signup_date + 30 days].
GROUP BY user_id — collapses to one row per user with the in-window event count.
HAVING >= 5 — filters out u2 (3 events). u1 (6) and u3 (5) pass.
ORDER BY DESC — u1 (6) ranks above u3 (5).

Output:

user_id	events_in_window
u1	6
u3	5

Why this works — concept by concept:

JOIN on user_id — pairs each event with its user's signup_date; without this, the query has no anchor for the window.
INTERVAL '30 days' boundary — PostgreSQL-native expression of the 30-day forward window; reads as "30 days from signup."
WHERE row filter — strips out-of-window events before grouping; doing this with HAVING would still work but reads worse and runs slower.
GROUP BY u.user_id — collapses the row stream to one row per user; the only non-aggregate column in SELECT matches the GROUP BY.
HAVING COUNT(*) >= 5 — filters groups; aggregate predicates have to live here.
O((|users| + |events|) + G log G) time — the JOIN dominates; G users with at least one in-window event get sorted at the end.

Inline CTA: Practice date-functions SQL problems and cohort-analysis problems on PipeCode.

SQL
Company — Square (SQL)
Square SQL practice problems

Practice →

SQL
Topic — date functions
SQL date-function problems

Practice →

SQL
Topic — cohort analysis
SQL cohort-analysis problems

Practice →

3. SQL Window Functions for Monthly Averages and Duplicate Detection

Window AVG OVER + ROW_NUMBER OVER for analytics in SQL for data engineering

"Calculate the monthly average rating for each product" is Block / Square's signature window-function SQL prompt (DataLemur Q1). The mental model: AVG(stars) OVER (PARTITION BY product_id, EXTRACT(MONTH FROM submit_date)) produces the monthly product average on every row, with no row-collapse. The mirror primitive is ROW_NUMBER() OVER (PARTITION BY ... ORDER BY ...) for duplicate detection — flag rows whose rn > 1 per partition. Same primitive powers any "row-level value vs group-aggregate" or "first-occurrence-per-group" question.

Pro tip: Window functions and GROUP BY answer different questions. GROUP BY collapses rows; window aggregates compute on the group without collapsing. Use GROUP BY when you want one row per group; use a window when you want every original row plus a per-group aggregate column.

Window AVG: `AVG(stars) OVER (PARTITION BY product_id, EXTRACT(MONTH FROM submit_date))`

The window-AVG invariant: AVG(expr) OVER (PARTITION BY ...) returns the average of expr across all rows in the same partition, attached to every row. Unlike GROUP BY AVG(...), the row count is preserved.

PARTITION BY product_id — one window per product.
PARTITION BY product_id, EXTRACT(MONTH FROM submit_date) — one window per (product, month).
OVER () — empty parens = global window across all rows.
ORDER BY submit_date ROWS UNBOUNDED PRECEDING — running average up to current row.

Worked example. Two reviews for product 50001 in June; window AVG returns 3.5 on both rows.

review_id	product_id	submit_date	stars	window_avg
6171	50001	2022-06-08	4	3.5
5293	50001	2022-06-18	3	3.5

Worked-example solution.

SELECT review_id, product_id, submit_date, stars,
       AVG(stars) OVER (
         PARTITION BY product_id, EXTRACT(MONTH FROM submit_date)
       ) AS monthly_avg
FROM reviews;

Rule of thumb: "show me each row alongside the group average" → window; "show me one row per group" → GROUP BY.

`ROW_NUMBER() OVER (PARTITION BY ... ORDER BY ...)` for duplicate detection

The duplicate-detection invariant: ROW_NUMBER() OVER (PARTITION BY <key_columns> ORDER BY <tiebreaker>) assigns 1, 2, 3, … to rows within each partition by the duplicate key; rows with row_number > 1 are duplicates.

PARTITION BY user_id, product_id — define what makes a duplicate.
ORDER BY submit_date ASC — tiebreaker; deterministic.
Filter — wrap in subquery / CTE: WHERE rn > 1 for duplicates only, WHERE rn = 1 for the first occurrence.
Alternative — COUNT(*) OVER (PARTITION BY ...) to count duplicates per row.

Worked example. Two reviews from same user for same product → duplicate.

review_id	user_id	product_id	submit_date	rn
6171	123	50001	2022-06-08	1
9999	123	50001	2022-06-15	2 (duplicate)

Worked-example solution.

SELECT *
FROM (
  SELECT review_id, user_id, product_id, submit_date,
         ROW_NUMBER() OVER (
           PARTITION BY user_id, product_id ORDER BY submit_date
         ) AS rn
  FROM reviews
) t
WHERE rn > 1;

Rule of thumb: ROW_NUMBER OVER (PARTITION BY <duplicate_key> ORDER BY <tiebreaker>) is the universal SQL idiom for "find duplicates" — no GROUP BY needed.

Window vs `GROUP BY`: row-level vs collapsed-row aggregations

The distinction invariant: window functions preserve the input row count and add per-group aggregate columns; GROUP BY collapses input rows to one row per group and replaces non-aggregate columns. Many candidates conflate them; interviewers grade the distinction.

Window — SELECT review_id, AVG(stars) OVER (PARTITION BY product_id) FROM reviews; — row count preserved.
GROUP BY — SELECT product_id, AVG(stars) FROM reviews GROUP BY product_id; — one row per product.
Mixed — wrap the GROUP BY in a CTE, then JOIN to the original; messy.
QUALIFY — Snowflake-only filter on window-result; PostgreSQL needs a wrapping subquery instead.

Worked example. Same data, two different shapes.

approach	rows returned
Window AVG	5 (same as input)
GROUP BY AVG	2 (one per product)

Worked-example solution.

-- Window: keep all rows
SELECT review_id, product_id, stars,
       AVG(stars) OVER (PARTITION BY product_id) AS product_avg
FROM reviews;

-- GROUP BY: collapse to one row per product
SELECT product_id, AVG(stars) AS product_avg
FROM reviews
GROUP BY product_id;

Rule of thumb: read the prompt — "for each row, show…" = window; "summarize per product" = GROUP BY.

Common beginner mistakes

Forgetting PARTITION BY — the window covers the whole table, producing the global average on every row.
Using GROUP BY when the prompt says "for each review, show the monthly product average" — collapses rows incorrectly.
Forgetting ORDER BY inside ROW_NUMBER OVER (...) — non-deterministic row numbering; the duplicate filter becomes random.
Using RANK() when ROW_NUMBER() is correct — RANK ties become same number; RANK > 1 doesn't reliably flag duplicates.
Filtering window results in WHERE — Postgres requires a wrapping subquery / CTE because window functions evaluate after WHERE.

SQL Interview Question on Monthly Average Product Ratings

Given reviews(review_id, user_id, submit_date, product_id, stars), write a query that returns, for each row, submit_date_month, product_id, and the monthly average rating for that product (avg_stars), sorted by month and product_id.

Solution Using `AVG ... OVER (PARTITION BY product_id, EXTRACT(MONTH FROM submit_date))`

SELECT
    EXTRACT(MONTH FROM submit_date) AS mth,
    product_id,
    AVG(stars)::decimal(10, 2) OVER (
        PARTITION BY product_id, EXTRACT(MONTH FROM submit_date)
    ) AS avg_stars
FROM reviews
ORDER BY mth, product_id;

Why this works: EXTRACT(MONTH FROM submit_date) pulls the month component from each row; AVG(stars) OVER (PARTITION BY product_id, EXTRACT(MONTH FROM submit_date)) partitions by (product, month) and averages stars within each partition; the cast to decimal(10, 2) rounds to two decimals; ORDER BY mth, product_id produces the canonical sort order. Each row keeps its identity — this is window, not GROUP BY.

Step-by-step trace for the DataLemur sample:

review_id	user_id	submit_date	product_id	stars
6171	123	2022-06-08	50001	4
7802	265	2022-06-10	69852	4
5293	362	2022-06-18	50001	3
6352	192	2022-07-26	69852	3
4517	981	2022-07-05	69852	2

EXTRACT(MONTH ...) — produces [6, 6, 6, 7, 7].
Partition by (product_id, month) — four partitions: (50001, 6) with 2 rows, (69852, 6) with 1 row, (69852, 7) with 2 rows.
AVG within partition — (50001, 6) → 3.5; (69852, 6) → 4.0; (69852, 7) → 2.5.
Cast + ORDER BY — five rows emitted, sorted by (mth, product_id).

Output:

mth	product_id	avg_stars
6	50001	3.50
6	50001	3.50
6	69852	4.00
7	69852	2.50
7	69852	2.50

Why this works — concept by concept:

EXTRACT(MONTH FROM submit_date) — PostgreSQL date-component function; returns the month as integer.
AVG(...) OVER (PARTITION BY ...) — window aggregate; preserves row count and computes per-partition mean.
Composite partition key — (product_id, EXTRACT(MONTH FROM submit_date)) ensures monthly granularity per product.
::decimal(10, 2) cast — rounds to 2 decimals; cleaner output than the default float representation.
ORDER BY mth, product_id — canonical sort order for the audit-readable result.
O(N log N) time — the planner sorts rows by partition keys once; no GROUP BY collapse.

Inline CTA: More SQL window-function problems and aggregation problems on PipeCode.

SQL
Company — Square (SQL)
Square SQL practice problems

Practice →

SQL
Topic — window functions
SQL window-function problems

Practice →

SQL
Topic — aggregation
SQL aggregation problems

Practice →

4. SQL `COUNT DISTINCT` and Status Filters for Payment-Flow Analysis

Payment-flow SQL with COUNT DISTINCT and status filters in SQL for Square data engineering

"Return the total unique successful transactions, unique senders, and unique recipients" is Square / CashApp's signature payment-flow SQL prompt (Medium / DataLemur Q6 staple). The mental model: filter rows with WHERE status = 'Success' (idempotency boundary), then aggregate with COUNT(DISTINCT col) over the columns you care about — payment_id, sender_id, recipient_id. Same primitive powers any "unique-X for filtered Y" pipeline — unique active users last week, unique trading symbols traded yesterday, unique ad creatives clicked today.

Pro tip: Always state your idempotency boundary out loud before writing the query: "I'm filtering to status = 'Success' because failed/pending transactions don't count toward unique-success metrics." Square interviewers grade this phrasing — it's the difference between "candidate writes correct SQL" and "candidate thinks like a data engineer at a regulated payments company."

`COUNT` vs `COUNT DISTINCT`: row count vs unique-value count

The distinct-count invariant: COUNT(*) counts rows; COUNT(col) counts non-null values in col; COUNT(DISTINCT col) counts distinct non-null values in col. The three are not interchangeable.

COUNT(*) — total rows in the result set.
COUNT(payment_id) — non-null payment IDs (often equal to COUNT(*) since payment_id is the PK).
COUNT(DISTINCT sender_id) — unique senders; collapses repeated sender_ids.
COUNT(DISTINCT col1, col2) — PostgreSQL syntax for distinct combinations.

Worked example. Five payments from 3 distinct senders.

payment_id	sender_id
p1	s1
p2	s2
p3	s1
p4	s3
p5	s1

metric	value
`COUNT(*)`	5
`COUNT(payment_id)`	5
`COUNT(DISTINCT sender_id)`	3

Worked-example solution.

SELECT COUNT(*) AS total_rows,
       COUNT(payment_id) AS total_payment_ids,
       COUNT(DISTINCT sender_id) AS unique_senders
FROM payments;

Rule of thumb: "how many rows" → COUNT(*); "how many unique entities" → COUNT(DISTINCT col).

`WHERE status = 'Success'` before aggregation

The idempotency-filter invariant: payment-flow analytics almost always filter to status = 'Success' (or whichever success state applies); failed and pending transactions do not contribute to success metrics. The filter goes in WHERE, before aggregation.

WHERE status = 'Success' — strictly successful payments.
WHERE status IN ('Success', 'Settled') — multiple success-equivalent statuses.
WHERE status NOT IN ('Failed', 'Cancelled') — exclusionary; semantically same when the universe is finite.
State the assumption — interviewers expect you to clarify which statuses count.

Worked example. Six payments, three successful.

payment_id	sender_id	status
p1	s1	Success
p2	s2	Failed
p3	s1	Success
p4	s3	Pending
p5	s1	Success
p6	s4	Failed

After WHERE status = 'Success': 3 rows (p1, p3, p5), unique senders = 1 (s1 only).

Worked-example solution.

SELECT COUNT(*) AS successful_payments,
       COUNT(DISTINCT sender_id) AS unique_successful_senders
FROM payments
WHERE status = 'Success';

Rule of thumb: the success-filter is non-negotiable for payment-flow questions; ask the interviewer which status values count if the prompt is ambiguous.

Multiple `COUNT DISTINCT` in one `SELECT`: sender / recipient splits

The compound-aggregate invariant: a single SELECT can compute multiple COUNT(DISTINCT col) aggregates over the same filtered row set; this is the common shape for payment-flow "uniques per role" questions.

Three aggregates in one SELECT — COUNT(DISTINCT payment_id), COUNT(DISTINCT sender_id), COUNT(DISTINCT recipient_id).
Same WHERE filter applies to all — single source row set drives every aggregate.
Senders ≠ recipients — even though both columns reference users, the distinct counts can differ.
COUNT(DISTINCT user_id) — pool sender + recipient with UNION ALL first if you want "unique users involved."

Worked example. Three successful payments; two distinct senders, three distinct recipients.

payment_id	sender_id	recipient_id
p1	s1	r1
p3	s1	r2
p5	s2	r3

metric	value
`COUNT(DISTINCT payment_id)`	3
`COUNT(DISTINCT sender_id)`	2
`COUNT(DISTINCT recipient_id)`	3

Worked-example solution.

SELECT COUNT(DISTINCT payment_id)  AS unique_transactions,
       COUNT(DISTINCT sender_id)   AS unique_senders,
       COUNT(DISTINCT recipient_id) AS unique_recipients
FROM payments
WHERE status = 'Success';

Rule of thumb: one query, multiple aggregates is the cleanest shape; never write three separate queries for sender / recipient / transaction counts.

Common beginner mistakes

Using COUNT(*) when the prompt says "unique" — undercounts (or overcounts, depending on whether duplicates exist).
Forgetting WHERE status = 'Success' — counts failed and pending transactions; the metric becomes meaningless.
Putting the status filter in HAVING instead of WHERE — works but is slow and reads worse.
Writing three separate queries for sender / recipient / transaction — composes into one SELECT with three aggregates.
Confusing "unique senders" with "unique users" — senders and recipients can be the same individual; pool them only if the prompt asks for "unique users."

SQL Interview Question on Payment-Flow Unique Counts

Given payments(payment_id, sender_id, recipient_id, amount, status, created_at) where status is one of 'Success', 'Failed', 'Pending', write a query that returns three columns: unique_transactions, unique_senders, unique_recipients — each a COUNT(DISTINCT) over rows where status = 'Success'.

Solution Using `COUNT(DISTINCT)` + `WHERE status = 'Success'`

SELECT
    COUNT(DISTINCT payment_id)   AS unique_transactions,
    COUNT(DISTINCT sender_id)    AS unique_senders,
    COUNT(DISTINCT recipient_id) AS unique_recipients
FROM payments
WHERE status = 'Success';

Why this works: WHERE status = 'Success' strips failed and pending payments before aggregation so every count reflects only successful transactions; three COUNT(DISTINCT col) aggregates over the filtered row set produce the three unique counts in a single pass; the result is one row with three columns — the canonical shape for "summary metrics for a filtered universe."

Step-by-step trace for a 6-row sample:

payment_id	sender_id	recipient_id	status
p1	s1	r1	Success
p2	s2	r2	Failed
p3	s1	r2	Success
p4	s3	r1	Pending
p5	s2	r3	Success
p6	s1	r1	Success

WHERE filter — strips p2 (Failed) and p4 (Pending); 4 successful rows remain.
COUNT(DISTINCT payment_id) — {p1, p3, p5, p6} = 4 (every payment_id is unique by definition).
COUNT(DISTINCT sender_id) — {s1, s2} = 2.
COUNT(DISTINCT recipient_id) — {r1, r2, r3} = 3.

Output:

unique_transactions	unique_senders	unique_recipients
4	2	3

Why this works — concept by concept:

WHERE status = 'Success' — idempotency-filter; payment-flow metrics never include failed or pending rows.
COUNT(DISTINCT payment_id) — counts unique transaction IDs; equal to COUNT(*) here since payment_id is the primary key, but spelled out for clarity.
COUNT(DISTINCT sender_id) — counts unique senders; senders can repeat across payments, so this collapses repeats.
COUNT(DISTINCT recipient_id) — counts unique recipients; same logic, different column.
Single-pass three-aggregate SELECT — one scan of payments filtered to Success, three aggregates in parallel; no need for separate queries.
O(|payments|) time / O(D) space — one linear scan; D = sum of distinct cardinalities held in three small hash structures.

Inline CTA: Drill more SQL filtering problems and the Square SQL practice page for the curated 2-problem set.

SQL
Company — Square (SQL)
Square SQL practice problems

Practice →

SQL
Topic — filtering
SQL filtering problems

Practice →

SQL
Topic — aggregation
SQL aggregation problems

Practice →

Tips to crack Square data engineering interviews

Square = Block — both names refer to the same loop

Square rebranded to Block Inc. in December 2021. The company name on offer letters is Block; the consumer products (Square seller tools, Cash App, Tidal, Spiral) keep their original brands. The data-engineering interview loop, the SQL bar, and the question shapes did not change with the rebrand. Search for both "Square data engineer interview" and "Block data engineer interview" — every external article you find under one name applies under the other.

Drill the four SQL primitives

The four primitives in this guide map directly to the curated 2 PipeCode SQL problems plus the two adjacent SQL primitives every Block / Square / CashApp SQL list rotates through: GROUP BY + COUNT + ORDER BY DESC + LIMIT N for top-N rankings (#45), DATEDIFF / INTERVAL '30 days' + GROUP BY + HAVING for cohort-retention queries (#217), AVG OVER PARTITION BY and ROW_NUMBER OVER window functions for monthly aggregates and duplicate detection (DataLemur Q1 + Q3), and COUNT(DISTINCT) + WHERE status = 'Success' payment-flow aggregations (Medium + DataLemur Q6).

CoderPad PostgreSQL is the live coding environment

Square / Block / CashApp interviews live SQL in CoderPad with PostgreSQL as the only available dialect. Drill PostgreSQL-flavored answers — EXTRACT(MONTH FROM ...), DATE_TRUNC('week', ...), INTERVAL '30 days', (d1 - d2)::int, COUNT(DISTINCT col), LIMIT N OFFSET M. Avoid Snowflake's QUALIFY, MySQL's DATEDIFF(d1, d2), BigQuery's DATE_DIFF, SQL Server's TOP N — these all parse-fail in PostgreSQL. Google search is allowed during the live interview, but stuttering on dialect signals weak SQL fluency.

Medallion architecture (Bronze / Silver / Gold) is the system-design house style

Square's data platform is structured around the Medallion architecture — Bronze for raw landed data, Silver for cleaned and enriched data, Gold for aggregated analytics-ready data. When you discuss pipeline design in the system-design or onsite panel rounds, frame the data flow as Bronze → Silver → Gold, name the orchestrator (Airflow or dbt), and mention data-quality gates (Great Expectations) between layers. This single framing carries weight at every Square data-engineering onsite.

Data quality is graded heavily — Great Expectations and validation gates

Square's interviewers probe data-quality concerns on every round. Mention specific tools you've used (Great Expectations, Soda, dbt tests), describe the validation gates between Bronze / Silver / Gold layers, and articulate the difference between "schema validation" (column types and nullability) and "semantic validation" (business-logic invariants like transaction_amount > 0 or payment_status IN ('Success', 'Failed', 'Pending')).

4-stage interview process — HR, Hiring Manager, Technical, Onsite Panel (NDA)

The Square data-engineering loop runs four stages: HR screen (15-30 min recruiter), Hiring Manager interview (~30 min, past experience and data-quality), Technical Screen (~1 hour, Python + SQL on CoderPad with medium-level complexity), and Onsite Panel (multiple 30-45 min rounds, NDA required, data modeling + analytical + cultural fit). Total comp data points are limited (only 2 reported), but the average base salary is $139,850, range $101K-$185K.

Where to practice on PipeCode

Start with the Square practice page and the language-scoped Square SQL practice page for the curated 2-problem set. Hit the company_topic Square — aggregation page for the only Square-tagged topic surface available. After that, drill the matching topic pages: ranking, aggregation, date functions, cohort analysis, window functions, filtering, group by. The interview courses page bundles structured curricula. For broader coverage, browse by topic, or pivot to peer guides — the Airbnb DE interview guide and the top DE interview questions 2026 blog. The SQL data types Postgres guide is a useful refresher because PostgreSQL is the CoderPad default.

Communication and approach under time pressure

Talk through the invariant first ("this is a top-N ranking problem with a tiebreak requirement"), the brute force second ("a self-join would also work but is O(n²)"), and the optimal third ("GROUP BY + ORDER BY + LIMIT is O(n log n) and idiomatic"). Interviewers grade process as much as the final answer. Leave 5 minutes for an edge-case sweep: empty input, ties at the cut, NULL handling, timezone bugs in date arithmetic, status values you didn't expect (Pending, Cancelled). The most common "almost passed" failure mode is correct happy-path code that crashes on edge cases — a 30-second sweep prevents it.

Frequently Asked Questions

What is the Square data engineering interview process?

The Square data engineering interview opens with a 15-30 minute recruiter HR screen, then a 30-minute hiring-manager interview focused on past experience and data-quality philosophy, then a 1-hour technical screen on CoderPad covering Python and SQL at medium-level complexity, then an onsite panel of multiple 30-45 minute rounds (data modeling, analytical thinking, cultural fit). The onsite panel requires an NDA. End-to-end the loop runs three to four weeks.

Is Square the same as Block? Which name should I search for?

Yes — Square rebranded to Block Inc. in December 2021. The legal entity is Block; the consumer brands (Square seller tools, Cash App, Tidal, Spiral) keep their original names. Search for both "Square data engineer interview" and "Block data engineer interview" — every external article you find under one name applies under the other. The DataLemur SQL guide is titled "Block SQL Interview Questions" and the IQ guide is titled "Square Data Engineer Interview Questions"; both refer to the same loop.

What programming languages does Square test in data engineering interviews?

Square tests SQL and Python in the technical screen and onsite. SQL is the heavier surface — the live coding round in CoderPad is PostgreSQL-only, with patterns like GROUP BY + COUNT + ORDER BY DESC + LIMIT, INTERVAL '30 days' cohort math, AVG OVER PARTITION BY window aggregates, and COUNT(DISTINCT col) over status-filtered payment data. Python is the lighter surface — typically 30 minutes of medium-difficulty data-manipulation questions (Pandas / dict / list comprehensions). Spark / PySpark show up in the onsite panel for senior roles.

How difficult are Square data engineering interview questions?

The curated Square practice set on PipeCode is 1 EASY + 1 MEDIUM, no hard. The EASY is a SQL ranking + aggregation problem (Top 10 Invoice Senders); the MEDIUM is a SQL aggregation + date-functions + cohort-analysis problem (Users with High Activity After 30 Days Signup). The DataLemur Block list adds 10 more SQL questions ranging from definition-style (joins, constraints, clustered vs non-clustered indexes) to scenario-grade (window AVG OVER, ROW_NUMBER for duplicates, click-through rate via LEFT JOIN + COUNT DISTINCT). Stuttering on the EASY is a stronger negative signal than struggling with the MEDIUM.

What is the Square data engineer salary range?

Square data engineer base salary ranges from $101K to $185K, with an average of $139,850 (median $140K) across 21 reported data points (per Interview Query). Total compensation data is sparse (only 2 reported points), so the total-comp average ($72K) is under-sampled and likely inaccurate; expect actual total comp to be substantially higher once equity refreshers and bonuses are factored in. Negotiation success rates are best-supported by competing offers and verified levels.fyi entries.

What tech stack does Square's data engineering team use?

Square's data engineering stack includes Python (heavy — Pandas, PySpark for large-scale transforms), SQL (PostgreSQL on CoderPad for interviews; production warehouses include Snowflake-equivalent platforms), Airflow + dbt for ETL/ELT orchestration, AWS or GCP for cloud infrastructure, Apache Spark for distributed data processing, the Medallion architecture (Bronze / Silver / Gold layers) for data organization, and Great Expectations for data-quality validation. The cultural emphasis is on data-quality-first pipelines, cross-functional collaboration with product / DS teams, and continuous learning.

Start practicing Square data engineering problems

Snowflake Data Engineering Interview Questions & Prep Guide

Gowtham Potureddi — Sun, 03 May 2026 05:20:29 +0000

Snowflake data engineering interview questions split into two distinct loops that share a name. The Snowflake-the-company SWE / DE interview is LeetCode-style Python: array iteration with set-based validation logic (the classic SET card-game rule, len(set(values)) in {1, 3} per attribute) and hash-table sliding-window counters over strings (Counter(substrings) driven by a rolling character-frequency dict). The Snowflake-as-tool data-engineering interview — for any DE role at any company that runs on Snowflake — is product-knowledge plus SQL: three-layer architecture, micro-partitions, clustering keys, Time Travel and Fail-safe, Zero-Copy Cloning, Snowpipe ingestion, and the LAG/LEAD/AVG ... OVER (PARTITION BY ...) window-function primitives that drive consecutive-streak detection, day-over-day deltas, and monthly-aggregate analytics.

This guide walks four topic clusters end-to-end, each with a detailed topic explanation, per-sub-topic explanation with a worked example and its solution, and an interview-style problem with a full solution that explains why it works. The mix matches a curated 2-problem set (1 EASY Python array + 1 MEDIUM Python hash-table-sliding-window) plus two adjacent primitives — SQL window functions and Snowflake architecture — that every Snowflake-flavored interview rotates through. Whether the candidate is interviewing AT Snowflake or for a data-engineering role at a company that builds on Snowflake, the four primitives below cover the bar.

1. Python Arrays and Set Validation for the SET Card Game

Array iteration with set-based validation in Python for data engineering

"Given three SET cards (each a 4-tuple of attributes from {color, shape, number, shading}), validate that across all three cards every attribute is either all the same or all different" is Snowflake's signature EASY Python prompt (SET Card Game Validation). The mental model: zip(*cards) produces one tuple per attribute index across all three cards; set(values) deduplicates that tuple; the SET game rule is satisfied when every attribute's set has cardinality 1 (all same) or 3 (all different) — never 2. Same primitive powers any "validate a row across N records" pipeline — schema-conformance checks, all-same-or-all-different label validation in ML datasets, "every replica reports the same status" health monitors.

Pro tip: len(set(values)) collapses three separate equality checks into a single primitive. Avoid the temptation to write (a == b == c) or (a != b and b != c and a != c) — the set-cardinality test is shorter, faster, and idiomatic. State the cardinality invariant out loud before writing code.

Iterating tuples by attribute index: `zip(*cards)`

The transpose-via-zip invariant: zip(*iterables) interleaves elements at the same index across all iterables; zip(*cards) where cards is a list of 3 tuples produces 4 tuples, one per attribute, each with 3 values. This is the standard Python idiom for column-wise iteration over a row-major data structure.

zip(*cards) — yields per-attribute tuples; * unpacks the outer list.
list(zip(*cards)) — materialize into a list if you need to iterate twice.
Equivalent loop — for i in range(len(cards[0])): values = tuple(c[i] for c in cards) — verbose but explicit.
Same pattern for matrix transpose — zip(*matrix) is the one-line transpose.

Worked example. Three cards as 4-tuples; transpose to 4 attribute tuples.

step	output
input	`[('R', 'oval', 1, 'solid'), ('G', 'oval', 2, 'solid'), ('P', 'oval', 3, 'solid')]`
`zip(*cards)`	`[('R','G','P'), ('oval','oval','oval'), (1,2,3), ('solid','solid','solid')]`

Worked-example solution.

cards = [('R', 'oval', 1, 'solid'),
         ('G', 'oval', 2, 'solid'),
         ('P', 'oval', 3, 'solid')]
attributes = list(zip(*cards))
# [('R','G','P'), ('oval','oval','oval'), (1,2,3), ('solid','solid','solid')]

Rule of thumb: zip(*rows) is the one-line Python transpose; reach for it whenever the question asks about "across all rows" properties.

All-same-or-all-different rule via `len(set(values)) in {1, 3}`

The cardinality invariant: set(values) deduplicates a sequence; len(set(values)) returns the number of distinct elements; for a 3-element input, that count is 1 (all same), 2 (mixed — invalid), or 3 (all different). The SET game rule accepts cardinality 1 or 3 and rejects 2.

len(set(vals)) == 1 — all values equal.
len(set(vals)) == 3 — all values pairwise distinct.
len(set(vals)) == 2 — invalid mixed pattern (rejected by the rule).
len(set(vals)) in {1, 3} — combined acceptance test in one expression.

Worked example. Three attribute tuples; one valid (all same), one valid (all different), one invalid (mixed).

values	set(values)	len	rule
`('oval','oval','oval')`	`{'oval'}`	1	all same ✓
`('R','G','P')`	`{'R','G','P'}`	3	all different ✓
`('R','R','G')`	`{'R','G'}`	2	invalid ✗

Worked-example solution.

def attribute_ok(values):
    return len(set(values)) in {1, 3}

attribute_ok(('oval','oval','oval'))  # True
attribute_ok(('R','G','P'))           # True
attribute_ok(('R','R','G'))           # False

Rule of thumb: the cardinality test (len(set(...)) in {1, 3}) is the one-line replacement for any "all equal OR all distinct" check; never expand it into a conjunction of pairwise comparisons.

`all()` composition over per-attribute checks

The composition invariant: all(predicate(x) for x in iterable) short-circuits on the first False and returns True only if every element passes. Combined with zip(*cards), the entire SET validation collapses to one expression: all(len(set(v)) in {1, 3} for v in zip(*cards)).

all(...) — returns True if every element is truthy; short-circuits.
any(...) — mirror image; returns True if any element is truthy.
Generator expression — (predicate(x) for x in iter) is lazy; pairs perfectly with all.
Avoid for + flag — ok = True; for ...: if not check: ok = False; return ok is verbose.

Worked example. Compose attribute_ok over the four attribute tuples of a valid SET.

attribute	values	ok
color	`('R','G','P')`	True
shape	`('oval','oval','oval')`	True
number	`(1,2,3)`	True
shading	`('solid','solid','solid')`	True

Worked-example solution.

def is_valid_set(cards):
    return all(len(set(v)) in {1, 3} for v in zip(*cards))

cards = [('R','oval',1,'solid'),
         ('G','oval',2,'solid'),
         ('P','oval',3,'solid')]
is_valid_set(cards)  # True

Rule of thumb: the entire problem is a one-liner once you compose all + set + zip(*cards); if your code needs more than three lines, you're missing a primitive.

Common beginner mistakes

Writing (a == b == c) or (a != b and b != c and a != c) instead of len(set(values)) in {1, 3} — slow, verbose, and easy to mistype.
Using cards[0][i], cards[1][i], cards[2][i] indexing instead of zip(*cards) — works but signals you don't know the idiom.
Treating cardinality 2 as valid — SET's whole rule is "no 2-distinct attributes allowed"; stuttering on this is graded as not understanding the prompt.
Hardcoding 3 cards / 4 attributes instead of letting zip and all derive the dimensions — the same code should work for any N-card / M-attribute generalization.
Returning 1 or 0 instead of True / False — the contract is a boolean.

Python Interview Question on SET Card Game Validation

Given three SET cards as 4-tuples of attributes from {color, shape, number, shading}, return True if every attribute is either all-same or all-different across the three cards, otherwise False.

def is_valid_set(cards):
    pass  # your code here

Solution Using `all`, `set`, and `zip(*cards)`

def is_valid_set(cards):
    return all(len(set(values)) in {1, 3} for values in zip(*cards))

Why this works: zip(*cards) transposes the 3×4 row-major card list into 4 attribute tuples of length 3; len(set(values)) counts distinct values per attribute and is 1 (all same) or 3 (all different) iff the SET rule is satisfied; len(...) in {1, 3} accepts both valid cardinalities and rejects the 2-distinct case; all(...) short-circuits to False on the first invalid attribute, returning True only when every attribute passes. The whole solution is one line, branch-free, and O(N · M) for N cards × M attributes.

Step-by-step trace for cards = [('R','oval',1,'solid'), ('G','oval',2,'solid'), ('P','oval',3,'solid')]:

Transpose — zip(*cards) → [('R','G','P'), ('oval','oval','oval'), (1,2,3), ('solid','solid','solid')].
Per-attribute set — {'R','G','P'} (len 3), {'oval'} (len 1), {1,2,3} (len 3), {'solid'} (len 1).
Cardinality check — 3 ∈ {1, 3} ✓; 1 ∈ {1, 3} ✓; 3 ∈ {1, 3} ✓; 1 ∈ {1, 3} ✓.
all — every check True → returns True.

Output:

attribute	values	len(set)	in {1, 3}
color	`R, G, P`	3	✓
shape	`oval, oval, oval`	1	✓
number	`1, 2, 3`	3	✓
shading	`solid, solid, solid`	1	✓
result			True

Why this works — concept by concept:

zip(*cards) transpose — turns row-major card data into column-major attribute tuples; the standard Python one-liner for "iterate by attribute across rows."
set(values) dedupe — collapses duplicates so len(...) returns the distinct count in O(N) hash-set inserts.
len(...) in {1, 3} cardinality test — the SET game rule expressed as a single set-membership check; rejects the 2-distinct case implicitly.
all(...) short-circuit — returns False on the first invalid attribute without scanning the rest; pairs naturally with the generator expression.
one-line composition — the entire solution is a single expression, branch-free, with no temporary state or accumulator.
O(N · M) time / O(M) space — N cards × M attributes; each per-attribute set holds at most N items, so space is O(M · N) worst case but typically just O(M) distinct sets.

Inline CTA: Drill the Snowflake Python practice page for the curated array problem and the array practice page for breadth.

PYTHON
Company — Snowflake (Python)
Snowflake Python practice problems

Practice →

PYTHON
Topic — array
Python array problems

Practice →

PYTHON
Company — Snowflake
Snowflake data engineering problems

Practice →

2. Python Hash Tables and Sliding Window for Maximum Substring Occurrences

Hash-table sliding-window for substring frequency in Python for data engineering

"Given a string s and integers k (substring length) and maxLetters (max distinct chars allowed), return the maximum number of times any length-k substring with at most maxLetters distinct characters appears in s" is Snowflake's signature MEDIUM Python prompt (Maximum Substring Occurrences). The mental model: a length-k window slides across s; a freq dict tracks per-character counts inside the window in O(1) per shift; a Counter tracks how many times each valid window-substring has been seen; the answer is the max value in that counter. Same primitive powers any "find the most frequent fixed-length pattern under a constraint" pipeline — find the most-repeated DNA k-mer with at most M distinct nucleotides, the most-repeated user-event sequence under a uniqueness cap.

Pro tip: Maintain the freq dict incrementally — increment for the entering character, decrement for the leaving character, and delete keys whose count hits 0 so len(freq) always equals the distinct character count in the window. Rebuilding freq from scratch each window is O(k) per shift and turns the algorithm O(n · k); the incremental update is O(1) per shift and the total runtime is O(n).

Char-frequency dict for the window: increment / decrement per character

The window-state invariant: the freq dict at any time reflects the multiset of characters in the current length-k window; updating from window [i..i+k-1] to [i+1..i+k] requires one increment (for s[i+k]) and one decrement (for s[i]). Cleanup the zero-count key so len(freq) always equals the distinct character count.

Increment — freq[c] = freq.get(c, 0) + 1.
Decrement — freq[c] -= 1; then del freq[c] if freq[c] == 0 (the cleanup step).
Distinct-char count — len(freq) (with cleanup) gives the number of distinct chars in the window.
defaultdict(int) — alternative; same outcome, slightly cleaner increment but you still need the zero-cleanup branch.

Worked example. Slide a length-3 window over 'abca'; show freq per position.

window	freq	distinct
`abc`	`{a:1, b:1, c:1}`	3
`bca`	`{b:1, c:1, a:1}`	3

Worked-example solution.

s, k = 'abca', 3
freq = {}
for i, c in enumerate(s[:k]):
    freq[c] = freq.get(c, 0) + 1
# freq == {'a': 1, 'b': 1, 'c': 1}; len(freq) == 3

# slide: out s[0]='a', in s[3]='a'
freq['a'] -= 1
if freq['a'] == 0: del freq['a']
freq['a'] = freq.get('a', 0) + 1
# freq == {'b': 1, 'c': 1, 'a': 1}

Rule of thumb: always pair the decrement with the zero-cleanup; otherwise len(freq) overcounts distinct characters and the constraint check breaks.

Distinct-char count via `len(freq)` and zero-cleanup

The distinct-count invariant: after every increment / decrement + cleanup, len(freq) equals the number of distinct characters currently in the window. This is the per-window check against maxLetters.

Valid window — len(freq) <= maxLetters.
Cleanup is non-negotiable — without del freq[c] if freq[c] == 0, a key with count 0 still inflates len(freq).
No set() rebuild — len(set(window)) is O(k) per check; the incremental dict gives O(1).
Snapshot only on demand — record the substring in the answer counter only when the validity check passes.

Worked example. Validate windows of 'abcabcab' with maxLetters = 2.

window	distinct	valid?
`abc`	3	no
`bca`	3	no
`cab`	3	no

Worked-example solution.

s, k, maxLetters = 'abcabcab', 3, 2
# every length-3 window has 3 distinct chars → no valid window → answer = 0

Rule of thumb: the distinct-char check is one comparison (len(freq) <= maxLetters); never recompute distinct count from scratch.

Substring counter for the answer: `Counter` or `dict.get(s, 0) + 1` over candidates

The answer-aggregation invariant: a separate Counter (or vanilla dict) keyed on the substring tracks how many times each valid window-substring has been seen; the final answer is the maximum value in that counter. The constraint filter happens before the counter increment, so only valid substrings ever land in the answer.

from collections import Counter — Counter()[k] += 1 is a clean increment.
Vanilla dict — answer[sub] = answer.get(sub, 0) + 1; same outcome, no import.
Final answer — max(answer.values(), default=0); the default=0 handles the empty-counter case.
Substring slice — s[i:i+k] is O(k) to create; one slice per valid window is the cost.

Worked example. Count valid windows for 'aababcaab' with k=3, maxLetters=2.

window	distinct	counter
`aab`	2	`{aab: 1}`
`aba`	2	`{aab: 1, aba: 1}`
`bab`	2	`{aab: 1, aba: 1, bab: 1}`
`abc`	3	(skipped)
`bca`	3	(skipped)
`caa`	2	`{aab: 1, aba: 1, bab: 1, caa: 1}`
`aab`	2	`{aab: 2, aba: 1, bab: 1, caa: 1}`

Worked-example solution.

from collections import Counter
s, k, maxLetters = 'aababcaab', 3, 2
freq, answer = {}, Counter()
# ... incremental loop ...
# answer == {'aab': 2, 'aba': 1, 'bab': 1, 'caa': 1}
# max(answer.values()) == 2

Rule of thumb: keep two state structures — one for the rolling window (freq), one for the answer (Counter); never conflate them.

Common beginner mistakes

Rebuilding freq from s[i:i+k] each iteration — O(n · k) instead of O(n).
Forgetting the zero-cleanup — len(freq) overcounts distinct chars and the validity check rejects valid windows.
Counting every window (not just valid ones) — pollutes the answer counter with substrings that violate maxLetters.
Using if sub in answer: answer[sub] += 1 else: answer[sub] = 1 — works but reads worse than Counter or dict.get(sub, 0) + 1.
Returning the first valid substring's count instead of max(answer.values()) — mis-reads the prompt.

Python Interview Question on Maximum Substring Occurrences

Given a string s and integers k and maxLetters, return the maximum number of occurrences of any length-k substring of s whose distinct-character count is at most maxLetters. If no such substring exists, return 0.

def maxFreq(s: str, k: int, maxLetters: int) -> int:
    pass  # your code here

Solution Using rolling `freq` dict + `Counter` over valid substrings

from collections import Counter

def maxFreq(s: str, k: int, maxLetters: int) -> int:
    if k > len(s):
        return 0
    freq, answer = {}, Counter()
    # initialize the first window
    for c in s[:k]:
        freq[c] = freq.get(c, 0) + 1
    if len(freq) <= maxLetters:
        answer[s[:k]] += 1
    # slide
    for i in range(1, len(s) - k + 1):
        out_c, in_c = s[i - 1], s[i + k - 1]
        freq[out_c] -= 1
        if freq[out_c] == 0:
            del freq[out_c]
        freq[in_c] = freq.get(in_c, 0) + 1
        if len(freq) <= maxLetters:
            answer[s[i:i + k]] += 1
    return max(answer.values(), default=0)

Why this works: the freq dict is updated incrementally — one increment (entering char) and one decrement-plus-cleanup (leaving char) per shift, total O(1) per window — so the rolling distinct-character count len(freq) is always correct in constant time; len(freq) <= maxLetters filters out invalid windows before they touch the answer counter; valid window substrings are tallied in a Counter; the final max(answer.values(), default=0) returns the most frequent valid substring's count, defaulting to 0 if nothing qualified.

Step-by-step trace for s = 'aababcaab', k = 3, maxLetters = 2:

i	window	freq (after slide)	distinct	valid?	answer
0	`aab`	`{a:2, b:1}`	2	✓	`{aab: 1}`
1	`aba`	`{a:2, b:1}`	2	✓	`{aab: 1, aba: 1}`
2	`bab`	`{a:1, b:2}`	2	✓	`{aab: 1, aba: 1, bab: 1}`
3	`abc`	`{a:1, b:1, c:1}`	3	✗	(unchanged)
4	`bca`	`{a:1, b:1, c:1}`	3	✗	(unchanged)
5	`caa`	`{a:2, c:1}`	2	✓	`{aab: 1, aba: 1, bab: 1, caa: 1}`
6	`aab`	`{a:2, b:1}`	2	✓	`{aab: 2, aba: 1, bab: 1, caa: 1}`

Final: max(answer.values()) == 2.

Output:

substring	occurrences
aab	2
aba	1
bab	1
caa	1
answer	2

Why this works — concept by concept:

Rolling freq dict — the per-window character-frequency map is updated in O(1) per shift via one increment and one decrement-plus-cleanup; total scan is O(n).
Zero-cleanup via del freq[c] if freq[c] == 0 — keeps len(freq) equal to the distinct character count; without it the constraint check would reject valid windows.
len(freq) <= maxLetters validity filter — single comparison; only valid windows enter the answer counter.
Counter over substrings — tallies how many times each valid substring has been seen; Counter short-circuits the missing-key branch.
max(answer.values(), default=0) — final aggregation; the default=0 handles the case where no window qualified.
O(n · k) time / O(n) space — the loop is O(n) shifts, each with a O(k) substring slice for the answer key; the answer counter holds at most O(n - k + 1) distinct substrings.

Inline CTA: More sliding-window problems and hash-table problems on PipeCode.

PYTHON
Company — Snowflake (Python)
Snowflake Python practice problems

Practice →

PYTHON
Topic — hash table
Python hash-table problems

Practice →

PYTHON
Topic — sliding window
Python sliding-window problems

Practice →

3. SQL Window Functions (`LAG`, `LEAD`, `AVG OVER`) for Snowflake Analytics

Window functions and CTE composition in SQL for Snowflake data engineering

"Find every contact who had a marketing touch in three or more consecutive weeks AND at least one trial_request" is the canonical Snowflake-flavored SQL prompt — the DataLemur-staple marketing-touch streak question that surfaces in nearly every Snowflake-as-tool DE interview. The mental model: DATE_TRUNC('week', event_date) snaps event timestamps to the week boundary; LAG(week_trunc) OVER (PARTITION BY contact_id ORDER BY week_trunc) reaches one row back inside the per-contact partition; the streak condition is lag_week = current_week - INTERVAL '1 week'. Same primitive powers any "consecutive-period" or "day-over-day" analytic — consecutive-month subscription renewals, hour-over-hour traffic deltas, week-over-week feature-adoption ramps.

Pro tip: Always ground your window function with both PARTITION BY and ORDER BY. Forgetting PARTITION BY contact_id makes LAG reach into the previous contact's events and produces meaningless deltas; forgetting ORDER BY week_trunc leaves the row order non-deterministic and the answer flaky. State both clauses out loud before writing the SELECT.

Window basics: `PARTITION BY group + ORDER BY ordering`

The window-function invariant: OVER (PARTITION BY <group> ORDER BY <ordering>) declares an independent ordered subset for every value of the group expression; the window function (LAG, LEAD, AVG, ROW_NUMBER, …) runs inside that subset only. PARTITION BY is the equivalent of GROUP BY for windows; ORDER BY fixes the row order so offset functions (LAG/LEAD) and ranking functions (ROW_NUMBER/RANK) are deterministic.

PARTITION BY is optional — omitted, the whole table is one window; useful for global running totals.
ORDER BY is required for offset / ranking functions — LAG, LEAD, ROW_NUMBER, RANK all need an explicit order.
AVG(...) OVER (...) — group-aggregate-on-row; returns the average across the partition for each row.
Frames — ROWS BETWEEN ... AND ... further bound the visible rows; default is fine for LAG.

Worked example. Two events for contact_id 1; LAG(week_trunc) reaches one row back inside the partition.

event_id	contact_id	week_trunc	LAG(week_trunc)
1	1	2022-04-11	NULL
2	1	2022-04-18	2022-04-11

Worked-example solution.

SELECT event_id, contact_id, week_trunc,
       LAG(week_trunc) OVER (PARTITION BY contact_id ORDER BY week_trunc) AS lag_week
FROM events_with_week;

Rule of thumb: always set PARTITION BY to the entity that owns the time series (contact_id, user_id, account_id); without it, day-1 of one entity leaks into day-N of another.

`LAG` vs `LEAD` vs `AVG OVER`: row-N-before vs row-N-after vs group-aggregate-on-row

The window-function-family invariant: LAG(expr, n) returns expr from n rows before; LEAD(expr, n) from n rows after; AVG(expr) OVER (...) returns the average of expr across the entire partition (not just the rows up to current). They share the OVER syntax but answer different questions.

LAG(week_trunc) — previous row's week_trunc.
LEAD(week_trunc) — next row's week_trunc.
AVG(stars) OVER (PARTITION BY product_id, EXTRACT(MONTH FROM submit_date)) — monthly average per product (DataLemur Q2 pattern).
ROW_NUMBER() OVER (PARTITION BY ... ORDER BY ...) — sequential rank inside each partition.

Worked example. Three rows for contact_id 1; both LAG and LEAD annotated.

event_id	week_trunc	LAG	LEAD
1	2022-04-11	NULL	2022-04-18
2	2022-04-18	2022-04-11	2022-04-25
3	2022-04-25	2022-04-18	NULL

Worked-example solution.

SELECT event_id, week_trunc,
       LAG(week_trunc)  OVER (PARTITION BY contact_id ORDER BY week_trunc) AS lag_week,
       LEAD(week_trunc) OVER (PARTITION BY contact_id ORDER BY week_trunc) AS lead_week
FROM events_with_week
WHERE contact_id = 1;

Rule of thumb: "delta vs prior" → LAG; "delta vs next" or "next-event distance" → LEAD; "row's value compared to group average" → AVG ... OVER.

CTE composition: `WITH consecutive_events AS (...)` for multi-step logic

The CTE composition invariant: a WITH ... AS (...) Common Table Expression names an intermediate result that subsequent SELECT statements can reference; multi-step window logic is far more readable as a CTE pipeline than as a deeply nested subquery. Snowflake supports CTEs natively, including recursive CTEs for hierarchical traversal.

WITH cte_name AS (SELECT ...) — defines the named intermediate.
Multiple CTEs — WITH a AS (...), b AS (...) SELECT ... FROM b JOIN a ... — chain steps.
Recursive CTE — WITH RECURSIVE cte AS (base UNION ALL recursive); rare in interview but Snowflake-supported.
Materialization — Snowflake may inline or materialize CTEs; behavior depends on the query optimizer.

Worked example. Compute lag_week and lead_week once in a CTE, then filter for the streak condition.

step	what	output
1	CTE	rows with current/lag/lead weeks
2	filter	rows where lag = current - 1 week OR lead = current + 1 week
3	join + filter	contacts with trial_request AND streak

Worked-example solution.

WITH consecutive_events AS (
    SELECT event_id, contact_id, event_type,
           DATE_TRUNC('week', event_date) AS current_week,
           LAG(DATE_TRUNC('week', event_date))
              OVER (PARTITION BY contact_id ORDER BY DATE_TRUNC('week', event_date)) AS lag_week,
           LEAD(DATE_TRUNC('week', event_date))
              OVER (PARTITION BY contact_id ORDER BY DATE_TRUNC('week', event_date)) AS lead_week
    FROM marketing_touches
)
SELECT DISTINCT c.email
FROM consecutive_events e
INNER JOIN crm_contacts c ON e.contact_id = c.contact_id
WHERE (e.lag_week  = e.current_week - INTERVAL '1 week'
    OR e.lead_week = e.current_week + INTERVAL '1 week')
  AND e.contact_id IN (
      SELECT contact_id FROM marketing_touches WHERE event_type = 'trial_request'
  );

Rule of thumb: use a CTE the moment your window logic needs more than one step; deeply nested subqueries with embedded OVER (...) clauses are unreadable.

Common beginner mistakes

Forgetting PARTITION BY contact_id — LAG reaches into the previous contact's events and produces meaningless deltas.
Forgetting ORDER BY week_trunc — row order is non-deterministic; LAG returns whichever row the planner happened to pick.
Using WHERE to filter aggregates — WHERE COUNT(*) > 1 is a parse error; use HAVING or wrap in a subquery / CTE.
Counting calendar weeks via (event_date - lag_event_date) / 7 instead of DATE_TRUNC('week', ...) — fails on edge cases like daylight-saving-time crossings.
Hardcoding the offset (LAG(volume, 1)) when the prompt says "vs 7 days ago" — read the spec and use LAG(volume, 7).

SQL Interview Question on Marketing-Touch Streak Detection

Given marketing_touches(event_id, contact_id, event_type, event_date) and crm_contacts(contact_id, email), return the email of every contact who had at least three consecutive-week marketing touches AND at least one event_type = 'trial_request'.

Solution Using `LAG` / `LEAD` window functions inside a CTE

WITH consecutive_events_cte AS (
    SELECT event_id, contact_id, event_type,
           DATE_TRUNC('week', event_date) AS current_week,
           LAG(DATE_TRUNC('week', event_date))
              OVER (PARTITION BY contact_id ORDER BY DATE_TRUNC('week', event_date)) AS lag_week,
           LEAD(DATE_TRUNC('week', event_date))
              OVER (PARTITION BY contact_id ORDER BY DATE_TRUNC('week', event_date)) AS lead_week
    FROM marketing_touches
)
SELECT DISTINCT contacts.email
FROM consecutive_events_cte AS events
INNER JOIN crm_contacts AS contacts
    ON events.contact_id = contacts.contact_id
WHERE (events.lag_week  = events.current_week - INTERVAL '1 week'
    OR events.lead_week = events.current_week + INTERVAL '1 week')
  AND events.contact_id IN (
      SELECT contact_id FROM marketing_touches WHERE event_type = 'trial_request'
  );

Why this works: the CTE truncates each event to its week boundary and uses LAG/LEAD to surface the adjacent weeks inside each contact's partition; the streak condition is satisfied when either the lag or lead week is exactly ±1 week away (which means the contact has at least two consecutive-week events, which combined with the current row means three weeks in a window); the IN subquery enforces the "at least one trial_request" requirement; the INNER JOIN resolves emails; SELECT DISTINCT collapses duplicates from contacts with multiple qualifying weeks.

Step-by-step trace for the DataLemur sample (contact_id 1 across April 17, 23, 30 + May 14):

event_id	contact_id	week_trunc	lag_week	lead_week	streak?
1	1	2022-04-11	NULL	2022-04-18	✓ (lead)
2	1	2022-04-18	2022-04-11	2022-04-25	✓ (both)
3	1	2022-04-25	2022-04-18	NULL	✓ (lag)

CTE materializes — three rows per contact_id with current/lag/lead weeks.
Streak filter — every row of contact_id 1 passes (lag or lead = current ± 7 days).
trial_request filter — contact_id 1 has event_id 2 = trial_request → passes.
Inner-join + DISTINCT — emits andy.markus@att.net once.

Output:

email
andy.markus@att.net

Why this works — concept by concept:

DATE_TRUNC('week', event_date) — snaps timestamps to the week boundary so consecutive-week comparisons are exact; works in Snowflake, Postgres, BigQuery alike.
LAG / LEAD window — surfaces adjacent rows inside the per-contact partition without a self-join; O(n log n) with one window sort instead of O(n²) with a self-join.
CTE composition — keeps the window logic separate from the streak filter; makes the query readable and the optimizer plan stable.
OR between lag and lead — accepts a row whose neighbor on either side is one week away, which combined with the chain of neighbors implies three consecutive weeks once any qualifying contact has at least three rows.
IN subquery for trial_request — enforces the second condition without a separate JOIN; the optimizer typically inlines this as a semi-join.
O((|events|·log|events|) + |events|·|contacts|) time — the window sort dominates; the join is hash-based.

Inline CTA: More SQL window-function problems and CTE problems on PipeCode.

SQL
Company — Snowflake
Snowflake SQL practice problems

Practice →

SQL
Topic — window functions
SQL window-function problems

Practice →

SQL
Topic — CTE
SQL CTE problems

Practice →

4. Snowflake Architecture: Micro-Partitions, Clustering, and Time Travel

Snowflake's three-layer architecture and product-knowledge primitives

"Design Snowflake clustering, virtual warehouse, and Time Travel retention for a 10TB events table queried frequently by date and region" is the canonical Snowflake architecture interview prompt. The mental model: Snowflake has three independent layers — Database Storage (immutable micro-partitions in S3 / Azure Blob / GCS), Query Processing (Virtual Warehouses, scale-up for compute, scale-out for concurrency), and Cloud Services (auth, metadata, optimizer, security). Performance comes from micro-partition pruning (the optimizer skips partitions whose min/max metadata excludes the WHERE predicate) and clustering keys that co-locate rows on chosen columns. Recovery comes from Time Travel (1–90 days of point-in-time queries) plus Fail-safe (7 additional days of Snowflake-managed recovery). Cloning comes from Zero-Copy Cloning (instant snapshot via metadata pointers, no storage duplication).

Pro tip: Snowflake is OLAP, not OLTP — designed for analytical workloads, not row-level UPDATE/DELETE traffic. It is also NOT an ETL tool — pair it with Airflow / dbt / Matillion for orchestration. Stating both framings unprompted in the interview signals product fluency and cuts off the most common follow-up traps.

Three-layer architecture: storage / query processing / cloud services

The architecture invariant: Snowflake separates storage, compute, and services so each scales independently — store petabytes while compute is off, spin up massive compute for a 10-minute job, and let the services layer manage metadata and optimization. Traditional warehouses (Teradata, Netezza) tightly couple storage and compute, forcing over-provisioning.

Database Storage — columnar micro-partitions in cloud storage; immutable; you don't manage files.
Query Processing — Virtual Warehouses; clusters of CPU/RAM/SSD; XS, S, M, L, XL, 2XL, 3XL, 4XL, 5XL, 6XL sizes.
Cloud Services — auth, RBAC, metadata, optimizer, security; the "brain"; serverless from your perspective.
Cross-cloud — runs on AWS, Azure, GCP; cross-cloud replication for disaster recovery.

Worked example. Sketch the layer responsibilities for a typical events table workload.

layer	responsibility	scaling lever
Storage	persist micro-partitions; immutable	grow with data volume
Query Processing	execute SQL on Virtual Warehouses	scale up (size) or out (clusters)
Cloud Services	auth, metadata, optimizer	serverless

Worked-example solution.

-- size a warehouse for the workload
CREATE WAREHOUSE events_wh
  WAREHOUSE_SIZE = 'LARGE'
  AUTO_SUSPEND   = 60
  AUTO_RESUME    = TRUE
  MIN_CLUSTER_COUNT = 1
  MAX_CLUSTER_COUNT = 4;

Rule of thumb: size the warehouse to the heaviest query, set AUTO_SUSPEND = 60 for cost control, and use MIN/MAX_CLUSTER_COUNT for concurrency rather than oversizing a single cluster.

Micro-partitions and pruning via min/max metadata

The pruning invariant: Snowflake stores per-column min/max (and other) metadata for every micro-partition; on a WHERE predicate, the optimizer reads metadata and skips partitions whose value range cannot satisfy the predicate. This is "automatic indexing" — no CREATE INDEX required.

Micro-partition — 50–500 MB compressed, columnar, immutable.
Min/max metadata — per-column, per-partition; enables WHERE date >= '2023-01-01' to skip partitions whose max date < 2023-01-01.
Clustering — ensures min/max metadata is selective by co-locating rows on the chosen columns.
Search Optimization Service — adds point-lookup acceleration on top of pruning for high-cardinality predicates.

Worked example. Pruning a query against a 1000-partition events table.

predicate	partitions scanned	pruned
`WHERE date = '2023-06-15'`	~3 (one per cluster region)	997
`WHERE region = 'US-EAST'`	~250 (with clustering on region)	750
`WHERE customer_id = 12345`	1000 (no clustering on customer_id)	0

Worked-example solution.

-- enable clustering for predictable pruning
ALTER TABLE events
  CLUSTER BY (DATE_TRUNC('day', event_date), region);

Rule of thumb: cluster on the columns you filter / aggregate by most often (typically date + a low-cardinality dimension); never cluster on a high-cardinality random column like user_id.

Clustering keys + scale-up vs scale-out + Time Travel + Zero-Copy Cloning

The composition invariant: Snowflake's full performance + recovery story stacks four primitives — clustering keys for storage layout, warehouse sizing for query speed (scale up) and concurrency (scale out), Time Travel for point-in-time recovery, and Zero-Copy Cloning for instant dev/QA environments. Each is independent; combine them per workload requirements.

CLUSTER BY (col1, col2) — co-locates rows on the chosen columns; benefits pruning.
Scale up — WAREHOUSE_SIZE = 'XLARGE'; doubles compute per cluster; speeds up one big query.
Scale out — MAX_CLUSTER_COUNT = 4; spins up additional clusters for concurrent queries.
Time Travel — DATA_RETENTION_TIME_IN_DAYS = 7; query past data with AT (TIMESTAMP => '...').
Zero-Copy Cloning — CREATE TABLE events_dev CLONE events; — instant, no storage duplication.

Worked example. Warehouse + clustering + Time Travel + clone for the events table.

concern	setting
query speed	`WAREHOUSE_SIZE = 'LARGE'`
concurrency	`MAX_CLUSTER_COUNT = 4`
pruning	`CLUSTER BY (DATE_TRUNC('day', event_date), region)`
recovery window	`DATA_RETENTION_TIME_IN_DAYS = 7`
dev environment	`CREATE TABLE events_dev CLONE events;`

Worked-example solution.

ALTER TABLE events
  SET DATA_RETENTION_TIME_IN_DAYS = 7
      CLUSTER BY (DATE_TRUNC('day', event_date), region);

CREATE TABLE events_dev CLONE events;

-- query a past state
SELECT *
FROM events
AT (TIMESTAMP => '2026-04-30 09:00:00'::timestamp);

Rule of thumb: set Time Travel retention to match your incident-response window (typically 7 days for production, 1 day for staging); the higher the retention, the higher the storage cost.

Common beginner mistakes

Calling Snowflake an ETL tool — it is a data warehouse; pair it with Airflow / dbt / Matillion / Airbyte.
Calling Snowflake OLTP — singleton UPDATE/DELETE traffic is poorly served; Snowflake is OLAP-first.
Asking "how do I create an index?" — Snowflake doesn't use traditional indexes; min/max metadata on micro-partitions handles pruning automatically.
Suggesting VACUUM / ANALYZE — Snowflake is fully managed; no manual maintenance.
Storing Time Travel as a backup substitute — Time Travel is for accidental drops within a window; Fail-safe is the 7-day Snowflake-managed safety net beyond Time Travel; for long-term backups use replicated databases or unloaded files.

SQL Interview Question on Snowflake Architecture Design

Design the Snowflake configuration for a 10TB events table that is queried daily by date and region, must support 4-day point-in-time recovery, and needs an isolated dev environment for testing schema changes. Specify the warehouse size, clustering keys, Time Travel retention, and clone strategy.

Solution Using clustering + multi-cluster warehouse + 7-day Time Travel + Zero-Copy Clone

-- 1. Warehouse: scale up (LARGE) for query speed; scale out (max 4 clusters) for concurrency
CREATE OR REPLACE WAREHOUSE events_wh
  WAREHOUSE_SIZE    = 'LARGE'
  AUTO_SUSPEND      = 60
  AUTO_RESUME       = TRUE
  MIN_CLUSTER_COUNT = 1
  MAX_CLUSTER_COUNT = 4;

-- 2. Clustering: co-locate rows by query predicates (date + region)
ALTER TABLE events
  CLUSTER BY (DATE_TRUNC('day', event_date), region)
  SET DATA_RETENTION_TIME_IN_DAYS = 7;

-- 3. Dev environment: instant zero-copy clone
CREATE OR REPLACE TABLE events_dev CLONE events;

-- 4. Verify pruning effectiveness on a sample query
SELECT *
FROM events
WHERE event_date = '2026-04-30'
  AND region = 'US-EAST'
LIMIT 10;

Why this works: the LARGE warehouse fits a 10TB scan into memory + local SSD without spilling; auto-suspend at 60s + auto-resume eliminates idle compute cost; clustering on (DATE_TRUNC('day', event_date), region) aligns the natural query predicates with the partition layout so pruning scans only ~3 partitions for a single-day single-region query (down from ~1000); 7-day Time Travel covers a 4-day recovery requirement with margin; Zero-Copy Cloning produces events_dev instantly, with metadata pointers — the dev table only consumes storage as it diverges from production.

Step-by-step trace of the design rationale:

Workload analysis — 10TB table; daily queries by date + region; 4-day recovery requirement; dev environment needed.
Compute layer — LARGE warehouse for the heaviest query; multi-cluster (1–4) for concurrency without oversize.
Storage layer — cluster on (date, region) so min/max metadata aligns with predicates → pruning skips ~99.7% of partitions on point queries.
Recovery layer — 7-day Time Travel covers 4-day requirement with safety margin; Fail-safe gives an additional 7 days of Snowflake-managed recovery.
Dev layer — Zero-Copy Clone produces instant dev table; storage diverges only on writes.

Output:

layer	setting	rationale
Compute	`WAREHOUSE_SIZE = 'LARGE'`, `MAX_CLUSTER_COUNT = 4`	scale up + scale out
Storage	`CLUSTER BY (DATE_TRUNC('day', event_date), region)`	maximize pruning on query predicates
Recovery	`DATA_RETENTION_TIME_IN_DAYS = 7`	covers 4-day requirement with margin
Dev	`CREATE TABLE events_dev CLONE events;`	instant zero-copy environment
Cost	auto-suspend = 60s	idle compute paid only in seconds

Why this works — concept by concept:

Three-layer separation — independent scaling of storage / compute / services; the compute warehouse can be 'LARGE' today and `XSMALL' tomorrow without touching data.
Micro-partition pruning — min/max metadata on each partition + clustering on predicate columns → the optimizer skips ~99.7% of partitions on a single-day single-region query.
Multi-cluster warehouse — MAX_CLUSTER_COUNT = 4 handles concurrency; instead of queueing 100 simultaneous queries on one cluster, additional clusters spin up.
7-day Time Travel — AT (TIMESTAMP => ...) queries any past state within retention; covers accidental drops, schema mistakes, and the stated 4-day recovery requirement.
Zero-Copy Cloning — CREATE TABLE ... CLONE ... is instant; storage is metadata-only until divergence; perfect for dev/QA from production.
Cost via auto-suspend — AUTO_SUSPEND = 60 charges only for compute-active seconds; storage and services are billed separately and remain on while compute sleeps.
O(partitions_scanned / partitions_total) pruning ratio — clustering on (date, region) aligns the natural query predicates with the partition layout, so a single-day single-region query scans ~3 of ~1000 partitions (≈ 0.3%); the remaining 99.7% are pruned via min/max metadata before any data is read.

Inline CTA: More dimensional-modeling problems and the Snowflake practice page for the full curated set.

SQL
Company — Snowflake
Snowflake SQL practice problems

Practice →

DATA-MODELING
Topic — dimensional modeling
Dimensional-modeling problems

Practice →

SQL
Topic — JSON / VARIANT
JSON / VARIANT SQL problems

Practice →

Tips to crack Snowflake data engineering interviews

Two audiences — pick yours and rebalance the four primitives

The phrase "Snowflake interview" splits cleanly. Audience A — interviewing AT Snowflake the company — the loop is LeetCode-style Python: the curated 2 problems (#24 SET Card Game Validation, #161 Maximum Substring Occurrences) hint at the bar, and the algorithm fluency in §1 and §2 carries the round. Audience B — interviewing for a data-engineering role at a company that uses Snowflake — the loop is SQL window functions plus product knowledge, and §3 plus §4 carry the round. Pick yours upfront and rebalance prep time accordingly.

Drill the four primitives

The four primitives in this guide map directly to the curated 2 PipeCode Python problems plus the two adjacent primitives every Snowflake-flavored interview rotates through: Python zip(*cards) + set + all for SET-style validation (Python EASY — #24), Python rolling freq dict + Counter over substrings for windowed counting (Python MEDIUM — #161), SQL LAG/LEAD/AVG OVER PARTITION BY plus CTE composition for streak detection and analytical aggregations (SQL — DataLemur staple), and Snowflake architecture (3 layers, micro-partitions + clustering + Time Travel + Zero-Copy Cloning). Each maps to a specific module: vanilla dict and collections.Counter for the Python primitives, OVER (PARTITION BY ... ORDER BY ...) for windows, and the Snowflake DDL surface (CLUSTER BY, DATA_RETENTION_TIME_IN_DAYS, CLONE, WAREHOUSE_SIZE) for architecture.

Memorize the OLAP-not-OLTP and NOT-an-ETL-tool framing

The single most common Snowflake interview trap is treating Snowflake as a traditional RDBMS or as an ETL tool. State unprompted: "Snowflake is OLAP — analytical workloads, not row-level UPDATE/DELETE traffic; it pairs with Airflow / dbt / Matillion for orchestration; it doesn't use traditional indexes — micro-partition min/max metadata + clustering does the pruning automatically." This single framing flip cuts off the most common follow-up traps before they fire.

LeetCode-style fluency for Snowflake-the-company

The 2 PipeCode problems hint at the actual Snowflake-the-company DE / SWE interview bar: algorithm fluency in Python, not Snowflake product trivia. The interview tests zip(*iter) + set composition and rolling freq dict + Counter patterns, not "explain virtual warehouses." Drill the Python practice page and the easy practice page until the canonical EASY-tier code rolls off your fingers in under three minutes.

Snowpipe + Time Travel + Zero-Copy Cloning are the signature features — name them out loud

For Snowflake-as-tool interviews, the three signature product features that separate fluent candidates from bluffers are Snowpipe (serverless continuous ingestion via cloud-event notifications), Time Travel (1–90 day point-in-time recovery + 7-day Fail-safe beyond), and Zero-Copy Cloning (instant dev/QA snapshots via metadata pointers). Name them unprompted when discussing ingestion, recovery, or dev environments — interviewers grade product fluency and these three are the highest-signal features.

Easy-Medium discipline matters

The curated Snowflake set on PipeCode is 1 EASY + 1 MEDIUM Python. EASY at Snowflake doesn't mean trivial — it means the interviewer expects zero hesitation, idiomatic Python, and an articulated invariant. A correct EASY answer with stuttering on zip(*cards) or len(set(...)) in {1, 3} is graded worse than a correct MEDIUM answer with the same flaw. Drill the Easy practice page and the Medium practice page until the canonical EASY-tier code rolls off your fingers in under three minutes.

Where to practice on PipeCode

Start with the Snowflake practice page and the language-scoped Snowflake Python practice page for the curated 2-problem set. After that, drill the matching topic pages: array, hash table, sliding window, string, window functions, CTE, aggregation, dimensional modeling, JSON. The interview courses page bundles structured curricula. For broader coverage, browse by topic, or pivot to peer interview guides — the Airbnb DE interview guide and the top DE interview questions 2026 blog.

Communication and approach under time pressure

Talk through the invariant first ("this is a sliding-window plus hash-table problem"), the brute force second ("a triple-nested loop over all length-k substrings would also work"), and the optimal third ("but the rolling freq dict gives O(1) per shift instead of O(k)"). Interviewers grade process as much as the final answer. Leave 5 minutes for an edge-case sweep: empty input, k > len(s), single-character strings, maxLetters = 0, cards with 0 attributes, NULL in a window-function partition. The most common "almost passed" failure mode is correct happy-path code that crashes on edge cases — a 30-second sweep prevents it.

Frequently Asked Questions

What is the Snowflake data engineering interview process like?

The Snowflake data engineering interview opens with a 30-minute recruiter screen, then a 60-minute technical phone screen with a live SQL or Python coding problem, then a 4-round virtual onsite: one or two coding rounds (Python algorithms or SQL window functions, depending on whether the role is at Snowflake the company or at a Snowflake-using company), one system-design round (designing data pipelines, warehouse architecture, or trade reconciliation), one data-modeling discussion (star vs snowflake schema, dimensional modeling), and a behavioral round. End-to-end the loop runs three to four weeks.

What programming languages does Snowflake test in data engineering interviews?

Snowflake data engineering interviews are bilingual — SQL and Python in roughly equal measure across the loop. Python concentrates on array iteration with set-based validation, hash-table and sliding-window patterns, and dict-counter idioms. SQL concentrates on window functions (LAG, LEAD, AVG OVER PARTITION BY), CTE composition, aggregation, and Snowflake's specific dialect features (DATE_TRUNC, DATEADD, QUALIFY, VARIANT / JSON queries). Snowpark (Python/Java/Scala on Snowflake compute) appears at senior backend-leaning DE roles but is rarely tested in the live coding rounds.

How difficult are Snowflake data engineering interview questions?

The curated Snowflake practice set on PipeCode is 1 easy + 1 medium, no hard. The EASY is a Python array + set validation problem (SET Card Game Validation); the MEDIUM is a Python hash-table + sliding-window problem (Maximum Substring Occurrences). At the onsite, system-design and architecture questions can reach L4-L5 level — designing the warehouse + clustering + Time Travel for a 10TB events table, choosing scale-up vs scale-out — but the live coding rounds stay in the EASY-MEDIUM zone. Stuttering on the EASY is a stronger negative signal than struggling with the MEDIUM.

How should I prepare for a Snowflake data engineer interview?

Solve the 2 problems on the Snowflake practice page end-to-end — untimed first, then timed at 25 minutes per problem — and broaden to 30 to 50 additional problems across the matching topic pages: array, hash table, sliding window, string on the Python side, and window functions, CTE, aggregation, dimensional modeling on the SQL side. Read the Snowflake architecture chapter of the official documentation for a week (three layers, micro-partitions, virtual warehouses, Time Travel, Snowpipe, Zero-Copy Cloning). Practice articulating idempotency and clustering choices when discussing warehouse design — those framings are graded heavily at any Snowflake-flavored interview, and they're the difference between strong "Snowflake interview questions and answers" and bluff-level definitions.

What Snowflake-specific topics show up most often in interviews?

Five Snowflake-specific topics dominate every interview list — Snowflake architecture (the three-layer split between Database Storage, Query Processing via Virtual Warehouses, and Cloud Services); micro-partitions and pruning (50–500MB columnar blocks with min/max metadata that the optimizer uses to skip irrelevant partitions); Snowflake window functions (LAG, LEAD, AVG ... OVER (PARTITION BY ...) plus QUALIFY for window-result filtering); Time Travel + Fail-safe (1–90 day point-in-time recovery plus 7 days of Snowflake-managed safety net); and Zero-Copy Cloning (instant snapshots for dev/QA via metadata pointers). Master these five and you can answer any Snowflake-product question on any interview list.

Is Snowflake an ETL tool, and how does that affect the interview?

No — Snowflake is not an ETL tool. It is a cloud data warehouse designed for storing and analyzing data; ETL/ELT orchestration runs in tools like Apache Airflow, dbt, Matillion, Coalesce.io, or Airbyte that load data into Snowflake. This distinction shows up in interview "Snowflake ETL interview questions" — the correct answer is to clarify the architecture (Snowflake as the destination warehouse, an external orchestrator for transformations) and walk through the typical loading patterns (COPY INTO for bulk historical loads, Snowpipe for serverless continuous ingestion, External Tables for in-place queries against cloud storage). Stating "Snowflake is not an ETL tool" unprompted is one of the highest-signal product-fluency moves a candidate can make.

Start practicing Snowflake data engineering problems

Robinhood Data Engineering Interview Questions & Prep Guide

Gowtham Potureddi — Sun, 03 May 2026 05:19:18 +0000

Robinhood data engineering interview questions are bilingual — SQL and Python in roughly equal measure — with a fintech-correctness edge that most generic interview-prep posts miss. Four primitives carry the loop: dict.get(s, 0) + 1 hash-table counters that aggregate stock-purchase events by symbol, INNER JOIN trades + users + GROUP BY + ORDER BY count DESC LIMIT N for top-N city / member-transfer rankings, LAG(volume) OVER (PARTITION BY stock_symbol ORDER BY trade_date) for day-over-day volume or balance change, and GROUP BY user_id HAVING SUM(notional) > limit for end-of-day threshold and notional-cap checks. The framings are everyday brokerage data engineering — count purchases per ticker, surface the top cities by completed trades, compute a daily volume percentage change, flag any account whose option exposure crosses a regulatory limit.

This guide walks through the four topic clusters Robinhood actually tests, each with a detailed topic explanation, per-sub-topic explanation with a worked example and its solution, and an interview-style problem with a full solution that explains why it works. The mix matches a curated 2-problem Robinhood set (1 EASY Python hash-table + 1 MEDIUM SQL joins) plus the two SQL primitives — window LAG and HAVING-aggregate — that show up on every Robinhood SQL question list and at every L4/L5 onsite. Robinhood pipelines also demand penny-perfect correctness, idempotency, and audit-aware thinking; brokerage candidates who frame their answers in those terms separate themselves from the generic-DE pile.

1. Python Hash Tables and Dict Counters for Stock-Purchase Aggregation

Hash-table dict counting for stock-purchase events in Python for data engineering

"Given a list of stock-purchase events, count purchases per symbol and surface the set of distinct buyers per symbol" is Robinhood's signature EASY Python prompt (Stock Purchases Count). The mental model: a hash-table counter is a dict keyed on the grouping attribute with a count value, updated in a single pass with counts[k] = counts.get(k, 0) + 1; a parallel dict[str, set] keyed the same way tracks distinct members per group. Same primitive powers any "count by category" or "unique-elements per bucket" pipeline — count clicks per user, distinct sessions per page, purchases per ticker.

Pro tip: The dict.get(k, 0) + 1 idiom is the workhorse — but collections.Counter is the idiomatic shortcut and collections.defaultdict(int) is the common middle ground. Pick the one your interviewer can read fastest: Counter for "count occurrences," defaultdict(set) for "track unique members per group," vanilla dict.get when you want zero imports. State which you chose and why.

Dict counter idiom: `dict.get(k, 0) + 1` vs `collections.Counter`

The dict-counter invariant: a dict keyed on the grouping attribute holds the running count for that group; on each event you read the current count (or 0 if absent), add one, and write it back. dict.get(k, default) returns default when k is missing, which avoids the KeyError that bare dict[k] would raise.

counts[k] = counts.get(k, 0) + 1 — vanilla dict, no imports, the universal pattern.
counts = Counter(events) — one-liner if events is the iterable of keys themselves (no extra fields).
counts = defaultdict(int); counts[k] += 1 — middle ground, handles missing keys via the factory.
Avoid try/except KeyError — works but is slower and harder to read than get or defaultdict.

Worked example. Count three purchase events for AAPL, TSLA, AAPL using the dict.get idiom.

step	counts
start	`{}`
`AAPL`	`{'AAPL': 1}`
`TSLA`	`{'AAPL': 1, 'TSLA': 1}`
`AAPL`	`{'AAPL': 2, 'TSLA': 1}`

Worked-example solution.

counts = {}
for symbol in ['AAPL', 'TSLA', 'AAPL']:
    counts[symbol] = counts.get(symbol, 0) + 1
# counts == {'AAPL': 2, 'TSLA': 1}

Rule of thumb: when the count is the only signal, Counter(iter) is shortest; when you also need other per-group state (a set of distinct buyers, a running max, a list of timestamps), reach for defaultdict(set) or vanilla dict.get with explicit logic.

Set-of-buyers per stock: dict-of-sets pattern

The dict-of-sets invariant: a dict keyed on the grouping attribute holds a set of distinct member ids; each event inserts the member id, and set semantics dedupe automatically. Inserting the same (symbol, user) pair twice is a no-op — no special handling required, no if member in s guard.

buyers.setdefault(symbol, set()).add(user) — one-liner that initializes the set on first touch.
buyers = defaultdict(set); buyers[symbol].add(user) — same outcome, slightly cleaner.
len(buyers[symbol]) — distinct-buyer count for any symbol.
Set membership — O(1) average; never use a list here, which would be O(N) for if x in lst.

Worked example. Track distinct buyers for AAPL across three events: (u1, AAPL), (u2, AAPL), (u1, AAPL).

step	buyers['AAPL']
start	`set()`
`u1`	`{'u1'}`
`u2`	`{'u1', 'u2'}`
`u1` (dup)	`{'u1', 'u2'}`

Worked-example solution.

from collections import defaultdict
buyers = defaultdict(set)
for user, symbol in [('u1', 'AAPL'), ('u2', 'AAPL'), ('u1', 'AAPL')]:
    buyers[symbol].add(user)
# buyers == {'AAPL': {'u1', 'u2'}}

Rule of thumb: defaultdict(set) collapses the "first-time init" branch — never write if k not in d: d[k] = set() when defaultdict exists.

One-pass aggregation in a single for-loop

The single-pass invariant: two parallel dicts updated in the same for loop produce both aggregates in O(N) total time and O(K) space, where K is the number of distinct symbols. Iterating twice — once for counts, once for buyer-sets — works but doubles the work; one pass is idiomatic and what interviewers grade.

One loop, two updates — counts[s] = counts.get(s, 0) + 1 and buyers.setdefault(s, set()).add(u) per event.
Avoid two passes — O(2N) == O(N) asymptotically but reads as wasteful.
Tuple unpack the event — for user, symbol, qty in events: is cleaner than indexing event[0], event[1].
Skip filters first — if the prompt says "only Completed trades," guard with if status != 'Completed': continue at the top.

Worked example. Aggregate the four-event stream into both dicts in one pass.

event	counts	buyers
`(u1, AAPL)`	`{'AAPL': 1}`	`{'AAPL': {'u1'}}`
`(u2, TSLA)`	`{'AAPL': 1, 'TSLA': 1}`	`{'AAPL': {'u1'}, 'TSLA': {'u2'}}`
`(u1, AAPL)`	`{'AAPL': 2, 'TSLA': 1}`	`{'AAPL': {'u1'}, 'TSLA': {'u2'}}`
`(u3, AAPL)`	`{'AAPL': 3, 'TSLA': 1}`	`{'AAPL': {'u1', 'u3'}, 'TSLA': {'u2'}}`

Worked-example solution.

from collections import defaultdict
counts: dict[str, int] = {}
buyers: dict[str, set] = defaultdict(set)
for user, symbol in [('u1', 'AAPL'), ('u2', 'TSLA'), ('u1', 'AAPL'), ('u3', 'AAPL')]:
    counts[symbol] = counts.get(symbol, 0) + 1
    buyers[symbol].add(user)

Rule of thumb: one loop, two parallel structures, zero if k in d branches — reach for defaultdict and dict.get first.

Common beginner mistakes

Using counts[k] += 1 on a missing key — raises KeyError. Use dict.get(k, 0) + 1 or defaultdict(int).
Using a list instead of a set for distinct buyers — O(N) membership checks blow up the asymptotic cost.
Iterating twice instead of once — wastes work and signals the candidate didn't think about cost.
Forgetting to skip filtered events at the top of the loop — counts include cancelled / failed trades and produce the wrong answer.
Returning dict.values() directly when the test expects a plain list — wrap with list(...) if the contract demands it.

Python Interview Question on Stock-Purchase Aggregation

Given a list of completed stock-purchase events (user_id, symbol, quantity), return a dict mapping each symbol to a tuple of (a) the total purchase count and (b) the set of distinct user_id values that bought it.

def stock_purchases(events):
    pass  # your code here

Solution Using `dict.get` and `defaultdict(set)` in one pass

from collections import defaultdict

def stock_purchases(events):
    counts: dict[str, int] = {}
    buyers: dict[str, set] = defaultdict(set)
    for user_id, symbol, _qty in events:
        counts[symbol] = counts.get(symbol, 0) + 1
        buyers[symbol].add(user_id)
    return {s: (counts[s], buyers[s]) for s in counts}

Why this works: the dict.get(symbol, 0) + 1 idiom keeps the count loop branch-free; defaultdict(set) keeps the buyer-set loop branch-free; one pass over events builds both structures in O(N) time. The final dict-comprehension stitches them together so each symbol maps to (count, set_of_buyers) in a single return value.

Step-by-step trace for events = [('u1', 'AAPL', 50), ('u2', 'TSLA', 10), ('u1', 'AAPL', 25), ('u3', 'AAPL', 5), ('u2', 'GOOG', 8)]:

step	event	counts	buyers
0	start	`{}`	`defaultdict(set)`
1	`(u1, AAPL, 50)`	`{'AAPL': 1}`	`{'AAPL': {'u1'}}`
2	`(u2, TSLA, 10)`	`{'AAPL': 1, 'TSLA': 1}`	`{'AAPL': {'u1'}, 'TSLA': {'u2'}}`
3	`(u1, AAPL, 25)`	`{'AAPL': 2, 'TSLA': 1}`	`{'AAPL': {'u1'}, 'TSLA': {'u2'}}`
4	`(u3, AAPL, 5)`	`{'AAPL': 3, 'TSLA': 1}`	`{'AAPL': {'u1', 'u3'}, 'TSLA': {'u2'}}`
5	`(u2, GOOG, 8)`	`{'AAPL': 3, 'TSLA': 1, 'GOOG': 1}`	`{'AAPL': {'u1', 'u3'}, 'TSLA': {'u2'}, 'GOOG': {'u2'}}`

After the loop, the dict-comprehension {s: (counts[s], buyers[s]) for s in counts} produces the merged output.

Output:

symbol	count	distinct_buyers
AAPL	3	`{'u1', 'u3'}`
TSLA	1	`{'u2'}`
GOOG	1	`{'u2'}`

Why this works — concept by concept:

Hash-table dict counter — counts[symbol] = counts.get(symbol, 0) + 1 updates per group in O(1) average; the dict.get default of 0 keeps the new-key branch implicit and unbranched.
defaultdict(set) factory — buyers[symbol].add(user_id) initializes a fresh set on first touch; subsequent .add calls dedupe at insertion time, so distinct counting is free.
Single-pass loop — two parallel updates inside one for keep total work at O(N); iterating events twice would double the work for the same answer.
Dict-comprehension merge — {s: (counts[s], buyers[s]) for s in counts} stitches the two parallel structures into the contract-shape return without an explicit accumulator.
O(N) time / O(K) space — N events, each touching two hash maps; K distinct symbols hold the aggregate state. No sorting, no nested scans.

Inline CTA: Drill the Robinhood Python practice page for the curated hash-table problem and the hash-table Python practice page for breadth.

PYTHON
Company — Robinhood
Robinhood data engineering problems

Practice →

PYTHON
Topic — hash table
Python hash-table problems

Practice →

PYTHON
Topic — dictionary
Python dictionary problems

Practice →

2. SQL Inner Join and GROUP BY for Top-N Trade Aggregations

Inner-join + group-by + top-N rankings in SQL for Robinhood data engineering

"Given a trades table and a users (or members) table, return the top three cities by completed trade orders" is Robinhood's signature MEDIUM SQL prompt — the same shape as PipeCode's #247 Member Transfer Records and DataLemur's #1 Cities With Completed Trades. The mental model: INNER JOIN the fact table to the dim table on the shared id, filter rows with WHERE status = 'Completed', group by the dim attribute, count or sum, then ORDER BY count DESC LIMIT N for the top-N. The same primitive powers any "rank entities by event volume" pipeline — top countries by signups, top sectors by trade flow, top symbols by buy-side volume.

Pro tip: Filter early with WHERE (per-row), not late with HAVING (per-group). WHERE status = 'Completed' removes cancelled trades before the GROUP BY, so the COUNT(*) is correct. Doing it after the group with HAVING requires an aggregate predicate and is slower; doing it not at all silently inflates the count with cancellations.

`INNER JOIN` cardinality: trades 1-to-many users

The join-cardinality invariant: INNER JOIN trades ON trades.user_id = users.user_id produces one output row per matching (trade, user) pair. Since each trade has exactly one user but a user can have many trades, the join expands the user table by trade count — the result has one row per trade. INNER JOIN drops trades whose user_id has no match in users (orphans).

INNER JOIN — only matching pairs survive.
LEFT JOIN — keeps trades with no user match (NULL city); use when you must report orphans.
FULL OUTER JOIN — also keeps users with zero trades; useful for reconciliation, not for top-N.
Pick the smallest table on the right — modern planners are smart, but explicit LEFT/INNER choice signals intent.

Worked example. Two-row trades, two-row users; INNER JOIN on user_id.

trade	user	city
100101 (u=111)	u=111	San Francisco
100259 (u=148)	u=148	Boston

Worked-example solution.

SELECT t.order_id, u.city
FROM trades t
INNER JOIN users u ON t.user_id = u.user_id;

Rule of thumb: INNER JOIN is the default for "this trade definitely has a user"; reach for LEFT JOIN only when orphans are part of the answer.

`WHERE` before `GROUP BY`: filtering rows vs filtering groups

The filter-order invariant: WHERE runs per-row before grouping; HAVING runs per-group after. WHERE status = 'Completed' strips out cancelled and pending trades so the subsequent GROUP BY city only counts what the question asks about. Putting the same predicate in HAVING requires HAVING SUM(CASE WHEN status='Completed' THEN 1 ELSE 0 END) > 0 — verbose, slower, and a "bad signal" in the round.

WHERE clause — row-level filter; runs first; uses =, <, IN, LIKE, IS NULL.
HAVING clause — group-level filter; runs after GROUP BY; uses SUM, COUNT, AVG, MAX.
Order in the query — SELECT ... FROM ... JOIN ... WHERE ... GROUP BY ... HAVING ... ORDER BY ... LIMIT.
Performance — WHERE-then-group reads less data from disk because the grouper sees fewer rows.

Worked example. Filter to status = 'Completed' before grouping by city.

input rows	after WHERE	grouped city → count
7	5 (2 dropped)	SF: 3, Boston: 2

Worked-example solution.

SELECT u.city, COUNT(*) AS total
FROM trades t
INNER JOIN users u ON t.user_id = u.user_id
WHERE t.status = 'Completed'
GROUP BY u.city;

Rule of thumb: push every row-level predicate into WHERE; reserve HAVING strictly for predicates that need an aggregate.

`ORDER BY count DESC + LIMIT` for top-N rankings

The top-N invariant: ORDER BY <metric> DESC sorts groups by the metric in descending order; LIMIT N returns only the first N. For ties at the cut, dialects differ — Postgres returns whichever order the planner prefers; Redshift / Snowflake similar. If ties matter, add a deterministic tiebreaker like , u.city ASC.

ORDER BY total_orders DESC — sorts groups by the aliased column.
LIMIT 3 — first 3 groups in sort order; standard across MySQL, Postgres, Snowflake, BigQuery (BigQuery uses LIMIT too).
OFFSET — pagination; LIMIT 3 OFFSET 3 returns ranks 4–6.
Tiebreakers — , u.city ASC makes ties deterministic and stable across runs.

Worked example. Sort 4 cities by count and take the top 3.

city	total	rank
SF	3	1
Boston	2	2
Denver	1	3
Austin	1	(cut)

Worked-example solution.

SELECT u.city, COUNT(*) AS total
FROM trades t
INNER JOIN users u ON t.user_id = u.user_id
WHERE t.status = 'Completed'
GROUP BY u.city
ORDER BY total DESC
LIMIT 3;

Rule of thumb: always alias the count column (AS total) and use the alias in ORDER BY; mixing literal COUNT(*) and the alias is style noise.

Common beginner mistakes

Using COUNT(*) when only completed trades should count — forgetting WHERE status = 'Completed' silently inflates totals.
GROUP BY user_id instead of GROUP BY city — confuses the dim level with the fact level; the question asks for cities.
LEFT JOIN when the question asks "completed trades" — orphan trades (no matching user) silently survive with NULL city and pollute the rank.
Forgetting the alias in ORDER BY — ORDER BY COUNT(trades.order_id) DESC works but reads worse than ORDER BY total_orders DESC.
Returning more than N rows by skipping LIMIT — graded as a wrong answer even when the top N are correct.

SQL Interview Question on Top Cities by Completed Trades

Given the tables trades(order_id, user_id, quantity, status, date, price) and users(user_id, city, email, signup_date), write a query that returns the top 3 cities by number of completed trade orders, in descending order. Output two columns: city and total_orders.

Solution Using `INNER JOIN + WHERE + GROUP BY + ORDER BY DESC + LIMIT`

SELECT
    u.city,
    COUNT(t.order_id) AS total_orders
FROM trades t
INNER JOIN users u
    ON t.user_id = u.user_id
WHERE t.status = 'Completed'
GROUP BY u.city
ORDER BY total_orders DESC
LIMIT 3;

Why this works: the INNER JOIN pairs each trade with its user (and drops orphan trades with no user match); WHERE status = 'Completed' removes cancelled trades before grouping; GROUP BY city collapses rows to one-per-city with COUNT(t.order_id) as the metric; ORDER BY total_orders DESC ranks cities; LIMIT 3 returns only the top three.

Step-by-step trace for the Robinhood-style sample data:

order_id	user_id	status	city
100101	111	Cancelled	San Francisco
100102	111	Completed	San Francisco
100259	148	Completed	Boston
100264	148	Completed	Boston
100305	300	Completed	San Francisco
100400	178	Completed	San Francisco
100565	265	Completed	Denver

Inner-join — every trade's user_id matches a row in users; the join produces 7 rows with (order_id, status, city) paired up.
WHERE filter — drop 100101 (Cancelled). 6 rows remain: 3 SF + 2 Boston + 1 Denver.
Group by city — collapse to three groups: SF=3, Boston=2, Denver=1.
Order by count desc — SF (3) > Boston (2) > Denver (1).
Limit 3 — returns all three (no cut needed).

Output:

city	total_orders
San Francisco	3
Boston	2
Denver	1

Why this works — concept by concept:

INNER JOIN cardinality — each trade pairs with exactly one user; orphan trades (none in this dataset) would be dropped, which is the right semantic when the question asks about completed-trade-by-city.
WHERE before GROUP BY — the row-level filter eliminates cancelled trades before grouping, so COUNT(*) reflects only completed orders; doing it after with HAVING would still work but reads worse and runs slower.
GROUP BY u.city — collapses the result to one row per city; the only non-aggregate column in SELECT must appear here (or be functionally dependent on it).
COUNT(t.order_id) — counts non-null order_id values per group; equivalent to COUNT(*) here because order_id is the trades primary key (never null).
ORDER BY total_orders DESC LIMIT 3 — sorts groups by the metric and slices the top three; the alias total_orders keeps the ORDER BY clean.
O((|trades| + |users|) + G log G) time — |trades| rows scanned for the join, G groups sorted (G ≤ |cities|); O(G) space for the group hash table.

Inline CTA: More SQL join problems and the Robinhood SQL practice page for the curated MEDIUM problem.

SQL
Company — Robinhood
Robinhood SQL problems

Practice →

SQL
Topic — joins
SQL join problems

Practice →

SQL
Topic — group by
SQL group-by problems

Practice →

3. SQL Window Functions and `LAG` for Daily Volume Change

Window LAG for day-over-day deltas in SQL for Robinhood data engineering

"For each day and each stock, compute the percentage change in trading volume vs the previous day for the same stock" is a Robinhood SQL staple — the same shape that scales up to "reconstruct an account balance from trade events" at the L4/L5 onsite. The mental model: LAG(value) OVER (PARTITION BY group ORDER BY ts) looks one row back inside the same partition and returns the prior row's value; partition keeps each stock's series independent, ORDER BY fixes the row order, and the day-over-day delta is (value − LAG(value)) / LAG(value). The same window primitive powers any "delta vs previous" query — running balance, running total via SUM, rank within a partition via ROW_NUMBER.

Pro tip: The first row in each partition has no predecessor, so LAG returns NULL. Decide upfront: filter out the NULL row with WHERE prev IS NOT NULL, or pass through with COALESCE(prev, value) to mark "no change." Stating this explicitly is the senior signal interviewers grade.

Window basics: `PARTITION BY group + ORDER BY ordering`

The window-function invariant: OVER (PARTITION BY <group> ORDER BY <ordering>) declares an independent ordered subset for every value of the group expression; the function (LAG, LEAD, SUM, ROW_NUMBER, …) is evaluated within that subset only. PARTITION BY stock_symbol builds one ordered series per ticker; ORDER BY trade_date fixes the row order inside each.

PARTITION BY is optional — omitted, the whole table is one window; useful for global running totals.
ORDER BY is required for offset functions — LAG, LEAD, ROW_NUMBER, RANK all need a defined order.
Frames — ROWS UNBOUNDED PRECEDING / BETWEEN ... AND ... further bound the visible rows; default is fine for LAG.
Multiple windows in one query — different functions can use different OVER (...) clauses on the same row.

Worked example. Two (date, symbol, volume) rows for AAPL; LAG(volume) OVER (PARTITION BY symbol ORDER BY date) returns the previous day's volume.

date	symbol	volume	lag(volume)
2022-07-01	AAPL	1000000	NULL
2022-07-02	AAPL	1500000	1000000

Worked-example solution.

SELECT date, symbol, volume,
       LAG(volume) OVER (PARTITION BY symbol ORDER BY date) AS prev_volume
FROM trading_volume;

Rule of thumb: always set PARTITION BY to the entity that owns the time series (stock_symbol, account_id, user_id); without it, day-1 of one symbol leaks across to compute deltas with a different symbol's day-N.

`LAG` vs `LEAD`: row-N-before vs row-N-after

The offset-function invariant: LAG(expr, n) returns expr from the row n positions before the current row inside the partition; LEAD(expr, n) returns from n positions after. Default n = 1. Both accept a third default argument that replaces NULL at partition boundaries.

LAG(volume) — previous row's volume (1 back).
LAG(volume, 7) — 7 rows back; useful for week-over-week deltas on daily data.
LEAD(volume) — next row's volume (1 ahead).
LAG(volume, 1, volume) — falls back to the current row's volume on the first row, so the % change is 0 instead of NULL.

Worked example. Volume on day 2 vs day 1.

trade_date	volume	LAG(volume)	LEAD(volume)
2022-07-01	1000000	NULL	1500000
2022-07-02	1500000	1000000	1800000
2022-07-03	1800000	1500000	NULL

Worked-example solution.

SELECT trade_date,
       volume,
       LAG(volume) OVER (PARTITION BY stock_symbol ORDER BY trade_date) AS prev_volume,
       LEAD(volume) OVER (PARTITION BY stock_symbol ORDER BY trade_date) AS next_volume
FROM trading_volume;

Rule of thumb: "delta vs yesterday" → LAG; "delta vs tomorrow" or "next-event distance" → LEAD. They are mirror images of the same primitive.

`NULL` on the first row: handling the edge with `COALESCE` or filtering

The boundary invariant: the first row of each partition has no preceding row, so LAG(expr) returns NULL; arithmetic on NULL propagates to NULL. The two clean strategies are: (a) filter out the NULL row (WHERE prev IS NOT NULL) so the result starts on day 2; (b) replace NULL upstream with LAG(expr, 1, default) or COALESCE(LAG(expr), 0) so day 1 has a defined % change (often 0 or NULL-as-string).

WHERE prev IS NOT NULL — drops the first-day row entirely.
LAG(volume, 1, volume) — first-day prev_volume becomes the same row's volume; % change becomes 0.
COALESCE(LAG(volume), 0) — first-day prev_volume becomes 0, which makes the % change blow up (/ 0); usually wrong.
NULLIF(LAG(volume), 0) — guards against zero-volume divisions; complementary to the boundary fix.

Worked example. Drop the first-day NULL on a 3-row series.

trade_date	volume	volume_change_pct
2022-07-02	1500000	50.00
2022-07-03	1800000	20.00

Worked-example solution.

SELECT *
FROM (
    SELECT trade_date,
           stock_symbol,
           volume,
           LAG(volume) OVER (PARTITION BY stock_symbol ORDER BY trade_date) AS prev_volume,
           (volume - LAG(volume) OVER (PARTITION BY stock_symbol ORDER BY trade_date))
             / NULLIF(LAG(volume) OVER (PARTITION BY stock_symbol ORDER BY trade_date), 0) * 100
             AS volume_change_pct
    FROM trading_volume
) sub
WHERE prev_volume IS NOT NULL;

Rule of thumb: state your boundary policy out loud — "I'll drop day 1" or "I'll show day 1 as NULL" — before writing the query; interviewers grade the explicit choice.

Common beginner mistakes

Forgetting PARTITION BY stock_symbol — LAG reaches into the previous symbol's series and computes a meaningless delta.
Forgetting ORDER BY trade_date — LAG returns whichever row the planner happens to pick; the answer is non-deterministic.
Dividing by LAG(volume) when the prior volume could be 0 — runtime divide-by-zero; use NULLIF(LAG(volume), 0).
Using LEAD when the prompt says "vs yesterday" — answers the wrong direction.
Hardcoding the offset (LAG(volume, 1)) when the spec is "vs 7 days ago" — read the spec and use LAG(volume, 7).

SQL Interview Question on Daily Volume Percentage Change

Given a trading_volume(trade_date, stock_symbol, volume) table with one row per (date, symbol), write a query that returns, for each row from day 2 onwards, the daily percentage change in volume vs the previous day for the same symbol. Output trade_date, stock_symbol, and volume_change_pct (rounded to 2 decimals).

Solution Using `LAG` window function

SELECT trade_date,
       stock_symbol,
       ROUND(
         (volume - LAG(volume) OVER (PARTITION BY stock_symbol ORDER BY trade_date))
         / NULLIF(LAG(volume) OVER (PARTITION BY stock_symbol ORDER BY trade_date), 0) * 100,
         2
       ) AS volume_change_pct
FROM trading_volume
QUALIFY LAG(volume) OVER (PARTITION BY stock_symbol ORDER BY trade_date) IS NOT NULL;

(In Postgres, replace QUALIFY with a wrapping subquery and WHERE prev_volume IS NOT NULL — Snowflake / BigQuery / Databricks support QUALIFY directly.)

Why this works: PARTITION BY stock_symbol keeps each ticker's daily series independent; ORDER BY trade_date fixes ordering inside the partition; LAG(volume) reaches one row back to fetch yesterday's volume; the formula (today − yesterday) / yesterday × 100 produces the % change; NULLIF(..., 0) guards against divide-by-zero on zero-volume days; QUALIFY ... IS NOT NULL filters the first-day boundary so output starts on day 2.

Step-by-step trace for AAPL across 5 days:

trade_date	stock_symbol	volume
2022-07-01	AAPL	1000000
2022-07-02	AAPL	1500000
2022-07-03	AAPL	1800000
2022-07-04	AAPL	1750000
2022-07-05	AAPL	1800000

Partition — single partition for AAPL; one ordered series of 5 rows by trade_date ascending.
LAG(volume) — produces [NULL, 1000000, 1500000, 1800000, 1750000] for the 5 rows.
Delta and division — row 2: (1500000 - 1000000) / 1000000 = 0.50; row 3: (1800000 - 1500000) / 1500000 = 0.20; row 4: (1750000 - 1800000) / 1800000 = -0.0278; row 5: (1800000 - 1750000) / 1750000 = 0.0286.
× 100 — convert to percent.
ROUND(..., 2) — round to 2 decimals.
QUALIFY LAG IS NOT NULL — drops row 1 (first day, no predecessor).

Output:

trade_date	stock_symbol	volume_change_pct
2022-07-02	AAPL	50.00
2022-07-03	AAPL	20.00
2022-07-04	AAPL	-2.78
2022-07-05	AAPL	2.86

Why this works — concept by concept:

PARTITION BY stock_symbol — declares one independent window per ticker; without it, AAPL's day-1 would compute a delta against TSLA's day-N.
ORDER BY trade_date — fixes the row order inside each window; offset functions need an explicit ordering or the result is non-deterministic.
LAG(volume) — retrieves the previous row's volume inside the partition; NULL on the first row by design.
NULLIF(LAG(volume), 0) — guards against divide-by-zero on zero-volume days; NULLIF(x, 0) returns NULL when x = 0, propagating cleanly through arithmetic.
QUALIFY LAG IS NOT NULL — drops the first-day row of each partition so the output starts on day 2 with a defined % change; the Postgres equivalent is a subquery WHERE prev_volume IS NOT NULL.
O(N log N) time — the planner sorts each partition once; O(N) space for the running window state. For Robinhood-scale daily volumes this is cheap.

Inline CTA: More window-function problems and SQL aggregation problems for the breadth tier.

SQL
Company — Robinhood
Robinhood SQL problems

Practice →

SQL
Topic — window functions
SQL window-function problems

Practice →

SQL
Topic — aggregation
SQL aggregation problems

Practice →

4. SQL Aggregation and `HAVING` for Threshold and Notional-Limit Checks

`GROUP BY` + `HAVING` for risk-threshold checks in SQL for Robinhood data engineering

"Find every user whose end-of-day option position exceeds the notional limit" is the canonical Robinhood risk / compliance SQL prompt. The mental model: aggregate per user with SUM(contract_count × strike_price × multiplier) to compute notional exposure, then filter the resulting groups with HAVING SUM(...) > limit. HAVING is "WHERE on aggregates" — it filters group rows, not source rows. Same primitive powers any "flag entities whose aggregate metric crosses a policy line" pipeline — flag accounts whose daily trade count exceeds a velocity cap, sectors whose monthly volume exceeds an exposure ceiling, users whose 24-hour withdrawals breach AML thresholds.

Pro tip: Brokerage data is decimal-precision-sensitive. Never store currency or notional as FLOAT / DOUBLE — the rounding errors compound and a 1¢ drift becomes a regulatory event. Use NUMERIC(18, 4) or DECIMAL(18, 4) for prices, quantities, and notionals. State this constraint when you write the schema; interviewers grade it.

`WHERE` vs `HAVING`: rows vs groups

The filter-stage invariant: WHERE filters source rows before grouping; HAVING filters group rows after grouping. WHERE cannot reference aggregates (SUM, COUNT, AVG); HAVING can. The execution order is FROM → WHERE → GROUP BY → HAVING → SELECT → ORDER BY → LIMIT.

WHERE status = 'Open' — row predicate; filters before grouping.
HAVING SUM(notional) > 100000 — group predicate; filters after grouping.
Both in one query — WHERE strips closed positions, HAVING keeps groups whose remaining sum breaches the limit.
Avoid duplicating logic — if a predicate doesn't need an aggregate, put it in WHERE for performance; if it does, HAVING is the only correct place.

Worked example. Filter open positions row-wise, then keep groups whose notional sum > $100K.

step	rows	groups
after WHERE	12 (open)	—
after GROUP BY	—	5 users
after HAVING > 100K	—	2 users

Worked-example solution.

SELECT user_id, SUM(contract_count * strike_price * 100) AS notional
FROM positions
WHERE status = 'Open'
GROUP BY user_id
HAVING SUM(contract_count * strike_price * 100) > 100000;

Rule of thumb: aggregate predicates always belong in HAVING; row predicates always belong in WHERE. Mixing them up is graded as a conceptual error.

Notional aggregation: `SUM(contract_count × strike_price × multiplier)`

The notional invariant: for an equity option, notional = contract_count × strike_price × 100 (one US equity option contract represents 100 shares). For futures, the multiplier varies by product (ES = 50, NQ = 20). Always express the multiplier explicitly in the SQL — never bake "× 100" silently into an upstream view.

Equity options — contract_count × strike_price × 100.
Index futures — multiplier per contract spec; ES = 50.
Equities (long) — qty × price.
Use NUMERIC columns — preserve cents through the multiply chain.

Worked example. Three positions for u2: 200 contracts at $80, all open.

user_id	contract_count	strike	notional
u2	200	80	200 × 80 × 100 = 1,600,000

Worked-example solution.

SELECT user_id, SUM(contract_count * strike_price * 100) AS notional
FROM positions
WHERE status = 'Open'
GROUP BY user_id;

Rule of thumb: spell out the multiplier every time and use a NUMERIC column for strike_price and contract_count; floats here will quietly lose pennies and the audit log will catch you.

Combining `HAVING` with `COUNT` and `SUM` thresholds

The composition invariant: HAVING accepts any boolean expression over aggregates and grouping columns. You can combine SUM, COUNT, AVG, MAX, and the grouping keys with AND / OR. Common compound predicates: "more than 10 trades AND notional > $100K", "MAX position older than 7 days", "average ticket size > $5K and at least 5 trades."

HAVING COUNT(*) > 10 — at least 11 trades in the group.
HAVING SUM(notional) > 100000 AND COUNT(*) >= 5 — both conditions must hold.
HAVING AVG(price) > 100 OR MAX(price) > 500 — either condition.
HAVING MAX(trade_date) < CURRENT_DATE - INTERVAL '7' DAY — group has been quiet for a week.

Worked example. Flag users with ≥5 trades AND notional > $100K.

user	trades	notional	flagged
u1	7	60,000	no (notional fail)
u2	12	1,600,000	yes
u3	3	200,000	no (count fail)

Worked-example solution.

SELECT user_id,
       COUNT(*) AS trades,
       SUM(contract_count * strike_price * 100) AS notional
FROM positions
WHERE status = 'Open'
GROUP BY user_id
HAVING COUNT(*) >= 5
   AND SUM(contract_count * strike_price * 100) > 100000;

Rule of thumb: compose the HAVING predicate exactly as the policy or rule reads; don't pre-aggregate into a CTE just to put WHERE on top — HAVING is the right tool.

Common beginner mistakes

Using WHERE SUM(notional) > 100000 — WHERE cannot reference aggregates; the parser rejects it. Use HAVING.
Computing notional in a subquery and then re-aggregating outside — works but is the "long way around"; one GROUP BY + HAVING is cleaner.
Storing strike_price as FLOAT — accumulates rounding error; use NUMERIC(18, 4).
Forgetting the × 100 multiplier on equity options — the notional comes out 100× too small.
Including closed positions — forgetting WHERE status = 'Open' inflates notional and produces false positives.

SQL Interview Question on Notional-Limit Threshold Check

Given a positions(user_id, contract_count, strike_price, status) table where status is 'Open' or 'Closed' and each row is one option position (equity-option multiplier = 100), write a query that returns every user_id whose total open notional exceeds $100,000. Output user_id and notional, ordered by notional descending.

Solution Using `GROUP BY` + `HAVING SUM(...) > limit`

SELECT
    user_id,
    SUM(contract_count * strike_price * 100) AS notional
FROM positions
WHERE status = 'Open'
GROUP BY user_id
HAVING SUM(contract_count * strike_price * 100) > 100000
ORDER BY notional DESC;

Why this works: WHERE status = 'Open' strips closed positions before grouping so the aggregate only reflects current exposure; GROUP BY user_id collapses to one row per user; the equity-option multiplier × 100 produces real-dollar notional inside SUM; HAVING filters groups whose summed notional crosses the policy limit; ORDER BY notional DESC ranks the breaches by severity.

Step-by-step trace for the sample:

user_id	contract_count	strike_price	status
u1	50	120	Open
u2	200	80	Open
u3	30	150	Open
u4	80	90	Open
u5	10	100	Open
u4	25	50	Closed

WHERE filter — drops the closed u4 row. 5 open rows remain.
Per-row notional — u1: 50×120×100 = 600,000 … wait — the worked-example diagram uses simplified numbers; recompute: u1: 50×120×100 = 600,000, u2: 200×80×100 = 1,600,000, u3: 30×150×100 = 450,000, u4: 80×90×100 = 720,000, u5: 10×100×100 = 100,000.
Group by user_id — already one open row per user; the SUM collapses each one to itself.
HAVING > 100,000 — strict greater-than; u5 at exactly 100,000 does not pass; u1 (600K), u2 (1.6M), u3 (450K), u4 (720K) do.
Order by notional desc — u2 > u4 > u1 > u3.

Output:

user_id	notional
u2	1,600,000
u4	720,000
u1	600,000
u3	450,000

Why this works — concept by concept:

WHERE row filter — strips closed positions before grouping; HAVING cannot do this work because the predicate is per-row, not per-group.
SUM(contract_count * strike_price * 100) — computes equity-option notional with the × 100 multiplier baked into the aggregate; spelled out so the reviewer can read the units.
GROUP BY user_id — collapses each user's positions to one row with summed notional; the only non-aggregate column in SELECT matches the GROUP BY.
HAVING SUM(...) > 100000 — filters group rows; aggregate predicates have to live here. Strict-greater means 100000 itself does not breach.
ORDER BY notional DESC — ranks breaches by severity so the on-call engineer can triage worst-first.
O(|positions| + G log G) time — one scan for the join-free GROUP BY, then O(G log G) to sort the group output where G is the number of users with open positions.

Inline CTA: More SQL HAVING-clause problems and SQL aggregation problems for breadth.

SQL
Company — Robinhood
Robinhood SQL problems

Practice →

SQL
Topic — having clause
SQL HAVING-clause problems

Practice →

SQL
Topic — aggregation
SQL aggregation problems

Practice →

Tips to crack Robinhood data engineering interviews

Bilingual profile — SQL and Python both, equally weighted

Unlike Cisco's Python-only loop, Robinhood splits the coding rounds between SQL and Python. The curated 2-problem Robinhood practice set is 1 EASY Python hash-table + 1 MEDIUM SQL joins — bilingual by design. Spending all your prep time on Python means losing the SQL round; spending it all on SQL means stuttering on the dict-counter Python prompt. Allocate roughly half-half across the SQL practice surface and the Python practice surface.

Drill the four primitives

The four primitives in this guide map directly to the curated Robinhood set plus the two adjacent SQL patterns every Robinhood SQL question list rotates through: hash-table dict counter (Python, EASY — Stock Purchases Count), INNER JOIN + GROUP BY + ORDER DESC + LIMIT (SQL, MEDIUM — Member Transfer Records / cities-completed-trades), LAG window function (SQL — daily volume change), GROUP BY + HAVING (SQL — notional-limit checks). Each maps to a specific module: vanilla dict and collections.defaultdict for the Python primitive, INNER JOIN and GROUP BY for the joins primitive, OVER (PARTITION BY ... ORDER BY ...) for window LAG, HAVING for the aggregate-threshold primitive.

Penny-perfect correctness is the bar

Robinhood is a regulated brokerage; every cent of every account balance is auditable. Float-typed currency, rounding-tolerant joins, and "approximately right" pipelines are downgrade signals in the round. Use NUMERIC(18, 4) for prices and notionals; spell out the equity-option × 100 multiplier; volunteer "I would store this as decimal, never float" when you write the schema. State idempotency requirements when designing pipelines — every job can be re-run without double-counting.

Event-sourced thinking — positions are folded from immutable trade events

Robinhood's distinctive architecture pattern is event sourcing: account balances and positions are not mutable rows you UPDATE; they are derived by folding an immutable log of trade and cash events. CRUD-style designs in interview answers underperform. Frame state as "I would write trades to an append-only Kafka log; the position table is a materialized fold of those events; corrections arrive as new events, not as UPDATE statements." This single framing flip unlocks the senior signal interviewers grade.

T+1 settlement and brokerage glossary as table stakes

Knowing the basics — settlement (T+1 since May 2024 in the US), corporate actions (splits, dividends, mergers), options assignment / exercise / expiration, FIFO / LIFO accounting, wash sales (same-security repurchase within 30 days disallows the loss) — gives a major edge. You don't need a finance degree; you do need to recognize the terms when the interviewer drops them. Spend a week with a brokerage glossary if your background is non-finance.

Easy-Medium discipline matters

The curated Robinhood set is 1 EASY + 1 MEDIUM. Easy at Robinhood doesn't mean trivial — it means the interviewer expects zero hesitation, idiomatic code, and an articulated invariant. A correct EASY answer with stuttering or a missing edge case is graded worse than a correct MEDIUM answer with the same flaw. Drill the easy practice page and the medium practice page until the canonical EASY-tier code rolls off your fingers in under three minutes.

Where to practice on PipeCode

Start with the Robinhood practice page for the curated 2-problem set. After that, drill the matching topic pages: hash table, dictionary, joins, group by, aggregation, window functions, having clause, filtering. The interview courses page bundles structured curricula. For a broader set, browse by topic or pivot to peer fintech with the Airbnb DE interview guide and the top DE interview questions 2026 blog.

Communication and approach under time pressure

Talk through the invariant first ("this is a LAG-on-partitioned-time-series problem"), the brute force second ("a self-join on date - 1 would also work"), and the optimal third ("but LAG is the idiomatic and faster move"). Interviewers grade process as much as the final answer. Leave 5 minutes for an edge-case sweep: empty input, single-row partitions, duplicate trade events, divide-by-zero on prior-day volume, decimal precision on the notional sum. The most common "almost passed" failure mode is correct happy-path code that crashes on edge cases — a 30-second sweep prevents it.

Frequently Asked Questions

What is the Robinhood data engineering interview process like?

The Robinhood data engineering interview opens with a 30-minute recruiter screen, then a 60-minute technical phone screen with one live SQL or Python coding problem, then a 4-round virtual onsite: a system-design round (commonly trade reconciliation, regulatory reporting, or position event-sourcing), a live coding round in the language you didn't see in the phone screen, a data-modeling discussion (often Type 2 SCD on financial dimensions), and a behavioral round. Robinhood interviewers grade integrity and compliance-mindedness heavily; bring a postmortem-style story about a hard call between speed and correctness. End-to-end the loop runs three to four weeks.

What programming languages does Robinhood test in data engineering interviews?

Robinhood data engineering interviews are bilingual — SQL and Python in roughly equal measure across the loop. SQL questions concentrate on joins, aggregations with GROUP BY and HAVING, window functions (LAG, LEAD, SUM OVER), and financial-precision patterns (account balance reconstruction, FIFO P&L, trade reconciliation). Python questions concentrate on hash-table counters, dict-of-sets aggregation, event-stream dedup with TTL, and event sourcing folds. Go and Scala appear at backend-leaning DE roles but are not expected in the coding rounds.

How difficult are Robinhood data engineering interview questions?

The curated Robinhood practice set on PipeCode is 1 easy and 1 medium, no hard. The EASY is a Python hash-table dict-counter problem (Stock Purchases Count); the MEDIUM is a SQL joins + filtering problem (Member Transfer Records / top-3 cities by completed trades). At the onsite, system-design and modeling questions reach L4-L5 level — trade reconciliation pipelines, regulatory CAT reporting, position event-sourcing — but the live coding rounds stay in the EASY-MEDIUM zone for IC2-IC3 hires. Stuttering on the EASY is a stronger negative signal than struggling with the MEDIUM.

How should I prepare for a Robinhood data engineer interview?

Solve the 2 problems on the Robinhood practice page end-to-end — untimed first, then timed at 25 minutes per problem — and broaden to 30 to 50 additional problems across the matching topic pages: hash table and dictionary on the Python side, joins, group-by, aggregation, window functions, and having-clause on the SQL side. Read a brokerage glossary for a week (settlement, corporate actions, options assignment, FIFO accounting, wash sales). Practice articulating idempotency and audit-log requirements when discussing pipeline design — those framings are graded heavily at Robinhood.

What is the Robinhood data engineer salary range?

Robinhood data engineer total compensation runs from roughly $170K (IC2, 2-4 years experience) to $620K (IC5, multi-org technical leadership). Senior Data Engineer (IC3) is the most common external hire at $240K–$370K total comp. Staff (IC4) sits at $330K–$500K. Average base salary across all levels lands around $163K with median $150K; total compensation averages around $221K when RSU refreshers and bonus are included. Negotiation success rates run 10–25% with competing offers per verified levels.fyi data.

What financial-domain knowledge do Robinhood interviewers expect?

Robinhood interviewers expect candidates to recognize and use brokerage-domain terms without flinching: settlement (T+1), corporate actions (splits, dividends, mergers — they require historical-position re-multiplication), options (assignment, exercise, expiration, the × 100 equity-option multiplier), accounting methods (FIFO vs LIFO vs average cost — FIFO is the default for tax lots), wash sales (loss disallowed if same security repurchased within 30 days), regulatory pipelines (CAT — Consolidated Audit Trail — is FINRA-mandated). You don't need a finance background to land the role, but you do need to converse fluently in these terms; a one-week brokerage-glossary sprint is enough.

Start practicing Robinhood data engineering problems

Bloomberg Data Engineering Interview Questions: Full DE Prep Guide

Gowtham Potureddi — Sat, 02 May 2026 06:17:38 +0000

Bloomberg data engineering interview questions sit at the intersection of three narrow, production-grade patterns: Python two-pointer and string manipulation that reverses words in a sentence using s.split() plus [::-1] plus ' '.join(...), production-quality OOP and abstract classes that subclass an ABC base with @abstractmethod load, transform, and write to stream a chunked CSV into line-delimited JSON without loading the whole file into memory, and SQL window functions for time-series and overlap analysis with SUM(volume) OVER (PARTITION BY symbol ORDER BY trade_date) for rolling totals and EXISTS subqueries with interval crosschecks for subscription-overlap detection. The schema you reason over feels like Bloomberg's own product (ticks, subscriptions, provider_feeds, corporate_actions), and the bar is fluency with two-pointer index arithmetic, from abc import ABC, abstractmethod, and PARTITION BY ... ORDER BY ... under tie-break and overlap edge cases.

This guide walks through the four topic clusters Bloomberg actually tests, each with a detailed topic explanation, per-sub-topic explanation with a worked example and its solution, and an interview-style problem with a full solution that explains why it works. The mix matches the curated 2-problem Bloomberg set (1 easy, 1 hard) plus one supplementary SQL section anchored on patterns Bloomberg's external interview reports surface heavily — a Python-and-SQL loop where dictation of the invariant out loud is half the score and the other half is typing production-quality code on the first try. Strong data engineer interview questions prep at Bloomberg is less about contest difficulty (the pass rate sits at 8% across recent samples) and more about clean class boundaries, deterministic ordering, streaming-friendly file I/O, and the kind of code review you would survive on day three of the job.

1. The Bloomberg Data Engineering Interview Process

The Bloomberg DE interview funnel from phone screen to final round

The Bloomberg data engineer interview process is a four-stage funnel that takes about thirty days end to end: a 45-60 minute technical phone screen (longer than the 20-30 minute non-engineering equivalent), an optional online assessment for some roles, a virtual onsite of three to five rounds covering coding plus system design plus behavioral plus fit, and a final decision call. Recent samples report an 8% pass rate across thirteen Bloomberg DE candidates — making it the most selective DE loop of any company in this prep series.

The two stages most candidates misread are the technical phone screen (40 minutes is usually enough only for one strong problem plus follow-ups, so move fast on the boilerplate) and the behavioral round (interviewers drill deep into past project technical details, not just culture-fit small talk).

Phone screen — 45 to 60 minutes of coding fluency

Detailed explanation. Bloomberg's engineering phone screen is conducted in a CoderPad-style live editor and runs 45-60 minutes for technical roles. The format converges on one medium-difficulty algorithmic problem (trees, graphs, two-pointer, sliding window, dynamic programming) plus follow-ups on edge cases and complexity. Bloomberg specifically values production-quality code — type hints, edge-case guards, and a few inline tests — over a clever-but-fragile one-liner. State your assumptions before typing: input bounds, null/empty handling, in-place vs return-new.

Worked example. When the interviewer hands you "reverse the order of words in a sentence", a clean opener is "I'll assume the input is a non-null string with words separated by single spaces, no leading or trailing whitespace; if the prompt allows multiple spaces I'll collapse them with split() since split() with no args treats any run of whitespace as one delimiter." That single sentence earns the round.

Rule of thumb: state assumptions, then type. Two minutes of clarification is worth twenty minutes of rework.

Online assessment and the take-home variant

Detailed explanation. Some Bloomberg DE roles route through an online assessment (Byteboard-style or HackerRank-style) instead of, or in addition to, the live phone screen. Format varies by team: a 60-minute timed assessment with two SQL questions and one Python question, or a 90-minute take-home with a small data-pipeline implementation graded on code quality and testability. The signal is the same as the live screen — production-quality code, edge-case awareness, deterministic output.

Worked example. A take-home variant might be: "Read events.csv, normalize the timestamps to UTC, deduplicate on (user_id, event_id) keeping the latest, and write the result as line-delimited JSON to events.ndjson." Strong candidates ship a typed function with a __main__ guard, an inline unit test, and a README.md with assumptions — even when the prompt does not explicitly ask for them.

Rule of thumb: on take-homes, ship a typed function plus one unit test plus a one-paragraph README of assumptions. Bloomberg grades on production-readiness, not just correctness.

Onsite — coding, design, behavioral, fit (3-5 rounds)

Detailed explanation. The onsite is a virtual loop of three to five rounds totaling four to five hours: a coding round (harder than the screen, often involving sliding-window aggregators or class-hierarchy refactors), a system design round (real-time market-data pipelines, dedup with sequence numbers, schema evolution, entitlements at read time), a behavioral round, and a fit round. Bloomberg's distinctive design moves include asking about exactly-once semantics under sink failures, late-arriving data with watermarks, and schema evolution across pipeline stages without breaking downstream consumers.

Worked example. "Design a pipeline that ingests trades from twenty venues, dedupes, enriches with reference data, and serves to downstream analytics consumers with a freshness SLA of 30 seconds." The clean answer routes through Kafka with idempotent producers, a Flink stateful operator keyed on (venue_id, sequence_number) for dedup, a side-input join against a slowly-changing reference table, and a CloudWatch alert on consumer lag. Naming idempotency, watermarks, and the SLA up front is what differentiates strong candidates.

Rule of thumb: every Bloomberg system-design answer should mention idempotency once, watermarks once, and the SLA / SLO once. Those three phrases earn the round.

Behavioral round — STAR plus deep-dive into past project tech

Detailed explanation. Bloomberg's behavioral round is conventional STAR — Situation, Task, Action, Result — but interviewers go technical-deep on the projects you mention. Bring two real projects you owned end-to-end with crisp numbers: rows per day, peak QPS, latency p99, cost per query, post-incident retro. Generic "I led a migration" stories without numbers will not pass. Recent first-person reports (Taro, October 2024) explicitly note: "The interviewer asked past project-related questions and went into depth about the technical aspects related to those projects."

Worked example. When asked "tell me about a time you handled a data corruption incident," a strong answer names the source (a downstream consumer reported NaN totals on the daily revenue dashboard), the diagnosis (a corrupt parquet file from a vendor partition that passed the schema check but had a sentinel -1 instead of NULL in a numeric column), the fix (added a Great Expectations contract test on the column's value distribution, plus a postmortem doc shared with the vendor), and the result (zero similar incidents over the next six months, two related vendors signed up for the contract).

Common beginner mistakes

Treating the phone screen as a sprint — Bloomberg grades production-quality code, not the cleverest hack.
Skipping README.md and inline tests on take-homes.
Naming idempotency without explaining the sink — "my Kafka producer is idempotent" is not a complete answer; the sink decides exactly-once.
Bringing one STAR story to the behavioral round (interviewers double-click and the story runs out fast).
Generic past-project answers without numbers (rows / day, p99, cost).

Practice: drill the Bloomberg DE panel before the live screen

COMPANY
Bloomberg — all DE problems
Bloomberg data engineering practice set

Practice →

PYTHON
Bloomberg — Python only
Bloomberg Python practice

Practice →

2. Python Two-Pointer and String Manipulation

Two-pointer iteration, string reversal, and split-rejoin in Python for data engineering

The first canonical Bloomberg Python pattern is string-level manipulation with two-pointer or split-rejoin primitives. The headline interview problem on the Bloomberg practice set, Reverse Words in String, is the textbook split-rejoin question: reverse the order of words in a sentence using only built-in string methods. The canonical answer is a single line — ' '.join(s.split()[::-1]) — but the interviewer is reading whether you can articulate the three primitives behind it (split, [::-1], join) and reach for the right tool when the constraints flip (immutable input, no built-ins, in-place character array).

Pro tip: Python's s.split() (with no argument) is not the same as s.split(' '). The no-arg form collapses runs of whitespace and ignores leading or trailing whitespace; the explicit ' ' form preserves empty strings between consecutive spaces. For "reverse words" prompts you almost always want the no-arg form because it handles ' hello world ' cleanly.

Splitting on whitespace — collapsing multiple spaces

The split invariant: s.split() with no argument splits on any run of whitespace and discards empty leading or trailing tokens; s.split(' ') with an explicit single-space delimiter preserves every empty token between consecutive spaces.

'a b c'.split() → ['a', 'b', 'c'].
' a b '.split() → ['a', 'b'] (leading and trailing whitespace dropped, runs collapsed).
' a b '.split(' ') → ['', '', 'a', '', 'b', '', ''] (every run preserved).

Worked example. Three quick splits on the same input.

input	call	output
`' hello world '`	`.split()`	`['hello', 'world']`
`' hello world '`	`.split(' ')`	`['', '', 'hello', '', 'world', '', '']`
`'hello world'`	`.split(' ')`	`['hello', 'world']`

Worked-example solution.

def words_collapsed(s: str) -> list[str]:
    return s.split()

Rule of thumb: default to s.split() (no arg) for natural-language input. Reach for s.split(' ') only when the prompt explicitly says preserve empty tokens.

Reversing a list with `[::-1]` vs `reversed()`

The reversal invariant: [::-1] is a slice that returns a new reversed list; reversed(seq) returns a lazy iterator; list.reverse() mutates in place and returns None.

words[::-1] — new list, original unchanged. O(n) time, O(n) extra memory.
list(reversed(words)) — new list via iterator, same cost.
words.reverse() — in-place mutation, returns None. O(n) time, O(1) extra memory.

Worked example. Three ways to reverse ['a', 'b', 'c'].

call	result	mutates original
`words[::-1]`	`['c', 'b', 'a']`	no
`list(reversed(words))`	`['c', 'b', 'a']`	no
`words.reverse()` (then read `words`)	`['c', 'b', 'a']`	yes

Worked-example solution.

def reverse_new(words: list[str]) -> list[str]:
    return words[::-1]

def reverse_in_place(words: list[str]) -> None:
    words.reverse()

Rule of thumb: [::-1] for "return a new reversed copy"; list.reverse() for "minimize memory and the prompt allows mutation."

Two-pointer in-place reversal of a character array

The two-pointer invariant: swap arr[left] with arr[right], then move left += 1 and right -= 1, until left >= right. This runs in O(n) time and O(1) extra memory and is the canonical follow-up when the interviewer says "now do it without [::-1] or reversed()".

Initialization — left = 0, right = len(arr) - 1.
Swap — arr[left], arr[right] = arr[right], arr[left] (Python's tuple assignment makes this one line).
Termination — left < right keeps you safe at the middle; left == right is a no-op on odd-length arrays.

Worked example. Reverse ['a', 'b', 'c', 'd'] in place.

step	left	right	array
start	0	3	`['a','b','c','d']`
swap, advance	1	2	`['d','b','c','a']`
swap, advance	2	1	`['d','c','b','a']`
stop (`left >= right`)	—	—	`['d','c','b','a']`

Worked-example solution.

def reverse_in_place_chars(arr: list[str]) -> None:
    left, right = 0, len(arr) - 1
    while left < right:
        arr[left], arr[right] = arr[right], arr[left]
        left += 1
        right -= 1

Rule of thumb: the left < right condition (strictly less than) is the safe termination — left <= right would over-swap on even-length arrays.

Common beginner mistakes

Using s.split(' ') on real-world input and getting empty-string tokens that confuse downstream logic.
Calling list.reverse() and then trying to use the return value (it returns None).
Two-pointer with left <= right — over-swaps on even-length arrays, restoring the original order.
Reaching for re.split when str.split is enough (no-libraries phone-screen variants reject re).

PipeCode's Python for data engineering interviews course drills these primitives across forty-plus problems, including the two-pointer and string-manipulation variants Bloomberg's screen reaches for.

Python Interview Question on Reverse Words in String

Write a function reverse_words(s: str) -> str that returns s with the order of words reversed. Words are separated by one or more spaces; treat any run of whitespace as a single delimiter. Leading and trailing whitespace in the input must not appear in the output, and the output must have exactly one space between words.

Solution Using split-reverse-join

def reverse_words(s: str) -> str:
    return ' '.join(s.split()[::-1])

Step-by-step trace (input s = ' the quick brown fox '):

step	expression	value
1	`s.split()`	`['the', 'quick', 'brown', 'fox']`
2	`s.split()[::-1]`	`['fox', 'brown', 'quick', 'the']`
3	`' '.join(s.split()[::-1])`	`'fox brown quick the'`

s.split() with no argument — splits on any run of whitespace, discards empty leading/trailing tokens. The two leading spaces, the three between quick and brown, and the two trailing spaces all collapse cleanly.
[::-1] slice — returns a new list with the words in reverse order; the original s is unchanged.
' '.join(...) — concatenates the reversed list into a single string with exactly one space between adjacent elements; returns 'fox brown quick the'.
Return — the function returns the joined string directly; no intermediate variable is needed.

Output:

input	result
`'the quick brown fox'`	`'fox brown quick the'`
`' hello world '`	`'world hello'`
`'one'`	`'one'`
`''`	`''` (empty input → empty output)
`' '`	`''` (whitespace-only input → empty output)

Why this works — concept by concept:

No-arg split collapses whitespace — s.split() (no argument) splits on any run of whitespace and silently drops empty leading or trailing tokens; this single call handles the "multiple spaces" and "leading / trailing whitespace" edge cases without any conditional logic.
Slice reversal returns a new list — [::-1] builds a new list in O(n) time without mutating the source; the function stays referentially transparent.
Join with a single space — ' '.join(...) concatenates with exactly one delimiter between elements and zero delimiters at the ends; the output cannot have leading or trailing spaces by construction.
Empty input is naturally handled — ''.split() returns [], [][::-1] returns [], and ' '.join([]) returns ''; the empty-input case requires no special branch.
Three primitives, one line — chaining split, slice, and join is the canonical Pythonic answer; reaching for re.split or manual character iteration is over-engineering.
Cost — O(n) time where n is the input length, plus O(n) extra memory for the new list and joined string; n is constrained by the line length in any real-world prompt.

PYTHON
Bloomberg — two pointers / string
Reverse Words in String (Bloomberg)

Practice →

PYTHON
Topic — two pointers
Two-pointer Python problems

Practice →

3. Abstract Classes, OOP, and File I/O in Python

ABC, abstract methods, and chunked file processing in Python for data engineering

The second canonical Bloomberg Python pattern is production-quality OOP with abstract base classes and streaming file I/O. The headline interview problem on the Bloomberg practice set, Chunked CSV to Line-Delimited JSON Processor, is exactly this shape: subclass an abstract RecordProcessor base, implement the load, transform, and write abstract methods, and stream a CSV file row-by-row into a line-delimited JSON output without loading the whole file into memory. This pattern shows up in roughly half of recent Bloomberg DE coding rounds — confirmed by external interview reports flagging "abstract DataProcessor framework" and "RotatingFileSink class hierarchy" prompts.

Pro tip: Python's abc module enforces the abstract contract at instantiation time, not at class definition time. class Foo(ABC): @abstractmethod def bar(self): ... lets you define Foo without implementing bar, but Foo() raises TypeError: Can't instantiate abstract class Foo with abstract method bar. Subclasses that miss any abstract method fail the same way. This is the language feature Bloomberg interviewers are reading you for.

`from abc import ABC, abstractmethod` — defining the contract

The abstract-class invariant: a class that inherits from ABC and decorates one or more methods with @abstractmethod cannot be instantiated directly; only subclasses that implement every abstract method can be instantiated.

from abc import ABC, abstractmethod — the standard import. ABC is a helper base class with ABCMeta as its metaclass.
@abstractmethod — decorator that marks a method as abstract. Place it on the method directly inside the class body.
Subclassing — class Concrete(Abstract): inherits all concrete methods and must override every @abstractmethod.

Worked example. Defining an abstract RecordProcessor with three abstract methods.

method	purpose	signature
`load(path)`	read the input	returns an iterable of records
`transform(record)`	reshape one record	returns the transformed record
`write(record, out)`	write one record	returns `None` (side effect)

Worked-example solution.

from abc import ABC, abstractmethod
from typing import Iterable, IO

class RecordProcessor(ABC):
    @abstractmethod
    def load(self, path: str) -> Iterable[dict]: ...

    @abstractmethod
    def transform(self, record: dict) -> dict: ...

    @abstractmethod
    def write(self, record: dict, out: IO[str]) -> None: ...

Rule of thumb: keep the abstract base small — three to five methods is the sweet spot. Abstract bases with twenty methods are a code smell.

Subclassing and implementing every abstract method

The subclass invariant: a concrete subclass must implement every method decorated with @abstractmethod in the parent (and every grandparent, transitively); missing even one keeps the subclass abstract.

class CsvToLdjsonProcessor(RecordProcessor): — declare inheritance.
Implement each abstract — load, transform, write all need concrete bodies (not pass or ...).
Concrete-only methods inherit unchanged — non-abstract methods on the parent are usable directly.

Worked example. A CsvToLdjsonProcessor that reads CSV and writes line-delimited JSON.

abstract method	concrete implementation
`load`	`csv.DictReader(open(path))`
`transform`	identity (return the row dict unchanged)
`write`	`out.write(json.dumps(record) + '\n')`

Worked-example solution.

import csv
import json
from typing import Iterable, IO

class CsvToLdjsonProcessor(RecordProcessor):
    def load(self, path: str) -> Iterable[dict]:
        with open(path, newline='') as f:
            yield from csv.DictReader(f)

    def transform(self, record: dict) -> dict:
        return record

    def write(self, record: dict, out: IO[str]) -> None:
        out.write(json.dumps(record) + '\n')

Rule of thumb: if your transform is the identity, leave it as the identity — interviewers reward not over-engineering as much as they reward implementing correctly.

Chunked CSV reading with `csv.DictReader` and line-delimited JSON output

The streaming invariant: csv.DictReader is a lazy iterator; iterating it yields one row at a time without loading the whole file. Pair it with a for loop that writes each transformed row to the output stream, and the program runs in O(1) extra memory regardless of input size.

csv.DictReader(f) — yields one dict per CSV row, using the first row as header keys.
yield from — the load method delegates iteration to DictReader without buffering.
for record in self.load(path): — the orchestrator iterates lazily; transform and write run per row.
Line-delimited JSON — json.dumps(record) + '\n' per row produces a file where each line is independently parseable.

Worked example. Streaming a 10-million-row CSV through the processor uses constant memory, not 10M-row memory.

stage	memory profile
`load`	one row at a time
`transform`	one row at a time
`write`	one row at a time
total extra memory	`O(1)`

Worked-example solution.

def run(processor: RecordProcessor, in_path: str, out_path: str) -> int:
    count = 0
    with open(out_path, 'w') as out:
        for record in processor.load(in_path):
            transformed = processor.transform(record)
            processor.write(transformed, out)
            count += 1
    return count

Rule of thumb: never for record in list(processor.load(path)) — wrapping a generator in list() materializes the whole file and defeats the streaming guarantee.

Common beginner mistakes

Using pandas.read_csv(path) instead of csv.DictReader — loads the whole file, defeats the streaming invariant.
Forgetting newline='' on open() — Python's csv module needs that to handle quoted multi-line fields correctly.
Implementing load to return a list instead of yielding — same issue as the pandas mistake, scaled.
Writing JSON without the trailing '\n' — produces concatenated objects on one line, not line-delimited.
Skipping the with block on the output file — the file may not flush on early termination.

Python Interview Question on Chunked CSV to LDJSON Processor

Implement a class hierarchy that reads a CSV file row-by-row and writes the rows as line-delimited JSON to an output file, without loading the whole CSV into memory. The base class RecordProcessor must be abstract with three @abstractmethod methods: load(path) (returns an iterable of dict), transform(record) (returns a dict), and write(record, out) (writes one record and returns None). Implement a concrete subclass CsvToLdjsonProcessor that satisfies the contract. Provide a top-level run(processor, in_path, out_path) function that orchestrates the streaming pipeline and returns the row count.

Solution Using a Concrete Subclass of an Abstract RecordProcessor

from __future__ import annotations
import csv
import json
from abc import ABC, abstractmethod
from typing import IO, Iterable

class RecordProcessor(ABC):
    @abstractmethod
    def load(self, path: str) -> Iterable[dict]: ...
    @abstractmethod
    def transform(self, record: dict) -> dict: ...
    @abstractmethod
    def write(self, record: dict, out: IO[str]) -> None: ...

class CsvToLdjsonProcessor(RecordProcessor):
    def load(self, path: str) -> Iterable[dict]:
        with open(path, newline='') as f:
            yield from csv.DictReader(f)

    def transform(self, record: dict) -> dict:
        return record

    def write(self, record: dict, out: IO[str]) -> None:
        out.write(json.dumps(record) + '\n')

def run(processor: RecordProcessor, in_path: str, out_path: str) -> int:
    count = 0
    with open(out_path, 'w') as out:
        for record in processor.load(in_path):
            transformed = processor.transform(record)
            processor.write(transformed, out)
            count += 1
    return count

Step-by-step trace (input events.csv with 3 rows):

name,age,city
alice,30,NYC
bob,25,SF
carol,40,LON

step	action	output side effect
1	`processor.load('events.csv')`	yields `{'name':'alice','age':'30','city':'NYC'}`
2	`transform(record)`	returns the same dict (identity)
3	`write(record, out)`	writes `{"name":"alice","age":"30","city":"NYC"}\n`
4	next iteration	yields `{'name':'bob','age':'25','city':'SF'}`
5	transform + write	writes `{"name":"bob","age":"25","city":"SF"}\n`
6	next iteration	yields `{'name':'carol','age':'40','city':'LON'}`
7	transform + write	writes `{"name":"carol","age":"40","city":"LON"}\n`
8	iterator exhausted	`count = 3` returned

Open the output file inside with — guarantees flush and close even on exceptions.
processor.load(in_path) — returns a generator; iterating it lazily reads one row at a time from the CSV.
processor.transform(record) — applies the per-record transformation (identity here).
processor.write(transformed, out) — writes one JSON line per record.
Increment count — after each successful write.
Return count — at end of iteration.

Output:

{"name": "alice", "age": "30", "city": "NYC"}
{"name": "bob", "age": "25", "city": "SF"}
{"name": "carol", "age": "40", "city": "LON"}

Plus return value 3.

Why this works — concept by concept:

ABC + @abstractmethod enforces the contract — RecordProcessor cannot be instantiated directly; any subclass missing load, transform, or write raises TypeError at construction. The interface is enforced by the language, not by convention.
csv.DictReader yields lazily — it is a generator that reads one row at a time; pairing it with yield from keeps the entire pipeline streaming.
Generator-based load — load returns an iterable, not a list; the caller iterates with for and never materializes the file.
Line-delimited JSON — json.dumps(record) + '\n' per row produces a file where each line is independently parseable, which is the canonical format for log streaming and incremental ingestion.
with-block on the output — the output handle flushes and closes deterministically; an exception mid-stream does not leave a partial file open.
Cost — O(1) extra memory regardless of input size (the generator holds at most one row), O(n) time where n is the input row count; no O(n) memory spike at any stage.

PYTHON
Bloomberg — OOP / abstract
Chunked CSV to LDJSON Processor (Bloomberg)

Practice →

PYTHON
Topic — OOP
OOP Python problems

Practice →

4. SQL Window Functions for Time-Series and Subscription Overlap

`PARTITION BY`, `LAG`, and rolling aggregates in SQL for time-series and overlap analysis

The third canonical Bloomberg interview pattern is SQL window functions on time-series data plus interval-overlap detection. Bloomberg's external interview reports converge here — a recent DataLemur compilation lists eight Bloomberg SQL questions covering rolling aggregates, monthly per-product averages, and ranking-tie semantics, and Interview Query flags Subscription Overlap as a flagship Bloomberg SQL question. The canonical primitives are SUM(volume) OVER (PARTITION BY symbol ORDER BY trade_date) for rolling totals, LAG/LEAD for adjacent-row comparisons, and EXISTS subqueries with interval crosschecks for overlap detection. PipeCode does not yet host a Bloomberg-tagged SQL problem, so this section is anchored on the topic-level SQL window functions practice page.

Pro tip: window functions evaluate after WHERE and before ORDER BY in the SQL pipeline. That is why you cannot filter WHERE rolling_total > 1000 in the same SELECT as the window — wrap the window in a CTE first, then filter in the outer query. This trips up most candidates on the first onsite SQL prompt.

`SUM` / `AVG` `OVER (PARTITION BY ... ORDER BY ...)` for rolling totals

The rolling-aggregate invariant: SUM(metric) OVER (PARTITION BY entity ORDER BY ts) returns a per-row running total scoped to each entity, ordered by time, including all rows up to and including the current row by default.

PARTITION BY symbol — resets the running aggregate at each symbol boundary.
ORDER BY trade_date — defines the time direction of the cumulation.
Default frame — RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW. Override with ROWS BETWEEN N PRECEDING AND CURRENT ROW for a fixed-window rolling sum.

Worked example. A ticks(symbol, trade_date, volume) table.

symbol	trade_date	volume	running_volume
AAPL	2026-04-01	100	100
AAPL	2026-04-02	150	250
AAPL	2026-04-03	200	450
MSFT	2026-04-01	80	80
MSFT	2026-04-02	120	200

Worked-example solution.

SELECT symbol, trade_date, volume,
       SUM(volume) OVER (
         PARTITION BY symbol
         ORDER BY trade_date
       ) AS running_volume
FROM ticks;

Rule of thumb: if the prompt says "running" / "cumulative" / "year-to-date" / "month-to-date" per entity, reach for SUM OVER (PARTITION BY entity ORDER BY ts) before anything else.

`LAG` / `LEAD` for adjacent-row comparison

The lag-lead invariant: LAG(col, n) OVER (PARTITION BY entity ORDER BY ts) returns the value of col from n rows earlier within the same partition, ordered by ts. LEAD is the symmetric "n rows later" variant.

LAG(volume) OVER (...) — previous row's volume, or NULL for the first row in the partition.
LAG(volume, 2) — two rows back.
LAG(volume, 1, 0) — default 0 instead of NULL for missing rows.

Worked example. Day-over-day volume change per symbol.

symbol	trade_date	volume	prev_volume	dod_change
AAPL	2026-04-01	100	NULL	NULL
AAPL	2026-04-02	150	100	50
AAPL	2026-04-03	200	150	50

Worked-example solution.

SELECT symbol, trade_date, volume,
       LAG(volume) OVER (PARTITION BY symbol ORDER BY trade_date) AS prev_volume,
       volume - LAG(volume) OVER (PARTITION BY symbol ORDER BY trade_date) AS dod_change
FROM ticks;

Rule of thumb: day-over-day deltas, week-over-week resets, and "did this row break a streak?" all map to LAG. Use LEAD for "is the next event within X minutes?".

Subscription overlap via interval crosscheck (`EXISTS` subquery)

The overlap invariant: two intervals (start_a, end_a) and (start_b, end_b) overlap if and only if start_a <= end_b and start_b <= end_a. This is the canonical interval-overlap formula; everything else (partial overlap, full containment, identical intervals) collapses to this two-condition check.

EXISTS subquery — for each row, check if any other row from the same table satisfies the overlap condition.
s1.user_id <> s2.user_id — exclude the row from comparing with itself.
End-date IS NOT NULL — only completed subscriptions count for overlap detection in Bloomberg's framing.

Worked example. A subscriptions(user_id, start_date, end_date) table with four users.

user_id	start_date	end_date	overlaps?
1	2026-01-01	2026-01-31	✓ (with user 2 and user 3)
2	2026-01-15	2026-01-17	✓ (with user 1)
3	2026-01-29	2026-02-04	✓ (with user 1)
4	2026-02-05	2026-02-10	✗

Worked-example solution.

SELECT s1.user_id,
       CASE WHEN EXISTS (
         SELECT 1 FROM subscriptions s2
         WHERE s2.user_id <> s1.user_id
           AND s2.end_date IS NOT NULL
           AND s1.start_date <= s2.end_date
           AND s2.start_date <= s1.end_date
       ) THEN 1 ELSE 0 END AS overlap
FROM subscriptions s1
WHERE s1.end_date IS NOT NULL;

Rule of thumb: the overlap formula is start_a <= end_b AND start_b <= end_a. Memorize it cold — every Bloomberg interval-detection prompt collapses to it.

Common beginner mistakes

Filtering WHERE running_volume > 1000 in the same SELECT as the window — windows evaluate after WHERE; wrap in a CTE first.
Forgetting PARTITION BY and getting one global running total across all symbols.
Using LAG(volume) without an ORDER BY inside the window — undefined ordering, undefined result.
Writing the overlap check as start_a < end_b AND start_b < end_a (strict) when the prompt allows touching boundaries — off-by-one on identical-day overlaps.
Forgetting s1.user_id <> s2.user_id — every row "overlaps with itself" and the answer is always 1.

SQL Interview Question on Detecting Subscription Overlap

Given a table subscriptions(user_id INT, start_date DATE, end_date DATE) where end_date IS NULL indicates an active subscription, return one row per user with overlap = 1 if the user's completed subscription window overlaps any other user's completed subscription, and overlap = 0 otherwise. Consider only rows where end_date IS NOT NULL. Use the canonical interval-overlap formula.

Solution Using EXISTS with Interval Crosscheck

SELECT
  s1.user_id,
  CASE WHEN EXISTS (
    SELECT 1
    FROM subscriptions s2
    WHERE s2.user_id   <> s1.user_id
      AND s2.end_date  IS NOT NULL
      AND s1.start_date <= s2.end_date
      AND s2.start_date <= s1.end_date
  ) THEN 1 ELSE 0 END AS overlap
FROM subscriptions s1
WHERE s1.end_date IS NOT NULL
ORDER BY s1.user_id;

Step-by-step trace (input subscriptions):

user_id	start_date	end_date
1	2026-01-01	2026-01-31
2	2026-01-15	2026-01-17
3	2026-01-29	2026-02-04
4	2026-02-05	2026-02-10

Filter to completed subscriptions — WHERE s1.end_date IS NOT NULL keeps all four rows here (every row has an end_date).
For user 1 (2026-01-01 to 2026-01-31) — check EXISTS. Compare against user 2 (2026-01-15 to 2026-01-17): 2026-01-01 <= 2026-01-17 ✓ AND 2026-01-15 <= 2026-01-31 ✓ → overlap. EXISTS returns true, overlap = 1.
For user 2 (2026-01-15 to 2026-01-17) — compare against user 1: 2026-01-15 <= 2026-01-31 ✓ AND 2026-01-01 <= 2026-01-17 ✓ → overlap. overlap = 1.
For user 3 (2026-01-29 to 2026-02-04) — compare against user 1: 2026-01-29 <= 2026-01-31 ✓ AND 2026-01-01 <= 2026-02-04 ✓ → overlap. overlap = 1.
For user 4 (2026-02-05 to 2026-02-10) — compare against every other user. User 1: 2026-02-05 <= 2026-01-31 ✗ (Feb 5 is after Jan 31). User 2: 2026-02-05 <= 2026-01-17 ✗. User 3: 2026-02-05 <= 2026-02-04 ✗. No overlap. overlap = 0.
ORDER BY s1.user_id for stability.

Output:

user_id	overlap
1	1
2	1
3	1
4	0

Why this works — concept by concept:

Interval overlap formula — two intervals (a_start, a_end) and (b_start, b_end) overlap iff a_start <= b_end AND b_start <= a_end; this two-condition check captures partial overlap, full containment, and shared boundaries in one expression.
EXISTS short-circuits — the subquery returns true the moment one matching row is found; for users with many overlapping peers, the engine does not evaluate every pair.
s1.user_id <> s2.user_id excludes self-overlap — without this guard, every row trivially overlaps itself and the answer is uniformly 1.
end_date IS NOT NULL filters incomplete rows — Bloomberg's framing only counts completed subscriptions; active subscriptions (open intervals) are excluded by the prompt.
CASE WHEN EXISTS ... THEN 1 ELSE 0 — produces a 0/1 flag column directly from the boolean predicate, no separate aggregation needed.
Cost — O(n²) worst case (every row compared against every other) without an index; with a B-tree index on (start_date, end_date) plus a stop-early plan, the average case drops to O(n log n) for typical interval distributions.

SQL
Topic — window functions
Window function problems (all companies)

Practice →

SQL
Topic — intervals
Interval / overlap problems

Practice →

Tips to crack Bloomberg data engineering interviews

These are habits that move the needle in real Bloomberg DE loops — not a re-statement of the topics above.

Practice with Bloomberg's data shape

Bloomberg's interview prompts model market data and subscription feeds: ticks, quotes, trades, subscriptions, provider_feeds, corporate_actions. Drilling on order-line ecommerce schemas wastes prep time. Stick to event-shaped tables with a per-symbol-per-day grain, and pull problems from the window-functions topic page for shapes that match.

Master `from abc import ABC, abstractmethod` cold

Half of Bloomberg DE coding rounds reach for an abstract base class — confirmed by PipeCode #508 plus two of four recent external Bloomberg DE problems (DataProcessor, RotatingFileSink). Type the boilerplate from memory until it is muscle memory: class Base(ABC): @abstractmethod def method(self): ... plus a concrete subclass with every abstract method implemented. Drill on the OOP topic page.

Type your Python with `from future import annotations`

Bloomberg grades production-quality code, and that includes type hints. Add from __future__ import annotations at the top, use list[dict] and Iterable[dict] rather than List[Dict], and annotate every function signature. Interviewers notice — and the type-checker catches the off-by-one before the interviewer does.

Drill window functions

PARTITION BY plus ORDER BY plus optional LAG/LEAD plus the ROWS BETWEEN N PRECEDING AND CURRENT ROW frame override is the Bloomberg SQL toolkit. Practice with SQL window-function problems until the syntax is reflex. PipeCode's SQL for data engineering interviews course drills these primitives across forty-plus problems.

Bring two STAR stories per behavioral round

Bloomberg's behavioral round goes technical-deep — interviewers drill past project details, not just culture-fit. Pick two real projects with crisp numbers (rows per day, p99, cost per query, post-incident retro) and rehearse the deep-dive Q&A. Generic teamwork stories will not pass.

Where to practice on PipeCode

Skill lane	Practice path
Curated Bloomberg practice set	/explore/practice/company/bloomberg
Bloomberg Python practice	/explore/practice/company/bloomberg/python
Two-pointer + string in Python	/explore/practice/topic/two-pointers/python
String-manipulation in Python	/explore/practice/topic/string-manipulation/python
OOP + abstract classes	/explore/practice/topic/abstract-classes
File I/O + CSV parsing	/explore/practice/topic/file-io/python
SQL window functions	/explore/practice/topic/window-functions/sql
Intervals / overlap	/explore/practice/topic/intervals
All practice topics	/explore/practice/topics
Interview courses	/explore/courses

Communication under time pressure

State assumptions before typing: "I'll assume the input CSV has a header row, no embedded newlines in fields, and one consistent delimiter." State grain: "One JSON object per CSV row, in source order." State edge cases: "If a row has missing keys, my transform returns the partial dict and write serializes whatever keys are present." Interviewers grade clear reasoning above silent-and-perfect.

Frequently asked questions

What is the Bloomberg data engineering interview process?

The Bloomberg data engineer interview process is a four-stage funnel: a 45-60 minute technical phone screen in CoderPad covering a medium-difficulty algorithmic problem with follow-ups, an optional online assessment for some roles (Byteboard / HackerRank format with SQL plus Python), a virtual onsite of three to five rounds covering coding plus system design plus behavioral plus fit, and a final decision call. Total elapsed time is typically about thirty days. The recent reported pass rate is 8% across thirteen samples — making it the most selective DE loop tracked in our blog series.

What programming languages does Bloomberg test for data engineering?

Bloomberg's data engineering interviews lean on Python and SQL, with Java or C++ appearing for some platform / infra roles. Python is production-quality — from abc import ABC, abstractmethod, csv.DictReader, type hints, generator-based streaming. SQL is at LeetCode-medium / DataLemur grade with a strong tilt toward window functions, rolling aggregates, and interval-overlap detection. PipeCode's Bloomberg Python practice anchors the Python side; the SQL flavor is covered by topic-level pages.

What SQL topics show up most in Bloomberg data engineering interviews?

The topics are narrow and consistent: window functions (SUM / AVG OVER (PARTITION BY symbol ORDER BY trade_date) for rolling totals), LAG/LEAD for adjacent-row comparisons (day-over-day change, streak detection), HAVING + COUNT for power-user filters, self-joins for pairwise comparisons, and EXISTS subqueries with interval crosschecks for overlap detection. PipeCode's window-function problems and interval problems cover these directly.

How difficult are Bloomberg data engineering interview questions?

Bloomberg data engineering interview questions are calibrated above generic LeetCode-medium for the algorithm half — closer to a production-code bar than a contest bar. The two-problem PipeCode set splits 1 EASY (Reverse Words in String) for warm-ups and 1 HARD (Chunked CSV to LDJSON Processor) for production-quality OOP. Reports describe the panel as friendly but rigorous; the 8% pass rate reflects code-review-grade judgment on edge cases, type hints, and streaming-friendly file I/O — not contest depth.

How should I prepare for a Bloomberg data engineering interview?

Solve the 2-problem Bloomberg practice set end to end — that maps the exact Python pattern coverage. Then back-fill: 10+ two-pointer / string-manipulation problems for the EASY phone-screen flavor, 10+ OOP / abstract-class problems for the HARD onsite Python round, and 20+ SQL window-function and interval-overlap problems for the SQL round. Add Bloomberg-specific behavioral prep — two real STAR stories with crisp numbers — and one read-through of the streaming stack (Kafka, Flink, Airflow, Parquet).

Does Bloomberg test object-oriented Python with abstract classes?

Yes — heavily. Half of recent Bloomberg DE coding rounds use an abstract base class (from abc import ABC, abstractmethod) and ask candidates to implement a concrete subclass. PipeCode's Chunked CSV to LDJSON Processor is the canonical PipeCode-Bloomberg problem in this family, and external interview reports confirm the pattern repeats with DataProcessor and RotatingFileSink framings. Master the ABC boilerplate and the streaming-generator pattern and this round becomes mechanical.

Start practicing Bloomberg data engineering problems

Reading patterns is not the same as typing them under time pressure. PipeCode pairs company-tagged Bloomberg problems with tests, AI feedback, and a coding environment so you can drill the exact Python OOP, two-pointer, and SQL window-function patterns Bloomberg asks — without the noise of generic algorithm prep.

Pipecode.ai is Leetcode for Data Engineering.

Browse Bloomberg practice →
Python for DE interviews course →

Rivian Data Engineering Interview Questions: Full DE Prep Guide

Gowtham Potureddi — Sat, 02 May 2026 06:14:30 +0000

Rivian Data Engineering Interview Questions: Full DE Prep Guide

Rivian data engineering interview questions sit at the intersection of three narrow, fluency-graded patterns: SQL aggregation with GROUP BY to summarize per-entity metrics like MIN(marks) and MAX(marks) per subject, aggregation joins that combine LEFT JOIN with SUM(fare) and ORDER BY total ASC LIMIT N to surface the lowest-earning locations on a ride-hailing dataset, and vanilla Python string padding that centers a string in a fixed-width line using len(), integer division (width - len(s)) // 2, and ' ' * pad + s + ' ' * pad — no str.center() shortcut. The schema you reason over feels like Rivian's own product (vehicles, drivers, rides, locations, telemetry_events), and the bar is fluency with MIN/MAX/SUM per group, JOIN-aggregate-order-limit composition, and string-arithmetic primitives — not contest-difficulty algorithms.

This guide walks through the four topic clusters Rivian actually tests, each with a detailed topic explanation, per-sub-topic explanation with a worked example and its solution, and an interview-style problem with a full solution that explains why it works. The mix matches the curated 3-problem Rivian set (3 easy, 0 medium, 0 hard) plus one process-and-prep section — a SQL-and-vanilla-Python loop where dictation of the invariant out loud is half the score and the other half is typing the right primitive on the first try. Strong data engineer interview questions prep at Rivian is less about contest depth and more about clean per-entity rollups, deterministic ordering, and arithmetic primitives that Snap to the AWS pipeline stack the company runs on.

1. The Rivian Data Engineering Interview Process

The Rivian DE interview funnel from recruiter call to onsite panel

The Rivian data engineer interview process is a four-stage funnel that takes about thirty days end to end: a recruiter screen, a technical phone screen that combines SQL and basic Python in a single 60-minute CoderPad session, a virtual onsite of four to five rounds covering coding plus system design plus behavioral, and an optional hiring-manager sync at the end. The technical bar is calibrated at LeetCode-medium for the algorithm half and "basic Python" for the data half — not contest difficulty.

The two stages most candidates misread are the technical phone screen (over-prepared on contest algorithms, under-prepared on SQL aggregation and cumulative-sum patterns) and the behavioral round — Rivian takes the Compass values seriously enough that candidates have been rejected despite strong technical performance for arrogance or for lacking a collaborative mindset. Both are addressed below.

Recruiter and hiring-manager calls

Detailed explanation. The recruiter screen is a 30-minute fit-and-routing chat: background, motivation, why Rivian, what excites you about the EV / adventure-vehicle space. Rivian recruiters specifically ask "Why Rivian?" and expect a real answer beyond generic enthusiasm — name a feature like Camp Mode, the R2 / R3 lineup, or the company's outdoor-mission stance. Hold your salary expectation until the offer stage; naming a number first usually leaves money on the table. The hiring-manager call (sometimes folded into the recruiter screen, sometimes its own block) is behavioral — past projects, problem-solving approach, how you collaborate cross-functionally with data scientists, ML engineers, and cloud-infrastructure teams.

Worked example. When the recruiter asks "why Rivian over a generalist data role?", a clean answer names the data shape — "Rivian's data is event-shaped, fleet-scale, and tied to a physical product. I want to design pipelines where the same telemetry feeds both the in-cab UI and the long-tail charging analytics — that's a different problem than warehousing for ad spend." Specific beats generic.

Rule of thumb: one Rivian-specific signal in the answer (Camp Mode, R2/R3, charging network, fleet telemetry) every minute of the recruiter call.

Technical phone screen — SQL plus basic Python

Detailed explanation. Roughly 60 minutes in CoderPad. The flavor that interview reports converge on is SQL (LeetCode-medium, often a cumulative-sum / rolling-average / aggregation question on a fleet-style schema) plus basic Python (a small simulation or string-processing problem solvable with len, slicing, dicts, and conditionals — no pandas, no re, no library shortcuts). Reports anchor the specifics: a Senior DE candidate in Canada (Nov 2023) was given "a cumulative sum type question in SQL" and "some SQL and basic Python" in the same hour. The signal Rivian is reading is decomposition + data-shape awareness, not library knowledge.

Worked example. A typical SQL prompt on Rivian-style data: "Given a rides(driver_id, fare, ride_at) table, return each driver's running total of fare, ordered by ride_at." The right primitive is SUM(fare) OVER (PARTITION BY driver_id ORDER BY ride_at) — a window-function cumulative sum. Saying "this is a cumulative-sum-per-entity, so window function with PARTITION BY driver_id ORDER BY ride_at" before typing earns the round.

Rule of thumb: if the prompt says "running total", "cumulative", "rolling X-day", or "as of each row", reach for SUM() OVER (PARTITION BY … ORDER BY …) before anything else.

Virtual onsite — coding, system design, behavioral

Detailed explanation. The onsite is four to five rounds over Zoom, totaling four to five hours: a coding round (CoderPad, similar to the screen but harder), one or two system-design rounds (vehicle telemetry pipelines, OTA update systems, charging-network backends), a behavioral round, and sometimes a domain deep-dive on a previous project. The Rivian-distinctive system-design move is asking "what happens when a vehicle loses cellular connectivity?" — design with offline-first / graceful-degradation in mind, not just cloud infrastructure. The AWS stack you should know cold: S3 for the data lake, EC2 for compute, Lambda + Kinesis for streaming, Glue for ETL, Airflow for orchestration, Great Expectations + CloudWatch for data quality and alerting.

Worked example. "Design a pipeline that ingests vehicle telemetry from a fleet of 100,000 trucks, surfaces per-vehicle daily summaries, and tolerates a vehicle being offline for up to 24 hours." The clean answer batches local telemetry on the vehicle side, syncs to S3 when the cellular connection returns, parses through Glue, lands the daily summaries in Snowflake or Redshift, and uses CloudWatch alerts on stale partitions. Naming offline-first up front is what differentiates strong candidates.

Rule of thumb: every Rivian system-design answer should say "offline-first" once and "graceful degradation" once — those two phrases earn the round.

Behavioral round — the Rivian Compass values

Detailed explanation. Rivian uses the Rivian Compass behavioral framework with three pillars: Stay Adventurous (take calculated risks, push boundaries), Lead the Way (drive outcomes, own the problem), and Bring People Together (collaborate cross-functionally, raise the team). Bring two STAR stories per pillar — Situation, Task, Action, Result — each tied to a real Rivian-relevant skill: shipping under hard product deadlines, owning a pipeline through a vendor migration, mediating between data scientists and platform engineers. Generic teamwork stories will not pass.

Worked example. For Bring People Together, a strong story names the friction ("data scientists wanted hourly granularity; the platform team budgeted daily"), the mediation ("I scoped a hybrid: hourly for the top three KPIs, daily for the rest"), and the result ("shipped on time, hourly partitions surfaced two outage windows in the first month").

Common beginner mistakes

Naming a salary number on the recruiter call (anchor low; let them name the band first).
Treating the technical phone screen as a contest-algorithm round (Rivian's screen is decomposition-grade, not LeetCode-hard).
Skipping Why Rivian? prep and answering with generic EV enthusiasm.
Forgetting to name offline-first in the system-design round on a vehicle-telemetry prompt.
Bringing only one STAR story per Compass pillar — interviewers double-click and a single story runs out fast.

Practice: drill the Rivian DE panel before the live screen

COMPANY
Rivian — all DE problems
Rivian data engineering practice set

Practice →

SQL
Rivian — SQL only
Rivian SQL practice

Practice →

2. SQL Aggregation and GROUP BY for Per-Entity Stats

Aggregation, GROUP BY, and per-entity MIN / MAX in SQL for data engineering

The first canonical Rivian SQL pattern is the per-entity summary — given a long event table, return one row per entity with one or more aggregates. The headline interview problem on the Rivian practice set, Range of Marks Scored, is a textbook "MIN and MAX per subject" prompt: collapse the per-student marks table down to one row per subject with the lowest and highest mark in that subject. The canonical answer is SELECT subject, MIN(marks) AS min_mark, MAX(marks) AS max_mark FROM marks GROUP BY subject — three primitives in one line.

Pro tip: MIN and MAX are NULL-safe — aggregates ignore NULL inputs, and a group of all-NULL returns NULL. That is exactly what you want when a subject has no graded marks yet — the result row carries NULL through as "no data" instead of needing a separate WHERE filter.

`MIN`, `MAX`, `SUM`, `AVG`, `COUNT` — the five core aggregates

The aggregate invariant: each function reduces a group of input rows to a single scalar value, ignoring NULL inputs by default.

MIN(col) — smallest non-NULL value in the group.
MAX(col) — largest non-NULL value in the group.
SUM(col) — sum of non-NULL values; returns NULL if every input is NULL.
AVG(col) — SUM(col) / COUNT(col) over non-NULL values; rounding depends on the engine.
COUNT(col) — number of non-NULL values; COUNT(*) counts every row including all-NULL.

Worked example. A marks(student_id, subject, marks) table.

student_id	subject	marks
1	Math	75
2	Math	92
3	Math	NULL
4	English	68

Apply each aggregate per subject: MIN(Math) = 75, MAX(Math) = 92, SUM(Math) = 167, AVG(Math) = 83.5, COUNT(Math marks) = 2 (NULL skipped), COUNT(*) = 3 (NULL counted).

Worked-example solution.

SELECT subject,
       MIN(marks)  AS min_mark,
       MAX(marks)  AS max_mark,
       AVG(marks)  AS avg_mark,
       COUNT(marks) AS marks_recorded
FROM marks
GROUP BY subject;

Rule of thumb: default to COUNT(col) when "graded" / "recorded" / "non-NULL" matters. Reach for COUNT(*) only when "every row" is the literal intent.

`GROUP BY` for per-entity rollups

The grouping invariant: GROUP BY entity partitions the table by entity and applies each aggregate inside each partition; the result has exactly one row per distinct entity value.

One column — GROUP BY subject collapses by subject.
Multiple columns — GROUP BY subject, year collapses by subject-year pair (composite grain).
HAVING vs WHERE — WHERE filters rows before grouping; HAVING filters groups after aggregation. WHERE marks > 50 is fine; WHERE MIN(marks) > 50 is illegal — use HAVING MIN(marks) > 50.

Worked example. From the same marks table, group by subject and apply MIN + MAX.

subject	min_mark	max_mark
Math	75	92
English	68	68

Worked-example solution.

SELECT subject, MIN(marks) AS min_mark, MAX(marks) AS max_mark
FROM marks
GROUP BY subject;

Rule of thumb: every column in the SELECT list that is not inside an aggregate must appear in GROUP BY. Postgres errors loudly when you skip this; MySQL silently picks an arbitrary value.

Including non-aggregated columns — the trap

The selection invariant: non-aggregated columns must be in GROUP BY or wrapped in an aggregate.

SELECT subject, student_id, MIN(marks) ... — illegal in standard SQL because student_id is neither aggregated nor grouped. Postgres raises a "column must appear in the GROUP BY clause or be used in an aggregate function" error.
Workaround 1 — put student_id in GROUP BY (changes the grain to per-subject-per-student).
Workaround 2 — wrap it in an aggregate (MIN(student_id) returns the smallest id per subject).
Workaround 3 — switch to a window function (ROW_NUMBER() plus WHERE rn = 1 carries the full row).

Worked example. "For each subject, return the lowest mark and the student who got it." MIN(marks) alone cannot pull student_id. Either group by (subject, student_id) (changes the grain) or use ROW_NUMBER() OVER (PARTITION BY subject ORDER BY marks ASC).

Worked-example solution.

WITH ranked AS (
  SELECT subject, student_id, marks,
         ROW_NUMBER() OVER (PARTITION BY subject ORDER BY marks ASC) AS rn
  FROM marks
  WHERE marks IS NOT NULL
)
SELECT subject, student_id AS lowest_scorer, marks AS lowest_mark
FROM ranked
WHERE rn = 1;

Rule of thumb: "the smallest value per group" → MIN. "The full row at the smallest value per group" → ROW_NUMBER() + PARTITION BY. Pick the cheap one when you only need the value.

Common beginner mistakes

Selecting non-aggregated columns alongside MIN/MAX — error in Postgres, undefined in MySQL.
Using WHERE to filter aggregates (use HAVING instead).
Forgetting NULL semantics — COUNT(col) skips NULL, COUNT(*) does not.
Treating AVG as integer arithmetic — AVG of all-integer columns returns a double in most engines; cast or ROUND if the prompt asks for an integer.

PipeCode's SQL for data engineering interviews course drills these aggregation primitives across forty-plus problems, including the cumulative-sum and rolling-average variants Rivian's screen reaches for.

SQL Interview Question on Range of Marks Scored

Table marks(student_id INT, subject VARCHAR, marks INT) records every student's mark per subject (marks is nullable for ungraded entries). Return one row per subject with the lowest and highest recorded mark in that subject. Output subject, min_mark, max_mark, ordered by subject.

Solution Using GROUP BY with MIN and MAX

SELECT subject,
       MIN(marks) AS min_mark,
       MAX(marks) AS max_mark
FROM marks
GROUP BY subject
ORDER BY subject;

Step-by-step trace (input marks):

student_id	subject	marks
1	Math	75
2	Math	92
3	Math	60
4	English	68
5	English	81
6	English	NULL

GROUP BY subject — partition the table into two groups: Math (rows 1, 2, 3) and English (rows 4, 5, 6).
MIN(marks) per group — Math group: MIN(75, 92, 60) = 60. English group: MIN(68, 81, NULL) = 68 (NULL is skipped, not treated as zero).
MAX(marks) per group — Math group: MAX(75, 92, 60) = 92. English group: MAX(68, 81, NULL) = 81.
Order final output — ORDER BY subject returns alphabetical stability (English before Math).

Output:

subject	min_mark	max_mark
English	68	81
Math	60	92

Why this works — concept by concept:

GROUP BY subject — partitions the table into one block per distinct subject value; aggregates evaluate inside each block independently.
MIN and MAX are NULL-safe — both functions ignore NULL inputs by default, so the ungraded English row does not skew the result.
One pass over the table — the engine scans once, hashing rows by subject and updating per-group min / max accumulators; no sort is required.
No window function needed — the prompt asks only for the per-group extremes, not the full row that produced them; MIN/MAX is the right tool.
ORDER BY at the end — the aggregate already produced one row per subject; ORDER BY subject is purely for human readability.
Cost — O(n) for the single hash-aggregate pass plus O(g log g) for the final sort, where n is row count and g is the number of distinct subjects (typically g << n).

SQL
Rivian — aggregations
Range of Marks Scored (Rivian)

Practice →

COMPANY
Rivian — aggregations
Rivian aggregation problems

Practice →

3. Aggregation and Joins for "Lowest-N per Dimension"

Aggregation, JOIN, and ORDER BY LIMIT in SQL for ride-hailing analytics

The second canonical Rivian SQL pattern is the lowest-N (or top-N) per dimension — combine JOIN to bring the dimension's metadata in with GROUP BY aggregation on the metric, then ORDER BY metric ASC LIMIT N. The headline interview problem on the Rivian practice set, Least Earning Locations for a Ride-Hailing Platform, is exactly this shape: join rides to locations, sum the fare per location, and return the three lowest-earning locations. The canonical composition is SELECT loc.name, SUM(r.fare) AS total FROM locations loc LEFT JOIN rides r ON r.location_id = loc.id GROUP BY loc.name ORDER BY total ASC LIMIT 3.

Pro tip: the choice between LEFT JOIN and INNER JOIN here is not cosmetic. LEFT JOIN keeps locations with zero rides (their SUM(fare) returns NULL, which sorts as the lowest in most engines). INNER JOIN silently drops them — and a location with zero rides is, by definition, the lowest earner, so dropping it produces the wrong answer.

`INNER JOIN` vs `LEFT JOIN` for "include zero-event entities"

The join invariant: every row from the left table survives LEFT JOIN; unmatched right-side columns come through as NULL. INNER JOIN keeps only matched pairs.

INNER JOIN — drops left rows with no right match (silent loss).
LEFT JOIN — keeps left rows; right columns are NULL when no match.
For "lowest-N" prompts — LEFT JOIN is almost always the right call because zero-event entities are the lowest by definition.

Worked example. A locations table with 3 rows and a rides table with rides in only 2 of them.

locations:

id	name
1	Downtown
2	Airport
3	Suburb

rides:

id	location_id	fare
100	1	25
101	2	60
102	1	35

INNER JOIN returns only Downtown and Airport rows. LEFT JOIN returns Downtown, Airport, and Suburb with NULL ride fields — Suburb is the location with zero rides.

Worked-example solution.

SELECT loc.name, r.fare
FROM locations loc
LEFT JOIN rides r ON r.location_id = loc.id;

Rule of thumb: if the prompt says "every location", "include locations with no rides", or "lowest-earning" / "least", you need LEFT JOIN. INNER JOIN is fine only when the prompt explicitly excludes empty entities.

`SUM` aggregation per dimension

The sum invariant: SUM(metric) GROUP BY dimension collapses event rows to per-dimension totals; NULL inputs are skipped, and a group of all-NULL returns NULL.

SUM(fare) GROUP BY location_id — total revenue per location.
COALESCE(SUM(fare), 0) — replace NULL totals with zero when a location has zero rides (often desirable for ranking).
Mixing aggregates — SUM(fare) plus COUNT(*) plus AVG(fare) in one query is fine; all evaluate over the same group.

Worked example. Apply SUM(fare) to the LEFT JOIN result above, grouped by location name.

location	total_fare
Downtown	60
Airport	60
Suburb	NULL

The NULL for Suburb is the signal we want — that location is the lowest earner. Wrap in COALESCE(SUM(fare), 0) AS total_fare if the downstream consumer prefers 0 over NULL.

Worked-example solution.

SELECT loc.name AS location,
       COALESCE(SUM(r.fare), 0) AS total_fare
FROM locations loc
LEFT JOIN rides r ON r.location_id = loc.id
GROUP BY loc.name;

Rule of thumb: if the prompt asks to "rank" or "order" the totals, prefer COALESCE(SUM(...), 0) so NULL does not produce an engine-specific sort surprise.

`ORDER BY ASC LIMIT` for the lowest-N pattern

The ordering invariant: ORDER BY metric ASC LIMIT N returns the N rows with the smallest metric value; ORDER BY metric DESC LIMIT N returns the largest.

ASC LIMIT N — bottom N (lowest-earning, slowest, smallest).
DESC LIMIT N — top N (highest-earning, fastest, largest).
Tie-break — add a deterministic secondary key (ORDER BY total_fare ASC, location ASC) so two locations with identical totals always sort the same way.

Worked example. ORDER BY total_fare ASC LIMIT 3 on the totals above returns Suburb (NULL / 0), then Downtown / Airport tied at 60, alphabetical tie-break.

Worked-example solution.

SELECT loc.name AS location,
       COALESCE(SUM(r.fare), 0) AS total_fare
FROM locations loc
LEFT JOIN rides r ON r.location_id = loc.id
GROUP BY loc.name
ORDER BY total_fare ASC, location ASC
LIMIT 3;

Rule of thumb: always pair LIMIT with a deterministic ORDER BY chain. Without a tie-break, LIMIT 3 over a tied set is non-deterministic.

Common beginner mistakes

INNER JOIN instead of LEFT JOIN for "every location" / "lowest-earning" — silently drops zero-event entities.
Skipping COALESCE and getting NULL ordering surprises (Postgres sorts NULL last by default; MySQL sorts it first).
ORDER BY total ASC LIMIT 3 without a secondary tie-break — non-deterministic results across runs.
Using WHERE SUM(fare) = 0 instead of HAVING SUM(fare) = 0 — WHERE filters before aggregation; HAVING filters after.

SQL Interview Question on Least Earning Locations

Tables locations(id INT, name VARCHAR) and rides(id INT, location_id INT, fare DECIMAL) log every ride on a ride-hailing platform. Return the three lowest-earning locations by total fare, including locations with zero rides. Output location, total_fare, ordered by total_fare ASC with a deterministic tie-break by location ASC.

Solution Using LEFT JOIN with SUM and ORDER BY LIMIT

SELECT loc.name AS location,
       COALESCE(SUM(r.fare), 0) AS total_fare
FROM locations loc
LEFT JOIN rides r
  ON r.location_id = loc.id
GROUP BY loc.name
ORDER BY total_fare ASC, location ASC
LIMIT 3;

Step-by-step trace (inputs):

locations:

id	name
1	Downtown
2	Airport
3	Suburb
4	Marina

rides:

id	location_id	fare
100	1	30
101	1	250
102	2	540
103	4	95

LEFT JOIN locations to rides — match every locations row to its rides rows. Downtown gets two ride rows, Airport gets one, Marina gets one, Suburb gets a single row with NULL on every rides column.
GROUP BY loc.name — collapse to one row per location.
SUM(r.fare) per group — Downtown: 30 + 250 = 280. Airport: 540. Marina: 95. Suburb: SUM(NULL) = NULL.
COALESCE(SUM, 0) — Suburb's NULL becomes 0.
ORDER BY total_fare ASC, location ASC — sort ascending: Suburb (0), Marina (95), Downtown (280), Airport (540). The secondary location ASC would only matter if two totals tied.
LIMIT 3 — keep the first three rows: Suburb, Marina, Downtown.

Output:

location	total_fare
Suburb	0
Marina	95
Downtown	280

Why this works — concept by concept:

LEFT JOIN preserves the dimension — every locations row survives the join; Suburb (zero rides) survives with NULL on the rides columns and is correctly identified as a candidate for "lowest-earning."
SUM is NULL-safe — aggregates ignore NULL inputs; a group of all-NULL returns NULL, which is then coerced to 0 by COALESCE for clean ordering.
GROUP BY on the dimension's name — collapses event rows to one row per dimension; the per-group aggregate evaluates inside each block.
Composite ORDER BY — the primary key total_fare ASC answers the prompt; the secondary location ASC is the deterministic tie-break that guarantees stable output across runs.
LIMIT 3 with a deterministic ORDER BY — LIMIT over a sorted set returns the bottom N rows; without the tie-break, ties produce non-deterministic results.
Cost — O(rides + locations) for the hash-join, plus one aggregation pass and a final sort over the per-location aggregate (typically tiny relative to rides); the LIMIT does not change the asymptotics.

SQL
Rivian — joins + aggregation
Least Earning Locations (Rivian)

Practice →

SQL
Topic — joins
Joins problems (all companies)

Practice →

4. String Padding and Centering in Vanilla Python

String length, slicing, and pad-and-center in Python for data engineering

Half of the Rivian technical phone screen is basic Python, and the explicit constraint candidates report is no library shortcuts: write the primitive yourself with len(), integer division, slicing, and string multiplication. The headline problem on the Rivian practice set, Centered Display Generator, is a textbook "pad-and-center" prompt — given a string and a target width, return the string centered inside that width using only spaces. The canonical answer is ' ' * left_pad + s + ' ' * right_pad, where left_pad = (width - len(s)) // 2 and right_pad = width - len(s) - left_pad (the leftover handles odd-width edge cases).

Pro tip: Python provides str.center(width) as a built-in shortcut, but Rivian's "basic Python" framing rewards writing the pad arithmetic by hand. The interviewer is grading whether you understand (width - len(s)) // 2 for left pad and width - len(s) - left_pad for right pad — the off-by-one logic on odd-width inputs is the test.

`len()`, integer division, and the centering math

The math invariant: total padding is width - len(s); left padding is total // 2; right padding is total - left_pad. This formula handles odd-width inputs correctly because right pad absorbs the leftover character.

Even pad case — width=6, s='HI', total=4, left=2, right=2 → ' HI '.
Odd pad case — width=7, s='HI', total=5, left=2, right=3 → ' HI ' (right side gets the extra space).
Why right gets the leftover — by convention, when a string cannot be perfectly centered, we left-align (give the extra to the right) so longer-string variants stay readable.

Worked example. Centering 'HI' in width 6.

step	value
`len(s)`	2
`width - len(s)`	4
`left_pad = 4 // 2`	2
`right_pad = 4 - 2`	2
result	`' HI '` (2 spaces + HI + 2 spaces)

Worked-example solution.

def centering_pads(s: str, width: int) -> tuple[int, int]:
    total = width - len(s)
    left = total // 2
    right = total - left
    return left, right

Rule of thumb: compute total pad first, then split — never compute left and right independently with two // divisions, because rounding can produce off-by-one drift.

Multiplying strings — the `' ' * n` primitive

The multiplication invariant: s * n repeats the string s exactly n times; n = 0 returns the empty string; n < 0 also returns the empty string (Python does not raise).

' ' * 3 — three spaces (' ').
'-' * 5 — five dashes ('-----').
'ab' * 2 — 'abab'.
' ' * 0 — empty string '' (useful when len(s) == width exactly).
' ' * -1 — empty string '' (Python does not raise on negative repeats).

Worked example. Build the centered output by concatenation.

component	value
`' ' * 2`	`' '` (two spaces)
`s`	`'HI'`
`' ' * 2`	`' '` (two spaces)
concatenated	`' HI '`

Worked-example solution.

def centered(s: str, width: int) -> str:
    total = width - len(s)
    left = total // 2
    right = total - left
    return ' ' * left + s + ' ' * right

Rule of thumb: when total could be negative (input longer than width), ' ' * negative returns '' quietly — the function returns just s, which is then truncated separately if the prompt requires it.

Truncating when `len(s) > width`

The truncation invariant: when the input string is longer than the target width, slice it to s[:width] to fit; the math for left/right pad then degrades to zero.

Slice notation — s[:width] returns the first width characters; safe when len(s) <= width (it returns the whole string).
Pair with the pad math — apply s = s[:width] at the top, then run the standard pad-and-center logic; both branches collapse to the same formula.
Why slicing is the canonical answer — slicing is O(width) and unambiguous; using if len(s) > width: return s[:width] as a special case adds a branch you do not need.

Worked example. s = 'HELLO', width = 3 → s[:3] = 'HEL', len(s[:3]) = 3, total = 0, left = 0, right = 0, result 'HEL'.

Worked-example solution.

def centered_safe(s: str, width: int) -> str:
    s = s[:width]
    total = width - len(s)
    left = total // 2
    right = total - left
    return ' ' * left + s + ' ' * right

Rule of thumb: slice first, pad second. The slice handles the over-long input case; the pad math handles the equal and short cases without a branch.

Common beginner mistakes

Reaching for str.center(width) despite the "basic Python" rule — interviewers note it as a signal you did not internalize the constraint.
Computing right_pad = total // 2 independently — drifts by one on odd-width inputs.
Forgetting the s[:width] truncation guard and producing output longer than width.
Using ' '.join([s]) instead of ' ' * n + s + ' ' * n — join does not pad; it inserts a separator.

Python Interview Question on Centered Display Generator

Write a function center_display(s: str, width: int) -> str that returns s centered inside a string of length width, padded with spaces. If len(s) > width, truncate s to width characters before centering. When the total padding is odd, the right side receives the extra space. Do not use str.center() or any standard-library helper.

Solution Using Pad-and-Center Without str.center

def center_display(s: str, width: int) -> str:
    s = s[:width]
    total = width - len(s)
    left = total // 2
    right = total - left
    return ' ' * left + s + ' ' * right

Step-by-step trace (input s = 'HI', width = 7):

step	expression	value
0	`s[:width]` (no truncation needed)	`'HI'`
1	`len(s)`	2
2	`width - len(s)`	5
3	`left = total // 2`	2
4	`right = total - left`	3
5	`' ' * left`	`' '` (two spaces)
6	`' ' * right`	`' '` (three spaces)
7	concatenated	`' HI '` (2 + 2 + 3 = 7 chars)

Truncate s to width — 'HI'[:7] returns 'HI' unchanged; the slice is a no-op when len(s) <= width.
Compute total padding — 7 - 2 = 5. The total amount of space to distribute around s.
Compute left pad — 5 // 2 = 2. Integer division rounds toward zero (or floor for non-negative inputs); the left side gets the smaller half on odd-width inputs.
Compute right pad — 5 - 2 = 3. The right side receives the leftover, satisfying the prompt's "right takes the extra" rule.
Build the output — ' ' * 2 + 'HI' + ' ' * 3 → ' HI '. Total length is 2 + 2 + 3 = 7, which matches width.

Output:

input s	input width	result
`'HI'`	6	`' HI '`
`'HI'`	7	`' HI '`
`'HELLO'`	3	`'HEL'` (truncated)
`''`	4	`' '` (four spaces, len(s) = 0)
`'EXACT'`	5	`'EXACT'` (no padding needed)

Why this works — concept by concept:

Slice first, pad second — s[:width] collapses the over-long input case to the equal-length case before any pad arithmetic runs; the rest of the function does not need a branch.
Total then split — computing total once and deriving left from total // 2 plus right from total - left guarantees left + right == total exactly, with no rounding drift on odd-width inputs.
Right absorbs the leftover — the formula right = total - left (rather than right = total // 2) sends the extra character to the right side on odd-pad cases, matching the prompt's specification.
String multiplication is safe at zero — ' ' * 0 returns '' quietly when no padding is needed; no special case for len(s) == width.
Concatenation order matters — ' ' * left + s + ' ' * right reads left-to-right as the visual output; reversing the order would mirror the result.
Cost — O(width) for the slice, O(left + right) = O(width) for the multiplication and concatenation; total O(width), independent of the input string's original length.

PYTHON
Rivian — string processing
Centered Display Generator (Rivian)

Practice →

PYTHON
Topic — string processing
String processing problems

Practice →

Tips to crack Rivian data engineering interviews

These are habits that move the needle in real Rivian DE loops — not a re-statement of the topics above.

Practice with Rivian's data shapes

Rivian's interview prompts model a vehicle / fleet / ride-hailing world: vehicles, drivers, rides, locations, charging_sessions, telemetry_events. Drilling on order-line ecommerce schemas wastes prep time. Stick to event-shaped tables with a per-entity grain, and pull problems from the aggregation topic page and the joins topic page for shapes that match.

Master `GROUP BY` and aggregation cold

The 3-problem PipeCode set is aggregation-heavy. Type SELECT entity, MIN(metric), MAX(metric), SUM(metric), AVG(metric), COUNT(metric) FROM events GROUP BY entity from memory until it is muscle memory. Layer in HAVING for group-level filters, COALESCE(SUM, 0) for ranking-friendly output, and ORDER BY metric ASC LIMIT N for lowest-N prompts.

Add cumulative-sum and rolling-average to your toolkit

Recent Rivian DE candidates have been asked cumulative-sum questions in SQL (Taro report, Nov 2023) and TechPrep names "Rolling Average SQL Query" and "Restaurant Growth" as recurring shapes. The primitive is SUM(metric) OVER (PARTITION BY entity ORDER BY ts ROWS BETWEEN N PRECEDING AND CURRENT ROW). Drill it on the window-functions topic page.

Write Python without library shortcuts

Rivian's "basic Python" framing means no str.center, no pandas, no re for these screens. Train yourself to reach for len, slicing, dicts, lists, and conditionals before any import. PipeCode's Python for data engineering interviews course drills the vanilla-Python primitives that match Rivian's expectations.

Know the AWS DE stack

Rivian's pipelines run on S3 (data lake), Lambda + Kinesis (streaming), Glue (ETL), Airflow (orchestration), and Great Expectations + CloudWatch (data quality and alerting). Even if the screen does not test these by name, the system-design round will. Be able to draw a vehicle-telemetry pipeline that ingests from a fleet of trucks, lands daily summaries in a warehouse, and tolerates a 24-hour offline window — in five minutes.

Map STAR stories to the Rivian Compass

The Compass is non-negotiable: Stay Adventurous, Lead the Way, Bring People Together. Bring two real STAR stories per pillar, each tied to a Rivian-relevant skill (shipping under a hard deadline, owning a vendor migration, mediating between data scientists and platform engineers). Generic teamwork stories will not stand out. PipeCode's behavioral interview prep course walks the STAR + values-mapping shape.

Where to practice on PipeCode

Skill lane	Practice path
Curated Rivian practice set	/explore/practice/company/rivian
Aggregation in SQL (Rivian-tagged)	/explore/practice/company/rivian/topic/aggregations
Joins for relational analytics	/explore/practice/topic/joins
Cumulative-sum / window functions	/explore/practice/topic/window-functions
String processing in vanilla Python	/explore/practice/topic/string-processing
All practice topics	/explore/practice/topics
Interview courses	/explore/courses

Communication under time pressure

State assumptions before typing: "I'll assume location_id is never NULL in the rides table and that we want zero-ride locations included in 'lowest earning.'" State grain: "One row per location after the aggregation." State edge cases: "If two locations tie on total_fare, my secondary ORDER BY location keeps the output stable." Interviewers grade clear reasoning above silent-and-perfect.

Frequently asked questions

What is the Rivian data engineering interview process?

The Rivian data engineer interview process is a four-stage funnel: a 30-minute recruiter screen, a 60-minute technical phone screen mixing SQL and basic Python in CoderPad, a virtual onsite of four to five rounds covering coding, system design, and behavioral, and an optional hiring-manager sync at the end. Total elapsed time is typically about thirty days. The curated Rivian practice set on PipeCode mirrors the technical-screen flavor (SQL aggregations + vanilla Python).

What programming languages does Rivian test for data engineering?

Rivian's data engineering interviews lean on SQL and Python for the screen, with AWS + Airflow + Kinesis + Great Expectations appearing in the system-design round. Python is "basic" — list, dict, str, len, slicing, conditionals — without pandas, re, or library shortcuts like str.center. SQL is at LeetCode-medium / DataLemur grade with a strong tilt toward aggregation, cumulative-sum, and joins on event data. PipeCode's string-processing topic page matches the Rivian Python flavor closely.

What SQL topics show up most in Rivian data engineering interviews?

The topics are narrow and consistent: aggregation with GROUP BY and MIN/MAX/SUM for per-entity rollups, LEFT JOIN plus aggregation plus ORDER BY ASC LIMIT N for lowest-N / top-N prompts, and cumulative-sum / rolling-average patterns using window functions like SUM() OVER (PARTITION BY entity ORDER BY ts). PipeCode's aggregation problems tagged to Rivian and the global joins problems cover these directly.

How difficult are Rivian data engineering interview questions?

Rivian data engineering interview questions are calibrated at LeetCode-medium for the algorithm half of the screen and "basic" for the data half — the bar is fluency, not contest difficulty. The Rivian PipeCode practice set is intentionally three EASY problems so candidates can build confidence on the exact patterns Rivian tests (aggregation, aggregation-with-joins, vanilla Python string processing). Reports describe the panel as friendly and conventional — no surprise loops, no whiteboard hazing.

How should I prepare for a Rivian data engineering interview?

Solve the 3-problem Rivian practice set end to end — that maps the exact pattern coverage. Then back-fill: 20+ aggregation problems for GROUP BY fluency, 10+ join-plus-aggregation problems for the lowest-N / top-N pattern, 10+ vanilla-Python string-processing problems, and a handful of cumulative-sum / rolling-average problems for the window-function half of the SQL screen. Add Rivian Compass behavioral prep — two STAR stories per pillar — and one read-through of the AWS DE stack (S3, Lambda, Kinesis, Glue, Airflow, Great Expectations).

Does Rivian's data engineering interview cover AWS and pipeline design?

Yes — the system-design round in the virtual onsite reaches deep into AWS (S3, EC2, Lambda, Glue, Kinesis), Airflow for orchestration, and Great Expectations + CloudWatch for data quality and alerting. Vehicle-telemetry pipelines, OTA update systems, and charging-network backends are the recurring framings. The Rivian-distinctive design move is offline-first — interviewers ask "what happens when a vehicle loses cellular connectivity?" and reward graceful-degradation strategies. PipeCode's ETL system design course walks the canonical pipeline architectures.

Start practicing Rivian data engineering problems

Reading patterns is not the same as typing them under time pressure. PipeCode pairs company-tagged Rivian problems with tests, AI feedback, and a coding environment so you can drill the exact SQL aggregation, join, and Python string-processing patterns Rivian asks — without the noise of generic algorithm prep.

Pipecode.ai is Leetcode for Data Engineering.

Browse Rivian practice →
SQL for DE interviews course →

#	Topic	Why it shows up in DE interviews
1	Bronze / silver / gold medallion zones	Progressive refinement is the single biggest lake-architecture concept; interviewers grade whether you know which transformations belong in landing/bronze vs refined/silver vs curated/gold and how SLAs differ per layer.
2	Ingestion → catalog → compute flow on object storage	Sources land into S3/GCS/ABS, register in a Hive/Glue/Unity catalog, and are queried by Spark, Trino, or warehouse external tables; the small-file problem, partition pruning, and schema evolution all live here.
3	Lake vs cloud warehouse vs lakehouse — and Iceberg / Delta / Hudi	The pattern-selection question is canonical; open table formats are what turn a lake into a lakehouse and bring ACID, time travel, and partition evolution to object storage.
4	Interview answer shape — grain, idempotency, lineage, reconciliation	Even system-design rounds reduce to a five-step template: clarify grain, separate landing from conformed, make loads idempotent, attach lineage keys, and reconcile aggregates against the source.

#	Topic	Why it shows up in SQL DE interviews
1	`INNER JOIN` vs `LEFT JOIN` and `LEFT JOIN ... IS NULL` anti-join	Combine rows from multiple tables without inflating cardinality; the `IS NULL` trick after a `LEFT JOIN` is the canonical "find rows in A with no match in B" pattern (orphan customers, churned users).
2	`GROUP BY` and `HAVING` for aggregates and duplicate detection	`WHERE` filters rows before grouping, `HAVING` filters groups after; the `HAVING COUNT(*) > 1` shape is the universal duplicate-finder.
3	Window functions — `ROW_NUMBER` vs `RANK` vs `DENSE_RANK`, `LAG`, `LEAD`	Per-partition ranking, top-N-per-group, second-highest salary, running totals, lookback for month-over-month deltas — all without collapsing rows.
4	CTEs, recursive CTEs, and correlated subqueries	`WITH` clauses make multi-step logic readable; recursive CTEs generate sequences and traverse hierarchies; correlated subqueries express row-by-row predicates against the same table.

#	Topic (sections 1–4)	Why it shows up at Facebook
1	Python array missing number — sum formula and XOR	Missing Number (EASY) — `n*(n+1)/2 - sum(arr)` arithmetic-series identity and its XOR self-cancellation alternative, the classic array primitive Meta phone-screens with.
2	Python arithmetic formula evaluator — array, math, bit, string parsing	Arithmetic Formula Evaluator (MEDIUM) — character-by-character tokenization plus two-pass evaluation (handle `*` and `/` first, then `+` and `-`), the parser primitive that powers any "given a string formula, return the value" question.
3	SQL window functions and `EXISTS` for monthly active user retention	Active User Retention — correlated `EXISTS` subquery with `EXTRACT(MONTH FROM ts - INTERVAL '1 month')` for month-over-month MAU, the SQL primitive that drives Meta product-analytics retention dashboards.
4	SQL CTE and self-join for post hiatus and friend recommendations	Average Post Hiatus + Friend Recommendations — `MIN/MAX(post_date)` per user with `HAVING COUNT > 1` and CTE-driven self-joins for pair-wise friend-rec queries, the SQL primitive that drives Meta social-graph analytics.

#	Topic (sections 1–4)	Why it shows up at Square
1	SQL ranking, sorting, and aggregation for top-N invoice senders	Top 10 Invoice Senders (EASY) — `GROUP BY sender_id + COUNT(*) + ORDER BY DESC + LIMIT 10`, the SQL primitive for any "top-N entities by event volume" question.
2	SQL aggregation, date functions, and cohort analysis for post-signup activity	Users with High Activity After 30 Days Signup (MEDIUM) — `DATEDIFF` / `INTERVAL '30 days'` + `GROUP BY user_id + HAVING COUNT(*) >= threshold`, the canonical retention-cohort pattern.
3	SQL window functions for monthly averages and duplicate detection	Monthly product rating + duplicate detection — `AVG(stars) OVER (PARTITION BY product_id, EXTRACT(MONTH FROM submit_date))` and `ROW_NUMBER() OVER (PARTITION BY ...)` for dedupe (DataLemur Block staples).
4	SQL `COUNT DISTINCT` and status filters for payment-flow analysis	Total unique successful transactions / senders / recipients — `COUNT(DISTINCT col) WHERE status = 'Success'`, the fintech-grade pattern that drives Square / Block / CashApp payment analytics.

#	Topic (sections 1–4)	Why it shows up at Snowflake
1	Python arrays and set validation for the SET card game	SET Card Game Validation (EASY) — `zip(*cards)` per-attribute iteration plus `len(set(vals)) in {1, 3}`, the all-same-or-all-different invariant that powers any "validate a row across N records" pipeline.
2	Python hash tables and sliding window for maximum substring occurrences	Maximum Substring Occurrences (MEDIUM) — rolling character-frequency dict over a length-`k` window plus a `Counter` keyed on the substring itself, the canonical pattern for "count distinct windows under a constraint."
3	SQL window functions (`LAG`, `LEAD`, `AVG OVER`) for Snowflake analytics	Marketing-touch streak detection — `LAG(DATE_TRUNC('week', event_date)) OVER (PARTITION BY contact_id ORDER BY ...)` plus CTE composition, the SQL primitive every Snowflake-flavored interview rotates through.
4	Snowflake architecture: micro-partitions, clustering, and Time Travel	Three-layer architecture (Storage / Query Processing / Cloud Services), micro-partition pruning via min/max metadata, clustering keys, scale-up vs scale-out, Time Travel + Fail-safe + Zero-Copy Cloning — the product-knowledge primitive every Snowflake-as-tool interviewer probes.

#	Topic (sections 1–4)	Why it shows up at Robinhood
1	Python hash tables and dict counters for stock-purchase aggregation	Stock Purchases Count (EASY) — `dict.get(symbol, 0) + 1` and dict-of-sets, the Python primitive for counting events and tracking distinct buyers.
2	SQL inner join and GROUP BY for top-N trade aggregations	Member Transfer Records (MEDIUM) — `INNER JOIN trades + users + WHERE status='Completed' + GROUP BY city + ORDER BY COUNT(*) DESC + LIMIT N`, the SQL primitive for top-N rankings on completed trades.
3	SQL window functions and `LAG` for daily volume change	Volume change percentage — `LAG(volume) OVER (PARTITION BY stock_symbol ORDER BY trade_date)` for day-over-day deltas on partitioned time series.
4	SQL aggregation and `HAVING` for threshold and notional-limit checks	Notional-limit check — `GROUP BY user_id HAVING SUM(contract_count * strike_price * 100) > limit`, the SQL primitive for "WHERE on aggregates" that flags risk and compliance breaches.

#	Topic (sections 1–4)	Why it shows up at Bloomberg
1	The Bloomberg data engineering interview process	Four-stage funnel — phone screen (45-60 min) → online assessment (some roles) → onsite of 3-5 rounds → final / fit. ~30 days end to end.
2	Python two-pointer and string manipulation	Reverse Words in String (EASY) — `s.split()` + `[::-1]` + `' '.join(...)`, or two-pointer in-place reversal of a character array.
3	Abstract classes, OOP, and file I/O in Python	Chunked CSV to Line-Delimited JSON Processor (HARD) — subclass an abstract `RecordProcessor`, implement `load`, `transform`, `write`, and stream `csv.DictReader` rows into `json.dumps(...) + '\n'` without loading the whole file.
4	SQL window functions for time-series and overlap	Rolling-sum aggregates and Subscription Overlap — `SUM(volume) OVER (PARTITION BY symbol ORDER BY trade_date)` for rolling totals, plus `EXISTS` interval crosscheck for overlap detection.

#	Topic (sections 1–4)	Why it shows up at Rivian
1	The Rivian data engineering interview process	Four-stage funnel — recruiter → tech screen (SQL + basic Python, 60 min) → onsite panel (4-5 rounds) → optional hiring-manager sync.
2	SQL aggregation and `GROUP BY` for per-entity stats	Range of Marks Scored (EASY) — `SELECT subject, MIN(marks), MAX(marks) FROM marks GROUP BY subject`.
3	Aggregation and joins for "lowest-N per dimension"	Least Earning Locations for a Ride-Hailing Platform (EASY) — `LEFT JOIN locations TO rides`, `SUM(fare) GROUP BY location`, `ORDER BY total ASC LIMIT 3`.
4	String padding and centering in vanilla Python	Centered Display Generator (EASY) — `(width - len(s)) // 2` left pad, `' ' * pad + s + ' ' * pad`, no `str.center()` shortcut.

DEV Community: Gowtham Potureddi

PostgreSQL SQL Data Types: Practical Column-Type Guide

1. Why column types matter

Storage, comparisons, indexes, and the cost of silent coercion

Storage footprint and on-disk layout

Equality and comparison semantics

Index operator classes and planner statistics

Common beginner mistakes

SQL Interview Question on Picking Types for an Orders Schema

Solution Using Tight Native Types + NUMERIC + TIMESTAMPTZ + CHECK Constraints

2. Numeric types

Integers for counts, NUMERIC for money, FLOAT for measurements

INTEGER / BIGINT — surrogate keys and counts

NUMERIC(p, s) — exact decimal for currency

REAL / DOUBLE PRECISION — binary floating point and rounding

Common beginner mistakes

SQL Interview Question on Reconciling a Drifting Invoice Total

Solution Using NUMERIC(14, 4) + a Generated line_total Column

3. Text and binary

CHAR vs VARCHAR vs TEXT, collations, and BYTEA

CHAR vs VARCHAR vs TEXT — pick TEXT unless you need fixed-width

Collations and locale-aware equality

BYTEA for binary blobs vs URL-in-SQL for files

Common beginner mistakes

SQL Interview Question on Reconciling Case-Sensitive Email Joins

Solution Using citext + Normalised Write-Time Email

4. Boolean and NULL

Three-valued logic and the WHERE flag trap

BOOLEAN literals, IS TRUE / IS FALSE / IS NULL

NOT col vs col = FALSE with NULLs

COALESCE and explicit NULL handling

Common beginner mistakes

SQL Interview Question on a Dashboard Missing 12% of Rows

Solution Using IS TRUE / IS NOT TRUE + a Schema-Level NOT NULL Fix

5. Date and time

DATE, TIME, TIMESTAMP, and TIMESTAMPTZ — instants vs wall clocks

TIMESTAMP without time zone — local wall-clock semantics

TIMESTAMPTZ — UTC instant, session display

AT TIME ZONE conversions and DATE_TRUNC pitfalls

Common beginner mistakes

SQL Interview Question on a Dashboard That Shifted 24 Hours After Deploy

Solution Using TIMESTAMPTZ + AT TIME ZONE in the Reporting View

6. Semi-structured and other types

JSONB, UUID, and arrays for flexible attributes

JSON vs JSONB — when binary indexing matters

UUID — opaque IDs for distributed systems

Arrays — INTEGER[], TEXT[], and the UNNEST pattern

Common beginner mistakes

SQL Interview Question on Searching JSONB Payloads at 50 M-Row Scale

Solution Using a GIN Index with jsonb_path_ops + @> Containment

7. Casting and comparison rules

Implicit coercion, explicit CAST, and index-friendly predicates

Implicit coercion — when PostgreSQL guesses

CAST(x AS type) vs x::type shorthand

Index-killing casts on indexed columns

Common beginner mistakes

SQL Interview Question on a Cross-Type Join Returning Zero Rows

Solution Using a Single-Type Schema + Explicit Literal-Side Cast

Choosing types (checklist)

Frequently asked questions

Should I use TEXT or VARCHAR(255)?

Is SERIAL still OK for primary keys?

Why is my join returning no rows when the IDs "look the same"?

When must I use NUMERIC instead of float?

Practice on PipeCode

PostgreSQL SQL Cheat Sheet — Clause Order, Joins, Aggregates, Windows

Top PostgreSQL SQL cheat sheet topics

1. PostgreSQL Logical Clause Order — FROM → WHERE → GROUP BY → HAVING → SELECT → ORDER BY → LIMIT

The seven-stage evaluation order every PostgreSQL query follows

FROM and JOIN — build the working row set

WHERE — row-level predicates before grouping

GROUP BY → HAVING → SELECT → ORDER BY → LIMIT

Common beginner mistakes

PostgreSQL Interview Question on Clause Order

Solution Using WHERE + GROUP BY + HAVING in the Right Slots

2. PostgreSQL Joins and Grain — INNER, LEFT, RIGHT, FULL, SELF, CROSS

Joins, anti-joins, and the grain-inflation trap in PostgreSQL

INNER JOIN — keep only matching pairs (no padding)

LEFT JOIN — keep every left row, pad the right with NULLs (anti-join trick)

FULL OUTER, SELF, and CROSS joins — the rarer shapes

Solution Using Tight Native Types + `NUMERIC` + `TIMESTAMPTZ` + `CHECK` Constraints

`INTEGER` / `BIGINT` — surrogate keys and counts

`NUMERIC(p, s)` — exact decimal for currency

`REAL` / `DOUBLE PRECISION` — binary floating point and rounding

Solution Using `NUMERIC(14, 4)` + a Generated `line_total` Column

`CHAR` vs `VARCHAR` vs `TEXT` — pick `TEXT` unless you need fixed-width

`BYTEA` for binary blobs vs URL-in-SQL for files

Solution Using `citext` + Normalised Write-Time Email

Three-valued logic and the `WHERE flag` trap

`BOOLEAN` literals, `IS TRUE` / `IS FALSE` / `IS NULL`

`NOT col` vs `col = FALSE` with NULLs

`COALESCE` and explicit NULL handling

Solution Using `IS TRUE` / `IS NOT TRUE` + a Schema-Level `NOT NULL` Fix

`DATE`, `TIME`, `TIMESTAMP`, and `TIMESTAMPTZ` — instants vs wall clocks

`TIMESTAMP` without time zone — local wall-clock semantics

`TIMESTAMPTZ` — UTC instant, session display

`AT TIME ZONE` conversions and `DATE_TRUNC` pitfalls

Solution Using `TIMESTAMPTZ` + `AT TIME ZONE` in the Reporting View

`JSONB`, `UUID`, and arrays for flexible attributes

`JSON` vs `JSONB` — when binary indexing matters

`UUID` — opaque IDs for distributed systems

Arrays — `INTEGER[]`, `TEXT[]`, and the `UNNEST` pattern

Solution Using a `GIN` Index with `jsonb_path_ops` + `@>` Containment

Implicit coercion, explicit `CAST`, and index-friendly predicates

`CAST(x AS type)` vs `x::type` shorthand

Should I use `TEXT` or `VARCHAR(255)`?

Is `SERIAL` still OK for primary keys?

When must I use `NUMERIC` instead of float?

1. PostgreSQL Logical Clause Order — `FROM` → `WHERE` → `GROUP BY` → `HAVING` → `SELECT` → `ORDER BY` → `LIMIT`

`FROM` and `JOIN` — build the working row set

`WHERE` — row-level predicates before grouping

`GROUP BY` → `HAVING` → `SELECT` → `ORDER BY` → `LIMIT`

Solution Using `WHERE` + `GROUP BY` + `HAVING` in the Right Slots

2. PostgreSQL Joins and Grain — `INNER`, `LEFT`, `RIGHT`, `FULL`, `SELF`, `CROSS`

`INNER JOIN` — keep only matching pairs (no padding)

`LEFT JOIN` — keep every left row, pad the right with `NULL`s (anti-join trick)

`FULL OUTER`, `SELF`, and `CROSS` joins — the rarer shapes

Solution Using `LEFT JOIN ... WHERE orders.order_id IS NULL`

3. PostgreSQL `GROUP BY`, `HAVING`, and Conditional Aggregates

`GROUP BY` with `HAVING`, `FILTER`, and `CASE` for one-pass metrics in PostgreSQL

`COUNT`, `SUM`, `AVG`, `MIN`, `MAX` — `NULL`-aware aggregates

`WHERE` vs `HAVING` — row filter vs group filter

`FILTER (WHERE …)` and `CASE` — conditional aggregates

Solution Using `GROUP BY email HAVING COUNT(*) > 1`

4. PostgreSQL Window Functions — `ROW_NUMBER`, `RANK`, `DENSE_RANK`, `LAG`, `LEAD`

`ROW_NUMBER` — unique sequential numbering per partition

`RANK` vs `DENSE_RANK` — tie semantics

`LAG`, `LEAD`, and running totals — lookback, lookahead, and `SUM(...) OVER (...)`

Solution Using `DENSE_RANK() OVER (PARTITION BY department ORDER BY salary DESC)` in a CTE

State the grain before any `JOIN`

Use `LEFT JOIN ... IS NULL` over `NOT IN` for anti-joins

Pick `DENSE_RANK` for "Nth distinct"; pick `ROW_NUMBER` for deduplication

Use `FILTER (WHERE …)` for one-pass conditional metrics

Always `ORDER BY` + tiebreaker; pair `LIMIT` with `ORDER BY`

Use PostgreSQL-specific helpers — `EXTRACT`, `DATE_TRUNC`, `INTERVAL`, `::DATE` cast

What is the difference between `WHERE` and `HAVING` in PostgreSQL?

What is the difference between `RANK`, `DENSE_RANK`, and `ROW_NUMBER`?

What does `FILTER (WHERE …)` do in PostgreSQL aggregates?

Why is `LIMIT 5` returning different rows on different runs?

Step 4 — Attach row-level lineage keys — `ingest_id`, `source_ts`, `pipeline_version`