DEV Community: Firat Celik

DuckDB 2.0 vs Polars vs Spark on One Machine: 150M Rows Under a 4.5 GiB Cap

Firat Celik — Thu, 22 Oct 2026 08:24:18 +0000

DuckDB 2.0.0 shipped on October 21, 2026 (announcement, PyPI). The next morning I ran the same single-machine benchmark I had already run on DuckDB 1.5.6: three engines, one 8 vCPU VM, Parquet in and Parquet out, a hard memory cap, and four queries that look like real data engineering work.

The thesis I started with, and that the numbers support:

DuckDB beats Spark-on-one-box on memory-bound DE work. It finished every run, at every size, 5 to 12 times faster than local PySpark at 150M rows, and it still finished the heavy queries with a 1 GB memory limit.
Polars hits a wall on window dedup. At 150M rows the ROW_NUMBER() style "latest row per customer" query never finished under the cap. Not at 4.5 GiB, not with an out-of-core budget, not at 9 GiB.
Polars is not slow. On filters, joins and a high-cardinality group by it ties DuckDB or beats it, when it fits.

Everything below is measured on that one box. Where I interpret or give an opinion, I say so.

The contenders

	DuckDB 2.0.0	Polars 2.0.0	PySpark 4.2.0
Model	In-process SQL engine	In-process DataFrame engine, lazy + streaming	JVM engine, `local[8]`
How I drove it	`COPY (SQL) TO 'out.parquet'`	`LazyFrame.sink_parquet()`	`spark.sql(...).write.parquet()`
Memory knob	`SET memory_limit = '3GB'`	default (plus one run with an out-of-core budget)	`spark.driver.memory = 3g`
Threads	8	8 (`POLARS_MAX_THREADS`)	8

Polars 2.0.0 was published on PyPI on October 6, 2026, PySpark 4.2.0 on July 14, 2026. DuckDB 2.0.0 wheels landed on PyPI on October 21, the same day as the release calendar entry.

Setup

Hardware and versions

VM: 8 vCPU Intel Xeon, 15 GiB RAM, no swap, Linux 6.12. It is a shared box, so other processes were running. Every run waited until at least 5 GiB was free and CPU busy was under 30% over a 3 second window.
Python 3.13.5, OpenJDK 21.0.12, pyarrow 26.0.0.
DuckDB 2.0.0, Polars 2.0.0, PySpark 4.2.0. The "before" numbers are DuckDB 1.5.6 on the same box and data, run on October 19.

Data. A seeded synthetic star schema written with pyarrow: a sales fact table and a 50,000-row products dimension. Three sizes: 10M rows (213 MB), 50M (1.1 GB) and 150M (3.2 GB of snappy Parquet, 30 files of 5M rows). About 10 orders per customer, so 150M rows means roughly 15M customers.

Queries. The same SQL for DuckDB and Spark, the same logic as Polars expressions:

q1_filter_agg: filter on a timestamp and status, group by store and status.
q2_join_agg: fact-to-dimension join, group by category and brand.
q3_window_latest: latest order per customer with ROW_NUMBER(). The classic dedup and SCD pattern.
q4_highcard_agg: one row per customer (about 15M groups at 150M).

-- q3_window_latest
SELECT customer_id, order_id, ts, qty * price AS amount
FROM (
    SELECT *, ROW_NUMBER() OVER (PARTITION BY customer_id
                                 ORDER BY ts DESC, order_id DESC) AS rn
    FROM sales
) t
WHERE rn = 1

Method.

Every run is a fresh process: interpreter start, import, session, read Parquet, compute, write Parquet. So no warm caches inside the engine, and Spark pays its JVM start every time.
Engines run strictly one at a time. The engine order rotates per repetition so no engine always goes first.
5 repetitions at 10M, 3 at 50M and 150M. I report medians.
A watchdog samples the RSS of the whole process tree every 50 ms and kills the run if it goes above 4.5 GiB. That is the cap. It is deliberately tighter than the box, because "fits in a small container" is the question I care about.
Wall is the whole process. Query is read, compute and write only. The difference is setup.
Correctness: a validation script compares row counts and rounded checksums of every output. Where an engine produced output, all three engines agreed, for every query and size.

Results at 150M rows

Query	DuckDB 2.0.0	Polars 2.0.0	PySpark 4.2.0
Filter + aggregate	0.87 s (3/3)	0.96 s (3/3)	10.77 s (3/3)
Join + aggregate	1.47 s (3/3)	1.46 s (3/3)	16.10 s (3/3)
Window dedup	7.92 s (3/3)	killed (0/3)	46.29 s (2/3)
High-cardinality group by	7.91 s (3/3)	4.21 s (1/3)	39.70 s (3/3)

Median wall seconds, runs that finished under the 4.5 GiB cap in brackets.

Peak RSS (median of finished runs) tells the other half:

Query	DuckDB 2.0.0	Polars 2.0.0	PySpark 4.2.0
Filter + aggregate	0.77 GiB	0.94 GiB	1.85 GiB
Join + aggregate	0.42 GiB	1.96 GiB	1.61 GiB
Window dedup	3.01 GiB	above 4.5 (killed)	3.74 GiB, one run hit 7.51
High-cardinality group by	3.01 GiB	4.49 GiB	4.40 GiB

Three things stand out.

Spark on one box pays twice. About 5 seconds of every Spark run is setup (median 5.0 to 5.8 s, JVM plus session) before it reads a byte. Then the query itself is slower: query-only time was 7.4x DuckDB's on the filter, 7.9x on the join, 5.2x on the window dedup and 4.3x on the group by. In wall time that is 5x to 12x. And spark.driver.memory = 3g did not keep the process at 3 GiB: one window dedup run reached 7.51 GiB of process-tree RSS and was killed.

Polars is fast right up to the cap. It tied DuckDB on filter and join and was the fastest engine on the high-cardinality group by (4.21 s against 7.91 s). But only 1 of 3 runs finished: that one peaked at 4.49 GiB, and the other two crossed the cap at 4.50 and 4.55 GiB. Fast and close to the edge is a fine place for a notebook and a bad place for a scheduled job.

The window dedup is where Polars stops. At 150M rows it was killed in all 3 runs, within a few seconds of starting, at 4.52 to 4.66 GiB.

Polars and the window wall

I gave Polars two more chances on the 150M window dedup:

Polars 2.0.0, 150M window dedup	Result
Default settings, 4.5 GiB cap (Oct 22)	0/3, killed at 4.52 to 4.66 GiB
`POLARS_OOC_MEMORY_BUDGET_MB=3072`, 4.5 GiB cap (Oct 22)	0/3, killed at 4.57 to 4.66 GiB
Default settings, 9 GiB cap (Oct 19, same Polars 2.0.0)	0/3, killed at 9.04 to 9.54 GiB

The out-of-core budget variable is one I found in the Polars 2.0.0 source, not a documented knob, so treat that row as "I tried", not as guidance. The same variable did help the group by: with it, Polars finished 3/3 at 4.67 s and 4.18 to 4.30 GiB.

At smaller sizes the dedup works, but the shape is already visible:

Window dedup	DuckDB 2.0.0	Polars 2.0.0	PySpark 4.2.0
10M rows	0.60 s, 0.65 GiB	1.83 s, 1.20 GiB	12.74 s, 2.94 GiB
50M rows	2.33 s, 2.19 GiB	10.36 s, 4.46 GiB	29.06 s, 3.77 GiB
150M rows	7.92 s, 3.01 GiB	killed	46.29 s, 3.74 GiB

At 50M Polars already used 4.46 GiB for a query DuckDB did in 2.19 GiB, and took 4.4x longer.

This is the Polars expression I used. It is the direct translation of the SQL:

import polars as pl

amount = pl.col("qty").cast(pl.Float64) * pl.col("price")
rn = pl.int_range(pl.len()).over(
    "customer_id", order_by=["ts", "order_id"], descending=True
)
(
    pl.scan_parquet("data/sales_150m/*.parquet")
    .filter(rn == 0)
    .select("customer_id", "order_id", "ts", amount.alias("amount"))
    .sink_parquet("out/polars/150m_q3_window_latest.parquet")
)

Fair caveat: I did not test other Polars formulations of the same dedup (a sort followed by unique(keep="first"), or a group_by().agg(...) with sort_by().last()). One of them may stream better. The point stands for the common case, though: the first thing a team writes is the window, and that one did not fit.

Update (Oct 22): after this went out, Victor Cid (@Vleyked) tested exactly that in engine-bench PR #1. His sort + unique(keep="first") version drops the in-memory window fallback from the plan and finished 3/3 at 150M rows under the same 4.5 GiB cap, peaking at 2.50 to 2.74 GiB, but only with POLARS_OOC_MEMORY_BUDGET_MB=1024. With default settings or a 3072 MB budget it still went over the cap. The price is speed: a 90.58 s median against 17.56 s for DuckDB on the same 4 vCPU runner, so roughly 5x slower. His output matched DuckDB row for row (14,999,347 rows). So Polars can do this dedup under the cap if you rewrite it and tune the spill budget, and with a smarter query it gets close to DuckDB (see below). His runs were on different hardware, so they are not in the tables above.

Second update: in engine-bench PR #2 he filtered each customer down to their max timestamp first and only sorted those rows to break ties. That brought the median down to 14.23 s at 150M rows with default Polars settings, peaking at 4.04 GiB under the same cap, with output matching DuckDB row for row. On his runner DuckDB's reference was a single 18.50 s run, so I'm not calling that a repeated cross-engine win. The real lesson is that the first way you write a Polars dedup can decide whether it finishes at all.

Third update (Oct 23): Ritchie Vink, the creator of Polars, pointed out that only DuckDB was told about the memory limit, so Polars did not know it had to stay under 4 GiB, and suggested setting POLARS_OOC_MEMORY_BUDGET_MB for a fairer comparison. I reran every Polars query at 150M rows on the same box under the same 4.5 GiB cap with that budget, next to a DuckDB rerun in the same session: the high-cardinality group by finished 3/3 at 6.17 s and 3.46 GiB with a 3072 MB budget, against 9.56 s for DuckDB (the box was busier than on Oct 22 and DuckDB ran slower too, so compare within this session, not with the tables above; the Oct 22 run with the same budget gave 4.67 s). The window dedup with over(order_by) still went over the cap in every run at 3072, 1024 and 512 MB, Victor's sort plus unique version fit only at 1024 MB (31.28 s, 2.29 GiB), and the max-timestamp rewrite finished 3/3 on default settings (5.61 s, 4.02 GiB). Ritchie says Polars main now spills the group by correctly and that this ships in the next release, which was not on PyPI yet when I ran this, so I will rerun when it ships. Raw results and output checks are in engine-bench PR #3 and in results/polars_ooc_budget_2026-10-23 on the polars-ooc-budget-2026-10-23 branch.

DuckDB under pressure: memory_limit = 1 GB

The interesting DuckDB question is not "how fast with room to spare" but "what happens when the box is small". Same 150M rows, memory_limit cut to 1 GB:

150M, `memory_limit = 1GB`	Wall (median)	Peak RSS	Finished
Window dedup	14.97 s	1.12 GiB	3/3
High-cardinality group by	13.94 s	1.08 GiB	3/3
Window dedup, `preserve_insertion_order = false`	14.94 s	1.12 GiB	3/3
Group by, `preserve_insertion_order = false`	13.88 s	1.08 GiB	3/3

With a third of the memory, DuckDB spilled to disk and still beat Spark's unconstrained run by about 3x on both queries, at a quarter to a third of Spark's peak RSS. The DuckDB docs list window functions with PARTITION BY ... ORDER BY among the blocking operators it can process larger than memory (How to Tune Workloads). This run is that sentence, measured.

The setup is three lines:

import duckdb

con = duckdb.connect()
con.execute("SET memory_limit = '1GB'")
con.execute("SET threads = 8")
con.execute("SET temp_directory = '/mnt/fast-local-disk/duckdb_tmp'")  # where spills go
con.execute("""
    COPY (
        SELECT customer_id, order_id, ts, qty * price AS amount
        FROM (
            SELECT *, ROW_NUMBER() OVER (PARTITION BY customer_id
                                         ORDER BY ts DESC, order_id DESC) AS rn
            FROM read_parquet('data/sales_150m/*.parquet')
        )
        WHERE rn = 1
    ) TO 'out/latest_per_customer.parquet' (FORMAT parquet)
""")

DuckDB 1.5.6 vs 2.0.0

Same box, same data, same scripts. 1.5.6 ran on October 19, 2.0.0 on October 22. Median wall seconds at 150M:

Setting	Query	1.5.6	2.0.0	Change
`memory_limit = 3GB`	Window dedup	9.09	7.92	-13%
`memory_limit = 3GB`	Group by	9.61	7.91	-18%
`memory_limit = 1GB`	Window dedup	17.79	14.97	-16%
`memory_limit = 1GB`	Group by	22.06	13.94	-37%
1GB, no insertion order	Window dedup	14.85	14.94	+1%
1GB, no insertion order	Group by	15.15	13.88	-8%

The useful finding is the last four rows together. On 1.5.6, SET preserve_insertion_order = false was worth 16% on the dedup and 31% on the group by under a 1 GB limit. On 2.0.0 the default already lands where the tuned setting did, and the flag changes almost nothing for these two queries. If you carry that setting in job configs as a performance trick, re-measure after upgrading. It still matters for what it was designed for: the docs recommend it when large imports or exports run out of memory, and it lets DuckDB reorder rows without an ORDER BY.

Not everything improved. The 50M window dedup was slower on 2.0.0 (1.94 s to 2.33 s), and peak RSS for the light queries went up (150M filter: 0.26 to 0.77 GiB, join: 0.23 to 0.42 GiB), while the heavy queries went down (window dedup: 3.47 to 3.01 GiB). I have not investigated why. The two runs were also three days apart on a shared box, so treat differences under about 10 to 15% as noise. The 1 GB group by drop is far outside that.

The release itself is mostly a feature release: Quack and CONNECT for client-server use, asynchronous I/O, a first-class VARIANT, triggers, a PEG parser by default (announcement). It is an LTS line supported until at least October 2027. One migration note matters for pipelines: database files written by 2.0 cannot be read by versions before 2.0. My benchmark only reads and writes Parquet, so it never touched that.

Where Spark still wins

None of this says "replace Spark". It says Spark on one box is the wrong comparison for small and medium jobs. Spark still wins when:

The data does not fit one node's disk and memory, or the deadline does not fit one node's cores. This test stops at 3.2 GB of Parquet on purpose. DuckDB spills well, but one machine is still one machine.
The platform is built around it. On Databricks and Fabric, Spark is where Unity Catalog or OneLake governance, Delta writes with concurrent writers, lineage, job orchestration and streaming already live. A DuckDB step outside that boundary is a step you have to secure, monitor and explain.
The cluster is already running. The 5 second JVM start I measured per run disappears in a warm interactive cluster or a long job. The query-only gap (4x to 8x here) does not.
The team is a Spark team. One engine everyone can debug at 02:00 is worth more than a faster engine one person understands.

My opinion as a Lead DE on Azure, Databricks and Fabric: the line is not the engine, it is the size of the job. Most jobs I see are tens of millions of rows dressed up as big data and started on a cluster. For those (file-to-file transforms before landing, validation and reconciliation checks, small dimension builds, CI tests for SQL logic) a single-node engine with a memory limit is the cheaper and more predictable choice. Anything that writes governed Delta tables stays on the platform.

Failure modes I hit, so you don't have to

Engine memory settings are not process memory. Spark's 3g driver heap still produced a 7.51 GiB process. DuckDB's whole process stayed close to its memory_limit: about 3.0 GiB peak at a 3 GB limit, 1.1 GiB at 1 GB. Size containers on measured RSS, not on the setting.
"Streaming" does not mean bounded. Polars streamed the filter, join and group by fine. The window dedup with an over(order_by=...) did not stay bounded, even with 9 GiB.
Close to the cap is a failure that hasn't happened yet. Polars' group by finished in 1 of 3 runs, peaking at 4.49 GiB against a 4.5 GiB cap. Same query, same data, same version. Schedule that and it fails some nights.
Shared hosts lie. One Spark run at 10M was killed because free memory on the whole box fell under my 1 GiB safety floor while Spark itself was at 1.6 GiB. That was the neighbours, not Spark. Gate each run on free memory and CPU, and record both.
Validate outputs, not just timings. Every engine produced the same row counts and checksums where it finished. Without that check, a fast wrong answer looks like a win.

Reproduce it

Everything is public at firfircelik/engine-bench: scripts, raw results, pinned versions and the query plans. Here are the parts that matter. The data generator is seeded (SEED = 20261019) and writes 5M-row chunks:

rng = np.random.default_rng([SEED, chunk_index])
t = pa.table({
    "order_id": np.arange(lo, lo + n, dtype=np.int64),
    "ts": pa.array(start + rng.integers(0, 270 * 24 * 3600, n), pa.timestamp("s")),
    "customer_id": rng.integers(0, rows // 10, n, dtype=np.int32),
    "product_id": rng.integers(0, 50_000, n, dtype=np.int32),
    "store_id": rng.integers(0, 500, n, dtype=np.int16),
    "qty": rng.integers(1, 10, n, dtype=np.int16),
    "price": np.round(rng.uniform(1.0, 500.0, n), 2),
    "status": rng.choice(STATUSES, n, p=[0.55, 0.20, 0.12, 0.08, 0.05]),
})
pq.write_table(t, f"sales_{rows // 1_000_000}m/part-{chunk_index:04d}.parquet",
               row_group_size=1_000_000, compression="snappy")

Spark session, the only non-default settings:

spark = (SparkSession.builder.master("local[8]")
         .config("spark.driver.memory", "3g")
         .config("spark.local.dir", TMP)
         .config("spark.ui.enabled", "false")
         .config("spark.sql.session.timeZone", "UTC")
         .getOrCreate())

And the watchdog, which is the part I would copy into any benchmark of your own. It measures the whole process tree, because JVMs and thread pools do not show up in the parent's RSS:

import psutil, subprocess, time

GIB = 1024 ** 3

def tree_rss(p):
    total = 0
    for q in [p] + p.children(recursive=True):
        try:
            total += q.memory_info().rss
        except psutil.NoSuchProcess:
            pass
    return total

proc = subprocess.Popen(cmd)
p, peak = psutil.Process(proc.pid), 0
while proc.poll() is None:
    rss = tree_rss(p)
    peak = max(peak, rss)
    if rss > 4.5 * GIB:
        for q in p.children(recursive=True) + [p]:
            q.kill()
        break
    time.sleep(0.05)

The takeaway

On one 8 vCPU machine with a 4.5 GiB cap, DuckDB 2.0.0 was the only engine that finished every run of every query at every size, and it did the heavy ones with 1 GB. Polars matched or beat it on the light queries and the group by, and could not finish the window dedup at 150M even with 9 GiB. Local Spark finished almost everything, 5x to 12x slower, and its memory setting did not bound its memory.

If your job fits on one machine, measure it on one machine before you start a cluster for it.

Sources

Announcing DuckDB 2.0.0 LTS "Cinnamon Teal", DuckDB team, October 21, 2026
DuckDB release calendar (2.0.0 LTS on 2026-10-21, 1.5.6 on 2026-09-28)
duckdb 2.0.0 on PyPI, polars 2.0.0 on PyPI, pyspark 4.2.0 on PyPI
DuckDB: How to Tune Workloads (larger-than-memory operators, preserve_insertion_order)
DuckDB: Order Preservation
All timings and memory numbers: my own runs on October 19 (DuckDB 1.5.6, and the 9 GiB Polars and Spark runs) and October 22, 2026 (DuckDB 2.0.0, Polars 2.0.0, PySpark 4.2.0).

FlipPrice: project report, side-hustle pricing copilot for vintage resellers

Firat Celik — Tue, 13 Oct 2026 21:47:16 +0000

FlipPrice

Can Open Models Price Second-Hand Goods Well Enough for Resellers?

Week 2 · Side Hustle Project Report

Repository: https://github.com/firfircelik/flipprice

Can an open-model stack turn a second-hand item into a usable listing, a price estimate, and estimated net proceeds — without relying on paid, per-request AI APIs?

That is the question behind FlipPrice, a lightweight pricing and listing assistant for vintage resellers and second-hand sellers.

The idea is simple: researching comparable sold items takes time, while per-listing AI fees accumulate. FlipPrice explores a local-first alternative that separates listing generation from price estimation.

1. The Problem

Resellers listing 20–50 items a week face two recurring challenges:

Pricing uncertainty: researching comparable sold items takes time, and asking prices do not necessarily reflect realised sale prices.
Accumulating AI costs: at an illustrative $0.05–$0.15 per listing, API costs can add up before a product sells.

Listings per week	Approx. listings per month	Illustrative monthly AI cost
20	87	$4.35–$13.05
50	217	$10.85–$32.55
200	867	$43.35–$130.05

Estimates use 52 weeks divided by 12 months. Actual costs depend on provider pricing and token usage.

The goal: generate a listing draft, estimate a selling price, and calculate expected net proceeds at near-zero marginal model-inference cost.

2. Architecture

FlipPrice uses a lightweight FastAPI service and separates language generation from numerical price estimation.

Listing generation: Gemma 3 + Ollama

Gemma 3 runs locally through Ollama and generates listing copy from a constrained prompt. Output stripping and a brand-first title rule help reduce unwanted preambles and improve consistency.

Pricing: Ridge with an optional TabPFN path

Ridge regression: the implemented fallback pricing path, using target-encoded features.
TabPFN: an optional tabular-model path. The code is implemented, but live verification is pending because the required license token is unavailable.
Training data: a 20-row seed dataset, currently suitable for early development rather than proving market-wide pricing accuracy.

The separation is intentional: a language model can write convincing copy without knowing the correct market price.

API, storage, and validation

FastAPI for the application layer.
Local JSON storage, with MongoDB Atlas as a possible upgrade.
Condition validation through an enum.
Non-negative cost validation.
Photo uploads capped at 2 MB.

These are baseline application safeguards, not a complete production security assessment.

3. Initial Results

Three example items were processed through the prototype using live Gemma 3 on an Apple M2 machine.

Item	Purchase cost	Estimated P50 sale price	Estimated net proceeds
Canon AE-1	$60.00	$286.34	$189.12
Levi's 501	$15.00	$108.85	$79.70
Sony WH-1000XM3	$40.00	$116.82	$61.63

These are recorded prototype outputs, not independently verified market values or guaranteed resale outcomes.

The experiment shows that the workflow can produce item-specific estimates. It does not yet demonstrate pricing accuracy. That requires independent sold-price data, a held-out test set, and explicit error metrics.

4. Latency

Operation	Observed latency
Listing draft generation	6.0 s
Pricing, cold run	5.2 s
Pricing, warm run	9 ms
Offline round trip	1.6 s

These are initial development-environment measurements. The number of repetitions, percentile distribution, and complete test procedure still need to be standardised.

The main observation is the difference between cold and warm pricing. Reusing loaded components and avoiding unnecessary initialisation appear to be important optimisation opportunities.

5. Debugging Notes

Random Forest inverted the expected condition relationship

With only 20 training rows, Random Forest returned:

like_new: 97.51
fair: 103.36

The model assigned a higher estimate to the fair-condition example. This exposed a weakness in the current model and dataset, not proof that Random Forest is inherently unsuitable for resale pricing.

Change: Ridge regression replaced the fallback model, producing a cleaner gradient in the tested examples. A regression test now guards against this specific failure.

TabPFN cold-start overhead

The TabPFN import path added approximately 15 seconds of cold-start overhead without a license token.

Change: The path was gated on token availability, reducing the observed cold-path time from approximately 17 seconds to 5 seconds in the measured setup. Live TabPFN pricing remains unverified.

Unwanted title preambles

Small models sometimes added introductory text to generated titles.

Change: Output stripping and a brand-first title rule improved consistency.

Tests

The automated suite reports 8/8 passing tests. This supports basic functional correctness but does not establish pricing accuracy or production readiness.

6. Unit Economics

Local inference removes the separate hosted-model inference fee for each listing, but it does not make the system completely free to operate.

Costs still include hardware, electricity, maintenance, storage, development, and any external services.

The business case therefore depends on two questions:

Can local inference reduce the marginal cost of preparing listings?
Can the price estimates improve reseller decisions enough to justify adoption?

The architecture supports the first hypothesis. The second remains unvalidated.

7. Limitations

Small dataset: 20 rows cannot adequately represent the diversity of second-hand markets.
No independent accuracy benchmark: MAE, median absolute error, category-level errors, and interval coverage have not yet been reported.
TabPFN is not live-verified: its code path exists, but the required license token is unavailable.
No external users: results are agent-run dogfooding simulations documented in FIRST_USER_TEST.md.
Offline operation needs broader verification: the complete workflow must be tested with network access disabled.
Commercial value is unproven: time saved, realised sale prices, and willingness to pay have not been measured.

8. What Comes Next?

1. Benchmark pricing accuracy

Build a representative dataset of completed sales and evaluate against a held-out test set. Report MAE, median absolute error, suitable percentage-error metrics, and performance by category and condition. Compare against simple baselines such as category medians.

Prevent duplicate and near-duplicate listings from leaking across training and evaluation.

2. Run a real reseller pilot

Recruit a small group of second-hand sellers and measure listing preparation time, draft acceptance, agreement with suggested prices, eventual sale prices, and willingness to pay.

3. Standardise performance tests

Document hardware, model configuration, repetition counts, and latency percentiles. Test offline operation with external network access disabled.

4. Validate TabPFN

If access becomes available, compare TabPFN, Ridge, and simple baselines on the same held-out dataset. Keep the fallback until the alternative is verified.

5. Test willingness to pay

Once pricing quality and time savings are demonstrated, test a simple commercial offer. The important question is whether the value delivered exceeds the cost and uncertainty of the seller's existing process.

9. Conclusion

FlipPrice demonstrates a local-first prototype that combines open-model listing generation with a separate tabular pricing layer.

The initial run produced listing drafts and item-specific price and net-proceeds estimates. Warm pricing latency reached 9 ms in the measured setup, and all eight automated tests passed.

These results support the feasibility of the implementation. They do not yet prove reliable market-price prediction or commercial demand.

The next milestone is evidence: validate estimates against real completed sales and test the workflow with actual resellers.

If FlipPrice can consistently save time and produce estimates sellers trust, its local-first architecture could provide a meaningful cost advantage. If it cannot, low inference cost alone will not make it a viable product.

That is what the next iteration needs to answer.

Project: FlipPrice on GitHub · License: MIT

FlipPrice is an experimental project. Estimates are informational and are not guaranteed resale values.

Python 3.15 Is Not a Speed Release for Data Teams. It Is an Operations Release.

Firat Celik — Mon, 12 Oct 2026 07:12:22 +0000

Python 3.15.0 shipped on October 9, 2026 (release post). The headline number most people will repeat is in the JIT section of What's New: a "7-8% geometric mean performance improvement" over the standard interpreter on x86-64 Linux, and 11-12% on AArch64 macOS.

For a data team, the tempting reading is "upgrade the job images, pipelines get faster." I installed 3.15.0 on a VM, ran a small pandas/pyarrow workload with the JIT off and on, and then spent most of my time on three features that do not touch throughput at all. Those three are the ones that change how you run Python next week.

The thesis

For data teams, Python 3.15 is an operations release, not a speed release. The features that matter are about startup (PEP 810 lazy imports), observability (the Tachyon sampling profiler and frame pointers on by default) and supply chain (PEP 829 .start files replacing import lines in .pth files).

Each of them also has a trap that the release notes do not spell out. I hit all of them on the VM, and they are documented below with the commands to reproduce them.

The antithesis, at full strength

The best counter-case is not silly. It goes like this:

The JIT is real now. What's New lists a new tracing frontend, basic register allocation, more constant propagation and reference count elimination. The pyperformance range is wide, "from roughly 15% slowdown to over 100% speedup", but the geometric mean is positive on both platforms.
Data engineers write more Python-level loops than they admit. Row-wise apply, per-record parsing, UDFs, validation code. That is exactly the code a JIT can see.
Free-threading got a stable ABI. PEP 803 adds abi3t: extensions that target the Stable ABI can now be compiled so they are compatible with free-threaded builds. That is the long road to real multi-core Python for CPU-bound pipeline code.

So the antithesis is: the startup and tooling features are nice, but the throughput work is what will show up on the cloud bill.

Here is what that looked like on the VM.

Setup

Everything below ran on one shared VM: 8 vCPU Intel Xeon, 15 GiB RAM, Debian 13, kernel 6.12. Python 3.15.0 is the uv-managed build (Clang 22, tail-calling interpreter, JIT compiled in but off by default, turned on with PYTHON_JIT=1). Libraries: numpy 2.5.4, pandas 3.0.6, pyarrow 26.0.0, all installed from cp315 wheels three days after the release. The VM is shared with other workloads, so I report medians of repeated fresh processes and repeat whole rounds.

One thing I did not expect: the VM had uv 0.12.24, and uv python install 3.15 installed 3.15.0rc3, not the final release. uv 0.13.0 installed 3.15.0. If your images resolve "3.15" through an older tool, check python -VV before you trust anything. That matters again in the profiler section.

JIT off vs on

Three workloads, timed in-process, 7 fresh processes per setting, two rounds:

Workload	JIT off (median)	JIT on (median)	Change
Pure-Python parsing loop, 1M rows	337.6 / 332.4 ms	309.7 / 304.6 ms	about 8% faster
pandas row-wise `apply`, 100k rows	1176.9 / 1183.6 ms	1126.8 / 1144.3 ms	about 3 to 4% faster
pandas vectorized `groupby().sum()`, 2M rows	63.5 / 76.3 ms	63.3 / 71.4 ms	within noise (single runs ranged 60 to 103 ms)

jit_bench.py (the three workloads)

"""Time three data-team-shaped workloads in-process. Run with PYTHON_JIT=0 or 1."""
import sys
import time

import numpy as np
import pandas as pd


def pure_python_parse(n=1_000_000):
    total = 0
    for i in range(n):
        line = f"{i},{i % 97},{i * 3}"
        a, b, c = line.split(",")
        if int(b) > 50:
            total += int(c)
    return total


def vectorized_groupby(n=2_000_000):
    rng = np.random.default_rng(42)
    df = pd.DataFrame({"k": rng.integers(0, 50_000, n), "v": rng.random(n)})
    return df.groupby("k")["v"].sum().iloc[0]


def rowwise_apply(n=100_000):
    df = pd.DataFrame({"email": [f" User{i}+spam@Example.com " for i in range(n)]})
    return df.apply(lambda r: r["email"].strip().lower().replace("+spam", ""), axis=1).iloc[0]


WORKLOADS = {"pure_python_parse": pure_python_parse,
             "vectorized_groupby": vectorized_groupby,
             "rowwise_apply": rowwise_apply}

fn = WORKLOADS[sys.argv[1]]
fn(10_000)  # warm-up
t0 = time.perf_counter()
fn()
print(f"{(time.perf_counter() - t0) * 1000:.1f}")

Run it as PYTHON_JIT=0 python jit_bench.py pure_python_parse and PYTHON_JIT=1 ..., several times each.

What the numbers support (interpretation): the JIT does what the release notes say for Python-level code. The pure-Python loop landed right in the 7-8% band. But a pipeline whose hot path is already in numpy, pandas or Arrow does not get faster, because the JIT never sees that code. And the row-wise apply is the honest middle: 3 to 4% is real, but the fix for a 1.2 second apply is to vectorize it, not to wait for a JIT.

The free-threading ABI is a packaging milestone. What's New says moving an extension to abi3t "usually requires some non-trivial changes to the source code", and it pays off only when the extensions you depend on ship those wheels and you run a free-threaded build (cp315t). That is worth tracking, not something you collect by bumping a base image.

So the antithesis is partly right: there is a free win for Python-level code. It is small, it is opt-in, and it is experimental. The three features below change your runbooks this week.

1. Lazy imports fix startup, not throughput

The wrong assumption: "lazy imports make Python faster."

The mechanics: lazy import pandas as pd binds pd to a lightweight proxy. The module is loaded the first time the name is used, a step PEP 810 calls reification. If loading fails, the error is raised at first use, and the traceback shows both places. Lazy imports are only allowed at module scope: inside a function, a class body or a try/except they are a SyntaxError. I confirmed the try case on 3.15.0, which rules out the classic try: import orjson as json fallback.

I wrote a toy pipeline CLI, pipectl, with the imports most data tools have at the top: numpy, pandas, pyarrow, requests. It has a version command, a check command that validates a JSON config, and a run command that aggregates 2M rows and writes Parquet. The lazy variant changes only the import lines:

lazy import json
import argparse
import sys

lazy import numpy as np
lazy import pandas as pd
lazy import pyarrow as pa
lazy import pyarrow.parquet as pq
lazy import requests

Each command was run 30 times in a fresh process after warm-up, in three clean rounds:

Command	Eager (median per round)	Lazy (median per round)
`python -c pass` (floor)	9.3 ms
`pipectl version`	351.4 / 381.2 / 344.4 ms	20.0 / 19.6 / 19.9 ms
`pipectl check cfg.json`	352.1 / 342.7 / 352.1 ms	21.2 / 21.1 / 20.8 ms
`pipectl run` (2M rows)	453.2 / 441.0 / 450.8 ms	400.5 / 392.8 / 400.5 ms

-X importtime shows where the eager 350 ms goes: pandas about 180 ms cumulative, numpy about 40 ms, requests about 35 ms (medians of three runs, and importtime adds its own overhead).

The version and check rows are the point of the feature: about 94% less wall time, because those commands never touch pandas. That is more than the "50-70% in practice" that PEP 810 cites for command-line tools, and it should be: my CLI is the extreme case where the cheap commands use none of the heavy imports.

The run row is the one to read twice. Lazy saved about 50 ms, so did it make the work faster? No. I ran a third variant, eager but with the import requests line deleted:

`pipectl run`, 20 runs each	Median
eager	454.9 ms
eager without `import requests`	410.4 ms
lazy	413.3 ms

The whole gain is the import that the run path never uses. Lazy imports remove the imports you don't use. They don't make the ones you do use any cheaper. If your job imports pandas and uses pandas, it pays the full 180 ms either way.

Where this pays off for data teams (interpretation): CLIs, --help in CI, pre-commit hooks, tests that import a package to run one function, notebook kernels, and anything that gets parsed often. The Airflow docs already warn that "import statements also count as top-level code" in Dag files and that expensive imports there hurt the scheduler (Airflow best practices). The lazy keyword looks built for that pattern. I did not test it inside Airflow, and you need an Airflow version that supports 3.15 first.

Trap 1: the global switch broke pandas

3.15 also has an interpreter-wide mode: -X lazy_imports=all or PYTHON_LAZY_IMPORTS=all makes every module-level import in every module potentially lazy, third-party code included. It is tempting to flip it in a job image and collect the startup win without touching code. On 3.15.0 with pandas 3.0.6 and numpy 2.5.4, that failed 5 out of 5 times:

$ PYTHON_LAZY_IMPORTS=all python -c "import pandas as pd; pd.DataFrame({'a': [1]})"
  File ".../pandas/_typing.py", line 401, in <module>
    TakeIndexer: TypeAlias = Sequence[int] | Sequence[np.integer] | npt.NDArray[np.integer]
                                                                    ~~~~~~~~~~~^^^^^^^^^^^^
TypeError: 'lazy_import' object is not subscriptable

The minimal trigger is npt.NDArray[np.integer] with numpy.typing imported under the global mode. numpy alone and requests alone worked. A tiny pure-Python package with the same re-export chain worked too, so this looks specific to how numpy.typing (which defines a module-level __getattr__) interacts with global laziness. I did not chase it further. The lesson does not depend on the root cause: the global mode changes the import semantics of code you did not write, and the lazy keyword in your own entry point does not.

Trap 2: side-effect imports silently stop happening

Plenty of data code imports a module only for what it does at import time: registering a pandas accessor, a SQLAlchemy dialect, a plugin. Linters flag these as unused, and they are the first thing someone will mark lazy:

lazy import pandas as pd
lazy import dq_accessors  # noqa: F401  (registers df.dq at import time)

df = pd.DataFrame({"a": [1, None, 3], "b": [None, None, 1]})
print(df.dq.null_ratio())

AttributeError: 'DataFrame' object has no attribute 'dq'. Did you mean '.eq' instead of '.dq'?

The name dq_accessors is never used, so it is never reified, and the registration never runs. The eager version prints {'a': 0.333, 'b': 0.667}. PEP 810 says this directly: side effects are "deferred until first use", and it recommends explicit initialization functions instead of import-time registration.

Trap 3: the PEP's kill switch does not exist in 3.15.0

PEP 810 describes a third mode, "none", to force every import eager. The shipped 3.15.0 accepts only normal and all, which matches the sys docs. Copying the PEP is not harmless:

$ PYTHON_LAZY_IMPORTS=none python -c pass
Fatal Python error: config_init_lazy_imports: PYTHON_LAZY_IMPORTS: invalid value; expected 'all' or 'normal'

Exit code 1, before any of your code runs. sys.set_lazy_imports("none") raises ValueError. The PEP page itself now says it is "a historical document". If you want an emergency "everything eager" switch, a filter that returns False does it (verified: json loads at the lazy import line):

import sys

sys.set_lazy_imports_filter(lambda importer, name, fromlist: False)

Runbook: lazy imports

Measure first: python -X importtime -m yourtool --help 2> imports.txt, then sort by the cumulative column.
Mark only the heavy imports in your own entry point lazy. Leave third-party internals alone.
Keep side-effect imports eager, or replace them with an explicit register() call.
Do not set PYTHON_LAZY_IMPORTS=all in a job image unless the full test suite passes under it.
If you must support 3.14 too, use __lazy_modules__ = ["pandas", ...] with normal import statements. 3.15 treats those imports as lazy, and older versions just see a module variable and import eagerly (checked on 3.15.0 and 3.13.5 with json).

2. Tachyon: attach to the stuck job by PID, and read the error rate

The wrong assumption: "a sampling profiler is free and always right."

The mechanics: profiling.sampling (PEP 799, named Tachyon) reads the target process's memory from outside. It can run a script, attach to a PID for a time window, or dump a single traceback-style snapshot of a live process. Output formats include pstats tables, collapsed stacks, flame graphs, Firefox Profiler and a live TUI. The docs list the constraints that matter in production:

Same Python minor version for profiler and target. If either side is a pre-release, both must be the exact same version.
Permissions. On Linux, root, CAP_SYS_PTRACE, or a Yama ptrace_scope of 0. The docs note the default ptrace_scope of 1 restricts ptrace to parent processes.
Non-blocking mode (the default) never stops the target, but "can occasionally produce incomplete or inconsistent stack traces". --blocking pauses the target for each sample.

I wrote the classic slow nightly job: a pandas row-wise apply with a per-row Python function, looping forever, and pretended I did not know why it was slow.

slow_job.py

"""A 'stuck' nightly job: a row-wise pandas apply that nobody profiled."""
import time

import numpy as np
import pandas as pd


def normalize_email(row):
    return row["email"].strip().lower().replace("+spam", "")


def enrich(df):
    df["email_norm"] = df.apply(normalize_email, axis=1)
    return df


def main():
    rng = np.random.default_rng(0)
    n = 200_000
    df = pd.DataFrame({
        "id": np.arange(n),
        "email": [f" User{i}+spam@Example.com " for i in rng.integers(0, 10**6, n)],
    })
    while True:
        enrich(df.copy())
        time.sleep(0.01)


if __name__ == "__main__":
    main()

The first thing you want during an incident is "what is it doing right now":

$ python -m profiling.sampling dump --blocking 440208
Stack dump for PID 440208, thread 440208 (main thread, has GIL; most recent call last):
  File "slow_job.py", line 31, in <module>
    main()
  File "slow_job.py", line 26, in main
    enrich(df.copy())
  File "slow_job.py", line 14, in enrich
    df["email_norm"] = df.apply(normalize_email, axis=1)
  ...
  File "slow_job.py", line 10, in normalize_email
    return row["email"].strip().lower().replace("+spam", "")
  File ".venv/lib/python3.15/site-packages/pandas/core/series.py", line 959, in Series.__getitem__
    return self._get_value(key)

No restart, no code change, and the culprit is on screen. On this kernel there is no Yama module (/proc/sys/kernel/yama/ptrace_scope does not exist), so attaching as the same unprivileged user just worked. On a host with the default ptrace_scope of 1, plan for root or CAP_SYS_PTRACE before the incident, not during it.

Then the part I did not expect. Without --blocking, dump failed 8 times out of 20 with RuntimeError: Failed to parse initial frame in chain. With --blocking it succeeded 20 out of 20. For attach, the profiler prints an Error rate line, the percentage of samples it could not read. I measured it on a fixed workload (median of six apply rounds per run, 1 kHz default rate, three repetitions):

Mode	Target round time	Error rate
No profiler	1170 / 1150 / 1186 ms
`attach` (non-blocking, default)	1179 / 1180 / 1150 ms	67.8% / 62.0% / 63.7%
`attach --blocking`	1606 / 1760 / 1657 ms	1.5% / 1.6% / 1.3%

Both halves of the docs are true at once. Non-blocking mode had no overhead I could measure on the target. And on this VM, 62 to 68% of its samples were unreadable in this test (46 to 50% in two earlier 5-second runs against slow_job.py, 38% against a pure-Python recursive function). Blocking mode was clean, and it made the job 37 to 53% slower while attached.

Interpretation: treat a non-blocking profile like a survey with a response rate. If 60% of samples failed, the remaining 40% may still point at the right function (here the row-wise apply dominated either way), but check the error rate before you trust percentages. Use dump --blocking for a one-shot snapshot, where a single pause costs nothing, and keep --blocking profiles short on anything user-facing.

And the version rule bit me right away. A job started on the 3.15.0rc3 that the older uv had installed could not be inspected from 3.15.0:

Can't attach to a pre-release Python interpreter from a process running a different Python version

From rc3 to rc3 it worked. Pin the profiler to the job image's exact interpreter, which in practice means running it from inside that image.

The quieter observability change is PEP 831: CPython is now built with frame pointers by default. What's New says this makes native stack unwinding "faster and more reliable" for system profilers, debuggers, crash analysis tools and eBPF-based tools. The uv build I used has -fno-omit-frame-pointer in its CFLAGS. The catch, in the release notes' words: "A single native component built without frame pointers can break stack unwinding for the whole Python process." Your own Rust or C extensions need the same flags if you want mixed native and Python profiles.

Runbook: profiling a stuck job

python -m profiling.sampling dump --blocking PID first. One snapshot, one short pause.
Then python -m profiling.sampling attach -d 30 --flamegraph -o job.html PID, and read Error rate before the flame graph.
Run the profiler with the job's exact interpreter. A pre-release on either side means exact versions only.
Decide now who is allowed CAP_SYS_PTRACE on worker hosts, and where.

3. `.pth` import lines become `.start` files: auditable, not safe

The wrong assumption: "a package can only run code when I import it."

The mechanics: at startup the site module reads every .pth file in site-packages. Lines are paths to add to sys.path, except lines that start with import, which are executed with exec(). Every Python process in that environment runs them: python -c pass, pip, pytest, the notebook kernel, the scheduler. PEP 829 calls this "a broad attack surface".

This is not theoretical. The advisory for the compromised litellm 1.82.7 and 1.82.8 releases (PYSEC-2026-2, published March 24, 2026) describes credential-harvesting malware, and its references point to a litellm_init.pth file inside the 1.82.8 package.

And it is not only malware. setuptools 84.0.0, installed fresh on the VM, still ships distutils-precedence.pth, whose single line starts with import os; ....

What PEP 829 and the site docs change in 3.15:

A new name.start file lists entry points as pkg.mod:callable. They run after all .pth path lines are applied.
If name.start exists next to name.pth, the import lines in that .pth are ignored.
Without a matching .start, import lines still run in 3.15, and the deprecation message appears only with -v. The docs say they will be "silently ignored in Python 3.18 and 3.19", with a warning in 3.20.
Errors on one line no longer abort the rest of the file. A failing entry point prints a traceback and startup continues.

On the VM, with a demo zz_demo.pth containing a path line and an import line:

$ python -c "import sys; print('/tmp/extra_path_dir' in sys.path)"
[zz_demo.pth] import line executed at startup
True

python -v shows the new deprecation notice for both the demo file and setuptools' file:

import lines in .../site-packages/distutils-precedence.pth are deprecated, use entry points in a .../site-packages/distutils-precedence.start file instead.

After adding zz_demo.start with one line, zz_demo_boot:init, the import line is suppressed and the entry point runs:

import lines in .../zz_demo.pth are suppressed due to matching .../zz_demo.start file.
Executing entry point: zz_demo_boot:init from .../zz_demo.start
[zz_demo.start] entry point zz_demo_boot:init called

Here is the part to be precise about. A .start file still runs code at every interpreter start. A malicious wheel can ship one just as easily as a .pth. The PEP says so: "The overall pre-start code execution attack surface is not eliminated by this PEP." What 3.15 gives you is a narrower, auditable shape: startup code is a named callable in an importable module, it goes through the import system so audit hooks (PEP 578) can see it, and "you can tell by listing the files in the site-dir, exactly where arbitrary code execution occurs."

That listing is the useful part, and you can do it today on any Python version:

startup_audit.py (runs on Python 3.9+)

"""List everything that can run code at interpreter start in this environment.

Run it with the interpreter you want to audit:
    .venv/bin/python startup_audit.py
"""
import site
import sys
from pathlib import Path

dirs = list(dict.fromkeys(site.getsitepackages() + [site.getusersitepackages()]))
found = 0
for d in map(Path, dirs):
    if not d.is_dir():
        continue
    for f in sorted(d.glob("*.pth")):
        lines = f.read_text(encoding="utf-8-sig", errors="replace").splitlines()
        code = [l for l in lines if l.startswith(("import ", "import\t"))]
        if code:
            found += 1
            shadow = f.with_suffix(".start").exists()
            note = "skipped on 3.15+ (matching .start), runs on older Pythons" if shadow else "RUNS on every start"
            print(f"{f}  [{note}]")
            for l in code:
                print(f"    {l[:110]}")
    for f in sorted(d.glob("*.start")):
        found += 1
        print(f"{f}  [.start entry points]")
        for l in f.read_text(encoding="utf-8-sig").splitlines():
            if l.strip() and not l.lstrip().startswith("#"):
                print(f"    {l.strip()}")
    for name in ("sitecustomize.py", "usercustomize.py"):
        if (d / name).exists():
            found += 1
            print(f"{d / name}  [runs on every start]")
print(f"{sys.version.split()[0]}: {found} startup code hook(s) found")

Output in the demo environment (with setuptools installed):

.../site-packages/distutils-precedence.pth  [RUNS on every start]
    import os; var = 'SETUPTOOLS_USE_DISTUTILS'; enabled = os.environ.get(var, 'local') == 'local'; enabled and __
.../site-packages/zz_demo.pth  [skipped on 3.15+ (matching .start), runs on older Pythons]
    import sys; sys.stderr.write("[zz_demo.pth] import line executed at startup\n")
.../site-packages/zz_demo.start  [.start entry points]
    zz_demo_boot:init
3.15.0: 3 startup code hook(s) found

Most .pth files are harmless. An editable install of a src-layout package with setuptools wrote a path-only .pth on the VM, with no code at all. The ones that execute code are few enough to put on an allowlist.

python -S disables both mechanisms (I checked), but it also skips site-packages entirely, so it is a debugging tool, not a production setting.

Runbook: startup code

Run the audit in your image build and fail the build on any startup hook that is not on an allowlist.
If you maintain a package with a .pth import line, move the code into a zero-argument function and add a matching .start file. Keep the .pth line as import pkg.mod; pkg.mod.callable() while you still support 3.14.
Treat a new .start file in a dependency diff like a new postinstall script in npm: a reason to read the release.

What to do this week

If you run...	Do this	Skip this
Python CLIs, CI tooling, notebook kernels	Measure with `-X importtime`, add `lazy` to your entry point's heavy imports	`PYTHON_LAZY_IMPORTS=all` in images
Long batch jobs on VMs or Kubernetes	Install the exact job interpreter in the image, rehearse `dump --blocking` and `attach` on a staging job, sort out `CAP_SYS_PTRACE`	Trusting a profile without reading `Error rate`
Anything that installs from PyPI	Add the startup audit to the image build, review new `.pth` and `.start` files in dependency updates	Assuming `.start` files make startup code safe
Python-heavy loops or row-wise UDFs	Try `PYTHON_JIT=1` in a benchmark, expect single digits	Counting on the JIT to fix a row-wise `apply`

The JIT will show up in your benchmarks as a few percent. Lazy imports, Tachyon and .start files will show up in your incident timeline: the CLI that starts in 20 ms instead of 350, the stuck job you can inspect without a restart, and the list of files that run code before your main() does. Upgrade for those, and measure the rest.

Kafka Just Learned to Be a Queue. Postgres Has Been One All Along.

Firat Celik — Mon, 05 Oct 2026 07:17:38 +0000

On February 17, 2026, Apache Kafka 4.2 shipped with this line in the release announcement: "Kafka Queues (Share Groups) is now production-ready."

That sentence quietly reopens one of backend engineering's oldest arguments. For years, "should my job queue be Postgres or Kafka?" had a lazy answer from both camps. The Kafka camp said Kafka scales. The Postgres camp said Kafka is not a queue: your consumer count is capped by your partition count, there is no per-message ack, and one poison message blocks everything behind it in its partition.

The second argument just lost most of its teeth. So this post rebuilds the comparison from primary sources and two small demos you can run in a few minutes: a Postgres SKIP LOCKED queue in Python, and a Kafka share group in Go.

The thesis

If a job is born inside a database transaction, Postgres is still the better queue in 2026.

Kafka share groups are real, well designed, and production-ready on the broker. But they answer a different question: "I already run Kafka, and now I also need queue semantics." They do not answer "I need a queue." Those two questions have different winners, and a team that does not already run Kafka would be adopting a whole platform to get one feature.

The antithesis, at full strength

Here is the best case against that thesis. It is a good one.

KIP-932 adds a new kind of group, the share group, and with it:

Consumers beyond partition count. "The number of consumers in a share group can exceed the number of partitions in a topic." The old trick of over-partitioning just to get parallelism goes away.
Per-record acknowledgement. A consumer can accept, release (retry), or reject (unprocessable) each record. Kafka 4.2 added a fourth option, RENEW, to extend the lock for slow work (KIP-1222).
Delivery counting with a poison-message limit. After the configured number of attempts (default 5, per the 4.3 broker configs), a record is archived instead of redelivered forever.
No queue depth limit and replay. In the KIP's own words: "queues done in a Kafka way, with no maximum queue depth and the ability to reset to a specific time for point-in-time recovery."
Many readers, one log. Several share groups can consume the same topic independently, next to ordinary consumer groups.

And the Postgres side has a well-known way to rot: MVCC. In the demo below, a single idle session holding an old snapshot made the median claim latency go from about 0.27 ms to about 15 ms on the same queue. That is not a theoretical concern. It is one forgotten BEGIN away.

So the antithesis is: Kafka now does queues properly, scales past one box, keeps history, and does not degrade because someone left a psql session open. The answer to that lives in the failure modes on both sides.

What actually shipped (facts, as of October 5, 2026)

	Kafka share groups (KIP-932)	Postgres `FOR UPDATE SKIP LOCKED` queue
Unit of work	Record in a topic partition	Row in a table
Claim	Broker-side acquisition lock, default 30 s (`share.record.lock.duration.ms`)	Row lock held by your transaction
Ack	ACCEPT / RELEASE / REJECT / RENEW	`COMMIT` (and delete or update the row)
Crash behavior	Lock expires (or the share session closes), record becomes available again	Transaction aborts, lock is released, row is visible again
Delivery guarantee	At-least-once (KIP-932)	At-least-once for external side effects; atomic with any writes in the same database
Ordering	Not guaranteed across batches, especially on redelivery	Whatever your `ORDER BY` gives you, best effort under concurrency
Poison messages	Delivery count limit, default 5	You write it (an `attempts` column)
Dead-letter queue	KIP-1191 listed as complete for 4.4.0, which is not released yet	You write it (a table)

Client support matters more than broker support, because your code talks to a client:

Java: KafkaShareConsumer is the reference client (javadoc).
Go: franz-go added "full support for KIP-932 share groups" in v1.21.0 (changelog). The demo below uses v1.22.1.
Python and C: librdkafka and confluent-kafka-python 2.15.0 (June 30, 2026) ship the share consumer as Preview, with an explicit note that it "should not be used in production environments" (confluent-kafka-python v2.15.0, librdkafka v2.15.0). Confluent's own GA post says non-Java client support is "targeted for the second half of 2026" (Confluent blog).

If your workers are Python on the mainstream Confluent client, Kafka queues are explicitly not for production today. That alone settles the question for a lot of data teams.

The Postgres queue is one SQL statement

Here is the entire claim step:

WITH next AS (
    SELECT id FROM jobs
    ORDER BY id
    LIMIT 50
    FOR UPDATE SKIP LOCKED
)
DELETE FROM jobs j USING next
WHERE j.id = next.id
RETURNING j.id, j.payload;

Run it inside a transaction, do the work, write the result, COMMIT. The commit is the ack. If the worker dies before committing, Postgres rolls the transaction back, the rows were never deleted, and another worker picks them up. That is the same contract as a share group's acquisition lock, except the "lock timeout" is "your connection died."

The Postgres docs are blunt about what SKIP LOCKED is for: "Skipping locked rows provides an inconsistent view of the data, so this is not suitable for general purpose work, but can be used to avoid lock contention with multiple consumers accessing a queue-like table" (SELECT docs).

Demo A: does SKIP LOCKED actually matter?

Eight worker threads drain 20,000 jobs in batches of 50. Each worker writes a row into a done table in the same transaction as the dequeue. The script asserts zero duplicates and zero lost jobs after every run. Each script execution does three alternating runs per cell and reports the median; the table shows the range of those medians across four executions:

Simulated work per batch (lock held)	`FOR UPDATE SKIP LOCKED`	plain `FOR UPDATE`
0 ms	14,400 to 17,600 jobs/s	13,200 to 32,800 jobs/s (about 32,000 in 3 of 4)
5 ms	14,000 to 17,600 jobs/s	7,100 to 7,200 jobs/s

Duplicates: 0. Lost: 0. In every run, for both variants.

The first row is the surprise: with near-zero lock hold time, plain FOR UPDATE was usually about twice as fast. A plausible explanation (interpretation, not measured): each SKIP LOCKED claim steps over the rows the other seven workers hold, while a blocked worker waits very briefly and gets its rows. The second row is the one that matters in production. Once workers hold the claim while doing real work, the blocking version collapses to about 7,000 jobs/s because everyone queues behind the head of the table, and SKIP LOCKED comes out 2 to 2.5 times ahead. Both variants lost nothing. SKIP LOCKED is about throughput under real work, not correctness.

One subtle thing the script surfaced: with SKIP LOCKED, a worker can get zero rows while jobs still exist, because everything left is claimed by someone else. It happened between 2 and 36 times per run. Treat an empty claim as "back off and poll again," never as "the queue is empty, exit."

Demo B: kill a worker mid-batch

A session claims 10 of 100 jobs and never commits. While it holds them, other sessions can claim 90. After pg_terminate_backend() on that session (the database-side equivalent of kill -9 on the worker), all 100 are claimable again. No lease column, no reaper job, as long as jobs are short (more on that below).

The real advantage: the job and the data commit together

This is the part Kafka cannot match without extra machinery. If your API inserts a user and enqueues "send welcome email" in the same transaction, the job exists if and only if the user exists. River, a Go job queue backed by Postgres, makes this its core argument (River docs): with a separate queue store you either enqueue after commit and lose jobs on a crash, or enqueue before commit and have workers race a row they cannot see yet.

The mainstream already voted on this. Solid Queue "is configured by default in new Rails 8 applications" and "leverages the FOR UPDATE SKIP LOCKED clause" (Solid Queue README).

Where the Postgres queue dies: one old snapshot

A queue table is the worst-case MVCC workload: every row is inserted once and deleted once. Postgres does not remove a deleted row immediately; the routine vacuuming docs explain that "the row version must not be deleted while it is still potentially visible to other transactions." If anything holds an old snapshot, vacuum cannot clean, and the dead rows pile up right where your claim query starts scanning.

Demo C: churn 200,000 jobs, then measure

The script pushes 200,000 jobs through the queue, enqueues 5,000 fresh ones, runs VACUUM (VERBOSE), and times 200 claims of 10 jobs each. Then it repeats the whole thing with one extra session sitting idle in transaction in REPEATABLE READ, which is exactly what a forgotten report query or a paused migration looks like.

	Median claim latency	`VACUUM VERBOSE` says
No old snapshot	0.269 ms / 0.272 ms	40,293 / 37,680 removed, 0 dead but not yet removable
One idle `REPEATABLE READ` session	15.754 ms / 14.961 ms	0 removed, 200,000 dead but not yet removable

(Two executions of the script, shown as first / second. In the no-snapshot case autovacuum had already cleaned most of the dead rows before the manual VACUUM ran.)

Same table, same query, same hardware. Roughly 55 to 58 times slower per claim, and the dead rows keep accumulating with every job processed while that session lives. Nothing errors. Your queue just gets slower until someone notices.

Things that hold old snapshots in real systems, all documented:

Long-running or idle-in-transaction sessions (vacuum docs tell you to find them via pg_stat_activity and backend_xmin).
Old replication slots (same page).
A hot standby with hot_standby_feedback on, which "can cause database bloat on the primary for some workloads" (replication config docs).

Mitigations that actually work:

Set idle_in_transaction_session_timeout for application and human roles, and transaction_timeout (added in PostgreSQL 17) where it is safe. Both are documented in client connection defaults, including the warning to be careful with connection poolers.
Alert on "dead but not yet removable" and on the age of the oldest backend_xmin, not just on queue depth.
Keep analytics off the primary. An extra test on the same box: an old snapshot held in a different database on the same cluster did not block cleanup of the queue table (50,000 removed), while the same snapshot in the same database blocked all of it (50,000 dead but not yet removable). That is a real escape hatch, but it has a price: a queue in its own database can no longer commit atomically with your application tables, which was the main reason to use Postgres in the first place.

The trap: your worker can be the long transaction

The claim pattern above holds the transaction open while the job runs. For a 50 ms job that is fine. For a 5 minute job, the worker itself is the old transaction. A quick check on the same box: a worker sitting idle in transaction after its DELETE ... RETURNING (plain READ COMMITTED, nothing exotic) left 50,000 freshly deleted rows in another table of the same database "dead but not yet removable." So long jobs need a lease instead: claim and commit in a short transaction, work outside any transaction, then ack only if you still own the lease.

-- claim: short transaction, commit right away
UPDATE jobs
SET locked_until = now() + interval '5 minutes',
    lease_token  = gen_random_uuid(),
    attempts     = attempts + 1
WHERE id IN (
    SELECT id FROM jobs
    WHERE locked_until IS NULL OR locked_until < now()
    ORDER BY id
    LIMIT 10
    FOR UPDATE SKIP LOCKED
)
RETURNING id, payload, lease_token, attempts;

-- ack: only the current lease holder can delete the job
DELETE FROM jobs WHERE id = $1 AND lease_token = $2;

Tested on the box: when a lease expired and a second worker re-claimed the job, the first worker's late ack deleted 0 rows and the second worker's ack deleted 1. Notice what you just built: an acquisition lock with a timeout and a delivery count. That is a share group, in SQL. You also gave up the "job commits with the data" property for the work itself, so this path needs idempotent jobs, exactly like Kafka.

A note on LISTEN/NOTIFY

If you use LISTEN/NOTIFY to wake workers, know the documented edges (NOTIFY docs): notifications are delivered only when the sending transaction commits, payloads must be shorter than 8000 bytes by default, and the notification queue (8 GB in a standard install) cannot be cleaned while a listening session sits in a long transaction. If that queue fills, "transactions calling NOTIFY will fail at commit." Use NOTIFY as a doorbell, never as the queue itself. PostgreSQL 19, with GA scheduled for October 29, 2026 (release dates), lists an improvement to "only wake up backends that are listening to specified notifications" in its draft release notes.

Where Kafka share groups bite

These are less about slow decay and more about semantics you have to design around.

1. The lock timeout is a duplicate-work timer. The default acquisition lock is 30 seconds. If your job takes 45, the broker hands the record to another consumer while the first one is still working. Use RENEW for long jobs, and make processing idempotent anyway. The franz-go kgo docs put it plainly: "Share group consumers are at-least-once regardless: any TCP hiccup can cause re-delivery, so user processing must be idempotent."

2. Ordering is gone. KIP-932: "The records in a share-partition can be delivered out of order to a consumer, in particular when redeliveries occur." If you need per-key ordering, you want a classic consumer group, not a share group.

3. A new share group starts at the end. share.auto.offset.reset defaults to latest (group configs). Create a share group on a topic that already has a backlog and, unless you set earliest, your workers will skip all of it.

4. Size-based retention can delete work you have not done. From the KIP: retention based on size "does potentially silently remove records that were eligible for delivery." A Postgres table does not delete your unprocessed jobs because the disk got full.

5. The unit of sharing is the record batch. The KIP says it directly: "the natural unit of sharing is the record batch." In the Go demo below, four consumers share one partition. With default fetch settings, one consumer got 949 of 1,000 jobs and two got nothing. With ShareMaxRecordsStrict() (the client side of the strict mode added by KIP-1206 in 4.2), the split was 246 / 251 / 252 / 251. Caveat: this ran against kfake, franz-go's in-process fake broker, not a real cluster, so treat the exact split as illustrative. The behavior matches what the KIP describes.

6. No atomicity with your database. Writing a row to Postgres and producing to Kafka are two systems. The standard fix is the transactional outbox: write the event to an outbox table in the same transaction, then let CDC (for example the Debezium outbox event router) publish it. Notice what that means: the reliable way to feed a Kafka queue from an OLTP app starts with a Postgres table.

7. In-flight work is capped per partition. group.share.partition.max.record.locks defaults to 2000 per share-partition (4.3 broker configs). Slow consumers holding locks can stall fetching on that partition until locks are acked or expire. And the delivery count "cannot be relied upon to be precise in all situations" (KIP-932), so do not build billing logic on it.

The Go side: a share group on one partition

The core loop, using franz-go v1.22.1:

cl, err := kgo.NewClient(
    kgo.SeedBrokers(addrs...),
    kgo.ConsumeTopics("jobs"),
    kgo.ShareGroup("email-workers"),
    kgo.ShareMaxRecords(25),
    kgo.ShareMaxRecordsStrict(), // without this one member may get a whole record batch
)
if err != nil {
    panic(err)
}
defer cl.Close()

for {
    pollCtx, cancel := context.WithTimeout(ctx, time.Second)
    fetches := cl.PollFetches(pollCtx)
    cancel()
    fetches.EachRecord(func(r *kgo.Record) {
        if err := sendEmail(r.Value); err != nil {
            r.Ack(kgo.AckRelease) // transient: try again, delivery count goes up
            return
        }
        r.Ack(kgo.AckAccept)
    })
}

In the full demo, every job whose number ends in 0 is released on its first delivery to simulate a transient failure. Three runs in a row: 1,000 of 1,000 jobs accepted, 100 on a second delivery, 0 accepted twice, four consumers on one partition. The headline feature works.

The decision rule

Put it on one line first:

Tasks that belong to a transaction go in the database that owns the transaction. Facts that many systems read go in a log. Share groups let the log also hand out tasks, which is great once you already have the log.

Then the checklist.

Choose a Postgres queue when:

The job is created in the same transaction as the business data it depends on.
Your workers are in Python, or any language without a production share consumer yet.
Your measured peak fits your Postgres with your payloads. Demo A did 14,000 to 17,600 tiny jobs/s with SKIP LOCKED on 8 vCPUs; River claims "tens of thousands of jobs per second." Measure, do not inherit someone's number.
Jobs are short enough to run inside the claim transaction, or you use leases for the long ones.
You can enforce idle_in_transaction_session_timeout, keep analytics off the primary, and alert on dead-but-not-removable rows.

Choose Kafka share groups when:

Kafka is already the system of record for these events, and the queue is one more reader.
Several independent consumers need the same stream, or you need replay and reset-to-timestamp.
Your workers are on Java or Go (franz-go), processing is idempotent, and ordering does not matter.
You are fine designing around lock timeouts, retention, and share.auto.offset.reset.

Use both when the job starts in your database but the work fans out to other teams: outbox table in Postgres, CDC into Kafka, share groups downstream.

Red flags that you picked wrong: you are adding an outbox just to feed a Kafka queue that only one service reads (you wanted Postgres), or your Postgres queue needs a second database to stay fast (you are halfway to wanting a log).

Run it yourself

Hardware and versions for every number above: an 8 vCPU, 15 GB RAM Linux VM (kernel 6.12, Debian 13), PostgreSQL 17.11 with default settings except max_connections=200, client and server on the same machine over a Unix socket, Python 3.13.5 with psycopg 3.3.6, Go 1.26.0 with franz-go v1.22.1 and kfake. These are toy payloads on one box. They show mechanisms and ratios, not capacity for your system.

pg_queue_demo.py (Python, needs a throwaway Postgres)

"""
Postgres-as-a-queue demo: SKIP LOCKED vs blocking FOR UPDATE, crash recovery, and the MVCC failure mode.
Needs a THROWAWAY Postgres (it drops tables named jobs and done).
  pip install "psycopg[binary]"
  DSN="host=/tmp port=54329 user=postgres dbname=postgres" python pg_queue_demo.py
"""
import os, time, threading, statistics
import psycopg

DSN = os.environ.get("DSN", "host=/tmp port=54329 user=postgres dbname=postgres")

CLAIM_SKIP = """
WITH next AS (
    SELECT id FROM jobs
    ORDER BY id
    LIMIT %(n)s
    FOR UPDATE SKIP LOCKED
)
DELETE FROM jobs j USING next
WHERE j.id = next.id
RETURNING j.id, j.payload
"""
CLAIM_BLOCKING = CLAIM_SKIP.replace("FOR UPDATE SKIP LOCKED", "FOR UPDATE")


def setup(conn):
    conn.execute("DROP TABLE IF EXISTS jobs, done")
    conn.execute("CREATE TABLE jobs (id bigserial PRIMARY KEY, payload jsonb NOT NULL)")
    # 'done' stands in for a side effect that lives in the SAME database as the queue
    conn.execute("CREATE TABLE done (job_id bigint NOT NULL, worker int NOT NULL)")
    conn.commit()


def enqueue(conn, n):
    conn.execute("INSERT INTO jobs (payload) SELECT jsonb_build_object('n', g) "
                 "FROM generate_series(1, %s) g", (n,))
    conn.commit()


def worker(wid, claim_sql, batch, stats, empties, work_ms):
    with psycopg.connect(DSN) as conn:
        while True:
            with conn.transaction():
                rows = conn.execute(claim_sql, {"n": batch}).fetchall()
                if rows:
                    if work_ms:
                        time.sleep(work_ms / 1000)  # "work" done while holding the claim
                    with conn.cursor() as cur:      # result commits atomically with the dequeue
                        cur.executemany("INSERT INTO done (job_id, worker) VALUES (%s, %s)",
                                        [(r[0], wid) for r in rows])
            if rows:
                stats[wid] += len(rows)
                continue
            # An empty claim does not always mean an empty queue (see the blocking variant).
            left = conn.execute("SELECT count(*) FROM jobs").fetchone()[0]
            conn.commit()
            if left == 0:
                return
            empties[wid] += 1


def run_workers(claim_sql, workers, batch, work_ms=0.0):
    stats, empties = [0] * workers, [0] * workers
    ts = [threading.Thread(target=worker, args=(i, claim_sql, batch, stats, empties, work_ms))
          for i in range(workers)]
    t0 = time.perf_counter()
    for t in ts:
        t.start()
    for t in ts:
        t.join()
    return time.perf_counter() - t0, sum(empties)


def experiment_throughput(n=20_000, workers=8, batch=50, repeats=3):
    for work_ms in (0.0, 5.0):
        print(f"\n== A. {workers} workers drain {n} jobs, batch={batch}, "
              f"simulated work per batch={work_ms} ms, {repeats} alternating runs")
        results = {"FOR UPDATE SKIP LOCKED": [], "FOR UPDATE (blocking)": []}
        for _ in range(repeats):
            for label, sql in (("FOR UPDATE SKIP LOCKED", CLAIM_SKIP),
                               ("FOR UPDATE (blocking)", CLAIM_BLOCKING)):
                with psycopg.connect(DSN) as conn:
                    setup(conn)
                    enqueue(conn, n)
                secs, empties = run_workers(sql, workers, batch, work_ms)
                with psycopg.connect(DSN) as conn:
                    total, distinct = conn.execute(
                        "SELECT count(*), count(DISTINCT job_id) FROM done").fetchone()
                    left = conn.execute("SELECT count(*) FROM jobs").fetchone()[0]
                assert total == distinct == n and left == 0, (total, distinct, left)
                results[label].append((n / secs, empties))
        for label, rs in results.items():
            print(f"{label:24s} median {statistics.median(r[0] for r in rs):7.0f} jobs/s | "
                  f"empty claims while work remained, per run: {[r[1] for r in rs]} | "
                  f"duplicates 0, lost 0")


def experiment_crash(n=100):
    print("\n== B. a worker dies mid-transaction")
    with psycopg.connect(DSN) as conn:
        setup(conn)
        enqueue(conn, n)
    victim = psycopg.connect(DSN)
    victim.execute(CLAIM_SKIP, {"n": 10}).fetchall()  # claimed 10 jobs, never commits
    with psycopg.connect(DSN, autocommit=True) as admin:
        q = "SELECT count(*) FROM (SELECT id FROM jobs FOR UPDATE SKIP LOCKED) s"
        print(f"victim holds its batch: claimable by others = {admin.execute(q).fetchone()[0]}")
        admin.execute("SELECT pg_terminate_backend(%s)", (victim.info.backend_pid,))
        time.sleep(0.5)
        print(f"victim session killed:  claimable by others = {admin.execute(q).fetchone()[0]}")
    try:
        victim.close()
    except Exception:
        pass


def median_claim_ms(rounds=200):
    lat = []
    with psycopg.connect(DSN) as conn:
        for _ in range(rounds):
            t0 = time.perf_counter()
            with conn.transaction():
                conn.execute(CLAIM_SKIP, {"n": 10}).fetchall()
            lat.append((time.perf_counter() - t0) * 1000)
    return statistics.median(lat)


def experiment_mvcc(churn=200_000):
    print(f"\n== C. churn {churn} jobs through the queue, then time claims, "
          f"with and without one old snapshot held open")
    for hold in (False, True):
        with psycopg.connect(DSN) as conn:
            setup(conn)
        holder = None
        if hold:
            # stands in for a long report query, a stuck migration, or an idle-in-transaction session
            holder = psycopg.connect(DSN, autocommit=True)
            holder.execute("BEGIN ISOLATION LEVEL REPEATABLE READ")
            holder.execute("SELECT 1").fetchone()  # first statement takes the snapshot
            with psycopg.connect(DSN, autocommit=True) as c2:
                st, xmin = c2.execute("SELECT state, backend_xmin FROM pg_stat_activity "
                                      "WHERE pid = %s", (holder.info.backend_pid,)).fetchone()
            print(f"   holder session: state={st!r}, backend_xmin={xmin}")
        with psycopg.connect(DSN) as conn:
            enqueue(conn, churn)
        run_workers(CLAIM_SKIP, 8, 100)            # drain everything, leaving dead rows behind
        with psycopg.connect(DSN) as conn:
            enqueue(conn, 5_000)                   # fresh work arrives behind the dead rows
        notes = []
        with psycopg.connect(DSN, autocommit=True) as conn:
            conn.add_notice_handler(lambda d: notes.append(d.message_primary or ""))
            conn.execute("VACUUM (VERBOSE) jobs")  # give vacuum every chance
        tuples_line = next((l.strip() for n in notes for l in n.splitlines() if "tuples:" in l), "")
        med = median_claim_ms()
        with psycopg.connect(DSN) as conn:
            size = conn.execute("SELECT pg_size_pretty(pg_total_relation_size('jobs'))").fetchone()[0]
        print(f"old snapshot held={str(hold):5s} median claim latency={med:8.3f} ms  "
              f"jobs table+index={size}\n   VACUUM VERBOSE: {tuples_line}")
        if holder:
            holder.execute("ROLLBACK")
            holder.close()


if __name__ == "__main__":
    with psycopg.connect(DSN) as c:
        print(c.execute("SELECT version()").fetchone()[0])
    experiment_throughput()
    experiment_crash()
    experiment_mvcc()

main.go (Go, franz-go share group against kfake)

// Share groups (KIP-932) in Go with franz-go: 4 consumers on ONE partition,
// per-record ack, and a transient failure that gets redelivered.
// Runs against kfake, franz-go's in-process fake broker, so no real cluster is needed.
package main

import (
    "context"
    "fmt"
    "strconv"
    "sync"
    "time"

    "github.com/twmb/franz-go/pkg/kfake"
    "github.com/twmb/franz-go/pkg/kgo"
)

const (
    topic   = "jobs"
    group   = "email-workers"
    total   = 1000
    members = 4
)

func main() {
    c, err := kfake.NewCluster(kfake.NumBrokers(1), kfake.SeedTopics(1, topic)) // ONE partition
    if err != nil {
        panic(err)
    }
    defer c.Close()
    // Default is "latest": a brand-new share group would skip everything already in the topic.
    c.SetGroupConfigs(group, map[string]string{"share.auto.offset.reset": "earliest"})

    producer, err := kgo.NewClient(kgo.SeedBrokers(c.ListenAddrs()...), kgo.DefaultProduceTopic(topic))
    if err != nil {
        panic(err)
    }
    for i := 0; i < total; i++ {
        producer.Produce(context.Background(), kgo.StringRecord(strconv.Itoa(i)), nil)
    }
    if err := producer.Flush(context.Background()); err != nil {
        panic(err)
    }
    producer.Close()

    var (
        mu          sync.Mutex
        accepted    = map[string]int{} // job -> times accepted
        perMember   = make([]int, members)
        redelivered int
    )
    ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
    defer cancel()

    var wg sync.WaitGroup
    for m := 0; m < members; m++ {
        wg.Add(1)
        go func(m int) {
            defer wg.Done()
            cl, err := kgo.NewClient(
                kgo.SeedBrokers(c.ListenAddrs()...),
                kgo.ConsumeTopics(topic),
                kgo.ShareGroup(group),
                kgo.ShareMaxRecords(25),
                kgo.ShareMaxRecordsStrict(), // without this the broker may hand one member a whole record batch
            )
            if err != nil {
                panic(err)
            }
            defer cl.Close()
            for {
                mu.Lock()
                done := len(accepted) == total
                mu.Unlock()
                if done || ctx.Err() != nil {
                    _ = cl.FlushAcks(context.Background())
                    return
                }
                pollCtx, pollCancel := context.WithTimeout(ctx, time.Second)
                fetches := cl.PollFetches(pollCtx)
                pollCancel()
                fetches.EachRecord(func(r *kgo.Record) {
                    n, _ := strconv.Atoi(string(r.Value))
                    if n%10 == 0 && r.DeliveryCount() == 1 {
                        r.Ack(kgo.AckRelease) // transient failure: give it back for another attempt
                        return
                    }
                    time.Sleep(2 * time.Millisecond) // pretend to send the email
                    r.Ack(kgo.AckAccept)
                    mu.Lock()
                    accepted[string(r.Value)]++
                    perMember[m]++
                    if r.DeliveryCount() > 1 {
                        redelivered++
                    }
                    mu.Unlock()
                })
            }
        }(m)
    }
    wg.Wait()

    dups := 0
    for _, k := range accepted {
        if k > 1 {
            dups += k - 1
        }
    }
    fmt.Printf("partitions=1 consumers=%d distinct jobs accepted=%d/%d\n", members, len(accepted), total)
    fmt.Printf("accepted per consumer=%v\n", perMember)
    fmt.Printf("accepted on a 2nd+ delivery (after RELEASE)=%d, accepted more than once=%d\n", redelivered, dups)
}

For the Go demo: go mod init sharedemo && go get github.com/twmb/franz-go@v1.22.1 github.com/twmb/franz-go/pkg/kfake@v0.0.0-20260927204940-b5a45ccfdf7e && go run . (the kfake module requires Go 1.26, which the go command downloads automatically if your toolchain is older).

Takeaway

Kafka 4.2 fixed the strongest technical objection to "Kafka as a queue." It did not change where your jobs are born. If they are born in a Postgres transaction, keep them there, and spend your effort on the one thing that kills Postgres queues: old snapshots. When the events already live in Kafka and many teams read them, share groups are now a legitimate queue, on Java and Go today, and on Python once the client leaves Preview.

Sources

Apache Kafka 4.2.0 release announcement (Feb 17, 2026): https://kafka.apache.org/blog/2026/02/17/apache-kafka-4.2.0-release-announcement/
Apache Kafka 4.3.0 release announcement (May 22, 2026): https://kafka.apache.org/blog/2026/05/22/apache-kafka-4.3.0-release-announcement/
KIP-932, Queues for Kafka: https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A+Queues+for+Kafka
Kafka 4.3 broker configs: https://kafka.apache.org/43/configuration/broker-configs/
Kafka 4.3 group configs: https://kafka.apache.org/43/configuration/group-configs/
KafkaShareConsumer javadoc (4.2): https://kafka.apache.org/42/javadoc/org/apache/kafka/clients/consumer/KafkaShareConsumer.html
Kafka 4.4.0 release plan (KIP-1191 status): https://cwiki.apache.org/confluence/spaces/KAFKA/pages/429064575/Release+Plan+4.4.0
Confluent, Queues for Kafka GA: https://www.confluent.io/blog/kafka-queue-semantics-share-consumer-ga/
franz-go changelog: https://github.com/twmb/franz-go/blob/master/CHANGELOG.md
confluent-kafka-python v2.15.0: https://github.com/confluentinc/confluent-kafka-python/releases/tag/v2.15.0
librdkafka v2.15.0: https://github.com/confluentinc/librdkafka/releases/tag/v2.15.0
PostgreSQL SELECT (locking clause, SKIP LOCKED): https://www.postgresql.org/docs/current/sql-select.html
PostgreSQL routine vacuuming: https://www.postgresql.org/docs/current/routine-vacuuming.html
PostgreSQL NOTIFY: https://www.postgresql.org/docs/current/sql-notify.html
PostgreSQL client connection defaults (timeouts): https://www.postgresql.org/docs/current/runtime-config-client.html
PostgreSQL replication config (hot_standby_feedback): https://www.postgresql.org/docs/current/runtime-config-replication.html
PostgreSQL 17 release notes (transaction_timeout): https://www.postgresql.org/docs/17/release-17.html
PostgreSQL 19 draft release notes: https://www.postgresql.org/docs/19/release-19.html
PostgreSQL 19 RC1 and GA dates: https://www.postgresql.org/message-id/915517c4-d0ee-49f9-bbeb-c683ef2b9d3f%40postgresql.org
River, transactional enqueueing: https://riverqueue.com/docs/transactional-enqueueing
Solid Queue README: https://github.com/rails/solid_queue
Debezium outbox event router: https://debezium.io/documentation/reference/stable/transformations/outbox-event-router.html

Two PySpark contracts Staff DEs learn on a bad Tuesday: foreachBatch restarts and applyInPandas skew

Firat Celik — Thu, 01 Oct 2026 06:16:50 +0000

Your streaming MERGE looked fine in the notebook. Your applyInPandas transform looked elegant. Then a job restarted and silver row counts jumped, or a hot customer key OOMed a worker while the rest of the cluster sat idle.

Neither bug is exotic. Both are documented contracts that happy-path snippets skip.

This article covers two production surfaces Staff and Lead data engineers own on Databricks / PySpark:

Streaming upserts via foreachBatch: at-least-once writes, watermark dedupe state, Delta MERGE idempotency (logical, not txnAppId), empty batches, and why you must fully consume a batch DataFrame.
Pandas Function APIs: groupBy().applyInPandas() loads an entire group into memory and does not honor spark.sql.execution.arrow.maxRecordsPerBatch; mapInPandas / mapInArrow do.

I will mark FACT (from primary docs) vs interpretation (engineering judgment). No invented latency percentages. No "exactly-once foreachBatch" fairy tales.

Primary sources: Databricks foreachBatch, Delta MERGE / streaming upsert, watermarks, state store reader, idempotent Delta writes, pandas function APIs, Apache Arrow in PySpark.

Part 1. Design for the restart: `foreachBatch`, watermarks, and MERGE

Contract 1: `foreachBatch` is at-least-once

FACT (Databricks): foreachBatch() provides only at-least-once write guarantees. You can use the batchId to deduplicate and reason toward exactly-once yourself. You still own end-to-end semantics.

Interpretation: A green micro-batch in the UI is not a guarantee that a retry will be a no-op. Treat every handler as if Spark will call it again with the same batch_id.

APP_ID = "silver-orders-upsert-v1"  # change if you reset the checkpoint


def process_batch(batch_df, batch_id: int) -> None:
    # FACT: Delta sources can deliver empty micro-batches (e.g. OPTIMIZE with
    # nothing to process, or predicate pruning). Code must tolerate them.
    if batch_df.isEmpty():
        return

    valid = batch_df.filter("order_amount > 0 AND customer_id IS NOT NULL")
    invalid = batch_df.filter(
        "NOT (order_amount > 0 AND customer_id IS NOT NULL)"
    )

    # FACT: txnAppId + txnVersion on DataFrameWriter make appends idempotent
    # for retries of the same batch_id. Binding txnVersion to batch_id is the
    # documented pattern.
    (
        valid.write.format("delta")
        .mode("append")
        .option("txnAppId", APP_ID)
        .option("txnVersion", batch_id)
        .saveAsTable("main.silver.orders")
    )

    if not invalid.isEmpty():
        (
            invalid.write.format("delta")
            .mode("append")
            .option("txnAppId", APP_ID)
            .option("txnVersion", batch_id)
            .saveAsTable("main.silver.orders_dlq")
        )


(
    spark.readStream.table("main.bronze.raw_orders")
    .writeStream.foreachBatch(process_batch)
    .option("checkpointLocation", "/Volumes/main/ops/checkpoints/orders")
    .start()
)

FACT (Databricks Delta streaming): If you delete the streaming checkpoint and restart with a new checkpoint, you must provide a different txnAppId. New checkpoints start at batch ID 0; reusing the old app id can make Delta skip writes you intended to redo.

FACT (Databricks): Prefer letting the query fail and letting Lakeflow Jobs / Airflow retry the batch, rather than building silent local retry loops that can duplicate or drop data.

Contract 2: watermark dedupe has a state lease (and you must finish the batch)

FACT (Databricks watermarks): dropDuplicatesWithinWatermark always deduplicates events that arrive within the watermark delay. Events outside the delay might still be deduped, but that is not guaranteed. To guarantee dropping all duplicates, set the watermark delay greater than the maximum timestamp distance between duplicate events.

FACT (Databricks foreachBatch): When you use stateful operators such as dropDuplicatesWithinWatermark, each foreachBatch iteration must completely consume the batch DataFrame. If you only peek (show(2), take a few rows) and leave rows unread, the streaming query can fail on the next batch.

from pyspark.sql.functions import col

deduped = (
    spark.readStream.table("main.bronze.events")
    .withWatermark("event_time", "10 hours")  # > max duplicate lag (FACT guidance)
    .dropDuplicatesWithinWatermark(["event_id"])
)


def bad_handler(batch_df, batch_id: int) -> None:
    # Anti-pattern: partial consume after a stateful operator
    batch_df.show(2)
    # remaining rows never consumed -> next micro-batch can fail


def good_handler(batch_df, batch_id: int) -> None:
    if batch_df.isEmpty():
        return
    # Full action that consumes the plan (write, count, foreach, etc.)
    (
        batch_df.write.format("delta")
        .mode("append")
        .option("txnAppId", "events-dedupe-v1")
        .option("txnVersion", batch_id)
        .saveAsTable("main.silver.events")
    )


(
    deduped.writeStream.foreachBatch(good_handler)
    .option("checkpointLocation", "/Volumes/main/ops/checkpoints/events")
    .start()
)

Debugging tip (FACT): You can inspect Structured Streaming state with the statestore reader. Metadata can show an operator named dedupeWithinWatermark; query that operatorId when you need to see what the lease still holds. See Read Structured Streaming state information.

Contract 3: MERGE inside `foreachBatch` is a logical idempotency problem

FACT (Delta Lake): You must use foreachBatch for Delta Lake merge operations from Structured Streaming. Restarts can apply the same batch multiple times, so your merge statement inside foreachBatch must be idempotent.

FACT (Databricks / Delta writer options): txnAppId and txnVersion are DataFrameWriter options for idempotent writes (append / save). Do not assume they wrap DeltaTable.merge(...). Design MERGE so re-running the same micro-batch yields the same table state.

FACT (Delta): A merge can fail if multiple source rows match the same target row (ambiguous update). For CDC, preprocess to keep the latest change per key before merging.

FACT (Delta): MERGE often reads the source multiple times. That can inflate reported streaming input rates. Cache the micro-batch DataFrame before merge and unpersist after if this becomes a bottleneck.

from delta.tables import DeltaTable
from pyspark.sql import functions as F

target = DeltaTable.forName(spark, "main.silver.customers")


def upsert_cdc(batch_df, batch_id: int) -> None:
    if batch_df.isEmpty():
        return

    # Collapse to latest change per business key (FACT: avoid multi-match MERGE)
    latest = (
        batch_df.select(
            "customer_id",
            F.struct("event_time", "payload", "deleted").alias("chg"),
        )
        .groupBy("customer_id")
        .agg(F.max("chg").alias("chg"))
        .select("customer_id", "chg.*")
    )

    # FACT: MERGE may scan the source more than once -> cache the batch
    latest = latest.cache()
    try:
        (
            target.alias("t")
            .merge(latest.alias("s"), "t.customer_id = s.customer_id")
            .whenMatchedDelete(condition="s.deleted = true")
            .whenMatchedUpdate(
                set={"payload": "s.payload", "updated_at": "s.event_time"}
            )
            .whenNotMatchedInsert(
                condition="s.deleted = false",
                values={
                    "customer_id": "s.customer_id",
                    "payload": "s.payload",
                    "updated_at": "s.event_time",
                },
            )
            .execute()
        )
    finally:
        latest.unpersist()


(
    spark.readStream.table("main.bronze.customer_cdc")
    .writeStream.foreachBatch(upsert_cdc)
    .option("checkpointLocation", "/Volumes/main/ops/checkpoints/customer_cdc")
    .start()
)

Interpretation: "Idempotent MERGE" here means latest-wins on a stable key (and deletes keyed the same way), not "Spark will skip the merge via txn metadata." If the same CDC events replay, the table should converge to the same rows.

Managed alternative (context, not a product pitch): Databricks processing-guarantees docs note that managed Delta-to-Delta flows and AUTO CDC can give stronger defaults; a custom foreachBatch edge is still at-least-once unless you add explicit idempotent write or dedupe logic. Use AUTO CDC when it fits; when you hand-roll MERGE, you own the restart contract above.

Part 1 checklist (print this next to the PR)

Checkpoint	Why
Empty-batch safe	Delta sources can emit empty micro-batches
Full consume after stateful ops	Partial `show`/`take` can fail the next batch
Watermark > max duplicate lag	Out-of-watermark dedupe is not guaranteed
Append path uses `txnAppId` + `txnVersion=batch_id`	Writer-level idempotency on retry
New checkpoint => new `txnAppId`	Else batch 0 collisions skip writes
MERGE is latest-per-key / safe to re-run	txn options do not wrap `DeltaTable.merge`
Cache around MERGE when metrics / cost hurt	MERGE multi-pass over the micro-batch
Fail fast to the orchestrator	Local retry loops are how you invent exactly-once bugs

Part 2. Arrow is not a free pass: `applyInPandas` vs `mapInPandas` / `mapInArrow`

Python DE craft is not only streaming. The other Tuesday failure mode is "we used pandas because Arrow is fast" and then a skewed group killed the executor.

Three APIs, three memory contracts

FACT (Databricks pandas function APIs / Apache Spark):

API	What you write	Memory / batching contract
`groupBy().applyInPandas(f, schema)`	One pandas DataFrame per group	All rows for the group load into memory before `f` runs. `spark.sql.execution.arrow.maxRecordsPerBatch` is not applied. Skew => OOM risk.
`mapInPandas(f, schema)`	Iterator of pandas DataFrames (batches)	Batch size controlled by `spark.sql.execution.arrow.maxRecordsPerBatch` (default 10,000 rows in Spark docs). Output length can differ from input.
`mapInArrow(f, schema)`	Iterator of `pyarrow.RecordBatch`	Same Arrow batch sizing idea; stay in Arrow without pandas round-trip when you can.

Cogrouped applyInPandas has the same whole-cogroup-in-memory warning.

Skew turns "elegant" into OOM

# Looks clean. Under a hot customer_id it is a bomb.
def per_customer_features(pdf):
    # pdf = ALL rows for one customer_id
    pdf = pdf.sort_values("event_time")
    pdf["rolling_amt"] = pdf["amount"].rolling(50, min_periods=1).sum()
    return pdf


# FACT: entire group must fit in worker memory; maxRecordsPerBatch does NOT help
out = (
    events.groupBy("customer_id")
    .applyInPandas(
        per_customer_features,
        schema="customer_id string, event_time timestamp, amount double, rolling_amt double",
    )
)

Interpretation: Use applyInPandas when the group is a true unit of work and you can bound group size (filter, sample hot keys, pre-aggregate, salt, or split whales). Do not use it as a generic "run pandas on Spark" hammer.

Prefer batch maps when you do not need whole-group semantics

spark.conf.set("spark.sql.execution.arrow.maxRecordsPerBatch", "5000")


def score_batches(iterator):
    # init once per Python worker task
    model = load_light_model()  # your code; keep it picklable / available on workers
    for pdf in iterator:
        pdf = pdf.copy()
        pdf["score"] = model.predict(pdf[["f1", "f2"]])
        yield pdf[["id", "score"]]


scored = events.mapInPandas(
    score_batches,
    schema="id long, score double",
)

Or stay in Arrow when pandas is unnecessary:

import pyarrow as pa
import pyarrow.compute as pc


def filter_arrow(iterator):
    for batch in iterator:
        mask = pc.greater(batch.column("amount"), 0)
        yield batch.filter(mask)


positive = events.mapInArrow(filter_arrow, schema=events.schema)

Decision table

Situation	Prefer	Watch out
Need whole-group window / asof / custom group ML	`applyInPandas`	Hot keys; no Arrow batch limit; cogroup same issue
Row/batch transform, arbitrary output length	`mapInPandas`	Tune `maxRecordsPerBatch` down if wide rows
CPU vector ops, avoid pandas copies	`mapInArrow`	Arrow type coverage / compute familiarity
Series-to-series column expression	classic `pandas_udf`	Output length must match input for many UDF shapes

FACT (Spark Arrow guide): Lower maxRecordsPerBatch when column count is large; Arrow batches can pressure JVM memory during conversion.

How the two themes connect

Both failures are the same Staff DE mistake: assuming the platform's happy path equals your failure domain.

Streaming: you assumed merge + checkpoint = exactly-once. Docs say at-least-once + you make MERGE idempotent; writer txn options are a different tool.
Pandas APIs: you assumed "Arrow optimized" = bounded memory. Docs say group maps ignore the batch limit.

Own the contract in the PR template, not in the postmortem.

Optional diagram (for a cover or inline figure)

Two panels:

Restart timeline: micro-batch N fails mid-MERGE -> orchestrator retries batch N -> arrows labeled "append: txn skip" vs "MERGE: must be latest-wins safe".
Skew funnel: many small customer_id groups sail through applyInPandas; one whale group expands to "all rows in RAM" with a red X; side path routes whales to salted / pre-aggregated / mapInPandas path.

References

I'm Firat Celik, Lead Data Engineer. I write about production data platforms, PySpark/Delta, and the boring contracts that keep pipelines honest.

Fabric Runtime 2.0 is becoming the default. Here is what actually breaks.

Firat Celik — Mon, 28 Sep 2026 06:44:13 +0000

Fabric Runtime 2.0 is becoming the default. Here is what actually breaks.

Late September 2026 is when Fabric Runtime 2.0 stops being an opt-in experiment and becomes the path of least resistance for new workspaces and environment items. If you treat that as a release note, you will discover the breakages in production.

I work as a Lead Data Engineer on Azure, Fabric, Databricks, and Spark. This is the migration checklist I wish teams ran before the default flipped under them.

Docs to keep open while you read:

1) Hook: the default is the migration

Microsoft's Runtime 2.0 page is blunt: GA and ready for production, but not the default yet. You opt in. The stated plan is to make Runtime 2.0 the default UX selection and the default for new workspaces and new environment items in late September 2026.

Today is 28 September 2026. That window is now.

Existing workspaces, jobs, and Environment items stay on whatever you configured. The trap is quieter. Every new workspace for a pilot, every Environment someone creates because "defaults are fine," starts landing on Spark 4.1 / Delta 4.2 / Python 3.13 once the plan executes.

Treat the flip as an incident rehearsal date. Pin what you care about. Validate 2.0 on a scoped Environment. Do not discover Python wheel breakage through a failed Monday job.

Runtime 1.3 still has runway. Secondary writeups put Microsoft's LTS window for 1.3 roughly from early October 2026 through March 2027. Verify against current lifecycle docs before you commit a program date. LTS means supported, not "ignore migration."

2) Wrong assumption: GA means notebooks just run

Runtime 1.3 to Runtime 2.0 is not a patch bump. It is a stack jump on nearly every component:

Component	Runtime 1.3 (typical)	Runtime 2.0
Apache Spark	3.5	4.1
Delta Lake	3.2	4.2
Python	3.11	3.13
Java	11	21
Scala	2.12	2.13
OS	(prior Fabric OS)	Azure Linux 3.0 (Mariner 3.0)

Industry writeups also call out Delta protocol defaults moving higher (reader/writer versions, Deletion Vectors in the 2.0 path). Treat protocol as a first-class migration risk, not a footnote. See Microsoft's Delta Lake table format interoperability guidance for what cross-experience readers can actually open.

Practical fallout I care about:

Python 3.11 -> 3.13. Environment-level libraries may migrate automatically. Validation does not. Custom wheels and pinned packages are your problem. Session-scoped %pip install and notebook Resources are a different blast radius than Environment items.
Scala 2.12 -> 2.13 / Java 11 -> 21. JARs and connectors that "worked forever" on 1.3 deserve a dual-run, not hope.
SparkR is deprecated in Spark 4.x and may be removed later. If you still have SparkR notebooks, the clock is already running.

GA means Microsoft ships it for production use. It does not mean your Environment's library graph survives the jump without you watching.

3) Wrong assumption: NEE = free 6x

Runtime 2.0 includes the Native Execution Engine (NEE). Microsoft's docs describe it as a vectorized path based on Apache Gluten (offload middle layer) and Velox (C++ engine from Meta). Supported operators leave the JVM and run columnar with SIMD-style acceleration on Parquet, Delta, and (now) CSV. Microsoft cites up to six times faster vs open-source Spark on TPC-DS. That is a vendor benchmark on a specific workload family. It is a ceiling story, not your median job.

How you enable it:

Environment -> Spark compute -> Acceleration -> enable native execution engine (preferred UX path), or
Spark conf: spark.native.enabled (also usable per notebook / SJD via %%configure)

Hard limits from the docs that matter in production:

Structured streaming is not supported. Use streaming? Plan for JVM (or keep NEE off for those jobs).
Unsupported ops fall back to classic Spark. Fallback is automatic. Your job keeps running. Your SLA may not.
Every boundary between native columnar and JVM row-oriented execution pays a conversion tax. That is the part marketing slides skip.
Fabric Spark Advisor can surface fallback alerts in notebook cell output. Use them. Do not ignore green "job succeeded" when Advisor is screaming about offload misses.

NEE also preserves AQE, cost-based rewrites, column pruning, and predicate pushdown when operators do offload. Good. That does not help if half the plan is bouncing in and out of Velox.

Blunt take: a vectorized engine that falls back on half your plan is not acceleration. It is a tax you opted into because a dashboard said "up to 6x."

4) How to see the truth in the plan

Do not argue with anecdotes. Read the plan.

In Spark UI / History Server, look for node names ending in:

*Transformer (for example ProjectExecTransformer, BroadcastHashJoinExecTransformer, RollUpHashAggregateTransformer)
*NativeFileScan
VeloxColumnarToRowExec

Those suffixes mean native offload happened for that node. Light-blue JVM nodes next to green native nodes (Microsoft's Gluten SQL / DataFrame tab uses that color language) are your fallback map. Every VeloxColumnarToRowExec (and the reverse path) is a place where you paid for format conversion.

Also useful: df.explain(), Fabric Spark Advisor fallback alerts, and the Gluten SQL / DataFrame tab's native-vs-JVM node counts.

Surgical disable for one cell (unsupported op, correctness edge, or A/B):

spark.conf.set('spark.native.enabled', 'false')

Re-enable afterward if later cells should use NEE again. Spark runs cells in order. Conf sticks.

5) Delta 4.x trap for multi-workload tables

This is the silent one.

Microsoft is explicit: Delta Lake 4.2-specific features are experimental and only work on Spark experiences (notebooks and Spark Job Definitions). If the same Delta tables must be used across multiple Fabric workloads, do not enable those features.

Why it hurts: SQL analytics endpoint, Power BI Direct Lake, and other non-Spark readers do not all speak the same protocol / feature set. Protocol upgrades (higher reader/writer versions, Deletion Vectors, experimental checkpoints) are effectively one-way doors. Recreate the table is the recovery path, not "uncheck the box." Shared bronze/silver tables that feed Spark and Direct Lake and SQL endpoint are exactly where someone enables a shiny 4.x feature on a Friday and breaks Monday's semantic model.

Rule I use: default to the most constrained reader in the estate. If Direct Lake or the SQL endpoint must read it, keep features inside what those readers support. Save experimental Delta 4.x toys for Spark-only sandboxes.

6) Practical migration pattern

This is the runbook shape I actually use.

1. Inventory

Workspaces and Environment items: which runtime are they on?
Jobs that create new Environments without pinning
Libraries: Environment wheels vs session %pip
Tables read by more than one Fabric experience
Streaming jobs (NEE: leave off or isolate)
SparkR notebooks (debt clock)

2. Baseline on 1.3

Duration, shuffle bytes, spill, failure rate, data quality checks
Capture plans for critical queries before you change anything
Note current Delta protocol / table features on shared tables

3. Dual-run on 2.0

Scoped Environment item first (surgical), not whole workspace
Same input snapshot, same checks, compare plans and SLA
Turn NEE on only after the JVM path is green on 2.0
Kill vanity actions that force row materialization (yes, that includes habit .show() in hot paths during timing runs)

4. Rollback in the runbook before you need it

Runtime: pin Environment / workspace back to 1.3 while LTS covers you
NEE only:

spark.conf.set('spark.native.enabled', 'false')

Put both in the on-call doc. A rollback you invent at 02:00 is not a rollback.

7) CI/CD angle: pin the runtime like a toolchain

You would not let python float to whatever the image shipped last Tuesday without a lockfile. Treat Fabric runtime the same way.

Pin runtime version on Environment items that production notebooks and SJDs attach to
Fail the pipeline if an Environment publishes without an explicit runtime
Gate promotion: 1.3 baseline metrics vs 2.0 dual-run metrics must be compared, not vibes
Separate "NEE enabled" as its own promotion flag. Runtime upgrade and native offload are two changes. Ship them as two changes.
Use Fabric runtime release channels (preview) to rehearse upcoming defaults on custom pools. Do not park production there.

If CI deploys notebook JSON and never asserts runtime + Acceleration settings, you are continuous-delivering entropy.

8) Close: benchmarks sell the ceiling, production pays the median

Workload type	Enable NEE?	Watch for...
Large Parquet/Delta batch ETL, heavy agg/join	Yes, after dual-run	Fallback density, conversion boundaries, Advisor alerts
Interactive lakehouse SQL / notebook exploration	Often yes	Date type mismatches on filters, wide UDF usage
Structured streaming	No (unsupported)	Accidental enable via Environment inheritance
JSON/XML heavy paths	Limited / often no	Format falls back to JVM; measure before celebrating
Shared Delta tables (Spark + SQL endpoint / Direct Lake)	NEE optional; Delta 4.x experimental features: no	Protocol / feature lockout of non-Spark readers
SparkR or exotic Scala libs	Maybe later	Language deprecation and binary incompat first

Takeaways

Pin Runtime 1.3 on anything you cannot afford to surprise, then opt into 2.0 on a scoped Environment before the default teaches your newest workspace the hard way.
Treat NEE as a plan shape problem, not a toggle. *Transformer and VeloxColumnarToRowExec tell the truth; "up to 6x" does not.
Never enable experimental Delta 4.x features on multi-experience tables. Shared tables inherit the weakest reader's constraints.

Benchmarks sell the ceiling. Production pays the median. Migrate like you believe that.

What is your scarier unpaid bill right now: Python 3.13 libraries, NEE fallback thrash, or a Delta feature someone enabled on a table Direct Lake still needs to read?

Sources

Runtime 2.0 in Fabric (Microsoft Learn)
Native execution engine for Fabric Data Engineering (Microsoft Learn)
Context (not a substitute for the docs): Fabric Runtime 2.0 two breakages (Rajaniesh Kaushikk, 2026-09-23)

Azure Synapse Databricks: A Migration Playbook That Survives Production

Firat Celik — Fri, 04 Sep 2026 10:11:59 +0000

Azure Synapse → Databricks: A Migration Playbook That Survives Production

A field-style guide for Lead / Senior Data Engineers moving Dedicated SQL Pools, pipelines, and BI off Synapse without burning trust, budget, or the on-call team.

Why this migration is harder than the slide deck
What you are actually migrating
Program shape: discovery → waves
Four parallel workstreams
Data movement: skip the JDBC tax
Code conversion: syntax is easy, semantics bite
Dimensional patterns on Delta (SCD2)
Reconciliation & cutover
Power BI / consumption cutover
30 / 60 / 90 plan
Anti-patterns I keep seeing
Field rules I will not compromise on
FAQ
Sources

1) Why this migration is harder than the slide deck

If you have lived in Azure Synapse for years, the marketing line sounds simple:

Move to Databricks. Unity Catalog. Lakehouse. Done.

In production programs, that is not the hard part.

The hard part is that Synapse is not one system. Under one brand you usually own years of Dedicated SQL Pool logic, ADF/Synapse Pipelines, serverless SQL over the lake, Spark pools, permission models, and Power BI datasets wired to Synapse endpoints.

Treat that estate as one workstream with one deadline, and the schedule slips. Every time.

I work as a Lead Data Engineer across Azure Synapse, Microsoft Fabric, Azure Data Lake, Databricks/Spark, SQL/T-SQL, and Power BI. This playbook is the version I want in the room before the first converted stored procedure lands in production.

Thesis: Migration success is not “all notebooks on Databricks.”

It is decision-ready data, trusted KPIs, and a platform your team can operate.

2) What you are actually migrating

Synapse piece	Typical complexity	What “done” looks like on Databricks
Dedicated SQL Pools	Highest — procs, distributions, CCI, years of tuning	Databricks SQL / Spark jobs, Delta tables, Liquid Clustering / predictive optimization
Serverless SQL	Medium — views & external tables over ADLS	Views / tables in Unity Catalog over the same lake paths (often simpler)
Spark pools	Lower — already Spark-shaped	Databricks Runtime notebooks / jobs (often few changes)
ADF / Synapse Pipelines	High — orchestration + connectors	Lakeflow Connect / Lakeflow Jobs / Workflows
Governance	High — SQL perms + Purview stitching	Unity Catalog (and stop dual-governing forever)
Power BI / semantic models	High politically	Databricks SQL Warehouse endpoints + dataset remaps

Dedicated SQL Pools usually consume most of the calendar. Orchestration and BI consume most of the organizational risk.

3) Program shape: discovery → waves

Do not start with “transpile everything.” Start with a program.

flowchart LR
  A[1 Discovery<br/>Lakebridge Profiler] --> B[2 Assessment<br/>Analyzer / T-SQL]
  B --> C[3 Design<br/>Hybrid lift + modernize]
  C --> D[4 Pilot<br/>Lighthouse E2E]
  D --> E[5 Waves<br/>Parallel workstreams]

Phase A — Discovery

Inventory what actually runs:

pools, databases, schemas, object counts
top queries / heavy procs by CPU & IO
pipeline inventory (ADF / Synapse)
downstream BI datasets and owners
unused objects (your cheapest scope cut)

Lakebridge Profiler (Databricks Labs) is built for Synapse estate profiling — metadata, utilisation, query patterns, baselines for a TCO conversation.

Phase B — Assessment

Classify the T-SQL estate:

low / medium / high complexity
unsupported constructs
dependency graphs (what breaks if this view moves)

Lakebridge Analyzer helps here. Start waves with low/medium complexity and clear business owners — not the ugliest proc on day one.

Phase C — Design (usually hybrid)

Three strategies show up in every steering committee:

Approach	When it works	Failure mode
Lift-and-shift	Hard deadline off Synapse	You recreate Synapse-shaped pain on a new bill
Big-bang rewrite	Greenfield team, tiny estate	Timeline fantasy
Hybrid	Most real enterprises	Needs discipline: automate bulk convert, modernize in waves

For most Synapse migrations, hybrid wins: automated conversion gets you off Synapse on a schedule; modernization (Delta patterns, Liquid Clustering, Lakeflow, UC) happens after workloads are stable.

Phase D — Pilot (lighthouse)

Pick one end-to-end slice:

source → ingest → transform → gold mart → Power BI

Prove:

storage + UC layout
job standards
reconciliation pack
BI remap
on-call runbook

The pilot’s job is not heroics. It is producing reusable assets for later waves.

Phase E — Waves

Each wave should ship a visible business win, not just “another schema moved.”

4) Four parallel workstreams

After the lighthouse, scale on four tracks together. Sequential “ETL first, BI last” is how stakeholders only feel pain.

flowchart TB
  P[Lighthouse Pilot]
  P --> I[Ingestion<br/>ADF/Synapse Pipelines → Lakeflow Connect]
  P --> T[Transformation<br/>T-SQL / procs → Databricks SQL / Spark]
  P --> O[Orchestration<br/>Schedules → Lakeflow Jobs]
  P --> C[Consumption<br/>Power BI → Databricks SQL Warehouse]

Track	From	To
Ingestion	ADF / Synapse Pipelines	Lakeflow Connect / Jobs (or open ingest into Delta)
Transformation	T-SQL / procs / views	Databricks SQL / Spark with clear contracts
Orchestration	Synapse schedules & deps	Lakeflow Jobs / Workflows
Consumption	Power BI / semantic models	Databricks SQL Warehouses

Sequencing tip that builds trust

A BI-aware sequence often works better than pure ETL-first:

Land or expose business-facing marts early (even interim / federation patterns where appropriate).
Put a trusted KPI dataset in front of users.
Modernize the uglier ETL underneath while the business already sees progress.

Enablement is a fifth unofficial track. Synapse-native teams underinvest here and then call the platform “not ready.”

5) Data movement: skip the JDBC tax

For large Dedicated SQL Pool tables, JDBC Synapse → Databricks is how you burn cluster hours and patience.

Prefer:

Synapse CETAS → Parquet on ADLS Gen2 → Databricks Spark read → Delta in Unity Catalog

flowchart LR
  S[Synapse Dedicated SQL Pool] -->|CETAS| L[ADLS Gen2 Parquet]
  L -->|Spark read / Delta write| D[Unity Catalog Delta]
  S -.->|JDBC slow path ❌| D

Why this path wins

Synapse writes columnar Parquet in parallel (CETAS).
ADLS is the shared landing zone both engines already understand.
Databricks reads Parquet with strong scan performance, then writes Delta (ACID, time travel, UC governance).
You avoid row-by-row JDBC serialization.

Example — CETAS from Synapse Dedicated SQL Pool

-- File format
CREATE EXTERNAL FILE FORMAT ParquetFF
WITH (
    FORMAT_TYPE = PARQUET,
    DATA_COMPRESSION = 'org.apache.hadoop.io.compress.SnappyCodec'
);

-- Landing zone on ADLS Gen2
CREATE EXTERNAL DATA SOURCE MigLanding
WITH (
    LOCATION = 'https://<storage>.dfs.core.windows.net/<container>',
    TYPE = HADOOP
);

-- Export a bounded slice (always partition/filter on first runs)
CREATE EXTERNAL TABLE ext.fact_sales_export
WITH (
    LOCATION = '/migration/fact_sales/ds=2026-09-03/',
    DATA_SOURCE = MigLanding,
    FILE_FORMAT = ParquetFF
)
AS
SELECT
    sales_id,
    customer_nk,
    product_nk,
    business_date,
    net_amount,
    quantity
FROM dbo.fact_sales
WHERE business_date >= '2024-01-01';

Example — land as Delta on Databricks

from pyspark.sql import functions as F

src = "abfss://<container>@<storage>.dfs.core.windows.net/migration/fact_sales/"

df = (
    spark.read.parquet(src)
    .withColumn("ingested_at", F.current_timestamp())
)

(
    df.write
    .format("delta")
    .mode("overwrite")
    .option("overwriteSchema", "true")
    .saveAsTable("sales_uc.gold.fact_sales")
)

Physical design: do not cargo-cult Synapse

In Synapse you spent years on:

DISTRIBUTION = HASH(...)
ROUND_ROBIN / REPLICATE
clustered columnstore choices

On Databricks / Delta, those directives do not travel. Mapping HASH(customer_id) to a Delta partition is a classic self-own: high-cardinality keys create tiny-file nightmares.

Prefer:

Liquid Clustering / CLUSTER BY AUTO
predictive optimization / compaction as an operating model
partitions only for coarse, stable predicates (e.g. business_date) when they truly help

Rule: Drop physical Synapse directives. Re-tune with lakehouse primitives after you have real query telemetry — not before.

6) Code conversion: syntax is easy, semantics bite

Automated conversion (Lakebridge transpile and friends) often handles a large share of syntax — commonly cited in the ~80–90% range for bulk translation. Your engineers live in the remaining 10–20%: cursors, dynamic SQL, weird error handling, and semantic mismatches.

Common remaps

T-SQL	Databricks SQL
`GETDATE()`	`CURRENT_TIMESTAMP()`
`ISNULL(a, b)`	`COALESCE(a, b)` / `IFNULL(a, b)`
`LEN(s)`	`LENGTH(s)`
`CHARINDEX(sub, str)`	`LOCATE(sub, str)`
`SELECT TOP 10 ...`	`SELECT ... LIMIT 10`
`CONVERT(INT, col)`	`CAST(col AS INT)`

Semantic trap #1 — case sensitivity

Dedicated SQL Pools are often case-insensitive by collation. Databricks SQL is case-sensitive by default.

-- May silently change results after migration
WHERE status = 'active'

-- Make intent explicit
WHERE LOWER(status) = 'active'

Build this into reconciliation: pick string dimensions and compare distinct counts before you argue about revenue.

Semantic trap #2 — stored procedures

Migrate structure first (parameters, control flow, set-based DML). Optimize second.

Rewrite candidates (do not pretend transpile “finished” them):

cursors / row-by-row loops
heavy dynamic SQL
Synapse-only performance hints
temp-table choreography that exists only to fight distribution skew

Semantic trap #3 — TRY/CATCH vs workflow design

Simple audit logging often stays SQL-shaped. Multi-step compensation and downstream signaling usually belong in Lakeflow Jobs (retries, quarantine tasks, alerting) rather than a 2,000-line proc that tries to be an orchestrator.

7) Dimensional patterns on Delta (SCD2)

Slowly Changing Dimensions are where Synapse estates accumulate snowflake uniqueness. Your goal is not to clone the old proc line-for-line. Preserve the business rule: history + current row queryability.

Example pattern — expire then insert (illustrative)

-- 1) Expire changed current rows
MERGE INTO dim_customer AS t
USING staging_customer AS s
ON t.customer_nk = s.customer_nk AND t.is_current = true
WHEN MATCHED AND (
     t.segment <> s.segment
  OR t.country <> s.country
) THEN UPDATE SET
  t.is_current = false,
  t.valid_to   = current_timestamp();

-- 2) Insert new current versions
INSERT INTO dim_customer (
  customer_nk, segment, country, valid_from, valid_to, is_current
)
SELECT
  s.customer_nk,
  s.segment,
  s.country,
  current_timestamp(),
  CAST(NULL AS TIMESTAMP),
  true
FROM staging_customer s
LEFT ANTI JOIN dim_customer t
  ON t.customer_nk = s.customer_nk
 AND t.is_current = true;

Delta’s ACID behavior makes multi-step SCD flows safer than “hope the warehouse lock ordering holds.” Still: wrap critical loads in jobs with clear quarantine paths.

8) Reconciliation & cutover

Budget reality: validation often costs more calendar time than conversion.

Minimum reconcile pack (every wave)

Row counts by business date / partition
Aggregates — revenue, quantity, distinct business keys
Hash / checksum compares on ordered business-key sets
Tolerance checks for floats / FX / rounded measures
String-dimension sanity (case folding surprises)

Lakebridge Reconcile supports structured comparison patterns across these dimensions.

Cutover pattern that keeps executives calm

Parallel run (Synapse + Databricks) for the lighthouse mart
Side-by-side Power BI validation with the business owner
Flip the dataset connection
Pause Synapse compute (do not delete yet)
Keep rollback for a defined soak window
Only then decommission

Early decommissioning is how “successful migrations” become weekend rollbacks.

9) Power BI / consumption cutover

Technical remap is usually:

Synapse endpoint → Databricks SQL Warehouse
refresh credentials / gateway assumptions
validate DAX measures against reconcile packs

Political remap is harder:

who signs off KPI parity?
which report is the source of truth during dual-run?
what is the communication plan when numbers differ by 0.3% due to case folding or timezone?

Treat consumption as a first-class workstream with named business owners, not a Friday afternoon connection string change.

10) 30 / 60 / 90 plan

Days 1–30 — foundations

[ ] Run Lakebridge Profiler + Analyzer on Dedicated pools
[ ] Cut dead objects from scope (document why)
[ ] Choose one lighthouse: pipeline → gold mart → one Power BI dataset
[ ] Stand up Unity Catalog layout + storage conventions
[ ] Define job naming, environments, and CI basics
[ ] Write the reconcile notebook/job v0

Days 31–60 — prove the path

[ ] CETAS → ADLS → Delta for large facts (no JDBC heroics)
[ ] Transpile + hand-fix the top procs for the lighthouse
[ ] Repoint one BI workspace to Databricks SQL Warehouse
[ ] Parallel run + signed business acceptance
[ ] Capture runbooks: failure modes, quarantine, reruns

Days 61–90 — scale without drama

[ ] Wave 2/3 domains with the same pack
[ ] Orchestration parity on Lakeflow Jobs
[ ] Enablement plan for Synapse-native engineers
[ ] Cost review (warehouse sizing vs old pool bills)
[ ] Only then: formal Synapse decommission proposal

11) Anti-patterns I keep seeing

Transpile-first, assess-never — migrating abandoned procs with confidence.
JDBC for multi-billion-row facts — slow, expensive, fragile.
HASH key → Delta partition — tiny files, sad Spark UI.
Big-bang BI cutover — no dual-run, no owner, no soak.
Recreating every Synapse knob — preserving constraints without preserving reasons.
Skipping enablement — platform is fine; team still thinks in distributions.
Decommission on day one of “green” — optimism is not a rollback strategy.

12) Field rules I will not compromise on

Hybrid > pure lift-and-shift > big-bang rewrite
Assessment before conversion
ADLS + Parquet/Delta over JDBC for big moves
Do not map Synapse HASH keys to partitions
Reconcile every wave
Do not decommission Synapse early
Enablement is a workstream, not a lunch-and-learn
Migrate outcomes (trusted KPIs), not just objects

13) FAQ

Q: Should we move to Fabric instead of Databricks?

A: Different target, overlapping Azure story. This article assumes a Databricks Lakehouse landing zone. If Fabric is in play, treat that as an explicit architecture decision — do not accidentally run two migrations under one slogan.

Q: Can we keep Synapse Serverless for “just a few views”?

A: Sometimes as a bridge. Long-term dual query planes usually recreate the governance tax you are trying to escape.

Q: How perfect must automated conversion be?

A: Good enough to move bulk syntax. Your senior engineers should own semantic diffs, SCD logic, and orchestration.

Q: What is the first metric of a healthy pilot?

A: Not “job duration.” It is signed KPI parity with a business owner plus a rerun runbook a junior can execute.

14) Sources

Databricks — Navigating a Synapse Migration to Databricks https://www.databricks.com/blog/navigating-synapse-migration-databricks
Databricks — Introducing Lakebridge https://www.databricks.com/blog/introducing-lakebridge-free-open-data-migration-databricks-sql
Lakebridge docs — Synapse Profiler https://databrickslabs.github.io/lakebridge/docs/assessment/profiler/synapse/
Lakebridge docs — Transpile https://databrickslabs.github.io/lakebridge/docs/transpile/
Practical large-table movement discussion (CETAS → ADLS → Delta) https://medium.com/@prabhakarankanniappan/copy-3-5-billion-rows-from-synapse-to-databricks-in-30-minutes-fda15e0ffca0

# go-intake: Go-Native Streaming Data Ingestion Toolkit

Firat Celik — Tue, 09 Jun 2026 10:01:42 +0000

Executive Summary

go-intake is a minimalist, streaming-first ETL toolkit for Go developers that transforms messy data into validated, record-oriented output. With zero third-party dependencies and a focus on composition over configuration, it provides enterprise-grade data pipeline capabilities in a single, embeddable library.

Architecture Overview

Core Abstraction Layers

Component	Interface	Responsibility	Pattern
Source	3-method interface (`Open`, `Read`, `Close`)	Data ingestion from CSV, JSONL	Stream until EOF
Transformer	`Apply(ctx, Record) (Record, error)`	Field normalization, type parsing	Immutable, returns copy
Validator	`Validate(ctx, Record) error`	Schema enforcement, business rules	Read-only
Quarantine	`Write(ctx, InvalidRecord)`	Rejected record capture with context	Structured error logging
Sink	3-method interface (`Open`, `Write`, `Close`)	Output to CSV, JSONL	Streaming write

Design Principles

┌─────────────┐    ┌──────────────┐    ┌─────────────┐
│   Source    │───▶│ Transformers │───▶│ Validators  │
└─────────────┘    └──────────────┘    └─────────────┘
                                            │
                               ┌────────────┴──────────┐
                               ▼                       ▼
                         ┌──────────┐           ┌───────────┐
                         │   Sink   │           │Quarantine  │
                         └──────────┘           └───────────┘

Memory Usage Comparison

Memory (MB)
   500 │                      
       │    ●●●●●●●●●●●●●●●●●●  go-intake (constant)
       │                      
   100 │                      
       │                      
    50 │    ████ goflow grows with dataset
       │    ██████            
    25 │    ████████████      
       │    ████████████████  streamz grows
       │                       
     0 └──────────────────────
        0K    50K   100K   150K
           Records Processed

Performance Benchmarks

Throughput Analysis

Dataset Size	Records Processed	Time Elapsed	Throughput
Small (10K)	10,000	~40ms	~250,000/sec
Medium (100K)	100,000	~410ms	~243,902/sec
Large (161K)	161,568	643ms	251,000/sec

Memory Efficiency: The streaming model ensures constant memory usage regardless of dataset size. Only one record exists in memory at any point during processing.

Comparative Performance

graph LR
    A[go-intake<br/>251K/s] --> B[Memory: O(1)]
    C[Gopherize<br/>125K/s] --> D[Memory: O(n)]
    E[goflow<br/>180K/s] --> F[Memory: O(n)]

Performance Scaling Chart

Records/sec
   300K │
        │
   250K │    ● (go-intake: 251K)
        │   ╱
   200K │  ╱
        │ ╱
   150K │● (goflow: 180K)
        │
   100K │● (streamz)
        │
    50K │● (Gopherize: 125K)
        │
     0K └─────────────────────

Feature Matrix vs Competition

Feature	go-intake	Gopherize	goflow	streamz
✅ Zero Dependencies	✓	✗	✗	✗
✅ Streaming Model	✓	Partial	✓	✓
✅ Structured Errors	✓	✗	Partial	✗
✅ Multi-validation Errors	✓	✗	✗	✗
✅ Schema Discovery	✓	✗	✗	✗
✅ Header Normalization	✓	✓	✗	✗
✅ Quarantine Sinks	✓	✗	✗	✗
✅ Context Cancellation	✓	✗	✓	✓
Lines of Code	~2,300	~8,500	~12,000	~6,200

Bundled Components

Sources

CSVSource: RFC 4180 compliant, configurable delimiter, empty line skipping
JSONLSource: Newline-delimited JSON, 16MB line buffer support

Sinks

CSVSink: Automatic header generation, deterministic column ordering
JSONLSink: HTML-escaping disabled, one object per line

Transformers

Function	Purpose	Use Case
`NormalizeHeaders(style)`	Case normalization	Column standardization
`TrimStrings()`	Whitespace removal	Data cleaning
`ParseFloat/Int/Bool/Date`	Type conversion	Schema enforcement
`Rename/Drop/Keep/Copy`	Schema evolution	ETL pipeline stages
`AddField`	Constant injection	Metadata enrichment
`MapField`	Custom transformations	Business logic

Validators

Function	Type	Error Aggregation
`Required(field)`	Presence	✅
`Min/Max/Between`	Numeric range	✅
`Regex(field, pattern)`	Pattern matching	✅
`Enum(field, values...)`	Allowlist	✅
`Email/URL`	Format validation	✅
`NotFuture(field)`	Temporal constraint	✅

Real-World Validation Results

Dataset 1: COVID-19 Aggregated Data (161,568 records)

Field Profiling Results:

Field	Type	Confidence	Null %
Confirmed	int	100%	0%
Country	string	100%	0%
Date	date	100%	0%
Deaths	int	100%	0%
Recovered	int	100%	0%

Pipeline Results:

Read: 161,568
Written: 161,568
Invalid: 0
Failed: 0
Runtime: 643ms

Dataset 2: Synthetic Messy Data (5,000 records with 2-5% error injection)

Statistical Validation:

Metric	Value
Total Records	5,000
Valid Output	4,772 (95%)
Quarantined	228 (4.56%)
Error Detection	100%

Error Distribution in Quarantine:

Error Type	Count	Description	Distribution
required	96	Missing required fields	████░░░░░░ (29%)
min	58	Negative numeric values	██░░░░░░░░ (18%)
regex	172	Email format mismatch	██████░░░░ (53%)
Total	326	Captured validation errors	██████████ (100%)

Use Cases

1. Data Migration Pipelines

p := intake.New().
    From(source.CSV("legacy_export.csv")).
    Transform(transform.NormalizeHeaders(transform.SnakeCase)).
    Validate(validate.Required("id")).
    To(sink.JSONL("normalized.jsonl"))

2. API Data Validation

p := intake.New().
    From(source.JSONL("api_incoming.jsonl")).
    Transform(transform.TrimStrings()).
    Validate(
        validate.Email("email"),
        validate.URL("website"),
        validate.Min("age", 18),
    ).
    OnInvalid(quarantine.JSONL("rejected.jsonl")).
    To(sink.CSV("clean.csv"))

3. ETL for Analytics

p := intake.New().
    From(source.CSV("raw_events.csv")).
    Transform(
        transform.NormalizeHeaders(transform.SnakeCase),
        transform.ParseDate("timestamp", time.RFC3339),
        transform.ParseFloat("value"),
    ).
    Validate(validate.Required("event_id")).
    OnInvalid(quarantine.JSONL("bad_events.jsonl")).
    To(sink.JSONL("analytics_ready.jsonl"))

4. Data Quality Inspection

src := source.CSV("unknown_dataset.csv")
profile, err := discover.InspectSource(ctx, src, discover.Options{
    SampleSize: 10000,
})

for _, f := range profile.Fields {
    fmt.Printf("%s: %s (confidence %.2f)\n", 
        f.Name, f.Type, f.TypeConfidence)
}
for _, issue := range profile.Issues {
    fmt.Printf("[%s] %s\n", issue.Severity, issue.Message)
}

Technical Specifications

Specification	Value
Go Version	1.23+
License	MIT
Dependencies	Zero (stdlib only)
Memory Model	Streaming (O(1))
Concurrency	Context-aware cancellation
Test Coverage	100% on public API
Race Detector	Clean

Non-Goals (Explicit Design Decisions)

❌ CLI/REPL interface (library-first)
❌ DAG engine or scheduler
❌ Distributed execution
❌ DataFrame abstraction
❌ Airflow/Airbyte clone
❌ Connector marketplace
❌ PDF/ML features

Installation

go get github.com/firfircelik/go-intake

Quality Gate

go test -count=1 ./...
go vet ./...
go test -race -count=1 ./...

All tests pass. Race detector clean. Zero external dependencies.

Introducing oteldoctor: a production-readiness analyzer for OpenTelemetry Collector configs

Firat Celik — Fri, 22 May 2026 14:28:36 +0000

I’ve just released oteldoctor v0.1.0, an open-source Go CLI that analyzes OpenTelemetry Collector configurations before they reach production.

Why I built it

OpenTelemetry Collector configs often start small.

Then they grow.

A few receivers.
A few processors.
A couple of exporters.
Some Kubernetes manifests.
A little bit of batching.
A debug exporter during testing.
A quick HTTP endpoint.
A few attributes added for convenience.

Eventually, that YAML file becomes production-critical infrastructure.

The problem is that a Collector config can be valid YAML, and even acceptable to the Collector, while still being risky for production.

For example:

telemetry may be dropped when exporters fail
the Collector may restart during memory spikes
debug endpoints may be exposed
hardcoded secrets may leak into version control
high-cardinality attributes may increase observability cost
service identity may be inconsistent across environments

That’s the gap oteldoctor tries to fill.

What oteldoctor checks

oteldoctor analyzes Collector configs across six categories:

Category	Examples
Structural	Undefined references, unused components, empty pipelines
Reliability	Missing `memory_limiter`, missing `batch`, retry/queue gaps
Security	Plain HTTP, hardcoded secrets, exposed debug endpoints
Cost / Cardinality	High-cardinality dimensions, missing sampling, debug in production
Semantic Quality	Deprecated attributes, missing service identity
Kubernetes Readiness	`GOMEMLIMIT`, resource limits, probes, exposure risks

Example usage

oteldoctor analyze ./deploy --profile production

Generate SARIF for GitHub Code Scanning:

oteldoctor analyze ./deploy --profile production --format sarif > oteldoctor.sarif

Render the Collector pipeline as a graph:

oteldoctor graph collector.yaml --format mermaid

Explain a rule:

oteldoctor explain OTEL-SEC-202

Install:

go install github.com/firfircelik/oteldoctor/cmd/oteldoctor@v0.1.0

What it is not

oteldoctor does not replace the OpenTelemetry Collector’s own configuration validation.

The Collector can tell you whether a config is syntactically valid and operationally acceptable.

oteldoctor focuses on production readiness: reliability, security, cost/cardinality, semantic convention quality, and Kubernetes deployment risks.

GitHub: https://github.com/firfircelik/oteldoctor

This is the first public release. I’d love feedback from anyone using OpenTelemetry Collector, especially around real-world configs and rule calibration.

LibKill: Scan Your Machine for Compromised npm, pip, and Bun Packages

Firat Celik — Wed, 13 May 2026 10:59:45 +0000

Over the last few weeks, supply chain security has once again become a very real problem for developers.

We keep seeing the same attack pattern:

CI/CD token stolen
→ malicious package version published
→ developers install or update dependencies
→ compromised code lands on real machines

And the worst part is simple:

You may already have one of those compromised packages on your machine without knowing it.

That is why I built LibKill.

What is LibKill?

LibKill is an open-source supply chain security scanner.

It checks your local development environment against a database of known compromised package artifacts.

Currently, it scans:

npm global packages
pip packages
Bun cache

Then it cross-references what it finds against 2,672+ known compromised package artifacts.

If something suspicious is detected, LibKill can help you remove it interactively.

Why I built it

Most developers hear about compromised packages after the damage is already done.

A package gets reported.

A malicious version is removed.

A GitHub advisory appears.

A security company publishes an analysis.

But then comes the real question:

“Is any of this already on my machine?”

That question should be easy to answer.

LibKill is my attempt to make that check fast, local, and simple.

Features

LibKill currently supports:

Scanning npm global packages
Scanning installed pip packages
Scanning Bun cache
Cross-checking against known compromised artifacts
Interactive removal when threats are found
Background daemon mode
Desktop notifications
macOS, Linux, and Windows support
Single binary
Zero runtime dependencies

It is written in Go and distributed as a small standalone binary.

Installation

You can install it with one command:

curl -fsSL https://raw.githubusercontent.com/firfircelik/libkill/main/install.sh | bash

Or build it manually:

git clone https://github.com/firfircelik/libkill.git
cd libkill
make install

Example usage

After installation, run:

libkill scan

To start background monitoring:

libkill daemon

If compromised artifacts are found, LibKill will show the results and offer interactive cleanup options.

Threat coverage

LibKill currently covers known compromised artifacts related to campaigns and threat clusters such as:

Mini Shai-Hulud
Contagious Interview
StegaBin
SANDWORM_MODE
CanisterWorm
W4SP
TeamPCP
and many others

The threat database is built from public security sources including:

Socket.dev
GitHub Advisory Database
OSV.dev
Public security research

Why local scanning matters

A lot of supply chain security tools focus on repositories, lockfiles, or CI pipelines.

That is useful, but it does not fully answer what is already installed locally.

Developers often install tools globally.

They test packages.

They use temporary scripts.

They work across multiple languages and package managers.

They forget what was installed weeks ago.

A compromised package does not need to be in your current production repository to be a problem.

It may already exist somewhere in your local environment.

Project details

Language: Go
License: MIT
Binary size: around 14 MB
Dependencies: zero runtime dependencies
Platforms: macOS, Linux, Windows

GitHub:

https://github.com/firfircelik/libkill

Final note

LibKill is still evolving.

My goal is to make it a practical local security tool for developers who work with open-source packages every day.

If you work with npm, Python, Bun, or modern JavaScript tooling, you should not have to guess whether a known compromised artifact exists on your machine.

You should be able to check it quickly.

That is what LibKill is for.

Feedback, issues, and contributions are welcome.

curl -fsSL https://raw.githubusercontent.com/firfircelik/libkill/main/install.sh | bash

How do you monitor third-party service outages?

Firat Celik — Tue, 12 May 2026 09:35:59 +0000

I’ve been thinking about a simple but painful problem:

When something breaks in production, how do you know if the issue is your own app or a third-party service?

Many apps today depend on services like AWS, GitHub, Vercel, Stripe, OpenAI, Anthropic, Cloudflare, Supabase, Firebase, Resend, SendGrid and many others.

If one of them has an outage or degraded performance, users usually do not care whose fault it is.

They just think your app is broken.

I built a small iOS app called StatusWatch to help with this.

It monitors 80+ public status pages and sends push alerts when services go down or degrade.

It is not meant to replace full observability tools.

It is more like a simple dependency health dashboard for developers, indie hackers and small SaaS teams.

The main question it tries to answer is:

“Is one of the services my app depends on having an incident right now?”

I would love feedback from other developers:

Do you currently monitor third-party status pages?
Would push alerts be useful?
Which services should be added?
Should this stay simple, or also include uptime monitoring?

Website: https://statuswatch.tech
App Store: https://apps.apple.com/us/app/statuswatch-service-status/id6761889971

How I Monitor 80+ Cloud Services in Real-Time (And Get Notified Before My Users Do)

Firat Celik — Sun, 26 Apr 2026 11:30:20 +0000

A deep dive into building StatusWatch; push notifications for service outages, built with Flutter and a real time status aggregation backend.

The moment that started it all

It was a Friday evening. A client messaged me: "The checkout isn't working."

Stripe had been down for 38 minutes. I had no idea.

I did what most developers do opened five browser tabs, checked Stripe's status page, scrolled Twitter looking for complaints. By the time I confirmed the issue and messaged the client back, 45 minutes had passed.

That was embarrassing. More importantly, it was preventable.

I started looking for a mobile app that would just... tell me when my stack breaks. Something that monitors the services I actually use and sends a push notification the moment something goes wrong. I couldn't find exactly what I wanted, so I built it.

This is the story of how StatusWatch works under the hood.

The core problem: status pages are not built for developers

Every major cloud provider publishes a status page. AWS has one. GitHub has one. Stripe has one. They're public, they're (usually) accurate, and almost nobody checks them proactively.

Why? Because checking 10 different status pages manually every time something feels off is not a workflow. It's a punishment.

The real need is simple: when something in my stack breaks, I want to know immediately not when a user tells me.

StatusWatch solves this by polling the official status APIs of 80+ services, normalizing the data into a unified format, and pushing a notification to your phone the moment an incident is detected.

Architecture overview

At a high level, the system has three layers:

[Status APIs] → [Aggregation Backend] → [Flutter Mobile App]

Let me walk through each one.

Layer 1: Polling the status APIs

Most major services expose their status data in one of two formats:

Atlassian Statuspage (the most common) used by GitHub, Stripe, Vercel, Cloudflare, Supabase, Linear, and dozens more. The API returns a structured JSON response with components, incidents, and scheduled maintenances.

GET https://www.githubstatus.com/api/v2/summary.json

A simplified response looks like this:

{
  "status": {
    "indicator": "minor",
    "description": "Minor Service Outage"
  },
  "components": [
    {
      "name": "Git Operations",
      "status": "degraded_performance"
    },
    {
      "name": "API Requests",
      "status": "operational"
    }
  ],
  "incidents": [
    {
      "name": "Increased error rates on Git push",
      "status": "investigating",
      "created_at": "2025-04-10T14:22:00Z"
    }
  ]
}

Custom status APIs — services like AWS, Google Cloud, and Azure have their own formats. AWS Health Dashboard, for example, returns RSS/Atom feeds per region and service. These require individual parsers.

The normalization challenge

Each service returns data in a slightly different shape. To display everything uniformly in the app, I normalize every response into a shared schema:

{
  "service_id": "github",
  "name": "GitHub",
  "overall_status": "degraded",
  "components": [...],
  "active_incidents": [...],
  "last_checked": "2025-04-26T09:15:00Z"
}

This normalization layer is what makes it possible to show a consistent green/yellow/red dashboard regardless of which service you're looking at.

Layer 2: Detecting state changes and triggering notifications

Polling alone isn't enough. The key logic is detecting when a service's status changes — from operational to degraded, from degraded to outage, and back to operational.

Previous state: operational
Current state:  degraded
→ Trigger notification: "GitHub is experiencing degraded performance"

Previous state: degraded  
Current state:  operational
→ Trigger notification: "GitHub is back to normal ✓"

Each service's last known state is stored server-side. On every polling cycle, the new state is compared against the stored state. If they differ, a notification is queued.

Polling intervals

Not all services are polled at the same frequency. The free tier uses a 5-minute polling interval. Pro users get near-instant detection — polling runs every 60 seconds, and for major services like AWS, GitHub, and Stripe, we watch for webhook-based updates where available.

Push notifications

Notifications are delivered via FCM (Firebase Cloud Messaging) for Android and APNs for iOS. When a state change is detected, the backend sends a targeted push to all users who have that service in their watchlist.

The notification payload is minimal and actionable:

Title: ⚠️ Stripe — Incident Detected
Body: Payment API is experiencing elevated error rates
Tap: Opens incident timeline in StatusWatch

Layer 3: The Flutter app

The mobile app is built with Flutter, which gives us a single codebase for iOS and Android with native performance.

The dashboard

The main screen is a flat list of your followed services, each showing:

Service name and logo
Current status (green dot, yellow dot, red dot)
Active incident summary if one exists

The status colors follow a simple convention: green = operational, yellow = degraded performance, red = major outage or partial outage.

Incident timeline

Tapping any service opens its incident timeline a chronological feed of status updates from "Investigating" through "Identified", "Monitoring", and finally "Resolved". This is pulled directly from the status API and gives you the full picture without opening a browser.

Uptime heatmap (Pro)

For Pro users, each service shows a 90-day uptime heatmap — a grid where each cell represents one day, colored by uptime percentage. This gives a quick visual sense of a service's reliability history.

State management

The app uses Riverpod for state management. Each service's status is a stream that updates when the backend pushes new data. The UI rebuilds reactively — no manual refresh needed.

The services we monitor

One of the most time-consuming parts of building StatusWatch was the initial service integration work. Not every service has a clean, documented status API. Here's a breakdown of the current coverage:

Category	Services
Cloud	AWS, GCP, Azure, DigitalOcean, Hetzner, Fly.io, Railway
DevOps	GitHub, GitLab, CircleCI, Render, Linear, Jira
Hosting	Vercel, Netlify, Heroku, Cloudflare, Firebase
Database	Supabase, MongoDB Atlas, Neon, Redis Cloud, Upstash
Payments	Stripe, Paddle, Twilio, SendGrid, Resend
Monitoring	Datadog, Sentry, Grafana Cloud, PagerDuty, New Relic
AI	OpenAI, Anthropic, Hugging Face, Together AI

80+ services total, and the list grows with every release.

What I learned building this

Status APIs are inconsistently documented. Some services have beautifully documented status APIs. Others you have to reverse engineer from their status page's network requests. A few don't expose machine-readable status data at all, which means falling back to HTML scraping fragile, but sometimes the only option.

Push notification timing is everything. A notification that arrives 20 minutes late is almost useless. The entire value proposition is knowing before your users do. This made the polling architecture a first class concern from day one, not an afterthought.

Normalization is harder than it looks. "Degraded performance" on Stripe is not the same operational severity as "degraded performance" on GitHub. Building a useful unified model required understanding each service's incident taxonomy not just their API shape.

Free tier design matters for conversion. The 5-service limit on the free tier is intentional. Most solo developers track 5–8 services. Hitting that limit naturally creates the moment where a user thinks "I need one more" which is where the Pro upgrade conversation starts.

What's next

A few things on the roadmap:

Android widgets glanceable status on your home screen without opening the app
More services always expanding the list based on user requests
Custom webhook endpoints for services not on the official list
Team accounts shared watchlists and notification routing for small DevOps teams

Try it

StatusWatch is free to download. The free tier covers 5 services with 5-minute notification delays enough to see if it fits your workflow before committing to Pro.

If your stack touches AWS, Stripe, Supabase, OpenAI, or Vercel, you'll probably want to know about outages before your users do.

[Download StatusWatch on the App Store]

Built by a Firat Celik. Feedback welcome drop a comment or find me on X [@Bi9x6].

Tags: flutter, devops, sre, devtools, buildinpublic, mobile, backend, cloudinfrastructure

DEV Community: Firat Celik

DuckDB 2.0 vs Polars vs Spark on One Machine: 150M Rows Under a 4.5 GiB Cap

The contenders

Setup

Results at 150M rows

Polars and the window wall

DuckDB under pressure: memory_limit = 1 GB

DuckDB 1.5.6 vs 2.0.0

Where Spark still wins

Failure modes I hit, so you don't have to

Reproduce it

The takeaway

Sources

FlipPrice: project report, side-hustle pricing copilot for vintage resellers

FlipPrice

Can Open Models Price Second-Hand Goods Well Enough for Resellers?

1. The Problem

2. Architecture

Listing generation: Gemma 3 + Ollama

Pricing: Ridge with an optional TabPFN path

API, storage, and validation

3. Initial Results

4. Latency

5. Debugging Notes

Random Forest inverted the expected condition relationship

TabPFN cold-start overhead

Unwanted title preambles

Tests

6. Unit Economics

7. Limitations

8. What Comes Next?

1. Benchmark pricing accuracy

2. Run a real reseller pilot

3. Standardise performance tests

4. Validate TabPFN

5. Test willingness to pay

9. Conclusion

Python 3.15 Is Not a Speed Release for Data Teams. It Is an Operations Release.

The thesis

The antithesis, at full strength

Setup

JIT off vs on

1. Lazy imports fix startup, not throughput

Trap 1: the global switch broke pandas

Trap 2: side-effect imports silently stop happening

Trap 3: the PEP's kill switch does not exist in 3.15.0

Runbook: lazy imports

2. Tachyon: attach to the stuck job by PID, and read the error rate

Runbook: profiling a stuck job

3. .pth import lines become .start files: auditable, not safe

Runbook: startup code

What to do this week

Kafka Just Learned to Be a Queue. Postgres Has Been One All Along.

The thesis

The antithesis, at full strength

What actually shipped (facts, as of October 5, 2026)

The Postgres queue is one SQL statement

Demo A: does SKIP LOCKED actually matter?

Demo B: kill a worker mid-batch

The real advantage: the job and the data commit together

Where the Postgres queue dies: one old snapshot

Demo C: churn 200,000 jobs, then measure

The trap: your worker can be the long transaction

A note on LISTEN/NOTIFY

Where Kafka share groups bite

The Go side: a share group on one partition

The decision rule

Run it yourself

Takeaway

Sources

Two PySpark contracts Staff DEs learn on a bad Tuesday: foreachBatch restarts and applyInPandas skew

Part 1. Design for the restart: foreachBatch, watermarks, and MERGE

Contract 1: foreachBatch is at-least-once

Contract 2: watermark dedupe has a state lease (and you must finish the batch)

Contract 3: MERGE inside foreachBatch is a logical idempotency problem

Part 1 checklist (print this next to the PR)

Part 2. Arrow is not a free pass: applyInPandas vs mapInPandas / mapInArrow

Three APIs, three memory contracts

Skew turns "elegant" into OOM

Prefer batch maps when you do not need whole-group semantics

3. `.pth` import lines become `.start` files: auditable, not safe

Part 1. Design for the restart: `foreachBatch`, watermarks, and MERGE

Contract 1: `foreachBatch` is at-least-once

Contract 3: MERGE inside `foreachBatch` is a logical idempotency problem

Part 2. Arrow is not a free pass: `applyInPandas` vs `mapInPandas` / `mapInArrow`