DEV Community: Melih Birim

For Structured Data, Retrieval Accuracy Isn't a Model Problem

Melih Birim — Thu, 13 Aug 2026 13:30:03 +0000

MongoDB shipped three retrieval features this week: context-aware embeddings, hybrid search, and a native reranker.

The pitch is the same one every vector database is making right now: retrieval accuracy is the thing that determines whether an agent's next step is right or wrong, and getting it wrong is expensive twice over, once for the bad action and again for the retry.

That framing is correct. It's also a framing built entirely around unstructured data, and it quietly assumes retrieval has to be approximate.

Why retrieval is probabilistic at all

Vector search returns the k passages whose embeddings are closest to the query's embedding.

"Closest" is a distance in a learned space, not a guarantee of relevance.

Two chunks can sit near each other in that space and still not answer the question, which is exactly why reranking exists as a second pass: a bigger model re-scores the same candidates because the first pass wasn't reliable enough on its own.

Hybrid search exists because vector similarity alone misses exact terms like a product SKU or an error code.

Context-aware chunking exists because splitting a document into pieces throws away the surrounding meaning each piece depended on.

All three of those are real, useful fixes. They're also all patches on the same underlying fact*: when your data is unstructured text, there is no query that deterministically identifies "the right answer." You can only get closer to it.*

A table doesn't have that problem

SELECT dept, AVG(salary) FROM employees WHERE region = 'West' GROUP BY dept doesn't return the four passages that are probably relevant.

It returns the rows that satisfy the predicate, with 100% recall and 100% precision, every time, because the question and the data share the same structure.

There's no embedding to be almost-right in the middle of. There's no reranking step, because there's nothing to rerank, the WHERE clause already decided.

This is why csvql's benchmarking has always focused on two things: does the query return the exact same result set DuckDB does, and how many tokens does an agent spend getting that result.

Accuracy was never the axis we had to optimize, because for tabular data, exact retrieval isn't a hard problem someone solved with a bigger model.

It's what a WHERE clause has always done.

Where the two worlds actually meet

None of this means vector search is wrong for what it's for.

A support ticket, a contract, a Slack thread, those are unstructured, and "closest passage" is a genuinely reasonable strategy when there's no schema to query against.

The mistake is applying that strategy to data that already has one.

A CSV of orders, a Parquet file of transactions, a SQL export of user events, these don't need semantic retrieval at all.

They need a query engine an agent can call directly, cheaply, and get back an exact answer.

The token cost of a wrong retrieval compounds the same way in both worlds, an extra round trip, a retry, a chain of tokens that never needed to be spent.

The difference is what fixes it.

For unstructured data, the fix is a better model.

For structured data, the fix already existed.

It's SQL.

One dev.to Comment on a Benchmark Post Turned Into 18 Filed Bugs

Melih Birim — Mon, 03 Aug 2026 21:00:00 +0000

I published a benchmark post comparing csvql against DuckDB on the NYC taxi dataset. Mads Hansen left this comment:

The most valuable result here may be the two correctness bugs, because differential testing gives you an oracle while the implementation is still small enough to reason about. I'd keep DuckDB in CI as a semantic reference and generate adversarial CSV fixtures: quoted newlines, escaped delimiters, empty versus NULL, Unicode, exponent notation, NaN/Infinity, timestamp offsets, and group cardinalities large enough to force spills. Compare normalized result sets and error classes, not just elapsed time.

I checked what csvql's existing correctness suite actually covered. It diffed csvql against DuckDB, but only on business-style data: names, cities, departments, salaries. No quotes, no embedded newlines, no unicode, nothing adversarial. Every check passed because nothing in the fixtures could fail.

What five test rows found

I built a small CSV with a quoted field containing an embedded newline, a doubled quote, and a unicode name, and ran it through csvql instead of the usual clean data. Two bugs showed up before I'd even finished writing the fixture.

LENGTH() on a field like "has ""quotes"" inside" came back wrong. The zero-copy parser found the correct closing quote but never collapsed the doubled "" down to a single ", so the stored value was longer than it should have been.

SUM() was worse. A row with an embedded newline in a quoted field caused the aggregate to silently drop that row's value from the total. No error, no crash, just a wrong number. Plain projection of the same column returned the correct value, so the bug was specific to how the aggregation scan split rows, not how it read them.

Both went into a CI job that runs DuckDB as an oracle on every push, so they can't come back unnoticed.

The second pass found more, and worse

Fixing those two led to a closer look at how CSV quoting was handled everywhere else in the codebase. The same class of bug showed up repeatedly, because quote-handling logic had been implemented separately in several places instead of shared:

The mmap and bulk scanners toggled quote state on any quote character, not just one at the start of a field, so a value like abc"def inside an unquoted field could throw off everything after it.
Header parsing didn't respect quotes at all when splitting column names.
IN (...) list parsing split on raw commas, breaking on any value like 'New York, NY'.
A fixed 8KB stack buffer for DISTINCT deduplication silently truncated longer rows and could falsely collapse two different rows into one.
parseIntFast had no overflow check, so a 20+ digit number could silently corrupt under a release build instead of failing.
An ORDER BY sort key was recorded as a live slice into a growable arena buffer, a genuine use-after-free once the arena reallocated mid-scan.
A duplicate header column name (two columns both called id) silently returned whichever one happened to be inserted into the lookup table last, not the first, with no indication anything was ambiguous.
A UTF-8 BOM, which Excel writes by default when you export "CSV UTF-8", glued onto the first column name and broke every reference to that column.

None of these needed a large file or a strange query. All of them needed data slightly outside what the original test suite had ever generated.

Where it stands

18 issues filed since that comment, 12 already fixed and shipped across two point releases. 6 are still open: a documentation-only issue about intentional differences from DuckDB (byte length vs character count, that kind of thing), a known gap where the four parallel worker threads used for files over 10MB still don't have the same quote-aware parsing as the single-threaded paths, and a handful of smaller ones (SUM/AVG returning 0 instead of NULL on an empty result set, a bare CASE WHEN outside an aggregate failing with a misleading error, a negative LIMIT silently being treated as unlimited).

The parallel-path gap is the one I'm least comfortable rushing. Fixing it means touching four independent threaded scan functions, each with its own I/O buffer boundary handling, and getting that wrong under load is worse than leaving it as a known, documented limitation for now.

The actual lesson

None of this was exotic. It was one reader pointing out that a correctness suite built entirely from clean, well-formed business data will pass no matter how many real bugs are sitting in the parser, because it never exercises the code paths those bugs live in. The fix wasn't a smarter test, it was a meaner one.

If you want to see the current adversarial fixtures or run them yourself:

git clone https://github.com/melihbirim/csvql
cd csvql && zig build -Doptimize=ReleaseFast
./bench/verify_correctness.sh

It runs on every push now. If you find a fixture it should have but doesn't, that's exactly the kind of comment that started this.

I Benchmarked My CSV Engine Against DuckDB on Its Own Dataset. It Found Two Bugs in Mine.

Melih Birim — Thu, 30 Jul 2026 07:03:05 +0000

csvql queries raw CSV ~2.8x faster than DuckDB on an 8 GB file, using ~6x less memory and zero extra disk — and the benchmark that proved it also surfaced two real bugs I'd never have found otherwise.

The setup: query the same raw CSV, on the same machine

DuckDB publishes its own NYC-taxi CSV benchmark. That makes it the perfect yardstick: I can't be accused of misconfiguring the competitor when I'm running the competitor's own dataset with the competitor's own queries.

The dataset lives on DuckDB's own blob storage (blobs.duckdb.org/data/nyc-taxi-dataset). Each file is 20 million rows, 51 columns, ~8 GB uncompressed. The four queries are the canonical "Billion Taxi Rides" aggregations — GROUP BY counts and averages over cab type, passenger count, year, and rounded trip distance.

The one rule that makes or breaks a benchmark like this: both engines must do the same work. DuckDB's headline numbers come in two flavors — "with storage" (after loading the CSV into DuckDB's native columnar format) and "without storage" (querying the raw CSV each time). csvql always queries raw CSV directly; it has no native store. So the fair fight is both engines reading the raw CSV, cold, every run:

csvql:  csvql "SELECT ... FROM 'trips.csv' ..."
duckdb: SELECT ... FROM read_csv_auto('trips.csv') ...   -- NOT a preloaded table

Comparing csvql-on-raw-CSV against DuckDB-on-preloaded-store would be dishonest, and anyone who opened the script would say so. Direct vs direct is the only comparison that survives scrutiny.

Machine: Apple Silicon, macOS. DuckDB: v1.4.2. Method: best-of-5 runs per query, warm OS page cache (both engines equally).

Speed: ~2.8x on 8 GB

Query	csvql	DuckDB	Speedup
Q01 `COUNT(*) GROUP BY cab_type`	1.29 s	3.55 s	2.8x
Q02 `AVG(total_amount) GROUP BY passenger_count`	1.41 s	3.81 s	2.7x
Q03 `COUNT(*) GROUP BY passenger_count, year`	1.36 s	4.01 s	2.9x
Q04 `GROUP BY passenger_count, year, ROUND(distance)`	1.38 s	3.99 s	2.9x

Query latency, lower is better:

        0s        1s        2s        3s        4s
Q01 csvql  ██████▌ 1.29
    duck   ██████████████████ 3.55
Q02 csvql  ███████ 1.41
    duck   ███████████████████ 3.81
Q03 csvql  ██████▊ 1.36
    duck   ████████████████████ 4.01
Q04 csvql  ██████▉ 1.38
    duck   ████████████████████ 3.99

Here's the counterintuitive part: the gap shrinks as files get bigger. On a 417 MB / 1M-row sample, csvql is ~10x faster. On 8 GB, ~2.8x. Why? On small files DuckDB's process startup and CSV-reader initialization dominate the wall clock, and csvql's lean startup wins big. On large files the actual parse-and-aggregate throughput dominates, DuckDB's parallel CSV reader catches up, and csvql settles into a steady ~2.8x. Both numbers are true; the honest thing is to publish the whole curve with file sizes attached, not cherry-pick the 10x.

Memory: ~6x less

Same 8 GB file, peak memory footprint per query:

Query	csvql	DuckDB
Q01	29 MB	178 MB
Q02	30 MB	210 MB
Q03	34 MB	208 MB
Q04	38 MB	219 MB

Peak memory (MB), 8 GB file
csvql   █▍                          ~30 MB
duckdb  ██████████████████████████  ~200 MB

csvql streams the file through a memory-mapped scan and keeps only the group-by hash table in memory — a few dozen groups. It never holds more than a sliver of the 8 GB at once. That's why a 30 MB footprint can chew through an 8 GB file.

Storage: the number that actually matters

This is where "raw CSV, no store" stops being a fairness caveat and becomes the whole point.

Extra disk required to answer these queries on the 8 GB file:

csvql   0 bytes        — queries the CSV in place, no ingest
duckdb  2.1 GB + 21.7s — must build a native store for its fast path

DuckDB can be faster than these raw-CSV numbers — but only via its "with storage" path, which first ingests the CSV into a 2.1 GB native store, a one-time cost of ~22 seconds for a single 8 GB file (and minutes for the full multi-file dataset). csvql needs zero extra disk and zero ingest to hit its numbers. For an ad-hoc query against a CSV that landed on your disk five minutes ago, "no ingest" is the difference between an answer now and an answer after a coffee break.

The honest summary: same answers, ~2.8x faster, ~6x less memory, zero extra disk, zero ingest.

The test I couldn't run — and why that's the honest part

You'll notice I didn't put csvql's numbers next to DuckDB's published figures (their blog reports Q01–Q04 at 2.45 / 3.89 / 5.21 / 11.2 s without storage). It would have looked great — csvql's ~1.3 s next to their 11.2 s on Q04 is a juicy 8x. It would also have been junk.

Two reasons:

Different dataset size. DuckDB's published query numbers are on the full ~1.1-billion-row dataset (all 65 files, ~500 GB of uncompressed CSV). Mine are on a single 20-million-row file. That's a 55x difference in rows. Putting them in the same table would be comparing csvql-on-20M against DuckDB-on-1.1B — the benchmarking equivalent of a fixed fight. (You can see the size gap in their own Q04: 11.2 s on 1.1B rows vs the ~4 s my local DuckDB takes on 20M.)
Different hardware. Their numbers are from an M1 Pro; mine are from a different Apple Silicon box. Borrowing a competitor's number measured on other silicon and dividing it by yours isn't a benchmark, it's a wish.

To legitimately cite DuckDB's published figures, csvql would have to run the identical ~1.1-billion-row dataset — which means ~500 GB of uncompressed CSV on disk. My machine has 275 GB free. It physically won't fit. So instead of borrowing numbers across hardware and data sizes, I ran both engines on the same 20 M rows on the same machine with the same OS cache state. That controls for everything the cross-hardware comparison doesn't. It's a smaller, less flashy result — and a far more trustworthy one.

If someone with a 1 TB disk wants to run the full 500 GB and report csvql against DuckDB's exact published table, the script is one flag away (bench_taxi.sh 65). I'd love to see it.

The plot twist: the benchmark found two bugs in my engine

I built the benchmark to measure speed. It ended up as the best test suite csvql ever had, because running real queries on real data immediately broke two of them:

Bug #1 — GROUP BY ROUND(trip_distance) crashed with ColumnNotFound. Grouping by a string-producing function (STRFTIME) worked; grouping by a numeric-producing one (ROUND) didn't — the group-key resolver simply had no case for it. Q04 wouldn't run at all until I added one.

Bug #2 — ORDER BY COUNT(*) DESC silently returned unsorted rows. No error, no crash — just wrong output. The GROUP BY code path sorted results by the internal hash key and ignored the ORDER BY clause entirely. This is the scariest class of bug: the kind that returns a confident, wrong answer. A benchmark that only checked timings would have sailed right past it; comparing output against DuckDB caught it in one diff.

Both are now fixed, each with a regression test, and the four canonical queries produce byte-identical results to DuckDB (modulo 1 vs 1.0 float formatting). The moral: a benchmark you check against a trusted oracle is a correctness test wearing a stopwatch. If I'd only ever run csvql against itself, both bugs would still be there.

Reproduce it yourself

Everything is in the repo. No 500 GB required — the sample mode pulls ~417 MB and runs in seconds:

git clone https://github.com/melihbirim/csvql
cd csvql && zig build -Doptimize=ReleaseFast

# quick: 1M-row sample (~417 MB)
./bench/bench_taxi.sh --sample

# full: 20M rows (~8 GB download from DuckDB's blobs)
./bench/bench_taxi.sh 1

# memory / CPU / storage instead of speed
./bench/bench_taxi.sh --resources 1

The script downloads DuckDB's own dataset, runs both engines on the raw CSV, and prints the tables above. It's ~130 lines of bash — read it before you trust it.

Takeaways

Benchmark direct-vs-direct or don't bother. The single most important line in the whole exercise was refusing to compare raw-CSV csvql against preloaded DuckDB.
Publish the whole curve. ~10x on 400 MB and ~2.8x on 8 GB are both true. Hiding one to headline the other is how benchmarks earn their bad reputation.
The interesting number isn't always speed. ~6x less memory and zero ingest is a bigger deal for ad-hoc CSV work than shaving two seconds.
Point your benchmark at a trusted oracle. It doubles as the correctness test you were too lazy to write. It found two bugs I'd have shipped.

csvql is open source (Zig, single static binary, github.com/melihbirim/csvql). If you run the full dataset, tell me what you get.

Numbers: Apple Silicon M2Pro, macOS, DuckDB v1.4.2, best-of-5, warm cache. Dataset and queries: DuckDB's NYC-taxi benchmark.