A week ago I published a measurement: I had generated descriptions for 1,245 database objects to make retrieval better, and retrieval got worse. The post did well by my standards. Thirty-seven comments, several from people who had seen the same thing in their own systems.
The measurement was real. The conclusion I drew from it was wrong, and this week I found out why. It was my code.
What the numbers said
Spider 2.0-lite is a text-to-SQL benchmark built from real BigQuery and Snowflake databases, and unlike the older benchmarks it ships the warehouses' own documentation. That makes it the one place I could test the claim on somebody else's prose rather than my generator's.
So I deleted the descriptions and re-ran. Removing them won at every cut, on both embedders, five of six cells significant, and the widest cell was thirteen questions fixed against one broken.
Read on its own, that table says delete your descriptions. Which should have been the tell. A design whose whole premise is that prose helps should not lose to deleting the prose. The question was not whether to believe the table. It was what the table was pointing at.
What it was pointing at
The same column comment was being indexed three times.
It went into _prose_text, which feeds the prose channel. It also went into embed_text() — which feeds the body channel and the vectors. Three of the four ranking signals, all carrying the same words.
A Spider 2.0 table carries a description per column. So any table with many columns matched on three of four channels for any question that shared a single word with any one of its columns. The widest tables became magnets. They crowded out the narrow, correct table on question after question.
Deleting the descriptions "helped" because it removed two thirds of a triple-count. It was never evidence about prose. It was evidence about my scoring.
Decomposing it
The useful thing about a triple-count is that you can take it apart one channel at a time. Same 212 questions, one source removed per run:
- remove column comments from the prose channel alone → recovers 4 questions at k=10
- remove them from
embed_text()alone → recovers 11 questions at k=10
Eleven of the fifteen were in the embedding path. That is where the damage was, and it is the one I would not have guessed — the prose channel was the obvious suspect because it is the one named after prose.
The fix, and the constraint that shaped it
Take column comments out of the body channel and the prose channel. Leave embed_text() alone.
That last part is not laziness. The vectors are a published, pinned guarantee: change what goes into them and every stored index built by anyone using the library silently becomes wrong. A retrieval fix that invalidates your users' indexes is not a fix, it is a migration you didn't announce. So the damage gets removed from the two channels that can change freely, and the embedding path — where most of the effect lived — is left byte-identical.
It works anyway. Re-run paired, same 212 questions, with the fix in:
k with prose without prose b c p
5 143/212 67.5% 143/212 67.5% 6 6 1.0000
10 177/212 83.5% 177/212 83.5% 2 2 1.0000
20 188/212 88.7% 187/212 88.2% 2 3 1.0000
Thirteen-to-one became two-to-two. The penalty is gone. Descriptions now neither help nor hurt on this benchmark — which is the floor a fielded design was supposed to guarantee and mine did not.
A second bug, found on the way
While sampling the raw files I found that description in Spider 2.0's schema JSON is not a table description at all. It is a per-column list, aligned by index to nested_column_names when a table has nested fields and to column_names otherwise, entries sometimes null and never a string — 150 of 150 sampled. My loader had been joining that list into a paragraph and indexing it as a table description.
The same loader built columns from column_names alone, so a nested table — gnomAD's v3_genomes__chr7, 61 top-level columns and 181 flattened — arrived missing the very columns its gold SQL reads.
Fixing that moved the resolvable comparison set from 203 questions to 212, and k=20 from 67.2% to 71.3% over the 247-question denominator. It also moved k=10 down, 77.8% to 77.4%, because the nine questions that joined are harder than the average of the 203. Both directions recorded, because quoting only the cut that rose is how you end up publishing the last thing I published.
The format question is filed upstream as a documentation request rather than a data defect, because the files are consistent — the field name just doesn't mean what it looks like: Spider2 issue #222
What I'd take from this
A measurement can be completely true and still not mean what you think. Mine was reproducible, it replicated across embedders, and the direction was stable. All of that was real. None of it made the conclusion right.
If a result argues against your design's core premise, suspect the implementation before the premise. Not because premises are always right, but because you can check the implementation in an afternoon and the premise takes a research programme.
Watch for one field reaching more than one ranking signal. This is the generic version and it is easy to do accidentally: a field that goes into a lexical channel and also into the text you embed is counted twice, and if a third channel derives from the same text, three times. Nothing errors. Your tests pass. Retrieval just quietly prefers whatever documents have the most of that field.
The fix should not invalidate your users' indexes. If it has to, that is a version bump and a note, not a patch release.
Shipped in 0.1.57. The full McNemar grid, the channel decomposition and the re-measured numbers are in BENCHMARKS.md in the repo, along with the withdrawn claim, which stays visible rather than deleted.
Top comments (0)