My LLM reranker did nothing, and every test passed

#debugging #llm #python

schemagate picks the few tables a text-to-SQL model should see out of hundreds. It ranks them with plain maths first (BM25, embeddings, name matching, fused with RRF). Then, if you give it an API key, a model can reorder the top 20. That second step is the reranker.

I was asked why it uses RRF and no LLM reranking. I said it has one, --rerank, and went to show it working. It didn't work.

What I saw

I ran the same three questions with and without --rerank on claude-haiku-5-5. The table order came out identical all three times.

That can happen if the maths already got it right. So I wrapped the provider to log every call and what came back. The model was being called. It just never gave an answer.

The bug

reply = provider.complete(SYSTEM, prompt, max_tokens=120)

120 tokens is plenty for a reply like 3, 1, 7, 2, 5. It was plenty when I wrote it.

Claude 5 models think before they answer, and that thinking counts against max_tokens. They used up all 120 tokens thinking and stopped before writing a single number. The provider raised an error. The reranker caught it, as designed:

except Exception:
    return list(docs)   # never worse than the maths alone

and handed back the maths order. No crash, no warning, just the same result you'd get without --rerank.

Why the tests didn't catch it

tests/test_rerank.py had good tests. A model that errors leaves the order alone. A model that replies with garbage leaves the order alone. A model can't promote a table the caller isn't allowed to see. All true, all passing.

Every one of them tested that failing is safe. None tested that the model actually got to answer. When the fallback is designed to be invisible, total failure looks exactly like success.

The fix

max_tokens is now 1024. The test suite now has a fake model that refuses to answer below 512 tokens, so if anyone lowers it again, a test fails. And the benchmark now logs whether the model answered on each question, not just the final score.

I checked it on the released package, not my working copy: 1.4.4 from PyPI got 1 of 3 model answers and changed 0 of 3 orders; 1.5.0 got 3 of 3 answers and changed 2 orders.

Was it worth fixing? The numbers

Once it actually ran, I measured it for the first time on public benchmarks. Every database from Spider dev (876 tables) and BIRD dev in one pool, no hint which database a question belongs to. A question counts as a hit if every table the correct SQL uses is in the top 5.

set     questions   maths order   reranked   only rerank right   only maths right
spider  1034        80.3%         96.5%      169                 1
bird    1534        88.6%         96.7%      125                 1

Paired McNemar test, p around 1e-49 on Spider and 1e-36 on BIRD. It's the largest single gain in the benchmarks file.

What this measures: the right tables reaching the prompt, not whether the final SQL was correct. One model. Spider and BIRD are public, so the model may have seen them in training. It only reorders tables the maths already shortlisted, and it sees table names and short summaries, not the answer SQL.

The lesson I'm keeping

If you build a "safe fallback", also log when it fires. Otherwise the feature can be 100% broken and your dashboard still looks fine.

Try it: pip install "schemagate[ai]", then schemagate demo "which customers owe us money" --rerank --provider anthropic --model claude-haiku-5-5. In the MCP server, set SCHEMAGATE_MCP_RERANK=1. Code and full benchmark: https://github.com/ashishsinha1602/schemagate

DEV Community