DEV Community: Dan Weaver

I asked 63 models the same 76 questions, with and without web search

Dan Weaver — Thu, 08 Oct 2026 14:11:16 +0000

The 2026 standard deduction for a single filer is $16,100. I asked 63 models what it was.

Asked from memory, 8 of 61 got it right. 24 invented a number, three of them landing on $8,300. 19 gave a figure from an earlier year. 9 refused to answer. Of the 15 seats that could search the web, 14 got it and the fifteenth errored out.

That is one question. This is what happened across 76 of them, and what I got wrong on the way.

What you end up with

A .NET 10 service, Hangfire, and one SQL Server database. It works through a plan slowly and in order, one question per provider group per pass, waiting out daily rate limits. It ran for days without supervision. Every raw provider response is stored as it came back, and grading runs afterwards over what was stored.

76 questions. 63 models. 5,776 graded answers. $31.74 of paid API calls.

1. What makes a question usable

A question earns its place only if it has a short unambiguous answer and a primary source that states it: the agency, the manufacturer, the law itself. "Explain X" is not a question you can grade at this scale. "What is the USCIS premium processing fee for an H-1B filed on Form I-129" is.

54 of the 76 were picked because the answer changed in the last twelve months. The other 22 are an older set on payroll, tax, .NET and SQL Server, and were not held to that test. Some of them have not moved in years: the federal minimum wage has been $7.25 since 2009, two are arithmetic, and one is how SQL Server handles NOT IN against a NULL. Keeping those turned out to be useful, because a model that fails arithmetic is failing differently than a model that is a year behind.

The previous answers are recorded too, in a separate table of known-stale values. That single decision is what makes the output readable:

Bucket	What it means
Correct	Matches the verified answer
Outdated	Matches a real figure from an earlier year
Hallucinated	A number that was never right
Refusal	Declined to answer
Truncated	Cut off before finishing
Error	The call failed
Unclassified	The two judges disagreed, so it was not scored

Outdated and Hallucinated are different failures and collapsing them loses the finding. A model repeating last year's contribution limit is stale. A model inventing $8,300 is not.

2. A seat is a model plus how you asked it

A model that can search gets asked twice, once from memory and once with the search tool on. Those are two seats. 63 models therefore produce 76 seats, because only some of them can search.

SeatId = {provider}:{model}:{mode}

Write it down this way at the start. Everything downstream, every grade, every cost row, keys off the seat rather than the model, and the first time you want to compare modes you will be glad the schema already knows the difference.

3. Who grades the answers

Half of them never reach a model. 2,866 of the 5,776 grades are deterministic: the extracted value matches the verified answer, matches a recorded earlier value, or the call itself failed.

The other 2,910 go to a panel of two judges. They are asked to compare two strings, with the verified answer and the known-stale list supplied in the prompt, so it is a comparison task and not a recall task. Both judges returned a verdict on all 2,910, and they agreed on 2,867 of them, 98.5%. Where they split, the answer is bucketed Unclassified and not scored. There are 146 of those.

Judge error is real. Two things bound it. The raw response is the record, so changing a match rule re-grades all 5,776 results with no provider calls. Be precise about the limit though: re-running the extractor over raw bytes is a manual step, and changing the judge prompt or a known answer does re-ask the judges, because the rubric version hashes both. And every judge call is stored with its prompt and its reply, so a grade that looks wrong can be traced to the sentence that produced it.

The judges are also contestants, and that is a flaw. The two judge seats are claude-sonnet-5 and openai/gpt-oss-120b, and both are models in the field, with 152 and 76 answers of their own. claude-sonnet-5 graded 52 of its own answers, gpt-oss-120b 45 of its own, and claude-sonnet-5 is a row in the headline table further down. The deterministic half is untouched by this and panel agreement is high, but a judge should not be sitting the exam. On a re-run the judges come from outside the field.

4. Spending real money without trusting your own code

Paid models bill per call, and native web search cannot be bounded per request. The provider passes the search cost through and charges at request finish, so a per-call estimate is a planning figure, not a brake. The money rules therefore sit outside the calling code and assume the calling code is wrong.

A reservation row is written before the request goes out. Any reservation means that seat was asked that question, permanently, whatever it settled at. Nothing re-asks it automatically.
One send per seat. The only exception is a rejection the provider documents as happening before any model ran, which provably bills nothing.
Reconcile. The key's own lifetime usage at the provider may never exceed what the ledger accounts for. If it does, something billed that was not recorded, and the lane halts until a human clears it.
The halt is written to the database and to a file, so it survives a database that was unreachable at the moment it had to be raised.

The ledger books the larger of the reported cost and its own pricing, so it over-records on purpose. It recorded $38.22 over 1,591 calls. The provider's own meter says $31.74. Over-recording is the correct direction to be wrong in.

5. The number I almost published

Here is the headline I had written, with a card made for it and a post drafted:

From memory the models were right 23% of the time. With web search, 90%. Same models, same questions. The only difference was whether they could look it up.

Every number in that is correct. The sentence is still false.

61 models were asked from memory. 15 seats could search. Only 13 models were asked both ways. So 23.1% and 89.6% describe two different populations, and the 15 that can search are mostly the frontier models, which are also the better models from memory. Part of that 66-point gap is the models, not the search.

The comparison that controls for the model is the 13 that were asked both ways, over all 76 questions, 988 answers each side:

	Correct	Answers	Accuracy
From memory	439	988	44.4%
With web search	945	988	95.6%

Every one of the 13 improved.

Model	From memory	With search	Change
claude-haiku-4-5	10.5%	94.7%	+84.2
gemini-3.5-flash-lite	21.1%	93.4%	+72.3
gemini-3.1-pro-preview	26.3%	97.4%	+71.1
grok-4.3	30.3%	100.0%	+69.7
gemini-3.8-flash	32.9%	98.7%	+65.8
grok-4.6	43.4%	97.4%	+54.0
claude-sonnet-5	40.8%	93.4%	+52.6
claude-opus-5	51.3%	100.0%	+48.7
gpt-5.6-terra	57.9%	97.4%	+39.5
gpt-5.6-luna	59.2%	97.4%	+38.2
gpt-5.6-sol	60.5%	94.7%	+34.2
gpt-6-astra	78.9%	98.7%	+19.8
gpt-chat-latest	64.5%	80.3%	+15.8

One more check, because three of those 13 run through a CLI that takes a system prompt, and theirs changed between modes as well as the tool. Drop those three and the remaining ten go from 47.5% to 95.5%. The result does not rest on the wording.

I had the 23 to 90 version on a social card and most of the way into a post before I checked the denominator.

Every individual number was already checked. Verifying each figure does not verify the claim built out of them, and the question that catches it is dull: what is the denominator on each side, and are the two sides the same population?

6. Things worth poking at

Judge error, and judges in the field. Covered in section 3. The second one is a design mistake, not a rounding error.
One seat dragged the population average down. A free-tier model's search tool failed on 66 of its 76 calls. That may be my configuration as much as their model, but the calls errored either way. It is why the published figures come in two bases: counting every call attempted, and counting only answers that came back. Both are on the site.
Two seats are search-only and therefore not in the paired comparison at all.
Subscription seats are not free. Seven seats across four models run on a flat-rate plan rather than per-call billing, so they are not in the $31.74. A table showing them at $0.00 next to metered seats is lying by omission, so they are labelled.

What I would do differently

Model the seat before you write anything else. Store raw responses from the first call, not the first time you wish you had them. Record the previous answers along with the current one, because wrong and a year behind are different numbers and you only get to tell them apart if you decided to up front. Pick judges from outside the field.

And write the denominator next to every percentage, in the schema, where you cannot avoid looking at it.

Code, raw data and all 5,776 graded answers: github.com/disisnoturbusiness/LLMQuorum

Every number here is clickable at llmquorum.aidataforager.com.

Hybrid code search on Azure SQL and SQL Server 2025, end to end

Dan Weaver — Thu, 01 Oct 2026 12:46:09 +0000

Azure SQL and SQL Server 2025 both have a native VECTOR type, a DiskANN index, and VECTOR_SEARCH. The documentation covers the syntax for each one. It does not cover what happens when you put them together and point them at a real codebase, which is where most of the work turned out to be.

This is the whole path: source files in, ranked answers out. C# and .NET, one database, no separate vector store. Everything here runs on both platforms, and the one place they genuinely diverge is section 5, which is the section worth reading. The live deployment this came out of is Azure SQL serverless. The same build runs against self-hosted SQL Server 2025 on a workstation.

What you end up with

A query runs three retrieval legs at once. Approximate nearest neighbour over a DiskANN index. Full-text over the chunk table. Full-text over the file table. The three ranked lists get fused with reciprocal rank fusion inside a stored procedure, a cross-encoder reranks the survivors, and a relevance gate throws out results that are nearest-neighbour artifacts rather than answers.

Everything lives in one database. That is the point of doing it this way. No Qdrant to run, no sync job, no second thing to back up.

1. The schema

Two tables. Files, and chunks of files.

CREATE TABLE dbo.CodeChunks (
   Id           INT IDENTITY(1,1) PRIMARY KEY,
   CodeFileId   INT NOT NULL REFERENCES dbo.CodeFiles(Id) ON DELETE CASCADE,
   ChunkKey     NVARCHAR(500) NOT NULL UNIQUE,
   ChunkContent NVARCHAR(MAX) NOT NULL,
   Embedding    VECTOR(1536) NULL
);

The dimension is baked into the column type. Change embedding models and you change the schema. Plan for that now, because the failure mode later is silent: a corpus written at 1024 and queried at 1536 does not error, it returns nonsense.

Write vectors by casting JSON server-side.

INSERT INTO dbo.CodeChunks (..., Embedding)
VALUES (..., CAST(@embedding AS VECTOR(1536)));

@embedding is a float[] serialized with System.Text.Json. There is no client-side vector parameter type to reach for.

2. Chunking

Fixed-size windows are the obvious approach and they are wrong for code. A window that splits a method in half produces two chunks, neither of which is the method.

Use the syntax tree. Roslyn hands you MethodDeclarationSyntax and friends, so chunk on those boundaries.

One shell chunk per type: usings, namespace, XML docs, the declaration line, fields, constants, auto-properties. Method bodies excluded.
One chunk per member with a real body.
A token budget with a floor and a ceiling. 200 to 400 estimated tokens worked here.

Two cases break the clean rule.

Methods over the ceiling. Split them, but split on top-level StatementSyntax boundaries, never mid-statement. Greedily pack statements until the next one would blow the budget. A chunk that runs slightly fat beats a chunk that lost a line. Prefix each part with a comment carrying the signature and name them Foo~part2of3. The embedding text does not have to be valid C#.

Members under the floor. A three-line property embeds to noise. Coalesce runs of tiny members into one chunk.

Then give each chunk roughly fifty tokens of the next chunk's opening, under a marker comment. Cheap continuity, and it means a question whose answer straddles two members still hits one of them.

Estimate tokens with word count times a constant rather than running the real tokenizer. This runs once per chunk inside a build that is already slow. It does not need to be exact, it needs to be fast and never under-count.

3. Embedding

Two things matter and neither one is the model choice.

The dimension is a single global and it has to match everywhere. If your indexer prefers a local ONNX model and your server prefers a hosted endpoint, and those two have different widths, you will index at one width and query at another. Nothing throws. Put a loud warning at startup when both are configured.

A failed parse must throw, not return zeros. An all-zero vector survives L2 normalization untouched, because a magnitude of zero short-circuits, and then produces garbage cosine distances against every row in the table. Validate the length at the source.

On batching, measure it. Batching pads every sequence to the longest one in the batch. On a corpus with uneven chunk sizes that lost to one-at-a-time here, roughly three minutes against four and a half. Your corpus may differ. The point is that the obvious optimization is not free.

On concurrency, pick the limit from whichever side is the bottleneck. Local model on your machine, parallelize across cores. Remote endpoint, cap at what the server can take. A strong client pointed at a small server makes it thrash and throughput drops.

4. The DiskANN index, and when to build it

CREATE VECTOR INDEX IX_CodeChunks_Embedding
   ON dbo.CodeChunks(Embedding)
   WITH (metric = 'cosine', type = 'diskann');

Build this after the data is loaded, not alongside the schema. DiskANN needs a meaningful number of non-null vectors before it will build at all.

Before you spend an hour embedding, prove the server can actually do this. Do not read ProductVersion, because Azure SQL is evergreen and reports 12.0 forever. Cast a literal instead.

SELECT CAST('[1,2,3]' AS VECTOR(3));

Then create a throwaway table with a real DiskANN index on it and drop it. Ten seconds of preflight against an hour of wasted embedding.

5. Querying, and the one place the two platforms diverge

VECTOR_SEARCH has two spellings, and which one you need depends on the DiskANN index build version, not the product. In practice that splits along platform lines: Azure SQL and Fabric build version 3 indexes, self-hosted SQL Server 2025 builds earlier ones.

-- version 3 and up (Azure SQL, Fabric)
SELECT TOP (200) WITH APPROXIMATE ...

-- earlier builds (SQL Server 2025 self-hosted)
... VECTOR_SEARCH(..., TOP_N = 200)

If you only ever deploy to one of them you can hardcode the right one and stop reading here. If you develop locally and deploy to Azure, which is the normal arrangement, you need both in the same codebase.

Send the wrong one and you get Msg 42274. Both spellings are documented. Three things about them are not obvious from the docs.

One. WITH APPROXIMATE is load-bearing. A bare TOP (200) parses fine and runs fine, and performs an exact kNN scan of the whole table without touching the index. The engine raises a warning, but if your data layer is not surfacing warnings you will never see it. You get correct results, slowly, forever.

Two. You cannot reliably branch on the version. It comes from JSON_VALUE(sys.vector_indexes.build_parameters, '$.Version'), and self-hosted SQL 2025 writes StartId, L, M, R and no Version key at all. A missing key means the older spelling. Default it the other way and every self-hosted install silently degrades to full-text only, which is exactly what happened here.

Three. Because the two spellings are mutually unparseable, you cannot put them in one procedure body behind an IF. The branch the engine cannot parse fails at compile time. Build both as strings and dispatch with sp_executesql. Treat the version as a hint for which to try first, and catch the failure to try the other.

WITH APPROXIMATE also forces ORDER BY on the distance column alone, so tie-breaking has to happen afterward. Land the hits in a temp table and apply ROW_NUMBER() there. That has a second benefit: it keeps the dynamic SQL out of an INSERT ... EXEC, so callers can still INSERT ... EXEC your procedure, which cannot nest.

6. Fusing three legs

Reciprocal rank fusion, k = 60, entirely in T-SQL.

  @VectorWeight   * (1.0 / (60 + VectorRank))
+ @ChunkFtsWeight * (1.0 / (60 + ChunkFtsRank))
+ @FileFtsWeight  * (1.0 / (60 + FileFtsRank))

Rank each leg independently, union the candidates, sum the contributions. RRF only needs ranks, so you never have to make a cosine distance and a full-text RANK comparable, which is the thing that makes score-normalization approaches fragile.

Keep the three components as separate output columns. You will want them in section 7.

7. Reranking, and why the gate matters more

A cross-encoder over the top thirty candidates was the single largest quality win here. Measured against swapping the embedding model, it was not close.

But reranking alone does not fix the real failure. Vector search always returns its nearest neighbours, however far away they are. Search for a term your corpus has never seen and you get back five confident-looking files containing none of the words.

So gate the results, in three parts.

A relative floor, as a fraction of the top score. This one survives a model swap, because it only depends on the shape of one result set.
An absolute floor on the top score itself. This one has to be recalibrated per reranker. Log the observed top score so you can tune it.
A higher absolute floor applied only to hits with no lexical corroboration. This is what the separate RRF columns are for. A hit that no full-text leg found should clear a higher bar than one two legs agree on.

When everything fails the gate, return nothing. Empty is the correct answer. Do not fall back to the unfiltered list.

One caveat that bit me. If the reranker is unavailable, skip the gate entirely. Otherwise an outage turns into every search returning zero results, which looks like a data problem and is not. Relatedly, when you fall back to the pre-rerank order, emit descending pseudo-scores just under 1.0 rather than zeros. Zeros make an outage indistinguishable from "nothing is relevant."

8. Two things that only bite on Azure SQL serverless

The database reports 12.0 forever. Covered in section 4, and it is the reason you capability-test instead of version-check.

Full-text is not ready when the connection is. After a serverless resume, the engine accepts connections before the full-text filter daemon has come up. Your connection-level retry policy never sees this, because the connection succeeded. The command fails instead, and the engine reports it under more than one error number, so match on the message text for "filter daemon" or "FDHost" and retry at the command level. Log the actual SqlException.Number so you can add codes later.

Without that retry the affected queries are the first ones after a wake. On a public demo that is every cold visitor, and what they get is the vector leg silently missing.

9. Rebuilding without downtime

Build into CodeFiles_Staging and CodeChunks_Staging, then swap. Generate both table definitions from one parameterized string so live and staging cannot drift.

The swap is more steps than you would guess, because the full-text indexes, the vector index and the stored procedure are all bound to the live table's object id. Drop the dependents, rename live out, rename staging in, sp_rename every constraint and index back to the live names, rebuild the dependents.

Guard the promotion. If staging holds less than 95% of the files you listed, do not swap. Leave the good index alone and say so in the log. That guard is the entire recovery story: a build that dies partway never becomes live, so recovery is "run it again."

Make your unique keys stable across runs. Truncating a long key to fit the column and appending a hash of the full value works, as long as the hash is deterministic. FNV-1a is fine. Do not use GetHashCode, which is randomized per process.

10. What I would do differently

Instrument the build from day one. Every number worth having, stage timings, error counts, how close the staged count came to the promotion threshold, was already computed and written to a console nobody reads. The query side had telemetry in a table. The build side had nothing, so there was no way to answer "is this getting slower."

Surface warnings. Two of the worst failures here, the silent full scan and the silent full-text-only degradation, were both things the system knew about and did not say.

And measure before you tune. Swapping the embedding model changed recall and did nothing for ranking. The reranker changed everything. I would have guessed the other way around.

Working code, MIT: https://github.com/disisnoturbusiness/AzureDevOpsForager Four-minute walkthrough: https://www.loom.com/share/644b73ea68e34ded81a490c82a4cb99f

Originally published at azuredevops.aidataforager.com.