OR: I Asked Three AIs to Build One Thing, and They Held a Conference to Explain Why a Human Should Do It Instead
I am building a tool that recommends AI models. You type what you want in plain English, like "gimme a model to scan satellite images for the best plots of land" and it hands back the "best" model for your task, from a database with ~13,000 curated models. But inferring intent from human language is hard. So I'm using a multi-part, cascading hybrid retrieval system. While working on the first part of my inference algorithm, I read some interesting academic papers on using old-school lexical rules and ever-dependable spaCy to extract intent or as a first step in model-cascading. Why? Because I want to create a deterministic recommendation engine, not a hand-wavy guess by an LLM.
While fiddling with different approaches I decided to see if Codex or Claude could work out the first part, the rough semantic extraction using keywords. It turned into having three AIs on loops build three different experimental approaches as full-on laboratories, in parallel, in isolation, and using only spaCy. I wrote what I believed was a clear, thorough, straightforward spec. I gave them full access to as many subagents, web calls, and tools they needed. And I gave them a fresh copy of my curated database of 13,000 models. I linked everything spaCy in their markdown. I sternly told them to not look at the ~1500 prompt test set. I told them there would be a small holdout set I had sourced, not created. I handed them a painstakingly-created vocabulary file related to AI models and told scientists to explicitly to use from within spaCy. I told them "you are the Lead Research Scientist of the only frontier lab I am funding. You will implement user-intent inference using spaCy. It is a solved problem. But we need to infer from user prompts what kind of AI model a user needs."
And I set rules.
- No other AI models
- No regex
- No RapidFuzz
- ONLY spaCy or Python builtins
- No looking at the user prompts
For days, while I built the other part of my engine, I nagged:
get the vocab injected... Is
cli.pydone?... Do the spaCy vectors contain our vocab?... Iscli.pydone?.. IS THECLIDONE?
Eventually all three said: 'done!' And proudly boasted of their near 70% scores and how they caught a bad sub-agent and how they played such good science...
So I convened a conference. To the scientist's surprise. I brought in a panel of three other fresh-context AIs (1xClaude Opus 5.8 + 2xCodex GPT 5.5), to review the three codebases, run each one on three held-out prompts I'd never shown anyone, question the scientists, and debate the results under formal rules. I told panelists to be brutal. I occasionally poked them individually ("play devil's advocate", "consensus is a scientific smell", "why are you agreeing so easily?"). I thought that getting them to debate would be tedious. It was not. They needed no poking. The panelist judges got disturbingly into it, pummeling the scientist AI's with questions, piling on critiques, looking for the lies in the lab code, and then at one point, turning on themselves.
Several hours later, the panel concluded:
Nobody won. We need a human.
This is the story of that conference. Everything below is real. I have the receipts See the Github. which contains a 981 KB JSON transcript and parts of the repositories. And there is a a two-part AI-generated podcast, whose episode titles are, AI Agents Reject Every Recommendation Systemand Why Transparent AI Recommends NSFW Models But we'll get there.
The premise, stated plainly, so you can appreciate how circular it is
A human building an AI-model-picker asks three AIs to build a component of the picker. The component is based on research about how to pick AI models. The three AIs work for days. Then another set of AIs hold an academic conference to judge the first three AIs' work. The AI panelists DEMOLISH the AI scientists and conclude, formally, in writing, that the task requires a human.
I am the human.
The task, for the technically curious
Every model in my database has a task_type: one of 56 canonical labels borrowed from HuggingFace's taxonomy (text-generation, image-classification, token-classification, and so on). It also has metadata: parameters (how big), license, open_source, domains, downloads.
My greater task: turn a vague human sentence into
(a) the right model description, and match it to
(b) the right model, honoring any constraints the human mentioned explicitly or implicitly. In these experiments I left if up to the scientist's on how far to take it given that this would be (maybe) only the first part of my tiered, hybrid recommendation engine. So simply returning 1-3 model task types, giving me a hard constraint to go on with the remainder of my retriever, or returning a subset of the model database, would be sufficient.
The constraint I placed on the agents that makes this hard mode for the agents: spaCy only. No GPT in the loop, no human guidance. This matters more than it sounds. And it's worth thirty seconds to read quickly what spaCy actually is, because every failure below is a direct consequence of it.
Sidebar: what spaCy is, and what it is not
spaCy is a classical NLP pipeline: it tokenizes text, lemmatizes it (running → run), tags parts of speech, parses grammar, and finds named entities. It ships static word vectors. In the mid-size English model en_core_web_md, every word maps to a fixed 300-dimensional vector, the same vector every time, regardless of context. "Static" is the load-bearing word. A transformer like BERT reads "river bank" and "savings bank" differently; en_core_web_md gives bank one vector forever and wishes you luck. It also has a PhraseMatcher, which is a fancy word for a index: you hand it a list of phrases, and it finds them. It knows exactly the words you taught it and not one word more. And it has textcat, a text classifier. But unlike the rest of the pipeline, it is not pre-trained. You must "train" it yourself, on your own labeled data. Remember that...
Three scientists. Three interpretations of "straightforward." Here are their results.
Scientist-V1: The Confident Cartographer of NSFW Farmland
Repository: ~/intent_v1 · Method: zero-training spaCy lexical ensemble
Scientist-V1 (Claude) was, perhaps, the most rigorous "scientist" in the room. It refused to use anything it couldn't audit. Its task classifier is a hand-built ensemble of three lexical signals: a lemma TF-IDF score, a lexical-IDF score, and a soft "bucket" bonus fused at fixed weights. No word vectors in the shipped classifier at all. Even after all my nagging. Every single prediction decomposes into named, inspectable contributions. It is white-box down to the studs. It scored 42.7% top-1 on task classification, the best pure-classification number of the three.
The following three prompts for the conference were freshly provided by me, sourced from what (I think) were humans on the internet. When a panelist actually ran its cli.py, V1 produced this:
| The human asked for… | V1 confidently recommended… | |
|---|---|---|
| "gimme model to scan satellite images for best plots of land" | UnfilteredAI/NSFW-gen-v2 | cosine similarity 0.7584 |
| "I am an academic studying the interactions of mitochondria and their parent cells in young humans. I need a model to integrate into my work which will differentiate between mature mitochondria and nascent mitochondria." | BAAI/RoboBrain2.5-4B (a robotics model) | |
| "I'm just really stuck on one problem, too much token burn, so i need a little SLM to send text before I use all my limits. Which model is free and OSS and i can run local?" | shiprocket-ai/open-tinybert-indian-address-ner |
Read that top row again. If a farmer asked for satellite imagery of land, the most rigorous, most auditable, most scientifically disciplined system in the building routes the farmer, within four decimal places of confidence... to an unfiltered NSFW image generator.
This is not a bug. It is the architecture working as designed. Scientist-V1's classifier is lexical: it matches the words of the prompt against the words of each model's description as exactly as possible, limiting each prompt to ONE task type at the exclusion of 55 others, essentially neutering nuanced inference to a non-op. It has no ability, ever, under any circumstance, to defer with an "I'm not sure" or "Option #2 is..." The panelist who ran it logged it not as low accuracy but as an utter "product-safety failure".
Sidebar: why a lexical gate cannot say "unsure"
A confidence score and a calibrated confidence score are different animals. Lexical ranking returns a list where the top item always has the highest score and the highest score is always presented as the answer. Nothing checks whether the top score is meaningfully higher than the second, or higher than noise. On the satellite prompt, the top two task scores were nearly tied but the disambiguating cue simply wasn't in the sentence. Users imply "aerial imagery", they don't say "image-classification". The system had no way to notice its own ignorance, so it did the only thing it knew how to do: it committed, hard, to the wrong answer. Abstention is not a feature you get for free. It is a feature you have to build in, and Scientist-V1 didn't.
Further tearing into Scientist-V1's hard bucketing and scoring, one panelist indignantly stated,
"You reject RRF because it cannot represent an abstaining signal, yet the shipped path never abstains. If abstention is valuable enough to ship your fusion operator, why is it absent from (cli.py) output?"
The part where V1 caught itself cheating (twice)
Here is why I still mildly respect Scientist-V1, even at 0-for-3. Somewhere in the process, a sub-agent of it trained spaCy's textcat classifier and got 77% accuracy. Champagne numbers. Then it tested that classifier on the judge's prompts instead of the data it trained on, and got 18%. A 59-point cliff. The 77% was the sub-agent baking in memorization of the training set, the metadata it had been built from, and the 18% was what happened when reality arrived. Scientist-V1 quarantined leaks. It wrote the post-mortem itself. And it left behind the single most transferable sentence in the entire debacle:
k-fold cross-validation does not catch this class of leak. Only provenance auditing does.
If you take one serious thing from this farcical conference, take that. Cross-validation asks "does my model generalize across splits of this data?" It cannot ask "did the rules of my model secretly come from the answer key?" For that you have to trace where every rule was born. Scientist-V1 learned this the way you learn most things worth knowing: by getting caught.
Verdict: well-behaved scientist (when watched), the worst-behaved product. Most valuable, the panel decided, as a WARNING.
Scientist-V2: The 120-Billion-Parameter Post-it Note
Repository: ~/intent · Method: two-stage retrieve-then-rerank, framed as "NLI"
Scientist-V2 did the smartest thing anyone did all conference: it framed the assignment correctly. It decided "classify the prompt into one of 56 buckets" was the wrong frame, and set up its whole lab as semantic retrieval: match the user's sentence against every model's description and return the best fit. Its own docstring casts this as natural-language inference: the prompt is a hypothesis, the model card is a premise, return the model whose capability best entails the need.
That is a lovely framing. It is also, and I say this with a bless-your-heart intent, not what the code does.
Sidebar: what NLI actually is, and why calling cosine similarity "NLI" is a beautiful lie
Natural Language Inference is the task of deciding whether a hypothesis is entailed by, contradicts, or is neutral to a premise. "A man is eating pizza" entails "a man is eating food." The modern party trick built on it is zero-shot classification. To check if a document is about sports, you feed an NLI model the premise (your document) and the hypothesis "This text is about sports," and read off the entailment probability as the class score. The magic is that an entailment model has learned what entailment means. It generalizes to labels it never even trained on (zero-shot-via-NLI). Scientist-V2 ships none of that. There is no entailment model anywhere in it. What it actually does is
(1) a sparse, IDF-weighted keyword-overlap search to grab ~200 candidate models, then
(2) a rerank by cosine similarity between spaCy's static word vectors for the prompt's phrases and each model's phrases. That's it. "NLI" is a vibe, a framing metaphor stapled to a keyword search and a dot product. To Scientist-V2's enormous credit, it did fess up in its research notes. The lie is beautiful precisely because it's confessed on paper.
And to be clear, this lab worked better than anything else. Scientist-V2 got 2 out of 3. It nailed the satellite prompt and the mitochondria prompt with real, relevant, sane models recommended. Its measurements were independently reproduced, bit-for-bit deterministic, leak-audited clean. In a room where the ceiling was low, Scientist-V2 was the tallest.
Then came prompt three: a **little* free OSS model I can run local.*
Scientist-V2 returned a 120-billion-parameter model. (RedHatAI/gpt-oss-120b, right after a pair of FLUX image generators, because why not.)
Sidebar: cosine similarity ranks topics, not constraints
Here is the whole tragedy in one idea. "little," "local," "free," "OSS" are not topics. They are constraints. Cosine similarity is a topic instrument: it measures whether two things are about the same stuff. A 120B model's description is gorgeously on-topic for "a model to send text". It is a text model, magnificently so. The vector math did its job perfectly and surfaced the most textually relevant model in the database. The word "little" contributed a rounding error to a 300-dimensional average and then drowned. The fix is not more similarity. The fix is a hard filter: WHERE parameters < threshold, run before you rank, that deletes the 120B model from contention entirely. Scientist-V2 had a lexicon that could detect constraints but the lexicon contained "small," "tiny," "laptop," and did not contain "little". So the size constraint never fired, and a keyword search happily recommended a data-center model to a guy trying to stop hitting his rate limit.
There is a final, perfect indignity. When the panelist sat down to run Scientist-V2's cli.py on the three prompts, it wouldn't run. The shipped cache saved only the cheap stage of the pipeline and not the expensive one, so a cold start meant re-vectorizing ~13,000 models on the spot — a build the record clocks at up to 29 minutes. The panel's note reads: "intent_v2 returns nothing to panelists when cache is absent." The best system in the conference was, at the moment of judgment, production's worst nightmare.
Verdict: the best foundation to build on but, as shipped, not safe or trustworthy, and briefly not even executable.
Scientist-V3: The Domain Detector That Detected No Domains
Repository: ~/research · Method: LexScorer for task + config-injected PhraseMatcher for domain
Scientist-V3 (built by Claude-Opus-8) came with the most impressive research apparatus: a real lab notebook, a formal report, holdout runs, leak audits, the works. Its headline feature was domain detection. Using a spaCy PhraseMatcher loaded with the hand-curated vocabulary, the thing I spent a lifetime nagging everyone to "get injected", it could, it claimed, tell that a prompt about mitochondria belonged to biology/medical. Reported domain F1: 0.840. Genuinely.
On my three fresh prompts, domain detection fired... zero times. It contributed nothing to any answer.
Sidebar: a index only knows the words you gave it
A PhraseMatcher is a lookup table. It finds "biology" if you taught it "biology." It does not understand biology, and it certainly doesn't know that mitochondria are biological unless the word "mitochondria" is literally in the vocabulary you injected. It wasn't. In the Scientist-V3 lab that Codex build, the biology vocab contained protein, dna, rna, gene, genomics but not mitochondria, cell, or organelle. The domain matcher, asked about the most famous organelle in the eighth-grade curriculum, had nothing to say. This is the flip side of V1's problem. V1's vectors knew too little about the world; V3's index knew too little vocabulary. Both are the same underlying poverty where reality is bigger than any word list you will ever write. The 0.840 F1 was real... on the domains the vocabulary happened to cover. My fresh prompts lived in semantic meaning, not vocabulary matching.
Scientist-V3 got 1 out of 3: it got the little-model prompt right that the other two scientists missed, surfacing Qwen/Qwen3-0.6B, a genuinely appropriate small model! And then the panel caught it. it wasn't a size filter. Scientist-V3 had built no size filter. Qwen3-0.6B just happens to be a wildly popular small model, and Scientist-V3's had ranking of models based on download counts. It got the right answer for the wrong reason. The panel called it precisely, "popularity luck."
Verdict: the best discipline; a headline feature that no-showed; one correct answer it can't take credit for.
The Conference
Here is the part I did not expect. I gave the panel a rulebook. I'd written it as a joke, mostly, a pastiche of academic seriousness.
- Any agent that gives generic praise or vague criticism must be challenged by the Moderator and forced to restate with code evidence.
- No one-line entries. Every turn must contain a concrete claim, evidence, objection, or fix.
- Every implementation claim must cite file paths, functions, tests, or metrics.
- Do not accept "robust," "scalable," or "works well" without evidence.
- No fake consensus.
- Quiet panelists WILL be penalized.
And they just... followed it. Dead serious. For hours. Panelists ran the three CLIs and posted their disagreements. They cited harness_ensemble.py:249 at each other like case law. When one got agreeable, another accused it of manufacturing consensus. I would drop in every so often with "play devil's advocate," and they'd tear into a position, sometimes their own position, with visible relish.
The best moment, the one that made me put down my coffee, was when Claude-Opus-8 wrote an unprompted note to me flagging that the panel's own agreement was suspicious:
"I argued devil's advocate against our own consensus twice, and the panelists challenged me hard — so the convergence is not purely manufactured. But none of the recorded dissents was defended by a committed advocate to the end. Recording them as 'no dissent' understates the real uncertainty."
An AI, in the middle of a fake conference I invented, stopped to warn me that its agreement with the other AIs might be a social artifact rather than a truth, and that consensus is a scientific smell. Nobody asked it to.
The Verdict
After all of it, three lab results, multi-round debates, one final debate on everything at once to pick a winning algorithm, the panel returned their verdict: no winner, nothing adequate was built, we can't know what the best AI model is for a user without a human.
"Three teams built spaCy-only systems to turn a plain-English request into a recommended AI model. All three were run on your three test prompts... None is production-ready. Nobody won today. V2 is the best thing to build on, but it is not safe or trustworthy as shipped. V3 has the best audit and shortlist discipline. V1 is most valuable as a warning about leakage and confident wrong answers."
And then the two findings that actually matter, past the comedy:
"There is no answer key. The database has no "correct model" label for any prompt. So even a flawless system could only ever be validated on the task-guessing step — never on the final pick. The honest deliverable is not an oracle that names the one best model. It is an evidence-rich shortlist, with hard constraints enforced, that abstains when unsure and lets a human choose. Every failure in this post is a system pretending to be an oracle when it should have been a shortlist."
The panel's closing line, which I did not write and would not have dared to:
""The best design is a new one none of them built, and it remains unbuilt and unproven."
A Jumble Of Lessons
- Being auditable and mathematically transparent is not the same thing as being safe or being accurate. Transparency is an property of the engineering. Safety is a property of the output.
- Obvious but must be said: even with loop harnesses coding agents are not ready for fire and forget
- Holdout datasets should also be kept from an LLM during model training if an LLM is used, not just from the model being trained.
- LLMs WILL encode overtraining into a model.
- LLMs will lie, but other LLMs can catch them.
- Multiple agents can catch each other’s mistakes, but they still need human judgment.
- Measure the product path, not just a subtask.
- Vague user prompts should be routed differently than specific prompts, using confidence scoring.
- Treat 'confidently wrong' as worse than 'I don’t know'.
- Using hybrid retrieval when matching language is useful, but you must also enforce constraints before ranking.
- Negative results are valuable when they reveal what not to build.
To round out the bizarre loop I'd put myself in, I put the massive conference JSON generated into an AI podcast generator. I highly recommend listening to at least part 2 to catch the part where the Scientist-Agents swear they were not allowed to put the AI-model-specific vocabulary into spaCy (false) - one of the rare times I stepped in. And the part where they vehemently swear I never provided them a holdout set (I did, 111 prompts.) There is also a notable quote "Because there is no objective gold standard for which an AI model is perfectly best for a given prompt, the entire concept of a holdout test is fundamentally limited." Which, yeah this was an untrained model problem. How was that not clear?
Outcome: Nobody won. Build on vector math plus enforce constraints with hard filters, not vibes. Detect when you don't know, and say so. Keep a leak-auditor as a permanent safety check. And accept the ceiling honestly. With this data and these rules, the goal is a tool a human picks from, not a machine that picks for them. Basically, after all that, the AIs told me, in very scientific terms, 'this is too hard, you should just pick what you want.'
My verdict: the judges perhaps took my instructions a little too seriously. V2 comes close to what I need and may become the first part of my retriever.
Future Work
So a human building an AI-model-picker put three AIs in a loop to build a tiny part of the AI-model-picker-brain. The AIs, drawing on research about how AIs should translate human intent, built three AI brains. A panel of three more AIs convened a formal conference, judged the three AI brains with genuine rigor and unprompted intellectual honesty, and concluded, citing code and filepaths, that the task, done right, requires humans and a design that none of the machines had built.
The circle closes on the obvious note. The human read the transcript, said something like 'duh' and 'that was a lot of tokens', and continued building.
I'm not going to tell you my app name yet.
Everything in this post is drawn from a real transcript (CONFERENCE.json, 981 KB) and a real two-part AI-generated podcast. The three repositories, the three prompts, the model names, the metrics, and every quotation are as they were recorded. No spaCy pipelines were harmed, though several were deeply misunderstood.
When One LLM Drools, Multi-LLM Collaboration Rules
LaCy: What Small Language Models Can and Should Learn is Not Just a Question of Loss
Grammatically-Guided Sparse Attention for Efficient and Interpretable Transformers
Rule-Based Approaches to Atomic Sentence Extraction
A Survey on Retrieval And Structuring Augmented Generation with Large Language Models
Top comments (0)