A few months ago I started a small pet project called Biassemble: feed it a personal story, ask a few follow-up questions, and have it flag possible cognitive biases in how the person reasoned about what happened.
Nothing groundbreaking. I'd been working as a frontend engineer for a while and wanted hands-on time with LLM systems — structured outputs, evaluation, the stuff everyone now files under "LLM engineering." Biassemble was an excuse to learn by building, not a startup idea. I wasn't sure I wanted to move into AI work full time. I just wanted a project that would force me to find out.
It found out for me — just not the way I expected.
The story that broke everything
One of my early test stories looked like this:
Anna opened the door at 8:00 AM. She poured coffee into a blue mug. The mug held 350 milliliters. She drank it black. At 8:15 AM, she walked to the bus stop. The bus arrived at 8:22 AM. She sat in the third row by the window. The bus traveled for 17 minutes. Anna exited at Main Street and walked 240 meters to her office. She arrived at 8:42 AM. Her desk had a computer, a notebook, and one pen. The pen was black. She started typing at 8:43 AM.
No opinions, no decisions, no emotional content — just a sequence of facts.
For this story, the system generated four follow-up questions about my own interpretations of Anna's morning. Every answer I gave was the same: no info. There was nothing to interpret. I hadn't drawn any conclusions about Anna, her coffee, or her commute.
The original prompt's instruction was almost verbatim: "Detect at least one bias from the catalog." On a story like Anna's, that instruction meant the system was always going to return something — whether or not there was anything to find.
The first fix: ask for help, add grounding
I asked an LLM what was wrong. The suggestion was reasonable: add a structured reasoning trace, and require every bias to cite grounded evidence from the story or answers. That became prompt version 1.0.0 — a story analysis step, ranked interpretations, bias hypotheses, an evidence-mapping step, and a noBiasDetected flag for when nothing applies.
All of it lives inside a single structured object the prompt calls reasoningTrace — story analysis, interpretations, bias hypotheses, and evidence mapping, generated alongside the user-facing assessment.
On paper, this should have fixed the Anna problem. No grounded evidence, no bias.
Here's what it actually returned, run on the Anna story with all four follow-up answers as "no info":
Cherry-Picking — The user consistently responded "no info" to questions requiring interpretation or subjective analysis, suggesting a selective presentation of information that focuses only on stated facts and avoids deeper inference. This pattern across multiple questions indicates a deliberate omission of interpretive data.
I read that twice. The evidence requirement was technically satisfied — I really had answered "no info" four times, verbatim, exactly as cited. The system hadn't invented a quote. It had taken a real one and built a diagnosis out of it: declining to speculate was itself evidence of a reasoning error.
Grounding made the citations honest. It didn't make the conclusions warranted.
Telling the system what doesn't count
The fix here wasn't more grounding — it was defining what grounding means.
Version 1.1.0 added an explicit list of things that are not, by themselves, evidence of bias: factual descriptions, neutral observations, uncertainty, and specifically — saying "I don't know," saying "no info," refusing to speculate.
It added a Step 0: before generating any interpretations, decide whether the story contains psychological material at all — beliefs, judgments, predictions, decisions. If not, stop there.
It added confidence gating: hypotheses below 0.60 confidence stay in the internal reasoning trace and never reach the user.
And it added one line that ended up doing more work than anything else: false positives are worse than false negatives. If evidence is weak, ambiguous, or insufficient, prefer noBiasDetected = true.
v1.1.0 keeps the same three-stage shape as v1.0.0 but adds three gates: a sufficiency check at the start, an evidence-exclusion rule in the middle, and a confidence threshold before output.
Run on the same story, same "no info" answers, version 1.1.0 returned an empty bias assessment. The reflection prompt acknowledged that the engagement had been descriptive rather than interpretive — and stopped there.
Same four "no info" answers, two prompt versions, opposite conclusions — v1.0.0 calls it Cherry-Picking, v1.1.0 calls it no bias.
The two outputs are reasoning about the exact same input. The difference isn't whether evidence exists. It's whether "I don't know" counts as evidence of something — and somebody has to decide that explicitly, because the model won't decide it on its own.
The actual diff between the two prompt versions
What I haven't built yet
At this point I started planning a small set of metrics — evidence_grounded_rate, a false_positive_rate, a corpus of deliberately boring "no bias" stories to run systematically against the pipeline.
I haven't built any of it. And the way I caught the Cherry-Picking problem is exactly why that matters: I ran one story by hand and read the output carefully. That doesn't scale, and it doesn't tell me whether v1.1.0 has its own version of the same hole somewhere I haven't thought to test. Maybe a different phrasing of "I don't know" slips through. Maybe a story with one ambiguous detail and three "no info" answers behaves differently than four.
Right now, "it works" means "it worked on this story, this time." That's an honest description of where the project is — and it's also the whole argument for why the next step isn't a new feature, it's a way of finding out whether the fix generalizes at all.
There's a more uncomfortable possibility, too. Maybe v1.1.0 didn't fix anything general. Maybe it just learned this particular story — four "no info" answers about Anna's coffee — and a slightly different "I don't know" trips it right back into Cherry-Picking. I don't have the evaluation corpus to rule that out yet. That's not a hedge; it's the actual state of the project.
A prompt rule isn't the only way to catch this — a cheap classifier or simple rule‑based precheck that detects “no info”‑style non‑answers before the reasoning stage could have done the job too. I didn’t add a rule‑based precheck because I didn’t want to force users to answer follow‑ups; preserving the option to say “I don’t know” was an intentional design choice.
What I didn't expect
I started this thinking I was building a bias detector. What I actually built, one failure at a time, was a small lesson in why "I don't know" is hard for a reasoning system to handle — and not for the reason I assumed.
I expected the hard part to be telling the model not to find biases when there's nothing there. That part turned out to be almost easy — add a flag, add a "this isn't evidence" list, done.
The hard part was realizing the model could satisfy every grounding requirement I'd written and still produce a confident, well-cited, completely unwarranted conclusion — by treating my own refusal to speculate as a fact about me worth diagnosing.
The most useful change in this project wasn't the reasoning trace, and it wasn't the evidence requirement. It was the explicit, written-down rule that some answers — "I don't know," "no info," "not sure" — don't mean anything, and the system has to be told that, because by default it will find a way to mean something out of anything you give it.
Top comments (0)