DEV Community: YuhaoLin2005

pip install self-audit: A Zero-Dependency CLI for AI Output Quality

YuhaoLin2005 — Sat, 27 Jun 2026 00:12:35 +0000

AI agents pass tests while producing sloppy thinking. They say "should work" without evidence. They present partial work as complete. They embellish.

I built a tiny tool that catches this. It checks any text across four dimensions: Completeness, Consistency, Groundedness, and Honesty.

Install

pip install self-audit

Usage

echo "Should work fine. Ready to ship." | self-audit --verbose

Completeness: FIXED

Groundedness: FIXED [should work fine]

FAIL

Zero dependencies. Python 3.8+. Stdlib only. 60 lines of core logic.

The Four Dimensions

Dimension	Question	What it catches
Completeness	Did I answer everything?	Missing requirements
Consistency	Did I contradict myself?	A-and-not-A patterns
Groundedness	Did I show evidence?	"should work" claims
Honesty	Am I honest about limits?	Embellishment, TODO stubs

The dimensions are grounded in Anthropic Constitutional AI framework — Completeness (helpfulness), Groundedness (harmlessness), Honesty (truthfulness), Consistency (rule alignment).

Try it on your own output

After any AI-assisted coding session, pipe the agent text through self-audit before shipping. You will be surprised what it catches.

GitHub: https://github.com/YuhaoLin2005/self-audit
Claude Code skill: https://github.com/anthropics/skills/pull/1361

Single-Modal LLMs Have a Blind Spot. Here's How to Fix It.

YuhaoLin2005 — Fri, 26 Jun 2026 19:56:45 +0000

If you use Claude Code, Cursor, or any AI coding agent, you know the problem: you ask the AI to review its own work, and it says "looks good." Every time.

The AI isn't being lazy. It's sharing the same mental model that produced the code. It literally can't see what's wrong — the blind spots are baked in.

Why "Review this as a senior engineer" Fails

Generic role-playing produces generic findings. The AI fills in what it thinks a senior engineer would say, which is usually:

"Consider adding error handling"
"Maybe add more comments"
"The variable name could be clearer"

Useful? Sometimes. But it misses the real bugs — the ones a human reviewer with a genuinely different perspective would catch.

What Actually Works: Named-Persona Review

Here's the method. It takes 5 minutes and costs nothing.

Step 1: Pick three named people, not generic roles.

Instead of	Use
"Review as a security engineer"	"Review as Linus Torvalds"
"Check for maintainability"	"Check as Ken Thompson"
"Think about the user"	"Think as Steve Jobs"

Why? Because Linus Torvalds has an actual documented philosophy: "Good taste is when the special case disappears. Eliminate the edge case by changing the data structure." That produces different findings than "check for security."

Step 2: Search for their real philosophy before role-playing.

Search: "Linus Torvalds engineering philosophy code review". Take 60 seconds per person. Extract 3-5 actual criteria from their documented words. Now role-play.

Step 3: Each person MUST find at least one issue.

No exceptions. If Ken Thompson finds nothing wrong with your code, you're not thinking like Ken Thompson. Go back and look harder.

Step 4: Run the Feynman check.

After the review, ask: "Would this person actually say what I just said, or am I projecting?" Feynman's actual rule: "The first principle is that you must not fool yourself, and you are the easiest person to fool."

Recommended Starter Pack

For your first try, use these three:

Linus Torvalds — Data structures, logic, correctness. Search for "good taste."
Ken Thompson — Architecture, API design, simplicity. Search for "do one thing well."
Steve Jobs — User experience, first impressions, clarity. Search for "design is how it works."

Two engineers + one product person. Five minutes. You'll find bugs you'd never catch reviewing as yourself.

Why This Works (The Science)

de Bono's Six Thinking Hats (1985) established that parallel multi-perspective thinking produces better decisions than single-perspective analysis. Kahneman's Thinking, Fast and Slow (2011) showed that forcing System 2 (analytical) thinking overcomes the biases of System 1 (intuitive).

Named-persona review is a System 2 forcing function. By grounding each perspective in real, searchable philosophy, it prevents the AI from defaulting to generic "looks good" mode.

This is especially valuable for single-modal LLMs — models that can only process text. They lack the diverse sensory input that helps humans notice different things. Multi-perspective review compensates for that gap.

Try It Now

Take any PR, code review, or document you're working on. Paste this prompt:

Review the following using Named-Persona Adversarial Review:

PERSONA 1: Ken Thompson (Unix philosophy)
- Search: "Ken Thompson Unix philosophy do one thing well"
- Find: 3-5 criteria from his actual words
- Review the code. MUST find >= 1 issue.

PERSONA 2: Linus Torvalds (Linux/git)
- Search: "Linus Torvalds good taste code review"
- Find: 3-5 criteria from his actual words
- Review the code. MUST find >= 1 issue.

PERSONA 3: Steve Jobs (Apple)
- Search: "Steve Jobs simplicity design principles"
- Find: 3-5 criteria from his actual words
- Review the code from a user's perspective. MUST find >= 1 issue.

OUTPUT: Structured report with CRITICAL/WARNING/NOTE findings.
PROMOTION: Issues caught by 2+ personas get severity upgrade.
HONESTY CHECK: Would these people actually say this?

<your code here>

That's it. You just ran a multi-perspective review. No API keys, no agents, no cost. Just better quality.

Go Deeper

Named-Persona Adversarial Review — Full Claude Code skill
open-source-flywheel — Methodology for turning personal tools into contributions
de Bono, Six Thinking Hats (1985) — Theoretical foundation
Kahneman, Thinking, Fast and Slow (2011) — Cognitive science backing

I Let 24 Famous Engineers Review My Methodology. Here's What Happened.

YuhaoLin2005 — Fri, 26 Jun 2026 19:52:04 +0000

I spent tonight building a methodology for turning personal tools into open source contributions. Before publishing it, I decided to let the methodology review itself.

The results changed how I think about AI-assisted code review.

The Method: Named-Persona Adversarial Review

The core idea is simple: instead of asking an AI to "review this code as a security engineer" (which produces generic, shallow feedback), you web-search actual engineers' documented philosophies and role-play as them.

Not "be a security auditor." Be Linus Torvalds, who said good code is when the special case disappears. Be Ken Thompson, who said each program should do one thing well. Be Richard Feynman, who said the easiest person to fool is yourself.

Two engineers + one product person per round. Three rounds minimum. Nine genuinely different perspectives.

The Experiment

I applied this to my own open-source-flywheel methodology. Eight rounds. Twenty-four persona views.

Round 1: Torvalds immediately found a flaw. "Your 'When NOT to Use' table repeats the same rules as your Pre-Flight checklist. Merge them. A good data structure eliminates the special case." He was right. Gone.

Round 2: Feynman asked the hardest question. "You only show successes. Where are the failures?" Without documented failures, it was unfalsifiable — the worst kind of cargo cult science. Added three failure cases.

Round 3: Kahneman called out overconfidence. "Self-assessment of originality is inherently biased. Ask someone else to answer your Pre-Flight questions." Added immediately.

Rounds 4-5: Carmack wanted time estimates. Musk wanted first-principles decomposition. Beck wanted respect for maintainers. All actionable, all added.

Round 6: Bret Victor, Don Norman, and Eric Ries independently flagged the same thing. The methodology had no feedback loop. Three legends of design and lean startup, across different decades, pointing at the exact same gap. When Norman, Ries, and Victor agree, you listen. Added LEARN step.

Rounds 7-8: Brooks confirmed we're not selling a silver bullet. Knuth confirmed the structure. Ritchie and Berners-Lee confirmed it was simple enough.

What Changed

v1 (8.3KB) ➜ v8 (~4KB) while gaining:

Fifth step: LEARN (feedback loop)
Pre-flight bias protection (ask someone else)
Bot-check (verify automated review)
Time estimates per step
Three documented failure cases
24-persona review badge

Why This Matters

Single-modal LLMs have a fundamental problem: reviewing their own output means sharing the same mental model, same blind spots, same tendency to say "looks good."

Named-persona review compensates. By grounding each review in real, searchable, citable philosophy, you force genuine context switches. Ken Thompson actually said "do one thing well." That constraint produces different findings than "review as a senior architect."

The Meta-Lesson

A methodology that can review itself is more trustworthy than one that can't. Mine found 8 bugs in itself before I published it.

Try it: open-source-flywheel. Or just search your favorite engineer's philosophy and role-play. The method is free.

24 personas, 8 rounds, 5 steps. Thanks to de Bono (Six Thinking Hats), alirezarezvani (adversarial-reviewer format), and Feynman (don't fool yourself).

Has Anyone Measured How LLM Output Quality Degrades Across Multiple Compactions?

YuhaoLin2005 — Fri, 26 Jun 2026 18:49:52 +0000

The Observation

After ~70 sessions with DeepSeek V4 (1M context), I noticed something odd. When Claude Code compacts my session, output quality doesn't just go down linearly. There's a moment — usually after the second compaction — where the model briefly gets better. Then it declines and never recovers.

Maybe I'm imagining it. Maybe it's specific to my model, my prompts, my workflow. But I can't shake the thought: what if context compaction has a curve, and nobody has mapped it?

What I Found (Not Much)

I searched for benchmarks that measure multi-round compaction degradation. Here's what exists:

RULER: Measures how performance drops as static input grows longer. Nothing about what happens after you compress and re-compress.
Context Rot (Chroma 2025): 18 models tested, all degrade with more tokens. Again, static.
Multi-turn evaluation: Tests whether models drift across conversation turns. Doesn't touch compaction.

Parameter compression (pruning, quantization) has well-mapped scaling laws. The Lottery Ticket Hypothesis (ICLR 2019) and Compression Laws for LLMs (2025) tell you exactly where the performance peak sits. Context summarization — the thing that happens every time your agent runs /compact — has no such curve.

Why This Might Matter

If the curve is real, you could:

Know exactly when to start a fresh session (before the decline hits)
Compare models on a new dimension: who maintains quality longest across compactions?
Give LLM providers a concrete target: "your compaction quality drops 20% faster than competitor X"

Right now, none of the major benchmark suites (MMLU, HELM, BigBench, RULER) include a "compaction persistence" metric. If context windows keep growing and sessions keep getting longer, this gap gets bigger every year.

What I'm Asking

I built a tiny monitor (compact-counter) and a rough experiment framework — 50 lines of Python, 10 benchmark tasks, 0-5 rubric. It's not polished. It's a starting point.

What I'd love:

Someone with a Claude Opus / GPT-5 / Gemini account to try reproducing this
Feedback on whether the methodology makes sense or is fundamentally flawed
If this is a real thing, ideas for how to measure it properly

I don't have the compute or the stats background to do this alone. But if enough people contribute data points across different models, we might find out whether this curve exists — and if it does, maybe it's useful to more people than just me.

References

Frankle & Carbin, "The Lottery Ticket Hypothesis" (ICLR 2019)
"Compression Laws for Large Language Models" (2025)
RULER: What's the Real Context Size of Your Long-Context Language Models? (COLM 2024)
Chroma Research, "Context Rot" (2025)