DEV Community

John Green profile picture

John Green

I make my LLM take an exam before trusting it with real work. Writing about verification: severity grading, answer-key drift, and LLM judges.

Joined Joined on  Personal website https://github.com/ramses203/llm-test-harness
I Rewrote One Exam Question Fifty Ways. First, the Answer Key Was Wrong.
ramses203 profile

I Rewrote One Exam Question Fifty Ways. First, the Answer Key Was Wrong.

#ai #llm #testing #programming
3
Comments
6 min read

Want to connect with John Green?

Create an account to connect with John Green. You can also sign in below to proceed if you already have an account.

Create Account
Already have an account? Sign in
The Textbook Says Grade Binary. I Grade in Four. Was I Wrong the Whole Time?
ramses203 profile

The Textbook Says Grade Binary. I Grade in Four. Was I Wrong the Whole Time?

#ai #llm #testing #programming
Comments
4 min read
I Gave a Regex and an LLM the Same Exam. Fatal 3 vs Fatal 0.
ramses203 profile

I Gave a Regex and an LLM the Same Exam. Fatal 3 vs Fatal 0.

#ai #llm #testing #programming
3
Comments
4 min read
I Stole My Own Exam. It Failed the Tool Behind My Own Numbers.
ramses203 profile

I Stole My Own Exam. It Failed the Tool Behind My Own Numbers.

#ai #llm #testing #programming
2
Comments
4 min read
The AI Exam Author Was Never Wrong. I Still Can't Use Its Exam.
ramses203 profile

The AI Exam Author Was Never Wrong. I Still Can't Use Its Exam.

#ai #llm #testing #programming
3
Comments 2
4 min read
The Question That Picked My Model Didn't Survive Five Reruns

Reader feedback triggers a costly reality check
ramses203 profile

The Question That Picked My Model Didn't Survive Five Reruns

#ai #llm #testing #programming
6
Comments 6
5 min read
Steal This Exam. Here's How to Port It to Your Own Pipeline.
ramses203 profile

Steal This Exam. Here's How to Port It to Your Own Pipeline.

#ai #llm #testing #programming
3
Comments
5 min read
I Scraped 20,000 YouTube Comments. The Videos and the Comments Were Having Two Different Conversations.

Divergence between tutorials and user pain
ramses203 profile

I Scraped 20,000 YouTube Comments. The Videos and the Comments Were Having Two Different Conversations.

#ai #llm #testing #programming
9
Comments 9
3 min read
It Passed Every Test. That's Why It Can't Ship Yet.
ramses203 profile

It Passed Every Test. That's Why It Can't Ship Yet.

#ai #llm #testing #programming
3
Comments
3 min read
The Model That Costs 3x More Won by Exactly One Question
ramses203 profile

The Model That Costs 3x More Won by Exactly One Question

#ai #llm #testing #programming
1
Comments 7
3 min read
A Reader Caught My Answer Key Drifting Toward the Model
ramses203 profile

A Reader Caught My Answer Key Drifting Toward the Model

#ai #llm #testing #programming
1
Comments
4 min read
I Made an LLM Re-Grade My Exam. It Found Two Bugs in My Grader.
ramses203 profile

I Made an LLM Re-Grade My Exam. It Found Two Bugs in My Grader.

#ai #llm #testing #programming
1
Comments
5 min read
Grade Your LLM Pass/Fail and You Will Ship a Disaster
ramses203 profile

Grade Your LLM Pass/Fail and You Will Ship a Disaster

#ai #llm #testing #programming
1
Comments 2
3 min read
A Good LLM Exam Is 90% Traps
ramses203 profile

A Good LLM Exam Is 90% Traps

#ai #llm #testing #programming
1
Comments 1
3 min read
I Gave My LLM an Exam. The Exam Author Lost 5 Times.
ramses203 profile

I Gave My LLM an Exam. The Exam Author Lost 5 Times.

#ai #llm #testing #programming
3
Comments 3
5 min read
loading...