Erik Hill

AI evaluation & testing engineer. I build the tests that catch AI when it's wrong: deterministic graders, not a model grading itself. 34 public repos; shipped a game to Google Play.

Charleston, SC Joined on Jul 15, 2026 https://erikhill.dev

Work

AI/ML Engineer at Epix Devo LLC

Pinned

Erik Hill

Aug 9

Your eval suite passes. Does it actually check anything?

#showdev #testing #ai #python

3 min read

Stats prove simple prompts beat complex panels

Erik Hill

Jul 25

I built a tool to prove my multi-agent harness was worth it. It told me it wasn't.

#showdev #ai #llm #testing

4 min read

Erik Hill

Jul 24

My LLM drift tracker flagged four regressions this week. All four were wrong.

#ai #llm #monitoring #testing

4 min read

Erik Hill

Jul 23

The AI Crash Test: adversarial LLM testing you can audit in the Network tab

#ai #llm #security #opensource

3 min read

Machine-checked proof and human-gated actions

Erik Hill

Jul 20

I built an AI dev harness that isn't allowed to trust itself

#showdev #ai #llm #programming

4 min read

Erik Hill

Oct 10

I wrote down which test would catch each bug. The one aimed straight at it was blind.

#debugging #java #softwareengineering #testing

5 min read

Want to connect with Erik Hill?

Create an account to connect with Erik Hill. You can also sign in below to proceed if you already have an account.

Create Account

Already have an account? Sign in

High false positive rates on real repositories

Erik Hill

Oct 8

I shipped a linter for checks that cannot fail. It had four of them.

#showdev #testing #cicd #python

8 min read

Erik Hill

Oct 3

The gate could not see what it was gating

#showdev #testing #ai

4 min read

Erik Hill

Sep 30

My arithmetic grader reads the first number in the sentence, not the answer

#showdev #testing #ai

5 min read

Erik Hill

Sep 23

Twice the data, less power: my stability rule got blinder the harder I looked

#showdev #testing #ai

4 min read

Erik Hill

Sep 23

My factual-recall tasks were scoring format, not facts

#showdev #testing #ai

4 min read

Erik Hill

Sep 23

My mutation score was 1.000 in two places and the denominators disagreed

#showdev #testing #python #devops

3 min read

Erik Hill

Sep 18

Three tools said my resume was clean. One of them was not installed.

#showdev #testing #bash #career

4 min read

Erik Hill

Aug 31

My board never scored an outage as a regression. My evidence couldn't prove it.

#evaluation #testing #llm #opensource

5 min read

Erik Hill

Aug 22

Five Wrong Root Causes, and the Fifth Was in the Post

#macos #debugging #wine #testing

4 min read

Erik Hill

Aug 16

One capital letter made my verifier call a failing score clean

4 min read

Erik Hill

Aug 14

I built an answer key for eval suites: six models broken on purpose, exactly.

#showdev #testing #llm #opensource

2 min read

Erik Hill

Aug 14

Your eval suite passes. I built the tool that checks whether it checks anything.

#showdev #testing #llm #opensource

2 min read

Erik Hill

Aug 4

My AI gate tests were green theater. The fix was to stub the wire — and nothing above it.

#testing #ai #playwright #typescript

5 min read

Erik Hill

Aug 1

My determinism test passed for months while the two builds played different games

#testing #java #android #webdev

5 min read

Erik Hill

Jul 22

301 duplicate IDs in the browser, 0 on the JVM: one real bug, end to end

#showdev #testing #java #webdev

5 min read

Erik Hill

Jul 21

Distinguishing wrong from absent

#machinelearning #llm #ai #testing

1 min read

Erik Hill

Jul 19

Set membership is not pairing: a property test that was green for the exact bug it was written to catch

#testing #python #softwareengineering #llm

4 min read

Erik Hill

Jul 16

The Cold-Context Critic: A Reviewer That Never Remembers Writing the Code

#ai #agents #codereview #softwareengineering

4 min read

Erik Hill

Jul 15

A differential oracle: making agentic code prove its own correctness

#ai

2 min read

DEV Community

Erik Hill

Badges

4 Week Community Wellness Streak

2 Week Community Wellness Streak

1 Week Community Wellness Streak

Writing Debut

Skills/Languages

Currently learning

Currently hacking on

Available for

Pinned

Your eval suite passes. Does it actually check anything?

I built a tool to prove my multi-agent harness was worth it. It told me it wasn't.

My LLM drift tracker flagged four regressions this week. All four were wrong.

The AI Crash Test: adversarial LLM testing you can audit in the Network tab

I built an AI dev harness that isn't allowed to trust itself

I wrote down which test would catch each bug. The one aimed straight at it was blind.

Want to connect with Erik Hill?

I shipped a linter for checks that cannot fail. It had four of them.

The gate could not see what it was gating

My arithmetic grader reads the first number in the sentence, not the answer

Twice the data, less power: my stability rule got blinder the harder I looked

My factual-recall tasks were scoring format, not facts

My mutation score was 1.000 in two places and the denominators disagreed

Three tools said my resume was clean. One of them was not installed.

My board never scored an outage as a regression. My evidence couldn't prove it.

Five Wrong Root Causes, and the Fifth Was in the Post

One capital letter made my verifier call a failing score clean

I built an answer key for eval suites: six models broken on purpose, exactly.

Your eval suite passes. I built the tool that checks whether it checks anything.

My AI gate tests were green theater. The fix was to stub the wire — and nothing above it.

My determinism test passed for months while the two builds played different games

301 duplicate IDs in the browser, 0 on the JVM: one real bug, end to end

Distinguishing wrong from absent

Set membership is not pairing: a property test that was green for the exact bug it was written to catch

The Cold-Context Critic: A Reviewer That Never Remembers Writing the Code

A differential oracle: making agentic code prove its own correctness