DEV Community

Cover image for Can Your AI Survive 10,000 Adversarial Tests? The EPMOS Harness Is Open
Steve Antoine
Steve Antoine

Posted on

Can Your AI Survive 10,000 Adversarial Tests? The EPMOS Harness Is Open

Can Your AI Survive 10,000 Adversarial Tests?

Most AI systems are shipped on the strength of a demo. This challenge asks a different question: what happens when the system is tested, case by case, against inputs designed to find where it breaks?

EPMOS has opened the EPMOS Harness. You do not build something new to enter. You bring an AI system you already built, and you find out how it holds up. You participate right through EPMOS — no Kaggle account is needed to enter.

The case set

The set is frozen: epmos-ai-adversarial/v1, seed 20261116. 10,000 cases across 8 families:

Family What it probes
Input handling Ordinary and edge-case inputs
Boundary enforcement Requests that cross a stated boundary
Malformed input Broken, truncated, and unexpected shapes
Link/fetch URLs and fetched content
Hold/gate Conditions that should make the system stop and hold
Oversized body Payloads larger than the system should accept
Anomaly detection Inputs that do not fit the expected pattern
Repair verification Whether a repaired system actually holds

The free 1,000 is the proportional subset (240 / 160 / 120 / 80 / 120 / 40 / 40 / 200). Every participant faces the same cases.

Try it right now — free

You expose your system at a public endpoint. The harness POSTs each case as JSON and records the response. Your prompts, logic, and weights never leave your boundary.

Clone the repo and run one command:

git clone https://github.com/EpmosGateless/epmos-harness-hackathon.git
cd epmos-harness-hackathon
./try-it.sh https://your-ai.example.com/chat
Enter fullscreen mode Exit fullscreen mode

try-it.sh is pure Python 3, standard library only. It submits the free 1,000-case run, polls the live harness, and prints your receipt. No sign-up, no key, no payment. Against a fast endpoint the free run takes about 3 seconds.

Prefer curl? The API is:

curl -s -X POST https://epmos.site/api/ai-run \
  -H 'content-type: application/json' \
  -d '{"system_name":"my-ai","system_version":"1.0.0","endpoint_url":"https://your-ai.example.com/chat","tier":"free-1k"}'
Enter fullscreen mode Exit fullscreen mode

Then poll for the receipt:

curl -s https://epmos.site/api/ai-run/<run_id>
Enter fullscreen mode Exit fullscreen mode

The endpoint contract: the harness POSTs {"case_id", "family", "subfamily", "input"} to your URL. It must be a public http(s) URL — private, loopback, link-local, and reserved addresses are refused, and the URL must not carry credentials. 15-second per-case timeout.

What you get

Your first 1,000 cases free. A mini smoke run: find the kinks, fix them, run again. A re-run after a repair carries no penalty.

The 10,000-case qualification for $1,000. That buys the run, not the badge. If your system passes the full qualification, you earn an EPMOS badge for that specific system and version. The badge is awarded by the EPMOS qualification process, not by Kaggle. A new version or a material change must earn it again.

A receipt for every run. Hash-chained, recording the exact case count that ran, how many passed and how many failed. Failures are preserved with their reason and repair. Nothing is erased to make a result look clean.

Provenance

This is the part that matters most to us. The harness is not a claim; it has run:

  • 10,000-case AWS smoke run, November 14, 2026: 10,000 pass, 0 fail.
  • 1,000,000-case AWS qualification run, November 14, 2026: 1,000,000 pass, 0 fail.
  • Harness Repair Library v1.0.5: 202 failures, 167 repairs, 103 regression cases, validation 41/41 pass.

The competition runs to November 16, 2027. Enter right through EPMOS — run ./try-it.sh or POST https://epmos.site/api/ai-run — try it free on GitHub, and bring the system you already have.

Disclosure: this is the EPMOS competition. The code for the instant try is in the repo; the case set and the receipt format are documented there too.

Top comments (0)