Can You Beat an LLM? Building Humans vs. Humanity's Last Exam

#ai #llm #hle #benchmark

Frontier models are crushing benchmark after benchmark...so I built a quiz to ask this simple yet humbling question: can a human still beat them?

Humans vs. HLE is a web-based quiz that pits you against frontier AI models on Humanity's Last Exam HLE is a benchmark of expert-level academic questions designed to be insanely challenging. You answer a series of multiple-choice questions (if you miss 4, you're out and the quiz ends!) and your score is stacked up both against how the best models perform on the same material as well as against other humans who have taken this quiz via a persistent human leaderboard.

Spoiler: it's harder than it looks.

What it does

Expert-level questions! Pulled from the HLE dataset, these are not trivia-night questions as they're designed to stump even frontier models.
You vs. the models. Your score is shown alongside frontier-model accuracy so you can see exactly where you stand.
A human leaderboard. Every score persists so there's a running ranking of how we mere mortals do!

Track it with Entire

Entire tracks prompts, outputs, and tool calls. You can see some of my prompts here.

You can see I used Claude code Opus 4.8, edited 27 files, and added 912 lines of code (and removed 239) for that checkpoint/feature.

How it's built

The entire (see what I did there) app runs on Cloudflare Workers. Here are a few decisions I want to call out.

Questions baked in at build time

Rather than fetch questions at runtime, they're curated offline from the gated cais/hle Hugging Face dataset into src/questions.json and then imported directly by src/questions.ts so the bank ships inside the Worker bundle. This means no database or external fetch, and that the deployed Worker never calls Hugging Face or agi.safe.ai at request time. This means the app is fast to serve and that nothing sensitive travels over the wire.

Keeping the answers un-cheatable

The interesting engineering challenge in a quiz app is: how do you stop people from just reading the answers in the network tab?

The approach keeps correct answers server-side the entire quiz lifecycle. GET /api/quiz/new shuffles each question's choices, strips the answers, and hands back an AES-GCM–encrypted answer-key token that's opaque to the browser. POST /api/quiz/answer decrypts it server-side to grade one question at a time. POST /api/quiz/submit returns an HMAC-signed result token — so a score forged in the client simply won't verify. The client gets what it needs to render a question, but never the answer key, and scoring happens where it can't be tampered with.

Durable Objects for the leaderboard

The leaderboard and result signing use Cloudflare Durable Objects, which are my go-to dev tool for consistent, stateful coordination (ie. a persistent leaderboard).

Built with Entire

The entire development process (again, see what I did there) was tracked with Entire, so every architectural decision (why encrypted tokens, why Durable Objects) has its reasoning captured alongside the code, not lost to a closed chat window.

Project structure

The pieces that matter, and what each one owns:

scripts/
  build-questions.py   # curate cais/hle → questions.json (offline)
src/
  questions.json       # 140 text-only MCQs, baked in
  questions.ts         # imports the JSON into the bundle
  index.ts             # Worker + the /api/quiz/* routes
  models.ts            # published HLE accuracy + AS_OF date

The API surface: /api/quiz/new, /api/quiz/answer, /api/quiz/submit, and /api/questions for the browse page.

Running it locally

Standard Workers setup — install, drop in two secrets, and go.

npm install
cp .dev.vars.example .dev.vars
# then set both values in .dev.vars, e.g.:
#   printf "QUIZ_TOKEN_KEY=%s\n" "$(openssl rand -base64 32)"
#   printf "RESULT_TOKEN_SECRET=%s\n" "$(openssl rand -base64 32)"
npm run dev

Tests:

npm run test

Deploy:

wrangler secret put QUIZ_TOKEN_KEY        # an openssl rand -base64 32 value
wrangler secret put RESULT_TOKEN_SECRET   # a different one
npm run deploy

One honest note on the scores

The AI numbers aren't the models sitting your exact ten questions live. They're each model's published overall accuracy on the full HLE benchmark, sourced from Scale AI's leaderboard (with an AS_OF date in models.ts). So it's you-on-a-sample versus them-on-the-whole-thing — close enough to sting, honest enough to admit.