How EvalPort's Grader System Works: 11 Types for LLM Evaluation

Adha AK — Tue, 04 Aug 2026 20:59:13 +0000

How EvalPort's Grader System Works

When designing EvalPort, the grader system was the hardest part to get right. Every eval framework has its own way of scoring LLM outputs — DeepEval uses metric classes, Promptfoo uses assertion objects, Inspect AI uses solver functions. We needed a system expressive enough to cover 90%+ of real-world eval needs, but simple enough that any framework could implement it.

The result: 11 grader types that carry their own semantics. A grader isn't just a name — it specifies its parameters, its model, its threshold. An eval suite is self-describing.

The 11 Grader Types

exact_match — Compare output to expected output, optionally ignoring case.
contains — Check if the output contains a substring.
regex — Match against a regular expression.
semantic_similarity — Embed output and expected output, compare cosine similarity against a threshold.
llm_judge — Use an LLM to evaluate the output against a prompt template. The most powerful grader.
json_schema — Validate that the output is valid JSON matching a JSON Schema.
json_path — Extract a value from JSON output using a JSONPath expression, then compare it.
code — Run a function to evaluate the output.
human — Defer to human review.
model_graded — Compare the output to a reference answer using a model.
custom — Escape hatch for graders not covered by built-in types.

How Graders Connect to Test Cases

A test case references graders by ID. Multiple graders can evaluate the same test case. The ResultSet records each grader's score separately.

Why This Design Works

Self-describing: An eval suite carries everything a framework needs to execute it.
Framework-agnostic: Any framework can implement any subset of grader types.
Extensible: The custom type lets frameworks bring their own graders.
Comparable: Results from different frameworks use the same grader IDs.

Try It

pip install evalport-sdk

npm install evalport-sdk

Spec: https://github.com/adhabnr-ux/evalport/blob/main/spec/SPEC.md

Repo: https://github.com/adhabnr-ux/evalport

OpenEval: Why LLM Evaluation Needs a Standard Format

Adha AK — Thu, 30 Jul 2026 03:40:11 +0000

Every LLM evaluation framework today invents its own test case format, its own grader definitions, and its own results schema. DeepEval, Promptfoo, Inspect AI, and lm-evaluation-harness all solve the same core problem (checking whether a model's output is correct) but none of them can read each other's eval datasets.

That means every time a team wants to compare frameworks, or move from a notebook prototype to a production eval pipeline, they end up hand-rewriting the same test cases over and over.

OpenEval is an attempt to fix that by defining a small, portable JSON Schema for eval test cases, graders, and results, plus tooling to move data between frameworks instead of retyping it.

What's in the repo:

A versioned JSON Schema spec for test cases, grader configs, and result records.

TypeScript and Python SDKs for reading and writing OpenEval-formatted datasets.

A CLI with validate, convert, init, and summarize commands.

Converters for popular frameworks so existing datasets can be brought in or exported out without a manual rewrite.

The project just published v1.0.0 to npm and PyPI, and issues are open on 17+ framework integrations if anyone wants to help wire up a converter for a framework not yet covered.

Repo: https://github.com/adhabnr-ux/openeval

Would love feedback from anyone who has hit this same portability problem while switching between eval tools.