DEV Community: Aditya Verma

AI Evals 101: How to Actually Know If Your AI Is Any Good

Aditya Verma — Wed, 05 Aug 2026 17:57:59 +0000

If you have ever shipped an AI feature and then crossed your fingers hoping it works, this post is for you. An eval is a structured test that measures how well your AI system performs: its quality, reliability, and correctness across a range of scenarios. Think of it as the unit test of the LLM world, except the thing you are testing is probabilistic, moody, and changes every time a new model drops.

Why Bother with Evals?

Teams that invest in evals see four big payoffs. First, they cut development time because you get rapid iteration cycles and can test locally across multiple LLMs instead of eyeballing outputs one at a time. Second, they reduce costs because automated evals replace slow manual review and let you release faster. Third, they enhance quality because real-time monitoring and compliance checks reduce risk and improve the customer experience. Finally, they scale teams because a good eval setup lets non-technical collaborators contribute to building the best possible product experience.

Greg Brockman captured the idea in a memorable line:

// Detect dark theme var iframe = document.getElementById('tweet-1733553161884127435-223'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1733553161884127435&theme=dark" }

The point is not that prompts, observability, or human judgment stop mattering. It is that once an AI system can produce plausible outputs, the hard part is often knowing whether your next change made it better.

The mental model has three pillars. Prompt and context engineering give you a place to prototype behavior. Evals answer the single most important question in AI development: did I just improve things, or did I regress? Depending on what you measure, a result might be a number, a pass or fail, a category, or written feedback. AI observability tells you what is actually happening in production, so when something breaks you can turn the failure into a reproducible test instead of relying on a guess.

A Useful Model for Most Evals

Most offline evals can be understood through the same three parts, even though frameworks give them different names.

The target is the code, prompt, model, or workflow you want to evaluate. Some tools call this a task, application, predict function, or system under test. It can be as small as one model call or as large as an entire agent workflow. The important part is that the same input can be run through different versions so you can compare their behavior.

The dataset is your collection of test cases. A case usually includes an input and may also include a reference answer, expected properties, a rubric, metadata, tags, or other context. The exact schema depends on the framework and the behavior you are testing. A reference is not always necessary: safety, style, latency, and many other criteria can be evaluated without one exact ideal output.

The evaluator is the logic that measures the target's behavior. Some frameworks call it a scorer, grader, metric, or judge. It can be deterministic code, a model-based judge, a human review, or a combination of them, and it can return a number, boolean, label, ranking, or structured feedback.

Offline vs. Online: Two Eval Mental Models

Two common modes are offline and online evaluation, and mature teams often use both.

Offline evals run on curated or historical test cases, usually during development, in CI, or before a release. The point is to catch regressions, compare versions, and understand tradeoffs under controlled conditions. You can run them with a local script, a test framework, a notebook, or an evaluation platform; the workflow is the same even when the interface changes.

Online evals measure selected production interactions as the system runs. They often operate on traces, logs, user feedback, or sampled traffic and focus on signals that do not require a known reference answer, such as safety, policy compliance, relevance, latency, or user satisfaction. Tracing and evaluation are related but different: tracing records what happened, while an evaluator judges some aspect of it. Interesting production cases should be reviewed and curated before they become offline test cases.

The Eval Framework: A Simple 2x2

Here is a mental framework worth memorizing. When you compare an output with an evaluator's result, there are four possibilities:

Output quality	Evaluation result	What it means
Good output	Positive	Everything is working
Bad output	Negative	The eval caught a real problem, so improve the AI app
Good output	Negative	The eval is too strict or measures the wrong behavior
Bad output	Positive	The eval rewards a failure and needs improvement

The interesting quadrants are the mismatches. A good output with a negative result, or a bad output with a positive result, means your evaluation criteria need improvement.

The practical takeaway is to create a baseline and start iterating immediately. Do not wait around for a perfect golden dataset that never arrives.

Target Patterns: From Model Calls to Agents

The simplest target is a single model call. Its prompt may combine instructions with variables such as the user's question, retrieved context, or conversation state. Template syntax varies across libraries, so the important contract is the data you pass in, not the braces used to represent it.

System: Answer using the supplied context.
User question: {question}
Context: {context}

Single-call targets are useful for testing instructions, examples, output formats, retrieval context, and model choices in isolation.

For multi-turn scenarios, evaluate a sequence of messages or an entire conversation. This is a good fit for measuring whether a chat experience remembers context, follows changing instructions, and stays coherent across turns.

Tool-using systems add another layer. The evaluator may need to inspect not only the final answer but also tool selection, arguments, permissions, intermediate results, cost, and latency. The tool can be implemented in any language or service; what matters is the observable contract between the model and the tool.

Agents and multi-step workflows combine model calls, tools, state, routing, and control logic. Evaluating them usually means scoring both the outcome and the trajectory: whether the system reached the right result in a safe, efficient, and reproducible way.

Datasets: Start Small, Never Stop

Datasets collect your test cases so you can run repeatable evaluations and track improvements over time. Three tips make the difference between a dataset that helps and one that rots:

Start small and iterate. Focus on building a feedback loop rather than a perfect dataset.
Never stop iterating. Use production logs to capture new edge cases and make your evals more holistic over time.
Implement human review to establish ground truth, especially when you rely on an expected-output field.

Evaluators: The Spec for Your Project

There are two common families of automated evaluators.

Code-based evaluators handle anything deterministic: exact matches, numeric comparisons, structured checks, or factual checks. Something as simple as the following counts:

output === expected ? 1 : 0

They can also run in your CI pipeline. Techniques like schema validation, exact match, string-similarity measures such as Levenshtein distance, numeric tolerances, and binary checks fit here too.

Model-based judges handle work that needs semantic or subjective interpretation, such as relevance, tone, completeness, or improvement across drafts. A judge prompt can be as simple as: "Does this response contain an apology? Return PASS or FAIL and briefly explain why."

Two common judge patterns are:

Direct assessment, where you design a rubric.
Pairwise comparison, where the judge picks the better of two outputs and a ranking algorithm sorts overall quality. This is particularly useful for subjective tasks.

A few hard-won tips for evaluators:

Use a judge capable enough for the criterion, and keep its configuration separate from the system being evaluated.
Treat evaluators like real judges evaluating intent, style, and overall quality, not just correctness.
Break evaluation into focused criteria such as accuracy, creativity, safety, and formatting so you can pinpoint exactly what broke.
Calibrate judge prompts against representative examples with human labels before trusting them.
Do not overload the judge with context. Keep it focused on the relevant input and output.

One of the most important ideas here is that evaluators are an executable part of the spec for your project. Writing criteria that reflect your users and failure modes is essential because generic metrics rarely capture everything your product actually needs.

Human in the Loop: When the Worst Case Matters

Not all AI mistakes are equal. In high-stakes industries like healthcare, finance, or legal tech, a single failure can mean regulatory violations or real user harm. That is why teams in these domains invest heavily in clear ground-truth definitions, use human subject-matter experts as annotators, and apply human-in-the-loop workflows to refine their scoring.

In these contexts, human review is not a nice-to-have. It is essential for catching hallucinations, establishing ground truth, and ensuring alignment with business, compliance, and user expectations. Automation misses nuance. Real people identify mistakes, label correct outputs, and make sure the final product actually meets human needs.

Human-in-the-loop work comes in two forms. Human review uses internal experts to manually label, score, or audit outputs. It is useful for building high-quality ground-truth datasets, auditing edge cases, and calibrating model-based judges. One example is legal subject-matter experts defining exactly how a legal AI assistant should respond.

User feedback is the implicit or explicit signal from end users during real usage. Thumbs up or down, flags, comments, corrections, and helpfulness ratings can trigger human review or flag production traces for dataset curation.

The formula is simple: automated testing for scale plus human judgment for nuance creates a north-star experience.

Spans: The Bridge Between Production and Testing

A trace records one end-to-end execution of your application. Within it, a span represents a unit of work such as a model call, retrieval step, tool invocation, or business-logic function. Spans can be nested to show how the smaller operations contributed to the overall result. The exact names and boundaries depend on how you instrument the application.

Why do traces matter for datasets? When your instrumentation records the relevant fields, traces can preserve the input, output, intermediate steps, and context behind real behavior. When something goes wrong, such as the AI rephrasing a user's question badly, you can save the relevant input and trace as the basis of a new test case. The failed production output is evidence of the problem, not the expected answer. Add a corrected reference, rubric, or explicit property that defines what success should look like, then use that curated case as a regression test.

The Flywheel Effect

This all comes together in a cycle worth internalizing:

Instrument the app and run it in production.
Collect traces, operational metrics, evaluator signals, and user feedback.
Find important examples, such as a user thumbs-down, policy failure, or unusual trace.
Curate those examples into test cases with a corrected reference, rubric, or expected property.
Fix the code, redeploy, and repeat.

Every turn of the flywheel makes your evals more realistic and your product more robust.

Best Practices: Playing Offense with Evals

The biggest mindset shift is to use evals to play offense. Evals are not just a safety net for catching regressions. They are what lets you launch better products, faster and with confidence.

A few principles follow from that. Great evals need to be intentionally designed, not bolted on. Treat datasets as maintained engineering assets: record provenance, version meaningful changes, deduplicate near-copies, protect sensitive data, and keep labels consistent as the collection grows. Prompt engineering is also evolving into context engineering. System instructions are only one part of what a production model sees; conversation history, retrieved documents, tool definitions and results, memory, and policy context can dominate the input. Evaluate the whole context assembly process, not just the prompt text.

Finally, remember that everything changes when a new model comes out. Model-agnostic evals are your defense against the nondeterministic nature of LLMs. They let you swap models and release efficiently and intentionally. Optimize the entire evaluation system, not just the prompts. Once your evals are solid, you can even close the loop entirely and use them to automatically improve your prompts.

Wrapping Up

Evals turn AI development from vibes into engineering. Start with a target, representative test cases, and evaluators that express what good means for your users. Run offline evals while you build and monitor selected quality signals after you ship. Curate production failures into regression cases, keep humans in the loop where the stakes are high, and treat your evaluation criteria as a living part of the product spec. Do that, and "did I improve or regress?" stops being a guess and becomes a question you can answer with evidence.

Building a Multimodal Agentic Pipeline for Educational Content Ingestion

Aditya Verma — Sat, 25 Jul 2026 07:28:13 +0000

Educational publishers have decades of valuable content locked inside worksheets, textbooks, and scanned assessment papers. Converting it into structured, interactive learning material is harder than it looks: a single document can mix questions in multiple languages, inconsistent mathematical notation, diagrams that are essential to the question, multi-column layouts, and imperfect OCR. Doing this manually is slow and error-prone, and plain OCR only solves the easy part: it extracts text without understanding question boundaries, formulas, answer choices, or educational intent.

The motivation is concrete. A structured question set ingested into Wayground works two ways: it can be printed as a proper classroom worksheet, or played live as a formative assessment. Publishers wanted both from the same source material, with every question tagged by Depth of Knowledge (DoK) so a teacher can control the rigor of what they hand out. That dual use is why structure is non-negotiable. A printed worksheet forgives loose formatting, but an interactive assessment needs clean question boundaries, machine-readable answers, and explicit metadata.

We built a publisher content ingestion system that handles two related jobs:

Extract existing questions from worksheets and assessment documents.
Generate new questions from textbooks and other reference material.

Both share the same foundation but need very different reasoning and quality control. This post walks through how each pipeline works and what we learned building them.

Why OCR Isn't Enough

A PDF is a visual representation of a document, not a semantic one. Consider a worksheet containing the expression:

(x^2 - 4) / (x - 2)

A basic OCR system might produce:

x2 - 4
------
x - 2

or:

x^2 - 4 / x - 2

These outputs are textually similar but mathematically different. The same ambiguity appears everywhere: chemistry superscripts and reaction arrows carry meaning, physics diagrams define the geometry needed to answer a question, and biology labels can be part of the question itself. Language adds another dimension: a Gujarati or Spanish worksheet may mix English scientific terminology, mathematical notation, and locally formatted instructions on one page.

A reliable ingestion system therefore needs more than OCR. It needs a sequence of specialized reasoning steps.

System Overview

The platform is a FastAPI application with LangGraph orchestrating the two workflows. LangGraph carries a typed state object through each pipeline, holding raw text, detected subjects, extracted questions, and quality scores. This makes runs easy to inspect, pause, and retry, and keeps agents from relying on information hidden inside prompts.

The supporting services:

Mistral Document AI for OCR and document parsing
GPT-4o for structured extraction, formula conversion, and image understanding
Portkey for model routing across providers
SymPy for deterministic mathematical verification
PostgreSQL for document, question, image, and processing metadata
A publishing layer that converts the internal representation into the platform schema

Part 1: Extracting Existing Questions

Extraction is not transcription. For each question, the system must reconstruct the author's intended structure: the stem, instructions, answer choices, correct answers, associated images, formulas, subject, language, and question type. We implemented this as an eight-stage sequential workflow.

1. OCR extraction

Pulls raw text, page structure, images, and language signals, and deliberately preserves as much source information as possible. Cleaning too early destroys useful signals like column structure and the relationship between an image and nearby text.

2. Structure analysis

Determines what kind of document this is (question numbering, section headings, multi-column layouts, repeating answer-choice formats) and picks a chunking strategy. Arbitrary token-based splitting can separate a question from its answer choices or detach a diagram from the text that references it. The useful question is not how many tokens fit in a context window, but what the smallest self-contained educational unit is.

3. Subject classification

Detects the subjects present, which drives later processing: a chemistry document needs different normalization rules than a biology worksheet. It also flags whether the document contains formulas, so text-only documents skip formula processing entirely.

4. Formula cleanup

Converts inconsistent OCR output into standardized LaTeX. This stage must be conservative: its job is to repair representation, never to change mathematical meaning. Where confidence is low, it preserves the original expression and flags it for review instead of inventing a correction.

5. Text normalization

Repairs broken whitespace, incorrect line breaks, split answer choices, repeated headers and footers, and encoding artifacts, while retaining references to the original source.

6. Structured question extraction

Converts normalized content into a validated schema:

{
  "question_text": "What is the value of x?",
  "question_type": "multiple_choice",
  "options": [
    { "id": "a", "text": "2" },
    { "id": "b", "text": "4" },
    { "id": "c", "text": "6" },
    { "id": "d", "text": "8" }
  ],
  "correct_answer": "b",
  "subject": "mathematics",
  "language": "en",
  "source_page": 4,
  "confidence": 0.91
}

Structured output here is critical. Free-form model responses are difficult to validate and dangerous to publish directly.

7. Language specialization

Corrects spelling and language-specific OCR errors without translating or changing meaning. This matters for documents that mix a regional language with English terminology or non-Latin scripts. The system currently supports Gujarati and Spanish alongside English.

8. Final validation

Checks that required fields are present, options are structurally valid, referenced images exist, LaTeX parses, questions are not empty or duplicated, and confidence meets the publishing threshold. The output is a set of structured questions with a quality report.

Why Multiple Agents Instead of One Big Prompt?

Sending the whole document to one large prompt is simpler, and nearly impossible to debug. When the output is wrong, you cannot tell whether the failure came from OCR, layout understanding, question boundaries, formula parsing, or schema generation.

Specialized stages give us three things: observability (every stage produces inspectable intermediate output), targeted retries (a failed formula conversion does not repeat OCR for the whole document), and independent improvement (the formula agent or validator can be upgraded without redesigning the pipeline).

That said, not every stage needs an LLM. Deterministic code is preferable whenever the task can be solved reliably with rules, parsers, or mathematical libraries.

Part 2: Generating New Questions

Generation has a different objective: use textbook content to create new questions that test understanding at specific cognitive levels. The workflow contains nine agents and four subject-specific generation modules, with a quality-gated retry loop: questions must score at least 75 against a rubric, with up to three generation attempts.

Content analysis

Beyond identifying topics, the analyzer extracts core concepts, relationships between them, important values, and, most valuably, common student misconceptions. A high-quality distractor should represent a believable reasoning mistake, not a random incorrect value. When testing order of operations, a plausible wrong answer comes from evaluating addition before multiplication, because it reveals what the learner misunderstood.

Cognitive-level planning

Before generating anything, the system plans a distribution of questions across cognitive levels. At publishing time, each question maps to the Depth of Knowledge tag Wayground uses, so a worksheet can be assembled against an explicit rigor target instead of an implicit difficulty:

Level	Intended capability
1	Recall a fact, formula, or definition
2	Apply a familiar procedure
3	Combine multiple concepts or reasoning steps
4	Analyze, compare, or solve a non-routine problem
5	Transfer knowledge to a novel situation

This planning step exists because without it, language models produce many questions that sound different but test the same shallow skill.

Subject-specific generation

A generic prompt is rarely sufficient across subjects, because correctness means something different in each:

Subject	What correctness requires
Mathematics	Symbolic verification: answers are checked with SymPy
Physics	Numerical results plus unit and dimensional consistency
Chemistry	Equation, reaction, and terminology constraints
Biology	Strong grounding in the source material, since most answers cannot be verified symbolically

Deterministic Answer Verification

Language models are useful generators, but they should not be treated as calculators. For mathematical questions, the generated answer is independently solved with SymPy:

The generator proposes; the verifier solves the problem independently. If they disagree, the question does not proceed to publishing. The same principle extends to unit validation, chemical equation balancing, and option uniqueness checks.

Distractors from Known Mistakes

This is my favorite part of the system. Instead of asking a language model to invent incorrect options, the system generates distractors by solving modified versions of the original problem with SymPy, where each modification encodes a specific mistake a student could plausibly make.

Consider:

f(x) = (3x^2 + 1)^4

The correct derivative requires the chain rule:

f'(x) = 4(3x^2 + 1)^3 * 6x = 24x(3x^2 + 1)^3

The distractor generator applies realistic mistake transformations (forgetting the chain rule, differentiating the inner function incorrectly, dropping the outer coefficient), and SymPy evaluates each altered procedure:

A. 24x(3x^2 + 1)^3    correct chain rule
B. 4(3x^2 + 1)^3      forgot the inner derivative
C. 12x(3x^2 + 1)^3    inner derivative taken as 3x instead of 6x
D. 6x(3x^2 + 1)^3     dropped the outer coefficient

The validator confirms each distractor is mathematically distinct from the correct answer and from the others, and does not simplify into the correct expression. The result is distractors that are realistic, deterministic, and traceable: the system can explain not only that an option is wrong, but which mistake produces it.

Quality-Gated Generation

Every generated question is scored against a rubric covering correctness, grounding in the source material, clarity, cognitive-level alignment, distractor quality, and language quality. Questions below the threshold are retried, but instead of a bare "generate again," the validator returns structured reasons:

{
  "score": 68,
  "passed": false,
  "issues": [
    {
      "category": "cognitive_alignment",
      "message": "The question tests direct recall instead of the requested level 3 reasoning."
    },
    {
      "category": "distractor_quality",
      "message": "Two answer choices are obviously unrelated to the concept."
    }
  ]
}

This feedback becomes input to the next generation attempt.

Extraction and Generation Are Different Products

Although they share infrastructure, the two workflows should not be treated as one generic operation:

Area	Extraction	Generation
Primary goal	Reconstruct existing questions	Create novel questions
Main risk	Losing or changing source meaning	Incorrect or shallow content
Validation	Fidelity and schema correctness	Correctness and cognitive alignment
Retry strategy	Reprocess the failed stage	Regenerate using rubric feedback
Deterministic tools	Formatting and structural checks	Symbolic answer verification

This distinction shapes prompts, validation, metrics, and failure handling throughout the system.

What We Learned

OCR is only the first step. Layout analysis, formula reconstruction, image association, and semantic extraction are separate problems.
Preserve provenance. Without page numbers and source spans, reviewing an incorrect question becomes a search problem.
Use models for interpretation, deterministic code for verification. Schema validation, math solving, deduplication, and retry limits do not need an LLM.
Validate intermediate representations. Each stage needs its own contract. Waiting until final output makes failures impossible to isolate.
Quality requires a feedback loop. Plan, generate, verify, score, and retry beats one-shot generation.
Correctness is subject-specific. Symbolic proof in mathematics, unit checks in physics, source grounding in biology.

What's Next

The current architecture is a solid foundation. The improvements I care most about:

Page-level provenance with bounding boxes, so a reviewer sees the exact source region beside each structured question.
Hard gates instead of one aggregate score, so a question can never pass because strong writing compensates for a wrong answer.
Confidence-based review routing: auto-approve the easy cases, send formula-heavy or low-confidence output to subject experts.
Semantic deduplication across editions, teacher copies, and answer keys, where questions differ only in formatting or values.
Cached document understanding and resumable execution, so a retry resumes from the failed stage instead of re-running OCR.
An evaluation dataset with human-reviewed ground truth, because the metric that matters is cost per accepted question, not cost per token.
Treating uploaded documents as untrusted input, with strict structured outputs so embedded text can never act as instructions.

Conclusion

Turning educational PDFs into reliable questions is not a single-model problem. It is a document-understanding, reasoning, verification, and orchestration problem. The most important architectural decision was treating extraction and generation as pipelines of specialized stages rather than one large prompt: models interpret, deterministic tools verify, and quality gates decide what reaches publishing. The payoff is publisher content that lands in Wayground ready for both lives it needs to lead: printed as a worksheet or played as a formative assessment, with a DoK tag on every question. And with provenance, evaluation, and reviewer feedback in place, the same system keeps improving with every document it processes.