DEV Community: deavgprime

Resuming an eval without changing the experiment

deavgprime — Sun, 01 Nov 2026 19:36:54 +0000

An eval stops halfway through. There is already an answer on disk, and another task has not finished.

What should happen when we run it again?

The easy part is skipping a file that already exists. The harder part is deciding whether that file still belongs to the experiment we are about to continue.

If the prompt changed, the grader changed, or a failed request quietly gets another chance, we may end up with a different experiment under the same directory name.

This is the next case study in the series, after What exactly are AI evals? and Before scoring a model, test the evaluator. This time, I am looking at the runner around the scorer in IaC Eval Bench.

On this page

What resume means here
Interrupting a two-task run
What must stay the same
A recorded failure is still an attempt
Try the checks locally
The boundary a checkpoint cannot solve

What resume means here

For this project, a resume does three things:

Checks that the run configuration, inputs, evaluator, and recorded validation runtime still match.
Validates and reuses the attempt records already saved.
Runs the missing attempts and rebuilds the reports.

A completed response can pass or fail the task. Either way, it is an attempt to keep. A recorded provider error is also retained.

A retry asks for another attempt. A resume continues the same planned set of attempts. Those are different decisions, and I do not want one to happen silently inside the other.

Interrupting a two-task run

I used two of the benchmark's tasks: a Deployment configuration and a Service configuration. Both used the project's saved reference fixtures.

These are constructed checks of the runner. They do not call a model or measure model quality.

For the interruption check, a controlled exception stops the fixture provider before the second attempt returns. The first attempt has already been written to disk. The runner is then called again with the same configuration and resume=True.

Here is what happened:

Step	Saved records	Fixture calls during this step
Before the controlled interruption	One completed attempt	Two calls entered; the second was interrupted
Resume the missing work	Two completed attempts	One
Resume the completed run again	The same two attempts	Zero

The first attempt file stayed byte-for-byte unchanged. After the first resume, both reference tasks passed. The second resume returned the same summary without calling the fixture provider again.

Figure 1. A controlled fixture example. The saved record stays; only missing work is continued.

The useful result is not the two passing grades. It is that continuing the run did not replace its earlier evidence or add an unplanned attempt.

What must stay the same

A directory name cannot tell us whether two runs mean the same thing.

IaC Eval Bench records a fingerprint derived from the configuration, task definitions, schema manifest, harness source, fixture contents where applicable, and Python and validation dependency versions. On resume, it recomputes that fingerprint and checks the saved manifest's provenance fields as well.

The source of this behavior is inspectable in the runner. It is a specific contract for this small project, not a requirement that every evaluation framework must use the same fields.

I checked four ways that continuing a saved run could go wrong:

Change before resuming	Observed result
Change the configured seed from 42 to 99	Rejected: configuration, inputs, or evaluator changed
Add a newline to a selected fixture file	Rejected: configuration, inputs, or evaluator changed
Change the saved manifest's harness version	Rejected: manifest provenance does not match
Change an attempt's score without updating its checksum	Rejected: attempt integrity check failed

The recorded observations include the continuation checks, rejection messages, and CLI exit codes.

The newline case is deliberately strict. The fixture means the same thing as JSON, but its bytes changed. This implementation compares content hashes rather than trying to prove that two inputs are semantically equivalent.

If I want to compare a new prompt or a changed grader, I should use a new output directory and record a new run. Keeping the old run lets me explain what changed.

The fingerprints also have limits. They do not establish which remote model weights a server actually loaded. A user-supplied model revision is recorded, not independently verified. A requested seed does not guarantee identical generation across machines or model servers.

A recorded failure is still an attempt

Suppose a provider request fails and the runner saves that error. On resume, should the runner try it again?

In this project, it does not. I checked that separately by injecting two provider errors, then resuming with the fixture function configured to fail the test if called.

The resumed run made zero provider calls and retained both errors.

There were no completed answers, so the quality pass rate was unknown. The operational pass rate was zero: none of the two planned attempts produced a passing result.

This distinction matters when reading a report. A failed request is not evidence that the model produced a wrong configuration. It is still evidence that the evaluation attempt did not succeed operationally.

An intentional retry belongs in a separately named run under this project's current interface. It should not erase the first failure or quietly improve the original run's headline result.

Try the checks locally

The previous case study includes the environment setup. Use the same pinned revision, 1dbabecd8cc377cef1082b2652131b6921c4c74b, with its locked dependencies.

From that checkout, create a small run:

iac-eval run --provider fixture --fixture-dir fixtures/reference \
  --task deployment-resources --task service-routing \
  --output runs/resume-demo

Run the same command with --resume:

iac-eval run --provider fixture --fixture-dir fixtures/reference \
  --task deployment-resources --task service-routing \
  --output runs/resume-demo --resume

This reuses the two saved attempts. It is a completed-run check, not a simulation of an interruption.

Now change the seed while keeping the same output directory:

iac-eval run --provider fixture --fixture-dir fixtures/reference \
  --task deployment-resources --task service-routing \
  --seed 99 --output runs/resume-demo --resume

The command exits with code 2 and reports:

error: resume rejected: configuration, inputs or evaluator changed

Use a fresh output directory if runs/resume-demo already contains evidence you want to keep.

For the controlled interruption and the related regression checks, run the existing runner tests:

python -m unittest discover -s tests -p test_runner.py -v

The tests cover partial-run continuation, duplicate generation, changed configuration, record integrity, locks, and recorded provider failures. The full suite has 51 test methods; it also covers the scoring cases from the previous post.

The boundary a checkpoint cannot solve

There is an important gap between making a provider request and saving its result.

If a process stops after the provider has produced an answer but before the attempt record is written, the runner has no completed record to reuse. A later resume can make that request again. The provider might have charged for the first request even though the local report never recorded it.

Writing attempt files atomically protects the local checkpoint transition. It does not make a remote request and a local file write one indivisible operation. This implementation does not guarantee exactly-once execution at the provider.

There are other practical boundaries too:

The directory lock stops competing local writers. An abrupt stop can leave a stale lock. Check that no runner is active before removing it.
Checksums detect a record changed without a matching checksum. They are not signatures and do not prove authenticity against someone who can rewrite both.
Rebuilding a report from saved attempts does not mean rerunning those answers through a different grader. A changed grader requires a new experiment under the current resume contract.

These checks used local fixtures and controlled faults. They establish the specific continuation and rejection behavior described above. They do not test a live provider outage or a machine losing power during a request.

The question I now ask of a resumable eval is simple: did it continue the same experiment, and can I still account for every attempt?

Before scoring a model, test the evaluator

deavgprime — Sun, 01 Nov 2026 19:07:46 +0000

One version of my evaluator passed a Kubernetes configuration whose selector contradicted its own pod labels. It also rejected a configuration just because an extra label had been added.

Both results were scoring errors.

If you are new to evals, start with What exactly are AI evals?. It explains the tasks, graders, and metrics used in this case study.

These came up while working on IaC Eval Bench, a small project for evaluating model-generated Kubernetes configurations. A scoring review found five mistakes in the evaluator, even though its original reference and negative-control examples were producing the expected totals.

That last part is worth explaining. A passing test suite tells us that the cases we wrote behave as expected. We still have to ask whether we chose enough useful cases.

On this page

How the evaluator works
A passing and a failing answer
Two mistakes the controls missed
All five scoring fixes
Run the checks
Limits and next checks

How the evaluator works

The benchmark has six tasks. They ask for configurations such as a Deployment with resource limits, a Service that selects the right pods, or a scheduled job with execution limits.

For each answer, the evaluator checks:

The response format. Can it read the expected JSON structure?
The Kubernetes schema. Do the fields and their types match the bundled schema?
The requested resources. Are the right resource types, names, and namespaces present?
The task requirements. Do the labels, ports, limits, and other settings satisfy the prompt?

A schema is a description of the allowed structure. For example, it can tell us that a replica count should be a number. That does not mean it checks every relationship between the fields.

The evaluator saves the answer and the outcome of its checks so a result can be inspected later.

Figure 1. The evaluator checks configuration data without deploying it. Task requirements are available when it scores the answer.

Start with a passing and a failing answer

Take a Service, which directs traffic to a set of pods. Kubernetes uses labels to decide which pods the Service selects.

Here is the relevant part of a matching example. These are excerpts, not complete manifests:

# Pod labels in the Deployment template
labels:
  app: api

# Service selector
selector:
  app: api

The Service asks for pods with app: api, and the pod template has that label.

Now change only the Service selector:

selector:
  app: unrelated

Both versions have the same field structure. The second selector does not match the intended pods. The evaluator needs to check that relationship.

This is a useful negative control: a constructed answer with a mistake we expect the evaluator to catch. The unchanged answer is the positive control.

The project has one reference answer and one targeted negative example for each task:

Constructed examples	Full task passes	Schema and resource checks passed
Six reference answers	6/6	6/6
Six answers with a targeted mistake	0/6	6/6

Those are the expected results. All six negative examples still get through the schema and resource checks. Each fails its intended task requirement.

Reading these results: These are constructed examples for testing the evaluator. No model produced them, so the pass counts do not measure model ability.

Two mistakes the original controls missed

A broken selector passed

The original examples all behaved as expected. Then the review tried changes beyond those six negative controls.

One change added this condition to a Deployment selector:

selector:
  matchLabels:
    app: api
  matchExpressions:
    - key: app
      operator: NotIn
      values: [api]

Read the two conditions together:

The first requires app to equal api.
The second requires app not to equal api.

They cannot both be satisfied. The selector also contradicts the Deployment's own pod template, which has app: api.

The old evaluator passed this configuration because it checked matchLabels and ignored matchExpressions.

The fix checks every declared selector condition against the submitted pod labels. This follows Kubernetes' rule that the conditions in a label selector must all be satisfied. Kubernetes selector documentation

A harmless label failed

Here is the other side of the problem:

labels:
  app: api
  team: platform

The required app: api label is still present. A selector asking for app: api still matches these labels.

The old label check compared the whole label map with the reference map. The extra team entry caused a failure.

That check was too strict. It now checks for the required label while allowing compatible additions. The task wording also makes those allowed additions explicit, so the prompt and the scoring rule agree.

This matters for model evaluation. If the evaluator rejects an acceptable answer, the model gets blamed for a scoring mistake. If the evaluator accepts a broken answer, the score looks better than it should.

Figure 2. The conflicting selector must fail; the compatible extra label must still pass. Both cases need regression coverage.

All five scoring fixes

The review recorded five scoring mistakes:

Change to the reference answer	Old	Corrected
Conflicting Deployment selector	Pass	Fail
Compatible extra pod label	Fail	Pass
Matching PodDisruptionBudget condition	Fail	Pass
Two readiness probe handlers	Pass	Fail
UDP workload port with TCP Service	Pass	Fail

A PodDisruptionBudget describes how many pods should remain available during voluntary disruptions. Like the Deployment, it uses a selector to identify the relevant pods.

The scoring audit explains the individual fixes and links the saved before-and-after observations.

The original controls were useful. They caught the mistakes they were designed to catch. But neither their old nor their corrected totals exposed these five scoring errors. Both versions reported six reference passes and zero negative-control passes.

The individual cases mattered more than those totals.

Run the checks yourself

The examples can run locally without a model endpoint or Kubernetes cluster. With Python 3.11 to 3.13 installed:

1. Get the version used in this article.

git clone https://github.com/deavgprime/iac-eval-bench.git
cd iac-eval-bench
git switch --detach 1dbabecd8cc377cef1082b2652131b6921c4c74b

2. Prepare the Python environment.

python3 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.lock
python -m pip install --no-deps -e .

3. Run the tests and constructed examples.

python -m unittest discover -s tests -v
python scripts/verify_baselines.py --output runs/article-check

The revision is pinned so the code stays the same when someone repeats the example. Use a fresh output directory if you run the baseline check again.

This revision has 51 test methods, including multiple cases within some tests. The baseline check should finish with:

Verified: 6 reference passes; all 6 targeted semantic mutations detected.

The output folders retain the inputs, individual check results, and reports. Open runs/article-check/reference/report.html or runs/article-check/mutated/report.html locally to inspect them.

Limits and next checks

These checks examine configuration data. They do not submit it to a Kubernetes API server, start a workload, or verify that an application serves traffic.

For example, a matching Service selector does not prove that the application is listening on the requested port. That needs a different kind of test.

The suite is also small: six public, synthetic tasks. Its controls and regression tests cover specific requirements and failure cases. They are not evidence that every valid configuration will pass or every invalid one will fail.

For my next evaluator change, I want to answer three questions before looking at a model score:

What answer should pass?
What specific mistake should fail?
What harmless change should still pass?

For this project, the third question caught a problem that the original pass/fail examples had missed.

Next in the series: Resuming an eval without changing the experiment. I test what happens when a run stops, which attempts are reused, and when resuming should be rejected.

What exactly are AI evals?

deavgprime — Sun, 01 Nov 2026 18:46:23 +0000

You ask an AI to write a configuration. It returns something that looks reasonable, explains what it did, and says the result is ready.

Would you use it?

Reading the answer might catch an obvious mistake. But if you change the prompt tomorrow, how will you know whether the new version is better? What if it fixes one example and breaks another?

That is where evaluations, usually shortened to evals, become useful.

I want to start this series with the basics. The case studies that follow will make more sense if we agree on what we are measuring, how we judge it, and what a score can leave out.

On this page

What an eval actually does
One answer, three different questions
The parts of an eval
How many types of evals are there?
How to design a useful first eval
Who checks the grader?
Why the same run can produce two percentages
Where our case study fits
The reading list for this series

What an eval actually does

An eval tests an AI system against a defined expectation and records what happened.

That expectation needs to be more useful than “give me a good answer.” For example:

Produce a configuration with exactly three replicas.
Answer using only the supplied document.
Fix a bug without breaking the existing tests.
Move a meeting to the requested time without changing its attendees.

The eval gives the system a task, collects its answer or actions, and checks them against the success criteria. A score is a result of that process, not the whole process.

For an agent that can take actions, its final message is only part of the evidence. Saying “I moved the meeting” does not establish that the calendar changed. We need to inspect the resulting state too. Anthropic makes this distinction between a recorded interaction and its actual outcome in its agent evaluation guide.

This also explains why an eval can contain ordinary software tests. A unit test may be one of its checks. The larger evaluation adds the chosen tasks, run conditions, repeated attempts where appropriate, and the records we need to interpret the results.

Running an eval does not, by itself, train or improve the model. We use its findings to decide what to change, then test again. The change might be in a prompt, a retrieved document, a tool, application code, or the model itself.

One answer, three different questions

Here is the running example for this article:

Create a Kubernetes Deployment with three replicas.

Kubernetes runs applications, and a replica is a copy of a workload. You do not need to know Kubernetes to follow the example: we asked for three copies.

Suppose the answer contains this excerpt:

{"spec": {"replicas": 2}}

It is readable JSON. But it says two, not three.

Now separate the questions:

Structure: Can we read the answer in the requested format?
Requirement: Does it contain the setting we asked for?
Behavior: Does the complete configuration work when deployed?

The excerpt answers the first two questions. It cannot establish the third. It is also not a complete Kubernetes manifest, so accepting this JSON does not establish full schema validity.

Figure 1. An illustrative example. Readable output can still miss a requirement, and neither observation proves runtime behavior.

An eval should say which of these questions it checks. Otherwise, “passed” is easy to overinterpret.

The parts of an eval

The terminology becomes easier when we attach each word to the same example.

Term	What it means here
Task or test case	The request, its inputs, and the criteria for success.
Dataset	The collection of examples we choose to test.
Evaluation suite	A group of evaluations covering selected behaviors or capabilities.
System under test	The model, application, or agent whose behavior we are examining.
Trial	One attempt at a case. Repeating a case creates more trials.
Grader or scorer	The method that checks an answer and assigns a result.
Score	A result for one case, such as pass/fail or partial credit.
Metric	A summary across results, such as a pass rate.
Harness	The software that runs the cases and saves the evidence.

Frameworks use these words somewhat differently. In Inspect AI, a task brings together a dataset, a solver that produces answers, and a scorer that grades them. Its task documentation shows how those pieces fit together.

A benchmark is usually a defined collection of tasks and a way to report performance. A public benchmark can be useful for comparisons. A project-specific eval asks whether a system meets the requirements of that project. Doing well on one does not automatically answer the other question.

How many types of evals are there?

There is no single, fixed number that covers all the ways people use the word. Different labels answer different questions.

I find it easier to sort them along three separate axes.

What are we evaluating?

A model: Can it solve a problem or produce the required response?
An application: Does the complete system, including prompts, retrieval, and other components, do its job?
An agent: Do its tool calls and actions produce the intended outcome under the allowed conditions?

A failure in an application does not immediately tell us which component caused it. The retrieved document might be wrong even when the model follows it correctly.

How are we grading it?

Code-based grading uses explicit checks: parse the output, compare a value, run tests, or inspect an environment. It works well when success can be stated precisely. It can also be wrong if the checks are incomplete or unnecessarily strict. Inspect's custom scorer examples show this kind of scoring logic.

Model-based grading asks another model to assess the response against instructions or a rubric. A rubric is a written set of judging criteria. This can help with open-ended answers, but the grader model can make mistakes too. Its judgments need comparison with trusted examples and human review. See Inspect's model-grading guide.

Human grading uses people with the relevant understanding to review the work. It is valuable for ambiguous cases and for checking automated graders. People also need clear criteria and a way to resolve disagreements.

These methods can be combined. We can use code to check structure, a model to flag a possible factual error, and a person to review disputed cases.

We do not always need one reference answer that every response must resemble. A code change can be judged by tests, and an explanation can be judged against a rubric. The expectation still needs to be defined even when several different answers could satisfy it.

When are we evaluating it?

Offline: Run saved or constructed cases without exposing users to the candidate system's actions.
Shadow: Observe how a candidate would behave on live inputs while its actions or answers are withheld from users.
Online: Assess the system during real use, for example through a controlled experiment and operational monitoring.

Offline does not mean “no network.” An offline eval may call a hosted model API. It describes how the experiment relates to users and deployment.

We can also name evals by their purpose: checking a new capability, catching regressions, probing safety requirements, or measuring reliability.

So our future case study can be an offline, code-graded evaluation of generated configurations, used to investigate scoring reliability. Those labels fit together. They are not competing categories.

How to design a useful first eval

Start with a decision you need to make. “Is this model good?” is too broad. “Did this prompt change improve the required configuration settings without increasing failures?” is something we can investigate.

Then work through these steps:

Define success before judging the answers. Identify the required behavior and acceptable variations. Refine unclear criteria when reviewing examples, then version the revised task and grader.
Choose cases that matter. Include ordinary requests, important edge cases, and failures you have observed. A few convenient examples are a starting point, not evidence of broad coverage.
Build the simplest appropriate checks. Use a direct assertion when a direct assertion answers the question. A model judge is not required for every eval.
Compare under matching conditions. Keep the task versions, tools, limits, and scoring rules consistent. Record what changed.
Inspect the failures. Classify whether the problem came from the system, the task wording, the environment, or the grader.

Repeat trials when the behavior varies. Keep each attempt rather than quietly replacing failures with successful retries. Repeating the same cases helps measure variation on those cases; it does not make the case collection more representative.

Correctness is not the only useful measurement. Record latency, token usage, cost, and resource use when they matter to the decision. Report these alongside the task results so readers can see what a successful attempt required.

An eval also needs records: the input, response, relevant actions, configuration, check results, and errors. Otherwise, the headline number may survive while the explanation disappears.

Hamel Husain's practical evaluation workflow connects this work to a useful development cycle: evaluate, inspect the problems, change the system, and evaluate again.

Who checks the grader?

This is the question that connects the introduction to our case study.

Suppose our grader only checks that replicas is present. The answer containing two replicas could pass even though the task asked for three.

We could get a clean report from a broken scoring rule.

Before using that grader to compare models, give it examples whose expected outcomes we understand:

Known-good: A correct answer should pass.
Known-bad: An answer with a specific mistake should fail.
Harmless variation: A different but acceptable answer should still pass.

Figure 2. Test the measuring instrument too. These constructed examples check the grader, not a model's ability.

The third case is easy to overlook. An extra label that does not break a requested selector should not automatically make a configuration wrong. A grader needs to catch mistakes while respecting the task's allowed variations.

These controls still cover only the situations we chose. A passing set of controls is useful evidence, not a guarantee that the grader is complete.

Why the same run can produce two percentages

Here is a made-up run, not a result from our benchmark:

Ten attempts were started.
Six produced passing answers.
Two produced answers that failed the checks.
Two timed out before producing an answer.

If we report passing answers among completed responses, the rate is 6/8 = 75%.

If we report successful attempts among everything started, it is 6/10 = 60%.

Figure 3. Same run, different denominators. Both calculations need their definitions beside them.

Neither rate should be presented without explaining what counts. There is also a difference between a model producing an invalid answer and a grader crashing while trying to inspect it. Inspect's scoring policy explains why those outcomes need separate treatment.

A partial score is another choice. An answer may satisfy several requirements and still fail the full task. “Most checks passed” and “the task passed” are different statements.

For comparisons, look beyond one percentage: case coverage, variation across runs, failure categories, and operational measurements can all matter. More trials and uncertainty estimates help describe the evidence we have; they do not repair a badly chosen dataset. Inspect documents several scoring metrics and uncertainty measures.

Where our case study fits

In IaC Eval Bench, we are studying model-generated Kubernetes configuration through offline structure and task checks. We retain the responses and individual scoring outcomes.

The upcoming case study examines the evaluator itself. A review found cases where it accepted a contradictory selector and rejected an acceptable extra label. It shows why the original positive and negative examples could produce expected totals while other scoring errors remained.

That work does not demonstrate runtime correctness. It does not establish a ranking of models. It gives us specific examples of how an evaluation can mislead us, and how to test the checks more carefully.

The three questions I want readers to carry into it are:

What are we claiming to measure?
What evidence supports the grade?
What could make the grade wrong?

The reading list for this series

I plan to follow this introduction with practical case studies: a reproducible example, the failure, the investigation, and the limits of the fix. Each should give readers something they can inspect or try themselves.

I will add links here as the case studies are published.

Before scoring a model, test the evaluator: five scoring errors in a Kubernetes evaluator and the tests that caught them.
Resuming an eval without changing the experiment: keeping saved attempts, continuing missing work, and rejecting changed inputs in IaC Eval Bench.

For the framework terminology and design choices behind this introduction, the linked Inspect, Anthropic, and Hamel Husain articles are useful places to continue reading.