Updated October 29, 2026. The earlier pseudocode relied on undefined scoring helpers. This replacement is a complete, local Python example for a deliberately narrow extraction workflow.
A small evaluation harness is easier to trust when every check has a precise meaning. Start with a bounded task and real failure handling before adding judgments about open-ended answer quality.
Quick answer: Represent a captured result, define separate checks, preserve pass/fail/unavailable outcomes and run reviewed fixtures. The example below checks source identity and extracted-body availability. It does not detect hallucinations, validate a summary or establish the quality of a real extraction service.
Define a narrow contract
The illustrative task is to retrieve the requested article. A fixture records the requested URL, returned source URL, HTTP status and extracted body. For this example, source URLs must match exactly; production applications may need a reviewed redirect policy instead.
A transport failure means later content checks are unavailable. A successful request with an empty body is a content failure. Those conditions should not collapse into the same result.
Run this standard-library example
Save the code as evaluate_extraction.py and run it with Python 3.9 or later. It makes no network calls and uses no API keys. All fixtures are synthetic.
from dataclasses import dataclass
from typing import Optional
import json
@dataclass(frozen=True)
class Extraction:
requested_url: str
returned_url: Optional[str]
status: Optional[int]
body: Optional[str]
def check(passed: Optional[bool], reason: str) -> dict:
state = "unavailable" if passed is None else ("pass" if passed else "fail")
return {"state": state, "reason": reason}
def evaluate(item: Extraction) -> dict:
transport_ok = item.status is not None and 200 <= item.status < 300
results = {
"transport": check(transport_ok, "Requires a recorded 2xx status.")
}
if not transport_ok:
results["source_matches"] = check(None, "No successful extraction.")
results["body_present"] = check(None, "No successful extraction.")
return results
results["source_matches"] = check(
None if item.returned_url is None else item.returned_url == item.requested_url,
"Exact source identity is the policy for this example.",
)
results["body_present"] = check(
None if item.body is None else bool(item.body.strip()),
"A missing field is unavailable; an empty field fails.",
)
return results
if __name__ == "__main__":
url = "https://example.com/article"
fixtures = {
"usable": Extraction(url, url, 200, "Synthetic article text."),
"empty": Extraction(url, url, 200, " "),
"wrong_source": Extraction(url, "https://example.com/other", 200, "Text."),
"missing_body": Extraction(url, url, 200, None),
"failed_request": Extraction(url, None, 503, None),
}
observed = {name: evaluate(item) for name, item in fixtures.items()}
assert observed["usable"]["body_present"]["state"] == "pass"
assert observed["empty"]["body_present"]["state"] == "fail"
assert observed["wrong_source"]["source_matches"]["state"] == "fail"
assert observed["missing_body"]["body_present"]["state"] == "unavailable"
assert observed["failed_request"]["body_present"]["state"] == "unavailable"
print(json.dumps(observed, indent=2))
Interpret only what the checks establish
A passing body_present check means a non-empty string was recorded. It does not prove that the extraction is complete, relevant or faithful to the page. Exact URL equality is also a deliberately strict policy, not a universal normalization algorithm.
For a real summarization workflow, add reviewed cases for incomplete extraction, relevant source material and unsupported summary claims. Those checks need additional evidence and a suitable rubric or reference. Do not replace them with a function named contains_hallucination that has no implementation.
Keep diagnostic results separate
This harness has no weighted average. A failed source-identity condition should remain visible rather than being canceled by a passing transport result. Preserve the application version, fixture selection and check configuration when comparing candidates.
Iām Shashank Agarwal, founder and CEO of Noveum AI and creator of API.market. The HTTP-success worked example explains why these layers matter; the eval-driven guide connects a reviewed regression set to development decisions.
Follow my work on LinkedIn. This refresh was drafted with AI assistance.
Top comments (0)