A validator test can pass while the surrounding application does the wrong thing. The warning is correct, but a retry discards useful text. Or the warning is wrong, and the application keeps asking the model to rewrite a sentence that was already acceptable.
At AETHERCORE, we build GenesisCore, which turns calculated astrology data into reports. An audit of a historical implementation reproduced a small but consequential mistake: a string-matching check could confuse the planet responsible for a house with the planet's location. Changing the order of an otherwise equivalent Japanese sentence changed the warning. The Japanese walkthrough covers that detector and its limitations.
The Japanese article examines why the warning appeared. Here, we test what the surrounding application does after receiving it. You can test that before paying for another model call.
Three policies for the same warning
The example below lets you choose between three behaviors:
-
audit: return the original text with its warnings. -
hold: retain the text for review, but do not deliver it. -
retry: request another candidate, up to three calls in total.
These are application policies. The validator supplies evidence; the caller decides how much authority it receives. A warning might represent an actual error, a false positive, or an uncertain judgment. This test does not pretend to tell them apart.
The code uses Python's standard-library Mock. Giving it a list through side_effect supplies one predetermined output per call, and call_count records how often the application asks for a candidate.
A complete test without an API key
Save this as validator_policy.py and run python validator_policy.py with Python 3.10 or later. It performs no network requests. All strings and warnings are synthetic. This is a teaching implementation, not a copy of GenesisCore's production runner.
"""Teaching example. Python 3.10+, standard library, no network calls."""
from unittest.mock import Mock
def generate_with_policy(generate, validate, policy, max_calls=3):
if policy not in {"audit", "hold", "retry"}:
raise ValueError("unknown policy")
if max_calls < 1:
raise ValueError("max_calls must be positive")
for _ in range(max_calls):
text = generate()
if not isinstance(text, str) or not text.strip():
raise ValueError("nonempty text required")
warnings = list(validate(text))
if not warnings or policy == "audit":
return {"state": "returned", "text": text,
"held_text": None, "warnings": warnings}
if policy == "hold":
break
return {"state": "held", "text": None,
"held_text": text, "warnings": warnings}
def run_tests():
scenarios = [
("retry", [[], [], []], 1, "returned", "candidate 1"),
("retry", [["warning"]] * 3, 3, "held", None),
("retry", [["warning"], [], []], 2, "returned", "candidate 2"),
("audit", [["warning"]] * 3, 1, "returned", "candidate 1"),
("hold", [["warning"]] * 3, 1, "held", None),
]
for policy, warnings, calls, state, expected_text in scenarios:
generate = Mock(side_effect=["candidate 1", "candidate 2", "candidate 3"])
validate = Mock(side_effect=warnings)
result = generate_with_policy(generate, validate, policy)
assert generate.call_count == calls
assert validate.call_count == calls
assert result["state"] == state and result["text"] == expected_text
assert result["warnings"] == warnings[calls - 1]
assert result["held_text"] == (f"candidate {calls}" if state == "held" else None)
for raw, limit, expected_calls in [("", 3, 1), ("unused", 0, 0)]:
generate = Mock(return_value=raw)
try:
generate_with_policy(generate, lambda _: [], "audit", limit)
except ValueError:
assert generate.call_count == expected_calls
else:
raise AssertionError("invalid input was accepted")
print("7 policy scenarios passed; external model calls: 0")
if __name__ == "__main__":
run_tests()
Expected output:
7 policy scenarios passed; external model calls: 0
The tests cover these outcomes:
| Situation | Policy | Calls to the fake generator | Text returned to the caller |
|---|---|---|---|
| No warning | retry | 1 | First candidate |
| Warning persists | retry | 3 | None; held |
| Warning clears on the second candidate | retry | 2 | Second candidate |
| Warning persists | audit | 1 | Original candidate plus warnings |
| Warning persists | hold | 1 | None; original retained for review |
| Generated text is empty | audit | 1 | Exception |
| Retry limit is zero | retry | 0 | Exception before generation |
The fourth row deserves attention. Audit-only avoids rewriting a possibly correct answer, but it can also return a genuinely wrong one. Holding the answer changes the user experience. Retrying changes the answer and may consume more calls. Each choice has a cost, even when the code behaves exactly as designed.
Why the caller belongs in the test
In the historical GenesisCore runner we inspected, three controlled scenarios produced one, three, and two calls to a fake model client. Persistent validation errors reached a fallback path. Those were control-flow tests against a fixed historical version, not measurements of live model quality, latency, or customer impact.
The example here deliberately holds after the retry limit instead of implementing that fallback. If your application has a fallback, add an assertion for its invocation and for the provenance of the returned text. A correct call count cannot tell you whether the fallback silently replaced useful content.
Similarly, checking only warnings == [] misses several important questions: did the original text survive, did a held result leak into the response, and did a supposedly advisory warning trigger generation? The return object makes those decisions visible with separate text and held_text fields.
Bring one real boundary into the fixture
For your own validator, start with two sentences that mean the same thing but use different word orders. Add an unmistakably incorrect sentence and an expression the detector does not understand. Run those through the real detector, then feed its results into policy tests like these.
Keep the two questions separate: does the detector recognize the problem, and does the application respond appropriately? A fake model answers the second question cheaply and repeatably. It cannot establish a model's factual accuracy.
Which assertion is missing from your current tests: generation count, preservation of the original text, or withholding a flagged response? Pick one you would add first; a concrete failure case is welcome.
This article is by AETHERCORE, the team developing GenesisCore. Our public astrology research is a separate body of work; it is not the repository for the private historical application code discussed here. The complete runnable teaching example is included above.
Top comments (1)
The count assertions are the strongest part, because a Mock with a three-item side_effect list also fails loudly if the loop ever makes a fourth call (StopIteration), so the cap is tested twice. Two cases I would add:
Your question about the missing assertion: for me it is the first one, whether the held text is the one you would want a reviewer to see.