You will download four JSON/Python files, run an offline checker, compare its complete stdout against a pre-published expected result, and then make a manual meaning decision on one deliberately altered case. No model requests are involved. No email is sent.
Prerequisites
You need Python 3 on your system and the curl command available in your shell. Create a fresh, empty working directory so the downloaded files do not collide with anything else.
Complete command sequence
curl -fLO https://hemmingway.io/downloads/email-rewrite-fidelity/cases.json
curl -fLO https://hemmingway.io/downloads/email-rewrite-fidelity/check_fidelity.py
curl -fLO https://hemmingway.io/downloads/email-rewrite-fidelity/worked-outputs.json
curl -fLO https://hemmingway.io/downloads/email-rewrite-fidelity/worked-review.json
python3 check_fidelity.py worked-outputs.json > actual-review.json
python3 - <<'PY'
import json
from pathlib import Path
actual = json.loads(Path('actual-review.json').read_text())
expected = json.loads(Path('worked-review.json').read_text())
assert actual == expected
print({key: row['status'] for key, row in actual['cases'].items()})
PY
The checker exits 0. The final printed mapping shows all six case IDs with status needs_manual_semantic_review:
{'deadline_request': 'needs_manual_semantic_review',
'conditional_commitment': 'needs_manual_semantic_review',
'refusal_and_alternative': 'needs_manual_semantic_review',
'amount_and_no_discount': 'needs_manual_semantic_review',
'timezone_and_duration': 'needs_manual_semantic_review',
'attachment_and_owner': 'needs_manual_semantic_review'}
The assertion actual == expected compares the entire parsed JSON object: every status, every missing-anchor list, every manual-invariant string, every null manual result, and every output string. It does not check a handpicked subset. If any field differs, the assertion raises. This is a reproducibility check for this fixed deterministic worked run, not a semantic score.
The one altered case
The other five entries in worked-outputs.json are exact copies of the source messages. They are identity controls: they confirm the checker accepts unchanged text and routes it to manual review. They are not model rewrites.
The refusal_and_alternative entry is the one deliberately altered case. Here is the source:
Hi Sam, I cannot take on the extra 6 reports this month. I can review 2 outlines next month, but I have not agreed to write them. Would that help? Thanks, Arjun.
And the altered output:
Hi Sam, I can take on the extra 6 reports this month. I can review 2 outlines next month, but I have agreed to write them. Would that help? Thanks, Arjun.
Two changes. cannot take on becomes can take on: the stated refusal changes to an assertion of availability or capability. That alone does not establish a promise. have not agreed to write them becomes have agreed to write them: this falsely asserts an agreement that was absent from the source. Would that help? remains unchanged and open.
The checker reports needs_manual_semantic_review for this output with no missing anchors. The declared anchors for this case are exactly ["Sam","6","2","Arjun"], and all four survive the alteration. The checker operates at the literal-screen level: it checks whether declared anchor strings appear in the output. It never validates overall meaning or assigns a model pass. The needs_manual_semantic_review status means the literal screen passed and a human must judge whether meaning changed.
The human decision is to reject this constructed output for altered meaning.
Six-row manual decision summary
| Case | Decision | Reason |
|---|---|---|
| deadline_request | Retain as unchanged control | Exact source copy; invoice requested, not acknowledged; deadline firm. |
| conditional_commitment | Retain as unchanged control | Shipment conditional on proof approval; aim remains tentative. |
| refusal_and_alternative | Reject for altered meaning | Refusal becomes asserted capability; false agreement to write added. |
| amount_and_no_discount | Retain as unchanged control | Fee excluding VAT; discount and revised date unapproved. |
| timezone_and_duration | Retain as unchanged control | Meeting move is a request, not a booking; 20 minutes remains. |
| attachment_and_owner | Retain as unchanged control | File not attached; Zara cannot approve budget; confirmation to Ayo. |
Five selected checker controls
Five deterministic tests were run against the live checker script to confirm selected behaviour:
| Control | Construction | Expected status | Observed |
|---|---|---|---|
| empty | deadline_request, empty string | missing_output | missing_output |
| identity | deadline_request, unchanged source | needs_manual_semantic_review | needs_manual_semantic_review |
| changed date | deadline_request, 20→21 November | literal_check_failed | literal_check_failed |
| meaning inversion | refusal source, two replacements above | needs_manual_semantic_review | needs_manual_semantic_review |
| number substring | refusal source, 6→16 reports | literal_check_failed | literal_check_failed |
All five matched. These verify selected checker behaviour, not comprehensive reliability and not model performance.
Adapting for a real model evaluation
When you later run an actual model, preserve the unchanged input cases and settings, collect every model output, run the same checker, and review the declared meaning rules against each output. The full JSON equality assertion above is a reproducibility check for this deterministic worked run. It is not a semantic score and should not be applied to unknown fresh model outputs as though matching a predetermined file proves writing fidelity.
Next action
Add a synthetic case to cases.json with anchors that cover a meaning you care about, construct a corrupted output, and review whether the literal screen catches it or whether the manual rule must carry the weight.
This tutorial is published by Hemmingway. The explanatory prose was drafted by Hemmingway-1 on Medium with editorial review.
Top comments (0)