DEV Community

Hemmingway
Hemmingway

Posted on Edited on

Reproduce a Complete Email-Rewrite Fidelity Check in Python

You will download four JSON/Python files, run an offline checker, compare its complete stdout against a pre-published expected result, and then make a manual meaning decision on one deliberately altered case. No model requests are involved. No email is sent.

Prerequisites

You need Python 3 on your system and the curl command available in your shell. Create a fresh, empty working directory so the downloaded files do not collide with anything else.

Complete command sequence

curl -fLO https://hemmingway.io/downloads/email-rewrite-fidelity/cases.json
curl -fLO https://hemmingway.io/downloads/email-rewrite-fidelity/check_fidelity.py
curl -fLO https://hemmingway.io/downloads/email-rewrite-fidelity/worked-outputs.json
curl -fLO https://hemmingway.io/downloads/email-rewrite-fidelity/worked-review.json
python3 check_fidelity.py worked-outputs.json > actual-review.json
python3 - <<'PY'
import json
from pathlib import Path
actual = json.loads(Path('actual-review.json').read_text())
expected = json.loads(Path('worked-review.json').read_text())
assert actual == expected
print({key: row['status'] for key, row in actual['cases'].items()})
PY
Enter fullscreen mode Exit fullscreen mode

The checker exits 0. The final printed mapping shows all six case IDs with status needs_manual_semantic_review:

{'deadline_request': 'needs_manual_semantic_review',
 'conditional_commitment': 'needs_manual_semantic_review',
 'refusal_and_alternative': 'needs_manual_semantic_review',
 'amount_and_no_discount': 'needs_manual_semantic_review',
 'timezone_and_duration': 'needs_manual_semantic_review',
 'attachment_and_owner': 'needs_manual_semantic_review'}
Enter fullscreen mode Exit fullscreen mode

The assertion actual == expected compares the entire parsed JSON object: every status, every missing-anchor list, every manual-invariant string, every null manual result, and every output string. It does not check a handpicked subset. If any field differs, the assertion raises. This is a reproducibility check for this fixed deterministic worked run, not a semantic score.

The one altered case

The other five entries in worked-outputs.json are exact copies of the source messages. They are identity controls: they confirm the checker accepts unchanged text and routes it to manual review. They are not model rewrites.

The refusal_and_alternative entry is the one deliberately altered case. Here is the source:

Hi Sam, I cannot take on the extra 6 reports this month. I can review 2 outlines next month, but I have not agreed to write them. Would that help? Thanks, Arjun.

And the altered output:

Hi Sam, I can take on the extra 6 reports this month. I can review 2 outlines next month, but I have agreed to write them. Would that help? Thanks, Arjun.

Two changes. cannot take on becomes can take on: the stated refusal changes to an assertion of availability or capability. That alone does not establish a promise. have not agreed to write them becomes have agreed to write them: this falsely asserts an agreement that was absent from the source. Would that help? remains unchanged and open.

The checker reports needs_manual_semantic_review for this output with no missing anchors. The declared anchors for this case are exactly ["Sam","6","2","Arjun"], and all four survive the alteration. The checker operates at the literal-screen level: it checks whether declared anchor strings appear in the output. It never validates overall meaning or assigns a model pass. The needs_manual_semantic_review status means the literal screen passed and a human must judge whether meaning changed.

The human decision is to reject this constructed output for altered meaning.

Six-row manual decision summary

Case Decision Reason
deadline_request Retain as unchanged control Exact source copy; invoice requested, not acknowledged; deadline firm.
conditional_commitment Retain as unchanged control Shipment conditional on proof approval; aim remains tentative.
refusal_and_alternative Reject for altered meaning Refusal becomes asserted capability; false agreement to write added.
amount_and_no_discount Retain as unchanged control Fee excluding VAT; discount and revised date unapproved.
timezone_and_duration Retain as unchanged control Meeting move is a request, not a booking; 20 minutes remains.
attachment_and_owner Retain as unchanged control File not attached; Zara cannot approve budget; confirmation to Ayo.

Five selected checker controls

Five deterministic tests were run against the live checker script to confirm selected behaviour:

Control Construction Expected status Observed
empty deadline_request, empty string missing_output missing_output
identity deadline_request, unchanged source needs_manual_semantic_review needs_manual_semantic_review
changed date deadline_request, 20→21 November literal_check_failed literal_check_failed
meaning inversion refusal source, two replacements above needs_manual_semantic_review needs_manual_semantic_review
number substring refusal source, 6→16 reports literal_check_failed literal_check_failed

All five matched. These verify selected checker behaviour, not comprehensive reliability and not model performance.

Adapting for a real model evaluation

When you later run an actual model, preserve the unchanged input cases and settings, collect every model output, run the same checker, and review the declared meaning rules against each output. The full JSON equality assertion above is a reproducibility check for this deterministic worked run. It is not a semantic score and should not be applied to unknown fresh model outputs as though matching a predetermined file proves writing fidelity.

Next action

Add a synthetic case to cases.json with anchors that cover a meaning you care about, construct a corrupted output, and review whether the literal screen catches it or whether the manual rule must carry the weight.

This tutorial is published by Hemmingway. The explanatory prose was drafted by Hemmingway-1 on Medium with editorial review.

Top comments (0)