<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Hemmingway</title>
    <description>The latest articles on DEV Community by Hemmingway (@hemmingway_ai).</description>
    <link>https://dev.to/hemmingway_ai</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4250615%2F5e4b450f-d9cc-4218-9d2e-e26b6b7af3b7.png</url>
      <title>DEV Community: Hemmingway</title>
      <link>https://dev.to/hemmingway_ai</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/hemmingway_ai"/>
    <language>en</language>
    <item>
      <title>Reproduce a Complete Email-Rewrite Fidelity Check in Python</title>
      <dc:creator>Hemmingway</dc:creator>
      <pubDate>Tue, 17 Nov 2026 22:40:22 +0000</pubDate>
      <link>https://dev.to/hemmingway_ai/testing-email-rewrite-fidelity-literal-anchors-vs-semantic-inversion-db3</link>
      <guid>https://dev.to/hemmingway_ai/testing-email-rewrite-fidelity-literal-anchors-vs-semantic-inversion-db3</guid>
      <description>&lt;p&gt;You will download four JSON/Python files, run an offline checker, compare its complete stdout against a pre-published expected result, and then make a manual meaning decision on one deliberately altered case. No model requests are involved. No email is sent.&lt;/p&gt;

&lt;h2&gt;
  
  
  Prerequisites
&lt;/h2&gt;

&lt;p&gt;You need Python 3 on your system and the &lt;code&gt;curl&lt;/code&gt; command available in your shell. Create a fresh, empty working directory so the downloaded files do not collide with anything else.&lt;/p&gt;

&lt;h2&gt;
  
  
  Complete command sequence
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;curl -fLO https://hemmingway.io/downloads/email-rewrite-fidelity/cases.json
curl -fLO https://hemmingway.io/downloads/email-rewrite-fidelity/check_fidelity.py
curl -fLO https://hemmingway.io/downloads/email-rewrite-fidelity/worked-outputs.json
curl -fLO https://hemmingway.io/downloads/email-rewrite-fidelity/worked-review.json
python3 check_fidelity.py worked-outputs.json &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; actual-review.json
python3 - &lt;span class="o"&gt;&amp;lt;&amp;lt;&lt;/span&gt;&lt;span class="s1"&gt;'PY'&lt;/span&gt;
import json
from pathlib import Path
actual &lt;span class="o"&gt;=&lt;/span&gt; json.loads&lt;span class="o"&gt;(&lt;/span&gt;Path&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'actual-review.json'&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;.read_text&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;
expected &lt;span class="o"&gt;=&lt;/span&gt; json.loads&lt;span class="o"&gt;(&lt;/span&gt;Path&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'worked-review.json'&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;.read_text&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;
assert actual &lt;span class="o"&gt;==&lt;/span&gt; expected
print&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="o"&gt;{&lt;/span&gt;key: row&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;'status'&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; key, row &lt;span class="k"&gt;in&lt;/span&gt; actual&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;'cases'&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;.items&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;&lt;span class="o"&gt;}&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;
PY
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The checker exits 0. The final printed mapping shows all six case IDs with status &lt;code&gt;needs_manual_semantic_review&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s1"&gt;'deadline_request'&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'needs_manual_semantic_review'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
 &lt;span class="s1"&gt;'conditional_commitment'&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'needs_manual_semantic_review'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
 &lt;span class="s1"&gt;'refusal_and_alternative'&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'needs_manual_semantic_review'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
 &lt;span class="s1"&gt;'amount_and_no_discount'&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'needs_manual_semantic_review'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
 &lt;span class="s1"&gt;'timezone_and_duration'&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'needs_manual_semantic_review'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
 &lt;span class="s1"&gt;'attachment_and_owner'&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'needs_manual_semantic_review'&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The assertion &lt;code&gt;actual == expected&lt;/code&gt; compares the entire parsed JSON object: every status, every missing-anchor list, every manual-invariant string, every null manual result, and every output string. It does not check a handpicked subset. If any field differs, the assertion raises. This is a reproducibility check for this fixed deterministic worked run, not a semantic score.&lt;/p&gt;

&lt;h2&gt;
  
  
  The one altered case
&lt;/h2&gt;

&lt;p&gt;The other five entries in &lt;code&gt;worked-outputs.json&lt;/code&gt; are exact copies of the source messages. They are identity controls: they confirm the checker accepts unchanged text and routes it to manual review. They are not model rewrites.&lt;/p&gt;

&lt;p&gt;The &lt;code&gt;refusal_and_alternative&lt;/code&gt; entry is the one deliberately altered case. Here is the source:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Hi Sam, I cannot take on the extra 6 reports this month. I can review 2 outlines next month, but I have not agreed to write them. Would that help? Thanks, Arjun.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;And the altered output:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Hi Sam, I can take on the extra 6 reports this month. I can review 2 outlines next month, but I have agreed to write them. Would that help? Thanks, Arjun.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Two changes. &lt;code&gt;cannot take on&lt;/code&gt; becomes &lt;code&gt;can take on&lt;/code&gt;: the stated refusal changes to an assertion of availability or capability. That alone does not establish a promise. &lt;code&gt;have not agreed to write them&lt;/code&gt; becomes &lt;code&gt;have agreed to write them&lt;/code&gt;: this falsely asserts an agreement that was absent from the source. &lt;code&gt;Would that help?&lt;/code&gt; remains unchanged and open.&lt;/p&gt;

&lt;p&gt;The checker reports &lt;code&gt;needs_manual_semantic_review&lt;/code&gt; for this output with no missing anchors. The declared anchors for this case are exactly &lt;code&gt;["Sam","6","2","Arjun"]&lt;/code&gt;, and all four survive the alteration. The checker operates at the literal-screen level: it checks whether declared anchor strings appear in the output. It never validates overall meaning or assigns a model pass. The &lt;code&gt;needs_manual_semantic_review&lt;/code&gt; status means the literal screen passed and a human must judge whether meaning changed.&lt;/p&gt;

&lt;p&gt;The human decision is to reject this constructed output for altered meaning.&lt;/p&gt;

&lt;h2&gt;
  
  
  Six-row manual decision summary
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Case&lt;/th&gt;
&lt;th&gt;Decision&lt;/th&gt;
&lt;th&gt;Reason&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;deadline_request&lt;/td&gt;
&lt;td&gt;Retain as unchanged control&lt;/td&gt;
&lt;td&gt;Exact source copy; invoice requested, not acknowledged; deadline firm.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;conditional_commitment&lt;/td&gt;
&lt;td&gt;Retain as unchanged control&lt;/td&gt;
&lt;td&gt;Shipment conditional on proof approval; aim remains tentative.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;refusal_and_alternative&lt;/td&gt;
&lt;td&gt;Reject for altered meaning&lt;/td&gt;
&lt;td&gt;Refusal becomes asserted capability; false agreement to write added.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;amount_and_no_discount&lt;/td&gt;
&lt;td&gt;Retain as unchanged control&lt;/td&gt;
&lt;td&gt;Fee excluding VAT; discount and revised date unapproved.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;timezone_and_duration&lt;/td&gt;
&lt;td&gt;Retain as unchanged control&lt;/td&gt;
&lt;td&gt;Meeting move is a request, not a booking; 20 minutes remains.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;attachment_and_owner&lt;/td&gt;
&lt;td&gt;Retain as unchanged control&lt;/td&gt;
&lt;td&gt;File not attached; Zara cannot approve budget; confirmation to Ayo.&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  Five selected checker controls
&lt;/h2&gt;

&lt;p&gt;Five deterministic tests were run against the live checker script to confirm selected behaviour:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Control&lt;/th&gt;
&lt;th&gt;Construction&lt;/th&gt;
&lt;th&gt;Expected status&lt;/th&gt;
&lt;th&gt;Observed&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;empty&lt;/td&gt;
&lt;td&gt;deadline_request, empty string&lt;/td&gt;
&lt;td&gt;missing_output&lt;/td&gt;
&lt;td&gt;missing_output&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;identity&lt;/td&gt;
&lt;td&gt;deadline_request, unchanged source&lt;/td&gt;
&lt;td&gt;needs_manual_semantic_review&lt;/td&gt;
&lt;td&gt;needs_manual_semantic_review&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;changed date&lt;/td&gt;
&lt;td&gt;deadline_request, 20→21 November&lt;/td&gt;
&lt;td&gt;literal_check_failed&lt;/td&gt;
&lt;td&gt;literal_check_failed&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;meaning inversion&lt;/td&gt;
&lt;td&gt;refusal source, two replacements above&lt;/td&gt;
&lt;td&gt;needs_manual_semantic_review&lt;/td&gt;
&lt;td&gt;needs_manual_semantic_review&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;number substring&lt;/td&gt;
&lt;td&gt;refusal source, 6→16 reports&lt;/td&gt;
&lt;td&gt;literal_check_failed&lt;/td&gt;
&lt;td&gt;literal_check_failed&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;All five matched. These verify selected checker behaviour, not comprehensive reliability and not model performance.&lt;/p&gt;

&lt;h2&gt;
  
  
  Adapting for a real model evaluation
&lt;/h2&gt;

&lt;p&gt;When you later run an actual model, preserve the unchanged input cases and settings, collect every model output, run the same checker, and review the declared meaning rules against each output. The full JSON equality assertion above is a reproducibility check for this deterministic worked run. It is not a semantic score and should not be applied to unknown fresh model outputs as though matching a predetermined file proves writing fidelity.&lt;/p&gt;

&lt;h2&gt;
  
  
  Next action
&lt;/h2&gt;

&lt;p&gt;Add a synthetic case to &lt;code&gt;cases.json&lt;/code&gt; with anchors that cover a meaning you care about, construct a corrupted output, and review whether the literal screen catches it or whether the manual rule must carry the weight.&lt;/p&gt;

&lt;p&gt;This tutorial is published by Hemmingway. The explanatory prose was drafted by Hemmingway-1 on Medium with editorial review.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>llm</category>
      <category>testing</category>
    </item>
  </channel>
</rss>
