DEV Community

Peng Zou
Peng Zou

Posted on Fully Autonomous

A small regression harness for citation-safe LLM rewriting

#ai

LLM rewriting can make a paragraph read more smoothly while changing what it claims. If you build a writing assistant, evaluate preservation separately from fluency.

I work on PaperHumanizer, an academic rewriting tool. This post describes a small evaluation pattern you can adapt. It is an illustrative harness, not a claim that every rewrite passes it.

Start with explicit invariants

Take a synthetic example:

In a pilot involving 42 participants, the intervention was associated with a 12% reduction in response time (Lee, 2025). The study did not establish causation.

A revision can change sentence rhythm. It should not silently change 42 to 420, 12% to 21%, or association to causation. Keeping the citation string is necessary but does not show that the surrounding claim still matches the source.

Use a fixture rather than another model score

Here is a dependency-free JavaScript example:

const fixture = {
  required: ['42', '12%', '(Lee, 2025)'],
  forbidden: ['proved that', 'established causation'],
};

function checkRevision(text, fixture) {
  const missing = fixture.required.filter(s => !text.includes(s));
  const introduced = fixture.forbidden.filter(s => text.includes(s));
  return { missing, introduced, needsReview: missing.length > 0 || introduced.length > 0 };
}

const bad = 'The intervention proved that response time fell by 21% (Lee, 2025).';
console.log(checkRevision(bad, fixture));
// missing: ['42', '12%']; introduced: ['proved that']; needsReview: true
Enter fullscreen mode Exit fullscreen mode

Write fixtures by hand for your examples. They can cover sample sizes, units, percentages, author-year references and domain terms. Avoid treating a generic regular expression as a complete citation parser: narrative references, numbered references and multiple studies in one parenthesis need different handling.

Include negative controls

A test suite that accepts the original paragraph proves little. Add deliberately broken outputs:

  • Swap a percentage or unit.
  • Remove a limitation sentence.
  • Keep the reference but attach it to a stronger claim.
  • Change a technical term to a near-synonym with a different meaning.
  • Return an empty response or an incomplete paragraph.

The substring check catches only some of these. In particular, it will miss a removed limitation unless the fixture explicitly requires it. It may also flag a harmless spelling or formatting change. Its purpose is to route a revision to review, not to certify accuracy.

Put the human comparison step in the product

For each paragraph, compare the source with the revision and ask:

  1. Is the population, method and result unchanged?
  2. Is uncertainty still expressed at the same strength?
  3. Does each reference still support the claim beside it?
  4. Were any facts or references added?

A clean automated result means only that the chosen invariants survived. Readability, factual consistency and source support still need review. A detector score does not answer those questions.

For an example workflow, PaperHumanizer's academic editor includes an original/rewrite comparison view, academic tone choices and text-file export. Rewriting requires sign-in and credits. Review the result against your sources and follow your institution's AI-use rules.

What preservation failures would you put in a fixture before shipping an LLM rewriting feature?

Disclosure: I work on PaperHumanizer. This article was prepared with AI assistance and reviewed before publication.

Top comments (0)