Fluent Is Not Faithful: Building a Safer AI Paraphrasing Pipeline

Luke M. — Tue, 21 Jul 2026 08:05:29 +0000

Large language models are very good at producing fluent text. That does not mean every fluent rewrite is a faithful rewrite.

While building Paraphraser AI, I kept coming back to a deceptively simple requirement:

Change the wording without changing the claim.

That requirement becomes difficult as soon as users ask for stronger edits, a different tone, several output versions, or the exact preservation of product names and SEO keywords. A model can satisfy the style request while quietly dropping a qualifier, changing a number, strengthening a cautious claim, or inventing a smoother transition that was never supported by the source.

This article describes the pipeline I use today, what it catches, and—more importantly—what it still cannot prove.

Start with a rewrite contract

I treat a rewrite request as structured data rather than a single text prompt:

type RewritePayload = {
  text: string;
  mode: "Standard" | "Natural" | "Formal" | "Academic" | "SEO" | "Humanize";
  strength: "Low" | "Medium" | "High";
  keepWords: string[];
  outputCount: number;
};

Each field represents a separate constraint:

text contains the facts and meaning that must survive.
mode describes the intended voice.
strength controls how far the syntax and vocabulary may move.
keepWords contains strings that should remain unchanged.
outputCount asks for one or more independently usable results.

Keeping these concerns separate makes the API easier to validate and the prompt easier to reason about. It also prevents a UI label such as “SEO mode” from becoming an undefined instruction that changes meaning from one model call to the next.

Validate before spending model tokens

The public endpoint validates the request before calling a provider:

const rewriteRequestSchema = z.object({
  text: z.string().trim().min(1).max(12_000),
  mode: z.enum([
    "Standard",
    "Natural",
    "Formal",
    "Academic",
    "SEO",
    "Humanize",
  ]),
  strength: z.enum(["Low", "Medium", "High"]),
  keepWords: z.array(z.string().trim().min(1).max(60)).max(12),
  outputCount: z.number().int().min(1).max(3),
});

The limits are not semantic safeguards by themselves, but they reduce several avoidable failure modes:

Extremely long input does not silently exceed the useful context budget.
Unknown modes cannot inject arbitrary instructions.
A request cannot ask for hundreds of expensive output variants.
Protected phrases have bounded size and count.

There is also a mode-specific rule: SEO mode requires at least one protected keyword. Without that rule, the product would claim to preserve keywords without knowing which terms matter.

Turn vague style labels into explicit instructions

Models respond more consistently when a mode is translated into a concrete instruction:

const modeInstruction = {
  Standard: "Write a clear, neutral paraphrase.",
  Natural: "Make the text sound fluent, human, and conversational.",
  Formal: "Use polished professional wording without sounding stiff.",
  Academic: "Use precise academic wording without adding citations or claims.",
  SEO: "Improve readability while preserving protected keywords.",
  Humanize: "Make the text sound natural and less machine-like.",
};

Strength receives the same treatment:

const strengthInstruction = {
  Low: "Make light edits, but still change wording in every sentence.",
  Medium: "Change sentence structure and vocabulary while keeping the same meaning.",
  High: "Restructure substantially while preserving facts.",
};

The final prompt combines those instructions with several invariants:

Preserve meaning and facts.
Do not return the original text unchanged.
Do not merely add a label such as “Paraphrased text:”.
Never invent facts, sources, citations, or statistics.
Keep these words or phrases exactly where reasonable: ...
Return strict JSON only.

This does not guarantee compliance. It creates a contract that can be checked after generation.

Ask for structured output, but distrust it

The provider returns a small JSON object:

{
  "outputs": ["The rewritten text goes here."],
  "meta": {
    "keywordsKept": ["Astro", "Cloudflare Workers"],
    "meaningPreserved": true
  }
}

Structured output makes downstream processing easier, but the metadata is not an independent evaluation. If the same model writes the rewrite and reports meaningPreserved: true, that boolean is only a model assertion.

The application therefore treats generated metadata as advisory. Deterministic checks decide whether an output is usable.

Enforce protected terms in code

Keyword preservation is one constraint that can be checked exactly:

function hasKeepWords(text: string, keepWords: string[]) {
  const normalized = text.toLowerCase();
  return keepWords.every((word) =>
    normalized.includes(word.toLowerCase())
  );
}

An output that drops a protected phrase is rejected, even if it is otherwise fluent.

This simple check has limitations. It does not handle inflection, Unicode normalization, or a requirement such as preserving a term only in a heading. For the current product, the UI promises literal phrase preservation, so literal validation is the most honest implementation.

The important design choice is that the model does not grade its own compliance when a deterministic validator can do the job.

Reject the two most common non-rewrites

Models sometimes return the original text unchanged. They also sometimes prepend a label and call that a rewrite:

Professional version: [the original text]

The first cleanup step strips common labels:

function stripRewriteLabel(text: string) {
  return text.replace(
    /^(paraphrased text|professional version|rewritten version|rewrite|output|version)\s*[:：-]\s*/i,
    "",
  ).trim();
}

The next step compares normalized word overlap:

function isTooSimilar(output: string, input: string) {
  const outputWords = normalize(output).split(" ").filter(Boolean);
  const inputWords = normalize(input).split(" ").filter(Boolean);

  if (outputWords.join(" ") === inputWords.join(" ")) return true;

  const overlap = outputWords.filter((word) =>
    inputWords.includes(word)
  ).length;

  return overlap / Math.max(outputWords.length, inputWords.length) > 0.82;
}

This is intentionally a coarse guardrail, not a quality metric. Word overlap cannot tell whether a changed output is good. It only catches outputs that clearly failed to perform a requested rewrite.

A production version should use token frequencies or n-grams rather than the simplified membership test above. Repeated words can otherwise distort the overlap score.

Expect malformed “JSON”

Even when instructed to return strict JSON, models may produce:

Markdown code fences
An explanation before the object
Truncated arrays
A JSON object encoded inside an output string
Valid prose instead of JSON

The parser first removes code fences and extracts the outermost object. It then validates that outputs is a non-empty array. A malformed response is never passed directly to the UI.

function extractJsonObject(text: string) {
  const trimmed = text
    .trim()
    .replace(/^```
{% endraw %}
(?:json)?\s*/i, "")
    .replace(/\s*
{% raw %}
```$/i, "");

  if (trimmed.startsWith("{") && trimmed.endsWith("}")) {
    return trimmed;
  }

  const start = trimmed.indexOf("{");
  const end = trimmed.lastIndexOf("}");
  return start >= 0 && end > start
    ? trimmed.slice(start, end + 1)
    : trimmed;
}

Repair should be conservative. Aggressively guessing the intended structure can turn a provider failure into apparently valid but incorrect text. If parsing or validation fails, it is safer to return a clearly marked fallback than to pretend the response is trustworthy.

Separate provider availability from product behavior

The application can use a Cloudflare Workers AI binding and fall back to an OpenAI-compatible provider when necessary:

async function rewriteWithAIProvider(env, payload) {
  if (env.AI) {
    try {
      return await rewriteWithWorkersAI(env, payload);
    } catch (error) {
      if (!env.OPENAI_API_KEY || !env.OPENAI_MODEL) throw error;
    }
  }

  return rewriteWithOpenAI(env, payload);
}

Provider fallback improves availability, but different models do not produce equivalent rewrites. A provider switch can change tone, formatting reliability, and semantic drift.

That means the evaluation suite must run against every configured model. “The endpoint returned 200” is not enough for a generative feature.

The missing validator: semantic faithfulness

The deterministic pipeline can verify schema, length, required phrases, and whether the output actually changed. It cannot prove that the rewrite preserved meaning.

Consider the source:

The treatment may reduce symptoms in some patients.

A fluent but unsafe rewrite might say:

The treatment reduces symptoms in patients.

Most words and the general topic remain similar, but two important qualifiers—“may” and “some”—have disappeared. The claim is now stronger.

A useful semantic evaluation set should include cases involving:

Negation
Quantifiers such as “some,” “most,” and “all”
Uncertainty words such as “may” and “suggests”
Dates, prices, measurements, and percentages
Named entities and product names
Comparisons such as “less than” and “at least”
Cause versus correlation
Quotations and attributed claims

I plan to score each case on at least four dimensions:

Required-term retention — a deterministic pass/fail check.
Factual invariant retention — numbers, entities, dates, and negations.
Semantic faithfulness — human review using a documented rubric.
Rewrite usefulness — whether the text changed enough to satisfy the request.

Embedding similarity can help prioritize suspicious cases, but it should not become the sole judge. Two sentences can be close in vector space while disagreeing on the exact qualifier that matters.

What I would change next

The current pipeline is a useful baseline, not a completed solution. My next improvements are deliberately narrow:

Extract factual invariants before generation and compare them afterward.
Add adversarial tests for negation, numbers, and uncertainty.
Replace the simple overlap calculation with an n-gram-based change score.
Record provider and model versions with evaluation results.
Stop presenting a model-generated meaningPreserved value as if it were proof.
Publish a small benchmark with inputs, outputs, failures, and the scoring rubric.

The broader lesson is that a generative feature needs two designs: a generation design and a rejection design. Prompt engineering controls what we ask for. Validation controls what we are willing to show.

Fluency is useful. Faithfulness is the actual product requirement.

What semantic-drift cases would you add to the test set?

DEV Community: Luke M.