A dialogue generator can produce five polished alternatives that all do the same thing: ask again, with different adjectives. If your writing tool labels these “more dramatic,” fluency has become a proxy for a change it never actually measured.
Here is a small, human-reviewed evaluation workflow for separating a wording change from a tactical change. It is a proposed test design, not a benchmark result or a claim about any product's implementation.
Start with an observable distinction
Use an invented scene. Mara needs her coworker Eli to sign a transfer request before the office closes. Eli wants to know who will take over his unfinished work.
Original line: “Please sign it before five.”
Candidate A: “I really need your signature before five.”
Candidate B: “If I take the Friday handover, will you sign it now?”
Candidate C: “The director already approved it. You are only holding everyone up.”
A intensifies the request. B offers a trade. C invokes authority and applies pressure—but invents an approval unless that fact already exists in the scene. A tactical change is not automatically an acceptable rewrite.
1. Make the fixture small enough to inspect
Keep the source exchange, known facts, and requested change together. For example:
{
"goal": "Mara wants Eli to sign before five",
"known_facts": [
"Eli is worried about his unfinished work",
"Mara can offer to handle Friday's handover"
],
"unknown_facts": ["Whether the director approved the transfer"],
"source_tactic": "direct request",
"requested_tactic": "offer a concrete trade",
"must_preserve": ["Eli has not agreed yet"]
}
Do not turn missing information into false certainty. “Unknown” means the evaluator should not accept a new claim as established fact; it does not mean the claim is necessarily false.
2. Ask for evidence, not a quality score
A useful review record has four fields:
- Observed tactic: What is the speaker trying to get the other person to do, and how?
- Evidence: Quote the exact words that support that interpretation.
- New commitments or facts: What did the rewrite add?
- Decision: Accept, revise, or uncertain—with one reason.
For B, the evidence is “If I take the Friday handover.” The added commitment is explicit and supported by the fixture. It is a reasonable candidate to keep, not proof of a good scene. Eli's response still has to make dramatic sense.
For C, the authority tactic is visible, but the approval is unsupported. Flag that separately rather than rewarding the stronger tone.
3. Test the evaluator with contrast pairs
Before relying on a reviewer model, include pairs where a fluent answer should not win:
- Same request, more emotional wording.
- A genuine trade versus an unrelated new offer.
- A change of tactic that preserves facts versus one that invents leverage.
- An indirect line with two plausible readings, where “uncertain” is appropriate.
Have a person label the pairs first. Compare the model's evidence and decisions against those judgments. If people disagree, keep the disagreement visible; do not hide it behind a single reference label.
This is a qualitative test set. With only a few examples, a pass percentage can create more confidence than the evidence deserves. Log concrete failure modes instead: unsupported leverage, paraphrase mistaken for strategy, or ambiguous subtext treated as certain.
Where automation should stop
Simple code can check whether required fields are present. It cannot establish that a character's offer feels honest, surprising, or dramatically appropriate. A second model can suggest a reading, but it is not independent proof that the first model improved the scene.
The final review question belongs to the writer: does the other character now have a different reason to answer?
Disclosure: I work on Laper. This article was generated with AI for our editorial account. The fixture and evaluation workflow above are illustrative, not a description of a shipped Laper feature or measured results.
Try the test on one exchange before expanding it into a pipeline. If every alternative gives the listener the same choice, you may have generated variety in wording—not variety in action.
Top comments (0)