DEV Community

Cover image for How to Test an AI Rewriting Tool Fairly: A Mini Experiment in Python
Maheen Razzaq
Maheen Razzaq

Posted on

How to Test an AI Rewriting Tool Fairly: A Mini Experiment in Python

Anyone can say an AI rewriting tool "works". Few people measure it. Here is a small, fair experiment you can run yourself in under an hour, with a Python script to do the counting.

The idea

Take the same input, run it through a tool, and compare the output against the original using the same measurements every time. The goal is not to prove a tool is "best". It is to see what actually changes.

Step 1: Build a small test set

Pick 5 to 10 short passages (80 to 150 words each) from different types of writing:

  • A technical explanation
  • A product description
  • A casual blog paragraph
  • A formal email

Save each one as a separate text file, for example samples/01_before.txt. Using different types matters, because a tool can do well on one style and badly on another.

Step 2: Run each sample through the tool

Paste each passage into the tool, copy the result, and save it as samples/01_after.txt. Use the same settings for every sample, and write them down.

Step 3: Measure what changed

This script compares each before/after pair on three simple signals.

import re
import statistics
import textstat  # pip install textstat
from pathlib import Path

def sentence_lengths(text):
    sentences = [s for s in re.split(r"[.!?]+\s", text) if s.strip()]
    return [len(s.split()) for s in sentences]

def report(text):
    lengths = sentence_lengths(text)
    spread = round(statistics.pstdev(lengths), 1) if len(lengths) > 1 else 0
    return {
        "words": len(text.split()),
        "flesch": round(textstat.flesch_reading_ease(text), 1),
        "spread": spread,
    }

for before_file in sorted(Path("samples").glob("*_before.txt")):
    after_file = before_file.with_name(before_file.name.replace("_before", "_after"))
    if not after_file.exists():
        continue
    b = report(before_file.read_text())
    a = report(after_file.read_text())
    print(before_file.stem.replace("_before", ""))
    print("  before:", b)
    print("  after: ", a)
Enter fullscreen mode Exit fullscreen mode

Step 4: Check what numbers cannot tell you

Numbers do not show whether the meaning stayed the same. For each pair, ask:

  1. Is every fact still correct?
  2. Did any important detail disappear?
  3. Does it still sound like the same writer?
  4. Is it easier to read aloud?

Give each pair a score from 1 to 5 for meaning and for readability. Write the scores in a simple table.

How to read the results

  • A higher Flesch score means easier reading, but a score that jumps a lot may mean the text became too simple.
  • A higher spread means sentence lengths vary more, which usually feels more natural.
  • A big drop in words can mean details were cut.

These are rough signals. They are useful for comparing tools on the same samples, not for judging quality on their own.

Keep the test fair

  • Use the same samples and settings for every tool.
  • Don't pick only the examples that look good.
  • Share your method, so others can repeat it.

If you want a tool to include in your own test, you can try RewriteAI alongside any others and compare the numbers side by side.

Final thought

A short, repeatable test tells you more than any marketing page. Try it on your own writing and see what changes.

What would you add to this test? Share your ideas in the comments.

Top comments (1)

Collapse
 
alexshev profile image
Alex Shev •

This is a sensible compact baseline because it separates repeatable measurements from human judgment. One addition worth making is a factual-entailment check with a small list of required claims per source passage; readability can improve while a qualifier, date, or constraint disappears. Reporting failures by writing genre would also show whether a tool is consistently safe or merely good at one style.