Build a Tiny False-Positive Auditor for AI Text Detectors

AI text detectors are probabilistic classifiers, not authorship proof; a tiny auditor shows how easily human writing gets flagged.

Recent LLM coverage often treats text watermarks as a solved detection problem. The missing part is false positives. A detector that flags a student's human essay has immediate costs, and no single score fixes that. This post builds a small Python auditor with a deliberately coarse detector, then uses an optional free model endpoint to show how style changes flip the verdict.

What you will build

• A standard-library Python script with a toy detector based on word length and unique-word ratio.
• An optional rewrite path through any OpenAI-compatible endpoint, including MonkeyCode's open-source project. Disclosure: This article was prepared as part of MonkeyCode's product outreach.
• Expected output: one technical human sentence is flagged, while two ordinary human samples are not.
• One false-negative input: a short machine-style sentence escapes the same rule.

Learning question: read the heuristic first and predict which sample fails.

Prerequisites

• Python 3.11 or newer.
• No third-party packages; the script uses only os, sys, statistics, json, and urllib.request.
• Optional: an OpenAI-compatible endpoint URL, API key, and model name. Set MONKEYCODE_BASE_URL, MONKEYCODE_API_KEY, and MONKEYCODE_MODEL if you want the rewrite path.
• MonkeyCode advertises 30 million free tokens and a free server option. Treat that as an operator-supplied testbed limit, not a permanent SLA. Check current docs before running a large batch.

The auditor script

Create audit_detector.py:

import os
import sys
import statistics
import json
import urllib.request

SAMPLES = [
    ('student_essay', 'The French Revolution began in 1789. It changed France. Many people were involved.'),
    ('technical_note', 'The implementation demonstrates deterministic ordering and guarantees eventual consistency across replicas.'),
    ('literary_style', 'The rain fell without hurry, and the leaves drank it.'),
]

def heuristic_score(text):
    words = [w for w in text.split() if any(c.isalpha() for c in w)]
    avg_len = statistics.mean(len(w) for w in words)
    unique = len({w.lower() for w in words}) / len(words)
    return avg_len, unique

def toy_detector(avg_len, unique):
    return avg_len > 5.4 and unique > 0.68

def rewrite_with_api(text):
    base = os.environ.get('MONKEYCODE_BASE_URL', '').rstrip('/')
    key = os.environ.get('MONKEYCODE_API_KEY', '')
    model = os.environ.get('MONKEYCODE_MODEL', '')
    if not base or not key or not model:
        return None
    payload = json.dumps({
        'model': model,
        'messages': [{'role': 'user', 'content': 'Rewrite this in clear plain English, keep all facts: ' + text}],
        'temperature': 0.7,
    }).encode()
    req = urllib.request.Request(
        base + '/chat/completions',
        data=payload,
        headers={'Authorization': 'Bearer ' + key, 'Content-Type': 'application/json'},
    )
    with urllib.request.urlopen(req, timeout=30) as response:
        data = json.load(response)
    return data['choices'][0]['message']['content']

def main():
    if len(sys.argv) > 1:
        text = ' '.join(sys.argv[1:])
        avg_len, unique = heuristic_score(text)
        print('input avg_len=%.2f unique=%.2f flagged=%s' % (avg_len, unique, toy_detector(avg_len, unique)))
        return
    for name, text in SAMPLES:
        avg_len, unique = heuristic_score(text)
        print('%s: avg_len=%.2f unique=%.2f flagged=%s' % (name, avg_len, unique, toy_detector(avg_len, unique)))
        rewritten = rewrite_with_api(text)
        if rewritten:
            r_avg, r_unique = heuristic_score(rewritten)
            print('  rewritten avg_len=%.2f unique=%.2f flagged=%s' % (r_avg, r_unique, toy_detector(r_avg, r_unique)))
            print('  rewritten: %s' % rewritten)

if __name__ == '__main__':
    main()

Run it with no arguments:

python audit_detector.py

Expected offline output:

student_essay: avg_len=5.25 unique=1.00 flagged=False
technical_note: avg_len=8.82 unique=1.00 flagged=True
literary_style: avg_len=4.20 unique=0.90 flagged=False

Try one false positive and one false negative

Run the technical note by itself:

python audit_detector.py "The implementation demonstrates deterministic ordering and guarantees eventual consistency across replicas."

Expected output:

input avg_len=8.82 unique=1.00 flagged=True

That is human technical writing, but the toy detector calls it AI because the surface register is long and varied.

Now run a terse machine-like reply:

python audit_detector.py "The API returned 429."

Expected output:

input avg_len=4.67 unique=1.00 flagged=False

The same rule misses a short generated-sounding sentence because it only looks at shape, not source.

What this shows

• Human technical writing gets flagged because the heuristic confuses formal register with machine output.
• A terse API-style response escapes because the rule ignores context and overweights word length.
• Real detectors use richer features, but they still face the same ground-truth problem: no detector can observe how a text was actually produced.
• OpenAI retired an earlier text classifier and documented its accuracy limits; that remains a useful warning: https://openai.com/blog/new-ai-classifier-for-indicating-ai-written-text

Where MonkeyCode fits

I used the optional rewrite_with_api path to ask a model to simplify the technical note. If the rewritten metrics drop below the threshold, the toy detector stops flagging the same facts. That is the lesson: detector output can track style rather than provenance.

MonkeyCode's open-source project advertises 30 million free tokens and a free server option, enough for repeated small rewrites while keeping the experiment disposable. I did not benchmark the full quota, uptime, or latency, so run your own check before planning a batch.

Common mistakes

• Using a detector score as final proof of authorship.
• Forgetting MONKEYCODE_MODEL; the script silently skips the rewrite path, so you may think the API ran when it did not.
• Comparing avg_len across different domains, languages, or text lengths.
• Reading old quota screenshots; current limits can change.
• Treating one flagged sample as evidence of overall detector accuracy.

Who should not use this

• Educators making academic-integrity decisions from a raw detector score.
• Teams expecting a free server option to replace paid production infrastructure.

Extension: turn the free quota into a measurement

• Add a token counter to rewrite_with_api using response['usage'] if the endpoint returns it.
• Run N samples and record prompt_tokens + completion_tokens per audit.
• Track total usage against the advertised 30 million token budget so the claim becomes testable instead of vague.
• Plot how often the same facts change flags when you vary the rewrite instruction.

A detector should start a conversation, not end one. If you want a disposable testbed for negative-result experiments, try the current free server limits and measure before relying on them.