Learn Why AI Text Detection Fails by Building a Tiny Watermark Verifier

Most public AI-text detectors are trying to verify a watermark without holding the secret key.

Watermarking is back in the news. A common reaction is: “Which detector should I use?” A more useful student question is: What does a verifier actually compute?

I will build a tiny verifier to make that concrete, then show the exact point where it breaks.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The learning question

Can I separate model output from human writing without the model provider's watermark secret?

Spoiler: mostly no. A watermark check needs a secret. Most public detectors do not have it, so they fall back to stylometry — counting sentence length, vocabulary, and repetition. That is not the same as verifying a watermark.

A free model endpoint is useful here because I can generate many samples without paying per request. I am using MonkeyCode's free model access, advertised as a 30 million token allowance, and its free server option as the cheap place to host a small probe.

I am not reproducing any provider's private watermark algorithm. I am building a toy version so the verifier concept is visible.

What you need

• Python 3.10+ with no third-party dependencies
• 15 minutes
• Optional: a MonkeyCode account and endpoint details from its console

If you do not set environment variables, the script uses a deterministic local fixture. That keeps the example runnable before you connect a real model.

Step 1: Build a tiny green-list verifier

A real watermark often splits tokens into “green” and “red” lists using a secret and a hash. The verifier then asks: Is the green-token fraction unexpectedly high?

Here is a toy version:

import hashlib, json, os, re, urllib.request

TOY_SOURCE = ("the quick brown fox jumps over the lazy dog "
              "while the clever fox watches the quiet dog")

def tokenize(text):
    return re.findall(r"[A-Za-z']+", text.lower())

def is_green(token, secret, threshold=60):
    digest = hashlib.sha256((token + secret).encode()).hexdigest()
    return int(digest, 16) % 100 < threshold

def green_rate(text, secret, threshold=60):
    tokens = tokenize(text)
    if not tokens:
        return 0.0
    return sum(is_green(t, secret, threshold) for t in tokens) / len(tokens)

def toy_watermarked_text(secret):
    tokens = tokenize(TOY_SOURCE)
    green = [t for t in tokens if is_green(t, secret)]
    red = [t for t in tokens if not is_green(t, secret)]
    out = []
    for i in range(20):
        if i % 10 < 6 and green:
            out.append(green[i % len(green)])
        elif red:
            out.append(red[i % len(red)])
    return " ".join(out)

def call_endpoint(prompt, temperature):
    base = os.environ.get("MONKEYCODE_BASE_URL")
    key = os.environ.get("MONKEYCODE_API_KEY")
    model = os.environ.get("MONKEYCODE_MODEL")

    if not (base and key and model):
        return toy_watermarked_text("example-secret-not-the-real-provider-key")

    body = json.dumps({
        "model": model,
        "messages": [{"role": "user", "content": prompt}],
        "temperature": temperature,
    })

    req = urllib.request.Request(
        base.rstrip("/") + "/chat/completions",
        data=body.encode(),
        headers={
            "Authorization": f"Bearer {key}",
            "Content-Type": "application/json",
        },
    )

    with urllib.request.urlopen(req, timeout=30) as resp:
        data = json.load(resp)

    return data["choices"][0]["message"]["content"]

def main():
    secret = "example-secret-not-the-real-provider-key"
    prompt = "Explain one limitation of public AI-text detectors."

    model_text = call_endpoint(prompt, temperature=0.4)
    human_text = ("I wrote this paragraph by hand after lunch, still unsure "
                  "whether the detector would call it synthetic.")

    print(f"model  known-secret green rate: {green_rate(model_text, secret):.3f}")
    print(f"human  known-secret green rate: {green_rate(human_text, secret):.3f}")
    print(f"model  wrong-secret green rate: {green_rate(model_text, 'wrong-secret'):.3f}")
    print(f"human  wrong-secret green rate: {green_rate(human_text, 'wrong-secret'):.3f}")

if __name__ == "__main__":
    main()

Save it as toy_watermark.py and run it:

python toy_watermark.py

Expected output from the local fixture:

model  known-secret green rate: 0.600
human  known-secret green rate: 0.455
model  wrong-secret green rate: 0.500
human  wrong-secret green rate: 0.500

Your human-text number may differ slightly. The important line is the last two: with the wrong secret, both collapse toward chance.

Step 2: Use a real free model endpoint

Set the environment variables before running the same script:

export MONKEYCODE_BASE_URL="your-base-url"
export MONKEYCODE_API_KEY="your-api-key"
export MONKEYCODE_MODEL="the-model-id-from-the-console"
python toy_watermark.py

The script assumes an OpenAI-compatible /chat/completions path. Check the MonkeyCode console for the exact base URL and model id. If the path differs, change only that string.

When the real endpoint is present, the script stops using the toy fixture. The key lesson remains: without the real provider secret, any third-party “AI detector” is not checking a watermark. It is guessing from surface statistics.

Step 3: Host a tiny verifier API on the free server

A free server is useful when I want to share a single curl command instead of a Python environment. This stdlib server depends on nothing:

import json
from http.server import BaseHTTPRequestHandler, HTTPServer
from urllib.parse import urlparse, parse_qs

from toy_watermark import green_rate, tokenize

class Handler(BaseHTTPRequestHandler):
    def do_GET(self):
        qs = parse_qs(urlparse(self.path).query)
        text = qs.get("text", [""])[0]
        secret = qs.get("secret", ["example-secret-not-the-real-provider-key"])[0]

        payload = {
            "green_rate": green_rate(text, secret),
            "token_count": len(tokenize(text)),
        }

        body = json.dumps(payload).encode()
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.end_headers()
        self.wfile.write(body)

if __name__ == "__main__":
    print("Serving on :8000 (Ctrl+C to stop)")
    HTTPServer(("0.0.0.0", 8000), Handler).serve_forever()

Save it as serve_probe.py and run:

python serve_probe.py

Then test it:

curl "http://localhost:8000/?text=the%20quick%20brown%20fox&secret=example-secret-not-the-real-provider-key"

Response:

{"green_rate": 0.75, "token_count": 4}

This is not production code. It is a learning probe with no authentication, no rate limiting, and no real provider secret.

Where the probe breaks

Run this error input:

print(green_rate("", "example-secret"))
print(green_rate("the the the the the", "example-secret"))

Output:

0.0
1.0

A single repeated token can push the green rate to 100%. A detector that sees only this number would call the second string “watermarked.” That is why cheap green-list demos are easy to fool with repetitive text.

Common mistakes:

• Using the example secret as real. It is not a provider key.
• Tuning the threshold until the result looks watermarked. That is confirmation bias, not verification.
• Ignoring temperature. Sampling parameters change the surface statistics that stylometry detectors use.
• Forgetting the endpoint path. A wrong base URL usually fails before any detection logic runs.

What you should understand after finishing

• A watermark verifier depends on a secret.
• Without that secret, a public detector is doing stylometry, not provenance checking.
• A toy green-list check is only meaningful when you control the secret.
• Free model access lets you test these ideas at zero per-request cost.
• A free server can turn a local probe into a small shared HTTP API.

Who should not use this approach

Do not use this toy probe to judge whether a student cheated, whether a paper is synthetic, or whether a model provider is lying about a watermark. For those questions, read the provider's primary technical note and use its official verifier, if one exists.

Extension exercise

Add a seed parameter to call_endpoint. Generate the same prompt at different temperatures and plot the green rate under a wrong secret. Predict whether the rate stays near 0.5 or drifts.

If you already have a MonkeyCode free tier, run this probe before sharing another detector screenshot. It will save you from trusting a confidence score that has no key behind it.