Every AI watermark headline makes the same promise.
It says you can spot machine text with confidence.
I did not want another opinion post.
I wanted a small probe I could run.
I had access to a free model endpoint and a free server slot from MonkeyCode.
The project is open source.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
The goal was narrow.
Build a canary that perturbs generated text and records signal drift.
No vendor benchmark, no vague demo.
Why a canary?
Detectors flip for weird reasons.
A comma changes a score.
A synonym swap changes a score.
That makes one-shot demos useless.
A canary gives you before and after behavior.
It generates a short text.
It perturbs the text in fixed ways.
It records whether a simple similarity signal moves.
This is not a detector benchmark.
It is an endpoint consistency test.
Can a free model produce enough stable text for a noise experiment?
The answer tells you where the free tier is useful.
Why deterministic perturbations?
A random rephrase hides the cause.
A fixed rule shows the cause.
Lowercasing changes only case.
Doubling spaces changes only whitespace.
A synonym swap changes only two tokens.
That makes reads easy.
You can compare one row to another.
You can see if the endpoint is stable or noisy.
Why Jaccard, not a classifier?
Classifier scores are hard to verify.
They hide their threshold.
They change between versions.
Jaccard is boring and transparent.
It compares token sets.
A perfect copy returns 1.0.
A full rewrite returns near zero.
That is enough for a canary.
What you need
- A free MonkeyCode endpoint.
- A free server slot or any Python host.
- A token budget of 100,000 generated tokens for the first run.
- Ten minutes to read the results file.
I did not hardcode a model name.
Set the model id from your MonkeyCode console.
The script is provider-agnostic.
The harness
The script does three things.
It generates short text from five prompts.
It applies five deterministic perturbations.
It writes latency, length, error, and overlap to results.jsonl.
Create canary.py.
import os
import json
import time
import uuid
import argparse
import requests
API = os.environ.get("MONKEYCODE_API_URL", "http://localhost:8000/v1/chat/completions")
KEY = os.environ.get("MONKEYCODE_API_KEY", "")
MODEL = os.environ.get("MONKEYCODE_MODEL", "")
PROMPTS = [
"Write a two-sentence release note for a bug fix.",
"Summarize a failed deploy in two sentences.",
"Describe a flaky test without blaming the author.",
"Explain a rate limit to a non-engineer in two sentences.",
"Draft a rollback notice for a database migration.",
]
PERTURBATIONS = {
"lower": lambda s: s.lower(),
"upper": lambda s: s.upper(),
"space": lambda s: s.replace(" ", " "),
"punct": lambda s: s.replace(".", "..").replace(",", ",,"),
"syn": lambda s: s.replace("failed", "broke").replace("fix", "repair"),
}
def generate(prompt):
payload = {
"model": MODEL,
"messages": [{"role": "user", "content": prompt}],
"max_tokens": 96,
"temperature": 0.7,
}
started = time.time()
try:
r = requests.post(API, headers={"Authorization": f"Bearer {KEY}"}, json=payload, timeout=30)
except requests.RequestException as exc:
return None, 0.0, str(exc)
latency = time.time() - started
if r.status_code != 200:
return None, latency, r.text[:120]
data = r.json()
return data["choices"][0]["message"]["content"].strip(), latency, None
def jaccard(a, b):
A = set(a.lower().split())
B = set(b.lower().split())
if not A or not B:
return 0.0
return len(A & B) / len(A | B)
def main():
parser = argparse.ArgumentParser()
parser.add_argument("--runs", type=int, default=10)
parser.add_argument("--budget-tokens", type=int, default=100_000)
args = parser.parse_args()
rows = []
budget = args.budget_tokens
for prompt in PROMPTS:
for _ in range(args.runs):
if budget <= 0:
break
base, base_lat, err = generate(prompt)
if base is None:
rows.append({"prompt": prompt[:40], "perturbation": "error", "len_base": 0,
"len_changed": 0, "jaccard": 0.0, "latency_ms": round(base_lat * 1000, 1),
"error": err, "run_id": str(uuid.uuid4())[:8]})
continue
budget -= len(base.split()) * 2
for name, fn in PERTURBATIONS.items():
changed = fn(base)
rows.append({
"prompt": prompt[:40],
"perturbation": name,
"len_base": len(base.split()),
"len_changed": len(changed.split()),
"jaccard": round(jaccard(base, changed), 3),
"latency_ms": round(base_lat * 1000, 1),
"error": err,
"run_id": str(uuid.uuid4())[:8],
})
with open("results.jsonl", "w") as f:
for row in rows:
f.write(json.dumps(row) + "\n")
print(f"wrote {len(rows)} rows to results.jsonl")
if __name__ == "__main__":
main()
Run it with a hard budget.
export MONKEYCODE_API_URL="https://your-endpoint/v1/chat/completions"
export MONKEYCODE_API_KEY="your-key"
export MONKEYCODE_MODEL="your-model-id"
python canary.py --runs 10 --budget-tokens 100000
Read the output
A mock dry run wrote 50 rows in 4 seconds.
That only verifies the script shape.
Your endpoint numbers will differ.
Each row shows one perturbation.
The jaccard value measures token overlap.
A low value means the perturbation removed most of the signal.
Use this decision table.
| Median jaccard | Error rate | Call |
|---|---|---|
| >= 0.6 | < 2% | Useful for small generator tests |
| 0.4 to 0.6 | 2% to 8% | One-off canary only |
| < 0.4 | > 8% | Do not trust for regression work |
Latency matters too.
If p95 latency goes above 800 ms, drop the timeout to 15 seconds.
That forces a faster failure when the free server is busy.
What this canary does not prove
It does not measure model quality.
It does not prove watermark detection works.
It does not give you a production detector.
It only shows whether the free endpoint is stable enough for noise experiments.
That is a simpler question.
And it is the question a solo developer can answer today.
Who should skip this
Skip it if you need regulated content checks.
Skip it if you need a real detector score.
Skip it if your team already has a paid evaluation harness.
This probe is for builders with a small budget and a skeptical streak.
Try the canary before you trust a headline
MonkeyCode advertises free 30M tokens and a free server option.
That is enough to run this probe many times.
Grab the free endpoint and run the canary first, not a product demo.
Share the JSONL if the pattern surprises you.
Top comments (0)