DEV Community: isaac

I Tried to Steal a Watermark Key Without Having It

isaac — Sat, 15 Aug 2026 10:39:00 +0000

I Tried to Steal a Watermark Key Without Having It | WMTrace

Sixty documents, no key, no model, no detector. One watermark leaked. The second seemed to fight back—until the evidence exposed a false lead.

gtesei.github.io

Anthropic is watermarking Claude's output. Here's how text watermarks actually work (with runnable code)

isaac — Sat, 15 Aug 2026 02:34:52 +0000

Anthropic announced this week that future Claude models will watermark their text output, using a version of Google DeepMind's SynthID-Text. Google already runs it in Gemini at production scale. Which means statistical text watermarks
just went from research curiosity to something you'll be reading every day — probably without noticing.

Most people, including most engineers, have never seen how one works. The mechanism is surprisingly simple, and you can hold all of it in your head. Let's build the intuition, then run a real detector.

## The core trick: a keyed coin flip over the vocabulary

An LLM produces text by choosing the next token from a probability distribution. At many positions, several tokens are all perfectly fine: team/group, short/concise, customer/client. A watermark hides inside that freedom.

The classic scheme (Kirchenbauer et al.'s "green list" watermark, the family SynthID-Text belongs to) works like this:

Take a secret key.
At each position, hash the key together with the recent context (say, the previous two words). The hash pseudo-randomly marks a fraction γ of the vocabulary as "green" — a different green set at every position.
When sampling, nudge generation to prefer green tokens when the choice doesn't matter much.

A human writer, or a model without the key, picks green tokens at the base rate γ. The watermarked model picks them far more often. Detection is then just counting:

  z = (X − γN) / sqrt(N·γ·(1−γ))

where N is the token count and X the green count. No access to the model needed — only the key.

## A worked example you can verify

Here's a toy configuration (from WMTrace, the open-source workbench this post is built on): lowercase word tokens, context = previous two words, γ = 0.25, and a token is green when CRC32(key + "|" + context + "|" + token) mod 4 == 0 under the published demo key demo-key-172118.

Original (human-written):

The project team reviewed the latest safety report during a quiet morning session. Members discussed the updated timeline, examined several open risks, and agreed to publish a concise summary before the next client meeting. …

Result: 15 green out of 49 tokens. Expected under the null: 12.25. z = 0.91, p = 0.23. Nothing.

Watermarked (same meaning, green-preferring word choices):

The initiative team examined the latest safety report during a quiet morning meeting. Participants discussed the current timeline, examined multiple outstanding risks, and
agreed to publish a concise summary before the next client session. …

Result: 25 green out of 49. z = 4.21, exact binomial p = 8.0×10⁻⁵. The text reads identically to a human — but under the key, it glows.

Run it yourself in four lines:

  from wmtrace.core import AnalysisContext, build

  result = build("kgw-toy").run(AnalysisContext(raw=your_text)).result
  print(result["green_tokens"], "/", result["usable_tokens"],
        "z =", result["statistic"], "p =", result["p_value"])

## Watch it light up, token by token

WMTrace ships a small web UI. pip install from the repo, then:

  wmtrace serve
  # open http://127.0.0.1:8177/?demo=1

?demo=1 generates a watermarked sample under the demo key and analyzes it live — every keyed-green token highlighted, exact binomial p-values, and Holm correction across detectors:

The Embed tab also does two lower-tech channels that predate LLM watermarks and still show up in the wild:

Zero-width Unicode payloads — bits hidden as invisible U+200B/U+200C characters. Fully invisible, trivially destroyed by normalization, and trivially forged.
Lexical codebooks — synonym pairs encoding bits (starts/begins = 0/1).

Both round-trip in the UI, and the Unicode inspector flags them (plus bidi controls and mixed-script confusables — useful far beyond watermarks).

## What a watermark detector can and cannot tell you

This is the part most coverage gets wrong, so it's baked into the tool's output format. Every result carries an evidence grade and an explicit scope statement. The rules that matter:

A negative result does not mean a human wrote it. Paraphrasing, translation, short text, or simply a different key all produce negatives.
A positive result means compatibility with a declared scheme + key + configuration. Nothing more. Under a published demo key it proves nothing about any provider.
You cannot detect Claude's or Gemini's production watermark yourself. Detection requires the provider's secret key. Any tool claiming otherwise is guessing — WMTrace deliberately has no anthropic detector and won't until an official verification API exists.
Unkeyed channels (zero-width, codebooks) are evidence of something, not of someone. Anyone can insert them, including someone trying to frame a text as AI-written.

That last point generalizes: as watermarks become policy-relevant (schools, courts, hiring), the difference between "this text is compatible with scheme X under key K" and "an AI wrote this" stops being pedantry and starts being due
process.

## Try it

gtesei / llm-watermark

Forensic detection, benchmarking, and explainable analysis of LLM text watermarks, provenance signals, and hidden-payload channels — with a live web UI

🔍 WMTrace

Forensic detection, benchmarking, and explainable analysis of LLM text watermarks, provenance signals, and hidden-payload channels.

WMTrace is a scheme-aware forensic workbench, not a monolithic "AI detector". It answers narrow, honest questions — does this text carry calibrated evidence for a declared watermark scheme, key, and configuration? — and it states, with every result, exactly what that result does not prove.

Important

A negative watermark result does not mean text was human-written. A recovered payload does not prove who embedded it. No result here supports provider attribution without trusted key provenance.

pip install -e ".[dev]"     # from a clone; Python 3.10+
wmtrace serve               # polished web UI at http://127.0.0.1:8177

The Analyze tab on a generated watermarked sample: summary pills per detector, calibrated statistics (z = 16.1, Holm-adjusted p = 3.5e-48), and the keyed green-token heatmap. Reproduce it locally with wmtrace serve → http://127.0.0.1:8177/?demo=1.

Design principles

Evidence,…

View on GitHub

Pure Python, no ML dependencies, Apache-2.0. The worked example above isn't just prose — it's pinned by golden tests in the repo, so the numbers in this post are executable. The docs/RESEARCH.md file covers the full scheme taxonomy
(SynthID's tournament sampling, distortion-free schemes, multi-bit payloads, C2PA provenance) and what each provider has actually deployed, with citations.

Feedback very welcome — especially from anyone working on watermark robustness, multilingual calibration, or provenance standards.

I sketched 29 agentic AI design patterns in a Da Vinci–style notebook (open source)

isaac — Sun, 31 May 2026 20:56:18 +0000

Spent the weekend turning every pattern in my agentic_design_patterns repo into a hand-drawn style diagram, using a sepia/parchment “bozza-scientifica” aesthetic — closer to a Leonardo da Vinci notebook page than a typical slide-style flowchart.

Repo: https://github.com/gtesei/agentic_design_patterns

Curious to hear feedback.

Turning OpenAPI Specs into Runtime LLM Tools in Pi

isaac — Thu, 14 May 2026 03:12:34 +0000

I built pi-openapi-tools — a Pi extension that ingests an OpenAPI/Swagger spec and dynamically registers one callable tool per API operation.

Instead of hand-writing wrappers for every endpoint, you point Pi at a spec URL and immediately start calling generated tools.

Why I built this

Most API-agent integrations still require too much manual glue:

mapping endpoints to functions
defining parameter schemas
maintaining naming consistency
wiring auth/token handling
repeating the same setup for every API

That friction slows experimentation and makes agent tooling brittle.

I wanted a runtime-first workflow:

spec → generated tools → live API calls

What the extension does

Given an OpenAPI or Swagger URL, pi-openapi-tools:

generates one Pi tool per operation
supports Swagger 2.0 and OpenAPI 3.x
builds structured parameter schemas automatically
resolves request bodies by declared media/content type
handles path, query, and header parameters
adds slash commands for managing generated toolsets

It also includes:

prefix-scoped registrations (/swagger-tools:list-prefixes)
prefix cleanup (/swagger-tools:remove-prefix)
tool inspection (/swagger-tools:describe)
token/auth helpers (/swagger-tools:auth)

60-second quickstart

Install:

pi install npm:pi-openapi-tools

Generate tools from Petstore:

/swagger-tools https://petstore.swagger.io/v2/swagger.json --prefix pet
/swagger-tools:list

You’ll get generated tools like:

pet_getstoreorderbyid
pet_postuser
etc.

Implementation details that mattered

Prefix-scoped additive registration
Multiple API toolsets can coexist cleanly.
Stable naming + collision-safe dedupe
Generated names are sanitized and safely deduplicated.
No unregister API in Pi? Stub instead.
Removed tools become non-executable guidance stubs rather than silently disappearing.
Content-type-aware serialization
Request body handling follows declared media types (JSON, multipart, urlencoded, text, etc.).

Docs + walkthrough

Tutorial: Petstore walkthrough
Repo: GitHub repository

If you try it on large or messy specs, I’d especially love feedback on:

naming ergonomics
auth edge cases
request body/media handling
discoverability in very large APIs

If this is useful, feel free to star the repo and share your API use case.