Alain Airom (Ayrom)

Posted on Sep 6

What is Text Watermarking?

#watermarking #bias #llm #bob

Implementing a didactical ‘water marking’ sample application (for self learning and explanation)

Since recently there is a lot of hype of gen ai tools implementing text watermarking for the text they provide. I wanted to learn what it is about and how it could be implemented and detected. So I made a didactical application for myself using Bob. The implementation follows.

Introduction & Theoretical Foundations

Text watermarking in Generative AI refers to the practice of embedding a hidden, detectable signal within AI-generated text. Unlike digital watermarks applied to continuous media like images or audio, a text watermark operates over a discrete domain of tokens without altering the fundamental meaning or readability of the text.

In autoregressive Large Language Models (LLMs), watermarking is integrated directly into the token sampling process rather than applied post-hoc.

The Green-List / Red-List Partitioning Algorithm (Kirchenbauer et al., 2023)

Explanations here: https://github.com/aairom/text-watermarking-test/blob/main/Docs/text_watermarking_algorithms_20260906.md && https://github.com/aairom/text-watermarking-test/blob/main/Docs/WatermarkingImplementation.md

The primary reference framework for LLM text watermarking relies on dynamic vocabulary partitioning:

Context Extraction: At step t, the system extracts a context window ct=(tt−h,…,tt−1) of the preceding h tokens.
Deterministic Pseudo-Random Seeding: The context sequence and a secret key k are hashed using SHA-256 to generate a pseudo-random seed st=SHA-256(ct∥k).
Vocabulary Partitioning: Using st, the vocabulary V is partitioned into a Green List Gt of size γ∣V∣ and a Red List Rt of size (1−γ)∣V∣.
Logit Biasing: During generation, a bias parameter δ>0 is added to the unnormalized logit zv for all green tokens v∈Gt:

Sampling: Softmax normalization is applied to z~v, biasing the model toward green-list tokens.

Statistical Hypothesis Testing & Detection

Detection reconstructs the sequence of green lists without needing access to the model logits or original prompt:

Null Hypothesis (*H*0): Text is human-written or unwatermarked. Each token lands in Gt with probability γ.
Test Statistic (*z*-score): For a sequence of length T with ∣G∣ green tokens:

Decision Rule: Reject H*0 (flag as watermarked) if *z≥z*threshold (typically *z≥4.0, corresponding to p≈3.17×10−5).

Didactical Application Architecture

To explore this concept hands-on, I built a local didactical full-stack application using IBM Bob.

System Flowchart

Data Flow Generation and Detection

Generation Sequence

The data flow generation shows how the application creates a "watermarked" text.

Detection Sequence

This is the architecture of "watermark" detection

Code Implementation & Excerpts

Deterministic Green List Generation

The implement code uses SHA-256 context hashing and random.Random Fisher-Yates shuffling:

def _hash_context_to_seed(context_ids: tuple[int, ...], secret_key: str) -> int:
    payload = secret_key.encode("utf-8") + struct.pack(f"{len(context_ids)}I", *context_ids)
    digest = hashlib.sha256(payload).digest()
    seed = struct.unpack(">Q", digest[:8])[0]
    return seed

def get_green_list(
    context_ids: tuple[int, ...],
    secret_key: str,
    vocab_size: int,
    gamma: float,
) -> frozenset[int]:
    seed = _hash_context_to_seed(context_ids, secret_key)
    rng = random.Random(seed)
    green_size = max(1, int(gamma * vocab_size))
    all_ids = list(range(vocab_size))
    rng.shuffle(all_ids)
    return frozenset(all_ids[:green_size])

Biased Token Sampling Loop

During token generation, δ is applied directly to logits of green-listed candidates:

def _sample_next_token(
    context_ids: tuple[int, ...],
    green_list: Optional[frozenset[int]],
    delta: float,
    temperature: float,
    rng: random.Random,
) -> int:
    prev_id = context_ids[-1] if context_ids else _COMMON_START_TOKENS[0]
    candidates = _BIGRAM_MODEL.get(prev_id, [])

    if not candidates:
        candidates = list(range(4, min(200, _VOCAB_SIZE)))

    pool = list(dict.fromkeys(candidates + list(range(4, 60))))[:80]

    logits = []
    for tid in pool:
        logit = 1.0
        if green_list is not None and tid in green_list:
            logit += delta  # Add bias before temperature scaling
        logits.append(logit / temperature)

    max_l = max(logits)
    exp_l = [math.exp(l - max_l) for l in logits]
    total = sum(exp_l)
    probs = [e / total for e in exp_l]

    r = rng.random()
    cumulative = 0.0
    for tid, p in zip(pool, probs):
        cumulative += p
        if r <= cumulative:
            return tid
    return pool[-1]

Detection Engine & Statistical Test

Detection evaluates z-scores and one-sided p-values via the error function:

def detect_watermark(
    text: str,
    secret_key: str,
    gamma: float = 0.5,
    context_window: int = 1,
    threshold_z: float = 4.0,
) -> DetectionResult:
    token_ids = _tokenize(text)
    T = len(token_ids)

    if T == 0:
        return DetectionResult(...)

    token_infos: list[TokenInfo] = []
    green_count = 0

    for i, tid in enumerate(token_ids):
        if i == 0:
            ctx = (_SPECIAL_TOKENS["<BOS>"],)
        else:
            ctx = tuple(token_ids[max(0, i - context_window): i])

        green_list = get_green_list(ctx, secret_key, _VOCAB_SIZE, gamma)
        seed_val = _hash_context_to_seed(ctx, secret_key)
        is_green = tid in green_list
        if is_green:
            green_count += 1

        token_infos.append(TokenInfo(
            token_id=tid,
            token_text=_ID_TO_TOKEN.get(tid, "<UNK>"),
            position=i,
            is_green=is_green,
            context_hash_seed=seed_val % (2 ** 31),
            green_list_size=len(green_list),
        ))

    expected_g = gamma * T
    z = (green_count - expected_g) / math.sqrt(T * gamma * (1 - gamma))
    p_value = 0.5 * math.erfc(z / math.sqrt(2))
    confidence_pct = round((1.0 - p_value) * 100, 2)
    verdict = "WATERMARKED" if z >= threshold_z else "NOT_WATERMARKED"

    return DetectionResult(
        text=text,
        token_infos=token_infos,
        total_tokens=T,
        green_tokens=green_count,
        expected_green=round(expected_g, 2),
        green_fraction=round(green_count / T, 4),
        z_score=round(z, 4),
        p_value=round(p_value, 6),
        confidence_pct=confidence_pct,
        verdict=verdict,
        threshold_z=threshold_z,
    )

REST API Endpoints (FastAPI)

The REST API using FastAPI implemented provides a full access through APIs to handle the actions like generation, detection…

@app.post("/api/generate", response_model=GenerateResponse, tags=["generation"])
def generate(req: GenerateRequest) -> GenerateResponse:
    wm = generate_text(
        prompt=req.prompt,
        secret_key=req.secret_key,
        gamma=req.gamma,
        delta=req.delta,
        temperature=req.temperature,
        max_new_tokens=req.max_new_tokens,
        context_window=req.context_window,
        watermarked=True,
        seed=42,
    )
    baseline = generate_text(
        prompt=req.prompt,
        secret_key=req.secret_key,
        gamma=req.gamma,
        delta=req.delta,
        temperature=req.temperature,
        max_new_tokens=req.max_new_tokens,
        context_window=req.context_window,
        watermarked=False,
        seed=42,
    )
    return GenerateResponse(watermarked=wm, baseline=baseline)

@app.post("/api/detect", response_model=DetectionResult, tags=["detection"])
def detect(req: DetectRequest) -> DetectionResult:
    return detect_watermark(
        text=req.text,
        secret_key=req.secret_key,
        gamma=req.gamma,
        context_window=req.context_window,
        threshold_z=req.threshold_z,
    )

Attack Simulator & Robustness Testing

To evaluate how well the watermark withstands real-world modifications, the application includes a dedicated Attack Simulator Panel (RobustnessPanel.tsx). Watermarks embedded during generation must remain detectable even after the text undergoes editing, paraphrasing, or noise injection.

Supported Attack Modes

The simulator allows testing four core text perturbation strategies at configurable intensity levels (intensity∈[0.01,1.0]):

Deletion Attack: Randomly strips a percentage of tokens from the text. This simulates aggressive summarization or partial content extraction.
Substitution Attack (Paraphrasing / Typo Injection): Replaces candidate tokens with synonyms or introduces adjacent character swaps. Because green-list selection depends on preceding context ct, modifying even a single token desynchronizes subsequent green-list lookups.
Insertion Attack: Inserts random filler words (e.g., "actually", "basically", "like") into the sequence, introducing local shift distortions into the context window.
Local Shuffle Attack: Randomly scrambles small token windows (3-grams) to disrupt local sentence structure without completely altering word frequency.

API Endpoint (POST /api/perturb);

@app.post("/api/perturb", response_model=PerturbResponse, tags=["robustness"])
def perturb(req: PerturbRequest) -> PerturbResponse:
    """
    Apply a text perturbation and re-run watermark detection.
    Shows watermark persistence / degradation under attack.
    """
    if not req.text.strip():
        raise HTTPException(status_code=422, detail="text must not be blank")

    perturbed = apply_perturbation(req.text, req.mode, req.intensity, req.seed)
    original_det = detect_watermark(
        req.text, req.secret_key, req.gamma, req.context_window, req.threshold_z
    )
    perturbed_det = detect_watermark(
        perturbed, req.secret_key, req.gamma, req.context_window, req.threshold_z
    )

    return PerturbResponse(
        original_text=req.text,
        perturbed_text=perturbed,
        mode=req.mode,
        intensity=req.intensity,
        original_detection=original_det,
        perturbed_detection=perturbed_det,
    )

Empirical Robustness Insights

Context Sensitivity: Because the pseudo-random seed st depends on ct=(tt−h,…,tt−1), a single token edit at position i invalidates the green-list match for the subsequent h tokens.
Sequence Length Resilience: For longer generated sequences (T≥150), the z-score margin is sufficiently high (z≥7.0) that the signal remains statistically significant even after 20–30% token degradation.

Theoretical Algorithm Deep Dive

The Algorithm Reference module (MathExplainer.tsx) provides an interactive breakdown of the mathematical properties governing the Kirchenbauer et al. framework.

Parameter Sensitivity & Trade-offs: the behavior and detectability of the watermark are governed by three primary hyper-parameters:

Parameter	Notation	Description	Impact of Higher Values
Green Fraction	γ	Ratio of vocabulary assigned to Gt.	Increases expected baseline green token count (γT), requiring longer sequences for clear detection.
Logit Bias	δ	Direct bias added to unnormalized logits of Gt tokens.	Increases detection confidence (z-score), but higher values (δ>3.0) can compromise text perplexity and fluency.
Context Window	h (or k)	Number of preceding tokens used as hashing context ct.	Increases resistance against naive token substitution, but makes detection more sensitive to context disruptions.

Mathematical Detection Limit & One-Sided p-Value

To determine the exact likelihood of false positives (a human-written text accidentally triggering a watermark detection), the detection engine converts the computed z-score into an upper-tail p-value using the complementary error function (erfc):

Where:

z≥4.0⟹p≈3.17×10−5 (99.997% confidence)

z≥6.0⟹p≈9.86×10−10 (99.9999999% confidence)

Conclusion

Text watermarking offers a robust, mathematically sound approach to establishing content provenance for large language models. By embedding statistical signals directly during sampling, models can output text that remains entirely natural to human readers while carrying a verifiable fingerprint.

Building this didactical application with IBM Bob highlighted the core mechanics of the algorithm:

Zero Model Overhead: Detection requires no knowledge of model weights or prompts—only the secret key and candidate text.
Controllable Quality Trade-offs: The logit bias δ and green fraction γ provide fine-grained control over detectability versus text quality.
Statistical Power: Even under minor perturbations, longer sequences (T≥200) maintain strong statistical significance against paraphrase attacks.

As generative AI continues to scale, watermarking frameworks like Kirchenbauer et al.'s Green/Red partitioning and DeepMind's SynthID will serve as critical foundational blocks for transparency, regulatory compliance, and content integrity. """

Thanks for reading #️⃣

DEV Community