Reverse Engineering GPTZero: Verdicts are Document-Level, Non-Additive, and Order-Dependent

Lovish Manchanda — Sun, 01 Nov 2026 06:17:55 +0000

Over the past month building Penned—a free AI humanizer and detector designed to rewrite AI drafts into natural prose while strictly preserving facts, names, and numbers—we ran into a fascinating, counter-intuitive reality when evaluating our candidate picker against GPTZero v3 API (version 2026-10-07-base).

Most people (and even many ML practitioners) imagine AI text detectors as sentence-level statistical classifiers: each sentence has a perplexity or burstiness score, and the document score is simply an average or a cumulative tally.

Our empirical benchmarks on research paper drafts revealed something completely different: GPTZero's verdicts are document-level, non-additive, and heavily order-dependent.

Here are 7 empirical findings from our evaluation logs (eval/gptzero/FINDINGS.md), and what they mean for anyone building or evaluating NLP systems today.

1. One verdict per document, with a sharp cliff

Near the 0.5 decision threshold, GPTZero exhibits an extreme non-linear cliff.

Adding just one extra paragraph or even a single table caption flipped an entire document's classification from human to ai. Once that threshold was crossed, the API highlighted every single sentence as AI-generated—including sections that scored 0% AI when evaluated in isolation.

The detector is not simply aggregating independent sentence probabilities; it appears to classify the whole document context jointly, and then distributes retroactive confidence across the text.

2. Non-additive and order-dependent

The most striking behavior was in section combinations:

Section 5.2 alone: 0% AI score
Section 5.3 (paragraph 1) alone: 0% AI score
Section 5.2 + 5.3 together: 100% AI score

Even wilder was the sensitivity to ordering. We took the results section of an academic paper:

In one section permutation, it scored 39% AI (classified as human).
In the paper’s actual narrative order, the identical text scored 100% AI.

Furthermore, swapping just one paragraph (section 4.3) for an alternative Penned-generated candidate flipped the entire real-order paper half from 100% AI down to 40% AI (human). This demonstrates that document-level classification is governed by non-linear transition dynamics rather than bag-of-words or simple token perplexity.

3. Sentence probabilities come in segmented blocks

When inspecting GPTZero's generated_prob at the sentence level, the scores are not smooth. GPTZero internally segments documents into structural blocks.

Even in a document overall classified as human (e.g. 39% AI), individual AI segments remain clearly visible:

Sections 4.1–4.3, 6, 7, 8 hovered near ~100% generated probability.
Sections 4.4 and 5.1 hovered near ~10%.

The global document score appears to weigh these segmented blocks against each other rather than averaging raw sentence scores.

4. Precision & numbers act as strong human anchors

Numerical specificity has an outsized effect on detection:

In our test paper, replacing empirical numbers with generic variable placeholders (e.g. replacing 42.8% with N%) turned a 39% AI human document into 100% AI.

AI detectors heavily associate dense, concrete domain measurements and precise numerical observations with authentic human scientific writing.

5. Formatting artifacts penalize authentic text

How text is pasted or extracted matters significantly:

Taking the exact same 39% AI text and copying it directly from a PDF with hard line breaks increased the AI detection score to 60% AI.
Interestingly, clean semantic paragraph breaks did not cause this jump—it was the erratic hard line wraps from PDF copy-pasting that tripped the detector.

6. Discussion/Conclusion vs. Results: The genre vulnerability

Summarizing findings is inherently dangerous under modern detectors:

Discussion & conclusion sections (which synthesize findings, restate implications, and provide broad takeaways) remained flagged as 100% AI under almost every rewrite strategy we attempted—including multiple candidate passes, restyled drafts, intentionally broken grammar, and strict fact-only sentences.
Results-style reporting (raw measurements, step-by-step procedural descriptions, and direct observations) is where humanization actually works reliably.

Because LLMs are trained to summarize and synthesize, detectors have learned that synthetic summaries look overwhelmingly "machine-like", even when written by humans.

7. Open-source detectors are completely blind to this

We benchmarked several in-house and open-source detection models against the exact same test suite:

DeBERTa (desklib): Scored 0–5% AI on texts that GPTZero flagged as 100% AI (including raw Gemini outputs).
ModernBERT: Showed almost zero correlation with GPTZero's labels.
Binoculars / Fast-DetectGPT (with Qwen2.5-0.5B): Failed to produce class separation.
ZeroGPT: Scored 0% across all test samples.

In short: local open-source detectors cannot see what proprietary v3 detectors see.

What this means for building AI Humanizers

When we started building Penned, our initial pipeline scored rewritten candidates paragraph-by-paragraph using local DeBERTa and a logistic regression GPT-2 feature checker.

These findings showed us that a greedy per-paragraph picker cannot work. Because the detector's decision boundary is non-additive and order-dependent:

Optimizing paragraph 1 in isolation can sabotage paragraph 4.
The candidate picker needs a document-level coordinate descent signal—iteratively testing candidate combinations across the whole document structure to find the global optimum.
Fine-tuning local candidate pickers requires calibrating directly on large batches of paired detector verdict labels.

We are currently building open benchmarks and testing these calibration pipelines in public.

Penned is free to test at penned-ai.vercel.app. Check it out and let us know your thoughts on our findings!

DEV Community: Lovish Manchanda