An LLM API Returned 4.17 Million Characters: Using MD5 Fingerprints to Prove It Was Duplication, Not Generation

A draft that should have been 3,000 characters came back at 4,169,336 — and splitting it by section and fingerprinting with MD5 proved that only 28 sections were ever generated and the other 2,037 were byte-identical copies.

In a WordPress AI article generation plugin I built, a draft that should have been around 3,000 characters was saved at 4,169,336 characters. I only noticed because the estimated reading time displayed as 8,712 minutes.

It is tempting to write this off as "the LLM ran away and generated the same thing over and over", but that does not add up with the billed token count. This article starts from that contradiction, uses section splitting and MD5 fingerprint analysis to separate "characters that were generated" from "characters that were copied", and ends with physical evidence that the cause was not on the client side.

The same technique lets you triage "the output looks weird" around LLM APIs with numbers instead of impressions.

Both of my first two hypotheses were wrong

When I first saw the failure, I formed two hypotheses. Both suspected the client side — my own code.

1. A bug in the continuation loop's stop condition — mishandling finishReason, failing to stop continuation requests, and accumulating the results
2. Duplicate appending in the body concatenation — appending the entire text so far on every chunk merge

Both plausible. That is in fact where I started. But the measured values in the API console made both look doubtful.

Here is what one runaway request actually measured:

Request count  : 7
Input tokens   : 13.86k
Output tokens  : 19.55k

19.55k output tokens is at most a few tens of thousands of characters in Japanese. Nowhere near 4.17 million.

If hypothesis 1 (continuation loop) were right, the request count would be far larger; if hypothesis 2 (duplicate appending) were right, output tokens would stay normal while only the body ballooned — the latter was still alive. So I went to look at what was actually saved.

Triage 1: How many times was it saved?

A WordPress post exposes date and modified if you hit the REST API read-only. If an appending loop had been running, the post would have been updated multiple times.

curl -s "https://example.com/wp-json/wp/v2/posts/<ID>?context=edit" \
  | jq '{date, modified, len: (.content.raw | length)}'

{
  "date": "2026-08-22T21:42:43",
  "modified": "2026-08-22T21:42:43",
  "len": 4169336
}

date and modified match exactly, down to the second. In other words, 4.17 million characters were written by a single wp_insert_post(). There is no appending loop and no continuation-fetch loop. Hypothesis 2 dies here.

I also audited the entire plugin code and found not a single place that concatenates a response body onto itself. The story that the client simply saved what it received became much stronger.

Still, that is only a claim of innocence. To prove it, I had to look inside those 4.17 million characters.

Triage 2: Split into sections and take MD5 hashes

This is the heart of it. Reading an enormous generated text by eye is impossible, but if you split it into structural units and hash them, repetition patterns become obvious at a glance.

The article body was divided by h2 headings, so I split on h2, took an MD5 fingerprint of each section, and counted occurrences.

import hashlib
import re
from collections import Counter

with open("draft.html", encoding="utf-8") as f:
    body = f.read()

# split on h2 boundaries (keep anything before the first heading as one section)
sections = re.split(r"(?=<h2)", body)
sections = [s for s in sections if s.strip()]

fingerprints = [hashlib.md5(s.encode("utf-8")).hexdigest() for s in sections]
counter = Counter(fingerprints)

print(f"Total sections  : {len(sections)}")
print(f"Unique sections : {len(counter)}")
for rank, (digest, n) in enumerate(counter.most_common(3), 1):
    idx = fingerprints.index(digest)
    print(f"  #{rank}  {n:>4} times  section length {len(sections[idx]):>5}")

The output looked like this (the actual digest values are environment-specific, so they are replaced with ranks):

Total sections  : 2065
Unique sections : 28
  #1   432 times  section length approx. 1700 chars
  #2   431 times  section length approx. 1700 chars
  #3   357 times  section length approx. 1700 chars

Of 2,065 sections, only 28 are unique. The most frequent section appears 432 times, and since the MD5 values match, those are completely identical strings — not one byte different.

Looking at the ordering as well, the pattern was "section 0 → one new section → fixed suffix", repeating aperiodically about 350 times. Because it is not periodic, it is also not a simple copy-paste loop.

Why this is provably "not generation"

This is the decisive part. The generation parameter was temperature = 0.7.

With sampling at temperature 0.7, the probability that 1,700 characters (several hundred tokens) reproduce byte-for-byte identically 432 times is effectively zero. When you sample each token from a probability distribution, repeating the same topic will always produce variation in wording. And indeed, the varied repetitions did appear as different hashes among the 28.

So the 4.17 million characters decompose like this:

Category	Reality
Actually generated	28 unique sections ≒ 19.55k tokens
Not generated	the remaining 2,037 sections = byte-level copies of those 28

And "the 28 sections that were actually generated" lines up cleanly with the 19.55k output tokens measured in the API console. The contradiction from the opening resolves here: billing and quota management were only counting the generated part, and the duplication happened downstream of it.

The conclusion is that the model generated a normal amount, and the layer that assembles and returns the response duplicated the same sections roughly 2,000 times, returning a body of about 4.2MB. As for the maxOutputTokens = 32768 cap itself, a reproduction test with the cap lowered to 50/200 confirmed that MAX_TOKENS stops correctly, so the generation limit was being respected. What was broken was not generation but response assembly.

I am not asserting that this behavior is permanent (it is one occurrence observed in my environment). What matters is that the client side could be cleared by physical evidence rather than by claims about the code.

The fix is not retrying — it is rejecting at the door

When the cause is not on your side, you cannot fix it by changing your code. All you can do is avoid ingesting a broken response. So I put three safety valves in place, all of them before parsing and before saving.

// (1) cut off by raw response size (before JSON parsing)
if ( strlen( $raw_body ) > 1500000 ) {
    return new WP_Error( 'acs_oversized_response',
        'Response size is abnormal (' . number_format( strlen( $raw_body ) ) . ' bytes). Generation aborted.' );
}

// (2) cut off by body length (before wp_insert_post)
if ( mb_strlen( $content ) > 100000 ) {
    return new WP_Error( 'acs_oversized_content',
        'Body is abnormally long, so saving was aborted (' . number_format( mb_strlen( $content ) ) . ' chars).' );
}

// (3) detect repeated identical headings
preg_match_all( '/<h2[^>]*>(.*?)<\/h2>/s', $content, $m );
$dupes = array_filter( array_count_values( array_map( 'trim', $m[1] ) ),
    static fn( $n ) => $n >= 3 );
if ( $dupes ) {
    return new WP_Error( 'acs_repeated_headings',
        'Repeated identical headings detected, so saving was aborted.' );
}

Three design points:

• Put (1) before JSON parsing. Handing a 4.2MB body to json_decode() first wastes memory and time. Byte length is the only indicator available before reading, so make it the first gate
• Do not swallow failures silently. All three surface as explicit errors. Quietly truncating turns into a different failure — "for some reason the article is short"
• Be selective about retrying. Only a failure inside a batch generation gets one automatic retry; there is no unconditional retry, because it may just fetch the same broken response again

For verification I used the actual 4.17-million-character data from the real incident as input. All 7 safety-valve tests passed and the 4 existing regression tests passed too. Normal generation against the real API completed in 43.1 seconds / 10,283 characters, and under a long-form condition 49.5 seconds / 6,948 characters. I also raised the timeout from 90 to 180 seconds (a long-form measurement stretched to 176.5 seconds).

The procedure you can take away

When you hit an output anomaly in a product built on an LLM API, this order gets you there fast:

1. Cross-check metadata against the console measurements — do request count and input/output tokens match the artifact in front of you, even by order of magnitude? If not, that is your shortest lead
2. Pin down how many times it was saved — if the timestamps (date / modified and friends) match, you can discard the whole family of append-loop hypotheses at once
3. Split into structural units and hash them — headings, paragraphs, chunks, anything. Just counting unique values and the top occurrence count separates "generation variance" from "byte duplication"
4. Cross-check against temperature — if temperature is non-zero and yet you see many byte-exact matches, that is not generation
5. If the cause is outside, reject at the door instead of fixing — drop it at three layers (size, length, structure), before parsing and before saving

Step 3 is the one that really works. It converts "I feel like it's writing the same thing over and over" into 28 unique / 2,065 total / 432 max occurrences, which is very hard to argue with. Discarding two hypotheses took a 20-line script.

---

I publish verification records and related tools on ACS Developer.

Originally published in Japanese on Zenn: https://zenn.dev/acs_developer/articles/llm-response-duplication-md5-forensics