A line in our own research file said four named humanizers preserve zero percent of your Markdown headings. One of the four was super-humanizer. It was ticked off as verified and marked as the centrepiece of the article I sat down to write, so I had no particular reason to look at it twice.
I recomputed it from the raw dataset anyway, mostly out of habit. The list of four was real. It was sitting in the wrong row. Those four tools are at zero for bold spans, and somewhere between the analysis and the summary that list had been carried across to the headings line.
On headings, super-humanizer is at 27 of 46, which is 58.7 percent. Here is one of its actual runs in the data: # Stainless-Steel Insulated Water Bottle goes in, # Reusable 18/8 Stainless-Steel Insulated Water Bottle comes out. It rewrote the wording and left the hash alone.
The tool that really is at 0 of 46 on headings is phrasly, which wasn't on our list at all. It took # Urban Housing Affordability: A Literature Review and gave back Housing affordability refers to the ability... as body text. No heading anywhere in the output.
So the version we almost shipped would have named a company for a failure its own published numbers disprove, and that data is CC BY 4.0 sitting in a public repo, so disproving it is a five minute job for anyone annoyed enough to try. The tool that actually did flatten every heading would have gone unnamed in the same sentence. Both halves wrong, from one list pasted one row too high. (We build a document rewriting tool ourselves, which is why we had a file full of competitor structure numbers to begin with. Read all of this as an interested party checking his own stick before swinging it.)
What bothers me is that the number didn't look wrong. Four tools, one metric, a percentage in a believable range, and a source that was our own. Internally consistent and wrong is a much worse failure mode than obviously wrong, because there is no seam for anyone to notice. I don't have a process fix beyond "recompute it before you publish it", which isn't a fix, it's just labour.
Recomputed per element: in the August 2026 HumanizerBench cycle, 291 of 552 Markdown headings survived rewriting across twelve tools, which is 52.7 percent, and per-tool survival ran the whole range from 0 to 100 percent. Bold spans came out lower, 319 of 792. undetectable, grammarly and humanize-ai-pro each returned 46 of 46 headings. Four tools returned 0 of 46.
Which kills a line I have written myself, in a draft, more than once: humanizers destroy your formatting. Three of the twelve keep every heading. That average is not describing the tool in front of you, it is averaging two groups that behave nothing like each other. Worth knowing why the split doesn't show up on the leaderboard: their composite weights detector bypass at 42 percent, meaning preservation at 32, readability at 16, consistency across writing categories at 10, then subtracts a penalty for things like meaning drift and length inflation. Formatting isn't in there anywhere. They do penalise a tool for handing back text that got longer. Nothing checks whether it came back shaped like a document.
Now the second half, which is what I actually wanted to know, and it is a different measurement. I want to say that plainly before I put any numbers near each other.
A Markdown heading is a hash and a space in a string. Every tool in that benchmark is paste-in, paste-out, so a string is the only place it can be measured. The files people hand us are .docx, where a heading isn't a character at all. It's a paragraph carrying a style id, and the table of contents, the navigation pane and every cross-reference point at that id rather than at the words. HumanizerBench counts # and ** in text. We counted styles and table objects inside a zip. The two numbers stack. They don't compare.
Counting turned out to be easy enough that I'm mildly embarrassed it took us this long. A .docx is a zip, the body is word/document.xml, and a heading is a non-empty w:p whose pPr/pStyle value starts with Heading (or 标题, if the file was authored in a Chinese copy of Word). Tables are w:tbl, rows are w:tr, cells are w:tc. Forty-odd lines, no dependencies.
Three before-and-after pairs from our own corpus: 345 paragraphs in and 345 out, 403 and 403, 405 and 405. Across the three, 152 headings went in and 152 headings came out, every one of them still a heading. The single table in the set kept all 7 rows and 14 cells.
Structure surviving is not the same as text surviving, and I would rather say so here than have someone open a file and catch it. On the 46-heading document, 28 headings came back with the wording untouched under the balanced setting and 41 under the high-strength one. The rest were reworded. Bring Your New Puppy Home—Without the Chaos came back as Bring the New Puppy to Your Home—Without the Chaos. Same Heading 2 style id, different words. A TOC entry and a cross-reference both point at the style id rather than at the words, so a rewording like that one doesn't break them. Though I should say plainly that not one of these three files contains a TOC field or a single cross-reference, so that part is how the format works and not something I measured. I'm not going to read anything into the gap between 28 and 41 either. One document, one run each.
Two things I can't tell you. There are no bulleted or numbered lists in any of those three files. Zero, which I took as a broken counter, so I pointed it at two documents I knew had numbered lists in them and it found 8 in one and 15 in the other. The counter fires. The corpus just has no lists, so I have nothing to report about what happens to one, and I'm not going to make it up.
Footnotes, TOC fields, cross-references and figure numbering: not measured, by them or by us. Those are the ones I'd worry about, because a footnote reference and its text live in different parts of the zip, joined by an integer.
The next corpus I count will have footnotes in it. If you rewrite prose inside real office files and you have already measured what a round trip does to footnote references, link me, I'd like to read it.
Dataset: HumanizerBench, August 2026 cycle, CC BY 4.0. I work on HumanPen.
Top comments (0)