DEV Community: Ren

The redaction you can't see is still in the text layer

Ren — Thu, 22 Oct 2026 13:57:35 +0000

You have probably heard this story before: someone "redacts" a PDF by drawing black boxes over names, and the names are still there, searchable and copyable, because the box only covers the picture. The text layer underneath never changed.

I found the same bug in my own code, in a document viewer I had written. This post is about what the hole looked like, and what it took to decide, character by character, whether text is actually visible on screen.

Background: a DocuWorks viewer in the browser

The viewer is for DocuWorks documents (.xdw / .xbd). DocuWorks is Fujifilm's document software, common in Japanese offices and local government; think of it as a PDF-like format you receive from a Japanese business partner or city office and cannot open without installing their viewer. My viewer runs entirely in the browser. The reader is ezpzfile-xdw (MIT, written in Rust, not affiliated with Fujifilm) compiled to WebAssembly, and pages are drawn on a canvas. There are two save buttons: "Save as PDF" and "Save the text".

While writing a help article, I read Fujifilm's support note DW10105 (in Japanese). Its title translates to roughly "When a DocuWorks file is converted to PDF, information hidden with annotations can be seen." That made me wonder whether my viewer had the same problem. I built a test document with one line of body text covered by a sticky note and by a black rectangle, and tried it. It did.

What was happening

Here is where the covered line ended up, by output:

Output	Covered line
Screen (canvas)	hidden
"Save as PDF": the page image	hidden
"Save as PDF": the invisible text layer	present
"Save the text"	present

The PDF was built as a page image (a 150 dpi JPEG) with an invisible text layer on top, for search and copy. The image uses the same drawing code as the screen, so the sticky note covers the line. But the text layer contained every piece of text on the page, covered or not. That is the classic black-box redaction failure, produced by my own export.

The people who get hurt are recipients. Someone opens a document they were sent, saves it as a PDF or text, and forwards it. Whatever the sender meant to hide goes along with it, and nobody notices.

Why: text extraction ignores what is drawn on top

A DocuWorks page is a display list: text, shapes, images and annotations (sticky notes, rectangles), drawn in order. The engine's text extraction (text()) and its PDF text layer (toPdf) simply collect every text item in that list. Neither looks at what gets drawn over the text afterwards.

According to DW10105, PDFs made by DocuWorks' own PDF exporter have a related problem: the original text, shapes and images and the annotations pasted on top are separate drawing elements. A slow viewer can show what is under a note for a few seconds, and PDF editing software can expose the covered text. The mechanism differs from mine (theirs keeps the annotations as separate, movable objects; mine put all text into the text layer), but Fujifilm's remedy is the same one I would reach for: produce an image-only PDF.

I could have dropped the text layer entirely. But being able to search a PDF is useful, and most documents have nothing hidden. So I decided to detect whether there is text the screen does not show, and ask before saving only when there is.

Attempt 1: look for what covers each character

The first version was straightforward. For each character, check whether a fill, a thick stroke or an opaque image drawn after it covers it.

Each character is sampled at 3×3 points: 20%, 50% and 80% across and down its box. A character counts as hidden when its centre and at least 7 of the 9 points are unseen, so a stamp over one corner does not count.

const F = [0.2, 0.5, 0.8];
for (const fy of F) for (const fx of F) {
  if (unseen(at, ink, ...point(fx, fy))) {
    hits++;
    if (fx === 0.5 && fy === 0.5) centre = true;
  }
}
if (centre && hits >= HIDDEN_POINTS) mask[n] = true; // HIDDEN_POINTS = 7

The character boxes are placed exactly the way the screen renderer (paint()) places them, because "hidden" should mean hidden on the screen.

Sticky notes, black rectangles, white rectangles, a black highlighter, a thick pen line, an opaque image: every sample I had made was caught.

Attempt 2: text the same colour as what is under it

Code review found shapes this missed. Draw a black rectangle first, then black text on top of it. Black text on a black background. White text on white paper. None of these can be read on screen, yet nothing is drawn over the text, so the check counted zero.

The first one is not contrived. Shade some words black in a word processor to "blank them out", print to DocuWorks, and you can end up with exactly that: black text on a black rectangle.

So I changed the question from "is anything on top of it?" to "can the text colour be told apart from the colour under it at this point?" At each sample point:

Composite everything drawn before the text (white paper, fills, strokes, image pixels) to get the background colour
Lay everything drawn after the text over both the text colour and the background
If the two colours differ by 60 or less (the sum of the absolute R, G and B differences, 0 to 765), the point is unseen

let under: Rgb = [255, 255, 255]; // white paper
let top = ink;                     // the text colour
for (const p of paintsNearThisPoint) {          // in drawing order
  const e = p.at_(x, y);
  if (!e) continue;
  under = blend(under, e);
  if (p.at > run) top = blend(top, e);          // paint after the text lands on the text too
}
return diff(top, under) <= CONTRAST;            // CONTRAST = 60

Attempt 1 turns out to be a special case of this: when an opaque fill lands on top, top and under become the same colour and the difference is 0.

Blending follows canvas. A highlighter is drawn with multiply, so black text under a yellow highlighter stays black and is not counted. Under a black highlighter it is black on black and is hidden. Translucent images are mixed by their opacity.

Some covers are made of glyphs: a run of █ typed over the text. Unicode's Block Elements (U+2580 to U+259F) define which part of the character cell they fill, so I treat each one as a filled rectangle. ■ (U+25A0) is a centred square, 0.1 to 0.9 of the box both ways. In both cases I measured the positions in Chromium with the viewer's fonts. A line covered with ▀ (upper half only) or ▓ (dark shade) is still readable on screen, and the check agrees.

The threshold of 60 is checked against pixels

I did not pick 60 by eye. Separately from the check, I wrote a script that compares pixels. It renders the page twice with the same paint(), once with a given text item and once without it, and if no pixel inside the character's box differs by more than 60 (the same three-channel sum), that character is not visible on screen. It redraws the page once per text item, so it is far too slow for the viewer; it is the ruler I measure the check against.

To pin down the boundary, I made samples whose colour difference is exactly 60 and 63, in three arrangements:

Sample	Difference	Check	Pixel comparison
#ebebeb text on white paper	60	hidden	hidden
#eaeaea text on white paper	63	visible	visible
#141414 text on a black rectangle	60	hidden	hidden
#151515 text on a black rectangle	63	visible	visible
#141414 highlighter over black text	60	hidden	hidden
#151515 highlighter over black text	63	visible	visible

A #202020 highlighter (a difference of 96 from black text) still lets you read the text in a photo of the screen, and the check says "visible".

"Unknown" is not "none"

The check runs over every page in a Web Worker right after the document opens. Each page answers with either "N characters, K hidden" or "could not be read", and a pure function sums up the document:

export function summarizeChecks(checks: PageCheck[]): HiddenSummary {
  const perPage: number[] = [];
  for (const c of checks) {
    if ('unreadable' in c) return { kind: 'unknown' };
    perPage.push(c.hidden);
  }
  const chars = perPage.reduce((a, b) => a + b, 0);
  return chars
    ? { kind: 'found', pages: perPage.filter((k) => k > 0).length, chars, perPage }
    : { kind: 'none' };
}

It did not start out like this. A page where the engine stopped (a WebAssembly trap) used to be added as "0 hidden". Code review caught that too. Adding it as 0 makes the whole document "none", and the viewer would save with text without asking. A page that could not be read is precisely the page whose contents you do not know, so one unreadable page makes the whole document "unknown".

The same rule applies elsewhere. If the caller forgets to pass the pixels of an image the check needs, it throws, which ends in "unknown". An image that is itself broken and cannot be decoded, on the other hand, covers nothing: it is not drawn on screen or in the PDF image either, so it really does not hide anything.

The UI keeps four states: checking, unknown, none, found. "Unknown" never looks the same as "none".

A notice on open, a dialog before saving

When the result is "found", a yellow notice appears right after the document opens:

Some text can't be seen on screen, for example because it is under sticky notes or filled shapes (22 characters on 1 page). When you save a PDF or the text, you can choose whether to include it.

Pressing a save button always uses the check's result. If the background check ended in "unknown", it runs once more. If there is hidden text, a dialog asks:

PDF: Save image-only PDF (primary) / Save with text / Cancel
Text: Save hidden text as ■ (recommended) / Save everything / Cancel

The tool cannot know whether the hiding was intentional. Some people just stick a note over their own scribbles. So I kept every choice, made the option that keeps hidden text out the primary button, and put the initial focus on it.

When the check could not be completed, the dialog says "Couldn't check for hidden text". For text, there is then no way to know which characters to replace with ■, so the only choices are "Save everything" and "Cancel". In that case the initial focus goes to Cancel, so pressing Enter does not save everything.

The image-only PDF is just one 150 dpi JPEG per page, drawn by the same paint() as the screen, with no text layer. Writing the PDF by hand took about 70 lines.

The ■ text cannot be made by rewriting the engine's text() output afterwards, because a plain string does not tell you which character came from which display item. I reimplemented the engine's text assembly in JavaScript (a new line when the baseline moves or the text is rotated, a space where items on a line are apart), and a test checks that without masking it matches the engine's text() character for character on every page of every sample.

Known limits

The check does not agree with the pixel comparison perfectly. Running about 60 samples through the same script, most disagreements were "the check says hidden, the pixels show a few visible pixels":

characters at the edge of a cover that only covers the left part of a line
covers made of a run of █ (thin gaps remain between glyphs)
a black bar covering 70 to 80% of the line height (the top edge of the characters peeks out)

Those over-count, so the cost is an extra notice and dialog. A few disagreements go the other way: the pixels say hidden, the check says visible. A hyphen in the shadow of a bar covering about half the line, and one character under a cover of bold ■. In each case, the other characters on the same line are readable on screen.

Things it does not look at:

text cut away by its own clip, and text too small to be drawn
dashed lines (they have gaps, so they do not count as covers)
"text" inside scanned images (it was never in the text layer, so it is out of scope)
fine detail in images (JPEG images are scaled down to 1,024 pixels on the long side before their pixels are read)

So the notice says what it found. It never says "no hidden text".

If you are the one sending the document

This check is a safety net for recipients, so they do not leak something by accident. It does not make the sender's redaction reliable. If you need to hide part of a document before sharing it, do not stop at drawing notes or boxes over it. Convert it to an image-only PDF first, as DW10105 recommends; in DocuWorks that is the image PDF output (画像PDF出力) of DocuWorks PDF, or the convert-to-image plug-in (画像に変換) in DocuWorks Desk. The same applies to any format: flatten to pixels, or use a real redaction tool that removes the text. Before you send it, search the finished PDF for the words you hid and make sure nothing comes up.

Wrapping up

The in-browser DocuWorks viewer now warns about hidden text and can save an image-only PDF or text with ■ in place of hidden characters. Files are never sent to a server.

If you find a way of covering text that the check misses, or a normal document where the notice shows up for no reason, I would like to hear about it. Every sample I tested with was built from scratch, so there are surely real-world covering methods I have not met yet.

Article for readers (the viewer is embedded): How to open an .xdw file
Tool: DocuWorks viewer

This article was written with the help of AI (Claude). The code and details come from the actual implementation and were checked before publishing.

Writing a TNEF parser so winmail.dat never leaves the browser

Ren — Tue, 20 Oct 2026 01:00:20 +0000

If you have ever received an email from Outlook with an attachment called winmail.dat, you know the feeling: no usable extension, nothing opens it, no hint of what is inside. When I went looking for a converter, every one I found said the same thing first: "upload your file to our server."

A winmail.dat is, by definition, an email somebody sent you. It might be a contract or an invoice. Handing it to a site you know nothing about just to find out what is in it gets the order backwards. So I implemented TNEF in TypeScript and made the whole thing run in the browser.

Some background, if you have been lucky enough never to meet it: when Outlook sends a message in Rich Text Format, it does not send normal MIME with normal attachments. It packs the subject, the formatted body, every attachment and a pile of Outlook metadata into one binary blob, encoded as TNEF (Transport Neutral Encapsulation Format), and attaches that as winmail.dat. Outlook on the other end unpacks it silently, so the sender never sees a problem. Everyone else gets the blob.

This post covers the three places where a straightforward implementation breaks, and two problems I only found after shipping: bodies that looked empty, and sender addresses stored in a form you cannot send mail to.

The container itself is simple

signature   uint32   0x223e9f78
key         uint16

repeat until EOF:
  level     uint8    1 = message, 2 = attachment
  attribute uint32   high 16 bits = data type, low 16 bits = attribute id
  length    uint32
  payload   byte[length]
  checksum  uint16

No tree, no index, no offsets to chase. It is a flat run of attribute records, so reading it is a loop. That part worked in an afternoon. The trouble was inside the MAPI property blocks.

1. Everything is 4-byte aligned

String and binary values are length-prefixed, but the length is the number of meaningful bytes, not the number of bytes written. Each value is padded to the next 4-byte boundary.

const len = d.readUint32LE();
const bytes = d.readBytes(len);
d.skip(pad4(len) - len);   // forget this line and everything after it is garbage

The nasty part is how it fails. You do not get a slightly odd filename. The stream slips by one to three bytes, and every property type you read after that is nonsense. It fails silently and completely, which makes it very easy to conclude "this file is corrupt" rather than "my reader is off by two."

2. Named properties have a variable-length header

This one cost me the most time. A property id of 0x8000 or above is a named property, and it is preceded by an extra header that is not mentioned anywhere near the description of the record layout.

if (propId >= 0x8000) {
  d.skip(16);                       // property set GUID
  const kind = d.readUint32LE();
  if (kind === 0) {
    d.skip(4);                      // named by numeric id
  } else {
    const nameLen = d.readUint32LE();
    d.skip(pad4(nameLen));          // named by string; length depends on the name
  }
}

Skip a fixed number of bytes here and it works on your test files, then breaks on somebody's real mail. Same failure as the alignment bug: total desync, no error.

3. Filenames live in four places, in a priority order

This is the one that matters in practice. The same attachment can carry its name in up to four fields, and they do not agree.

const name =
  a.mapiLongFilename ||   // 0x3707, UTF-16LE, full name
  a.mapiFilename ||       // 0x3704
  a.mapiDisplayName ||    // 0x3001
  a.legacyName ||         // attAttachTitle, ANSI, often 8.3
  'attachment';

attAttachTitle sits right next to the attachment bytes, so it is the easiest one to find. But it is an "ANSI" string in whatever legacy code page the sender's Windows used, and it is often truncated to 8.3. On a Japanese system that code page is Shift_JIS (code page 932), the two-byte encoding Windows still uses for non-Unicode text in Japanese locales. A parser that reads only this field produces filenames like 8E9095948E.PDF instead of what the sender typed, which is exactly the mojibake people complain about in winmail.dat tools.

The encoding is declared, not guessed

The encoding of ANSI strings is declared by attribute 0x9007 (the OEM code page): 932 is Shift_JIS, 936 is GBK, 949 is EUC-KR, 950 is Big5, the 125x range is the Windows Latin family, 65001 is UTF-8.

It governs every ANSI string that comes after it, so the parse is order-dependent. You have to read and apply the code page before decoding the subject. Process attributes in whatever order is convenient and you will quietly decode a Japanese subject line as Windows-1252.

Found later: the body often exists only as compressed RTF

For a while after launch, many winmail.dat files showed up as "no body". The body was there. It was stored as Outlook's compressed RTF (PR_RTF_COMPRESSED, 0x1009), and in a winmail.dat that is often the only copy of the body.

The compression (MS-OXRTFCP) is a small LZ77 variant. Its 4,096-byte circular dictionary starts out holding 207 bytes of common RTF:

// MS-OXRTFCP 2.1.2.1
const RTF_DICTIONARY_PREFIX =
  '{\\rtf1\\ansi\\mac\\deff0\\deftab720{\\fonttbl;}{\\f0\\fnil \\froman \\fswiss \\fmodern \\fscript ' +
  '\\fdecor MS Sans SerifSymbolArialTimes New RomanCourier{\\colortbl\\red0\\green0\\blue0\r\n' +
  '\\par \\pard\\plain\\f0\\fs20\\b\\i\\u\\tab\\tx';

Each bit of a control byte says whether the next token is a literal byte or a reference into the dictionary, eight tokens per control byte. A reference can read into the very bytes it is writing, so the copy has to go one byte at a time. The compressed data carries a CRC, so you know it is damaged before you decompress it.

The RTF that comes out is usually not RTF anyone wrote. When Outlook stores an HTML message this way, it wraps the HTML in RTF (MS-OXRTFEX). The tags go into {\*\htmltag …} groups, and whatever Outlook added for RTF readers sits between \htmlrtf and \htmlrtf0. Drop that part, join the rest, and the original HTML comes back. It then goes through the same treatment as any other HTML body from a stranger: sanitized with DOMPurify against an allowlist, and rendered in a sandboxed iframe whose content security policy blocks network loads, so remote tracking images never fire.

The encoding lesson applies here too. The RTF declares its own code page (\ansicpg932), and a double-byte character arrives as two separate escapes, \'82\'a0. If you decode each escape on its own, you get garbage; collect the bytes first and decode them together.

Also found later: the sender's address is stored in a form you cannot send to

Since winmail.dat is created by the sender's Outlook settings, the recipient cannot stop it from happening. So under the result I show a short "here is what to ask the sender" message and a mailto: link that opens it in the user's mail app. I wanted the sender's address in the To field. That turned out to be harder than it sounds.

The sender is recorded in two places:

MAPI sender properties (PidTagSenderName / PidTagSenderEmailAddress, plus the PidTagSentRepresenting… pair for sending on behalf of someone)
The TNEF attribute attFrom (MS-OXTNEF 2.1.3.3.3): an 8-byte header (0x0004, the structure's length, the name's length, the address's length), then the name, then the address as type:address, then 8 zero bytes

For mail sent from inside an Exchange organization, both addresses are often X.500 paths like EX:/O=EXCHANGELABS/OU=…/CN=RECIPIENTS/CN=…. No mail app can send to that. The internet address lives in yet another pair of properties: PidTagSenderSmtpAddress (0x5D01) and PidTagSentRepresentingSmtpAddress (0x5D02).

// Use the first candidate that is an internet address.
const isInternetAddress = (a: string) => /^[^\s@/<>]+@[^\s@<>]+$/.test(a);
const address = [smtp, mapiEmail, attFromEmail].find(isInternetAddress);

If only the X.500 path exists, the From line still shows it (that is what the file holds), but the mailto: link leaves the To field empty for the user to fill in.

Delegated sending adds one more trap. When an assistant sends on behalf of their manager, the "sender" and "sent representing" names differ. Name and address have to come from the same person, or you build a recipient with the manager's name and the assistant's address. The name in attFrom is ANSI, so characters can arrive as ?; I only compare names when both are readable.

Wrapping up

The result is a winmail.dat viewer that runs entirely in the browser. You can check that nothing is uploaded: open the Network tab in your browser's developer tools and convert a file.

It does not yet extract emails attached inside the winmail.dat, or OLE objects.

If you have a winmail.dat it opens wrong, I would like to hear about it. This format has more corners than one person's inbox can cover.

Tool: winmail.dat viewer
For non-developers (the viewer is embedded in the article): How to open winmail.dat
The longer write-up (HTML sanitizing, what is not supported): Writing a TNEF parser so winmail.dat never leaves the browser

This article was written with the help of AI (Claude). The code and details come from the actual implementation and were checked before publishing.