Extracting Attachments Metadata From an Email Export

#ai #llm #machinelearning #programming

Email has no attachment header. What it has is a tree of MIME parts, some conventions about which ones a mail client chooses to show as paperclips, and a filename parameter with three competing encodings. Everything hard about this extraction follows from there being no single authoritative answer to “what was attached”.

An attachment is a position in a part tree

A message with a body and two files is typically a multipart/mixed whose first child is the body and whose remaining children are the files. But the body itself is usually a multipart/alternative holding a plain-text and an HTML version, and if the HTML references embedded images the whole thing is wrapped in a multipart/related. So a routine message with a logo in the signature and one real attachment is three levels deep and contains five parts, only one of which a user would call an attachment.

multipart/mixed
├── multipart/related
│   ├── multipart/alternative
│   │   ├── text/plain
│   │   └── text/html            references cid:logo@example
│   └── image/png                Content-ID: <logo@example>
│                                Content-Disposition: inline
└── application/pdf              Content-Disposition: attachment;
                                   filename="Invoice 4471.pdf"

You have to walk the tree. A flat scan for parts with a filename finds the logo too, and a flat scan for non-text parts finds it as well. The discriminators are Content-Disposition and Content-ID, and they only mean anything relative to where the part sits.

One nesting case deserves its own handling: a part of type message/rfc822 is a whole email attached to an email. Its children are a complete message with its own attachments. Decide deliberately whether your records are per top-level attachment or flattened across nesting, because “the export contains 4,120 attachments” means two different numbers otherwise, and recurse with a depth cap so a maliciously nested archive cannot exhaust you.

The filename is in one of three places

The canonical location is the filename parameter of Content-Disposition, defined by RFC 2183. The older location is the name parameter of Content-Type, which that RFC effectively supersedes but which clients still emit and which is sometimes the only one present. Read Content-Disposition first, fall back to Content-Type, and record which one you used.

Non-ASCII filenames are where this gets genuinely messy, because two incompatible mechanisms are in circulation. The correct one is RFC 2231, which encodes a parameter as filename* with a charset and language prefix, and allows a long value to be split across numbered continuations filename*0*, filename*1* and so on. The incorrect but extremely common one is an RFC 2047 encoded-word — the same construction used for Subject lines — dropped into a parameter value where the standard does not permit it. A parser that only implements RFC 2231 will hand you a literal string beginning with a question mark and an equals sign for a large fraction of real mail.

Prefer filename* when both it and filename are present; they can disagree, and the extended form is the one that carries the charset.
Reassemble numbered continuations in index order before decoding, not after.
Detect an encoded-word in a plain filename and decode it anyway. Being right about the standard and wrong about the file helps nobody.
Sanitise before you touch a filesystem. Filenames in mail contain directory separators, leading dots and control characters, and this is a path-traversal vector as old as mail itself. Store the original string as data and generate your own storage key.
Expect duplicates. Five replies each attaching scan.pdf are five different documents, so the filename is not an identity. Hash the decoded bytes if you need one.

Size is not a header

There is no size field. RFC 1806 defined an optional size parameter for Content-Disposition and it is essentially never populated in practice, so the only way to know how big an attachment is is to decode it and measure. What you have before decoding is the encoded length, and you can derive the rest.

Base64 encodes three bytes as four characters, so the encoded payload is four-thirds of the original, rounded up to a multiple of four with padding. MIME also line-wraps, conventionally at 76 characters, and each break costs two bytes for the carriage return and line feed. Taking a one-mebibyte file — 1,048,576 bytes, an assumed round number, not a measured one — the payload encodes to 1,398,104 characters, which at 76 characters per line is 18,396 lines and therefore about 36,792 bytes of line breaks. The part is roughly 1,434,896 bytes on the wire, near enough 37 per cent larger than the file.

Two consequences. First, if you are estimating storage or a transfer budget from an export’s byte count, the attachments in it are about a third smaller than they look. Second, if you report the encoded length to a user as the file size you will be consistently and confusingly wrong, and the error is large enough to notice. Decode, measure, store both numbers, and store the transfer encoding you found — base64, quoted-printable, 7bit, 8bit or binary — because it explains the difference.

Inline parts that are not attachments

The signature logo is the canonical false positive, and at scale it dominates: a company with a graphical signature contributes two or three image parts to every message in the archive. The rule that works is a conjunction rather than a single test. A part is not a user-facing attachment when its disposition is inline, it carries a Content-ID, and that id is referenced by a cid: URL in the HTML body part — the scheme RFC 2392 defines for exactly this purpose. All three conditions, because plenty of real attachments are marked inline by clients that treat the disposition as a display hint.

Conversely, a part with no Content-Disposition at all and a type of application/pdf is an attachment, whatever the absence of the header implies. Default to attachment for non-text types outside a multipart/related, and record a classification_reason so the rule is inspectable when somebody disputes a count.

One more case defeats all of this. When an Outlook client sends in its proprietary rich-text format, the attachments and formatting are packed into a single part conventionally named winmail.dat with a type of application/ms-tnef. To anything else it is one opaque attachment; the real files are inside it. Microsoft publishes the format in its Exchange protocol documentation, and there are libraries that unpack it. Detect the type and either unpack or flag it — silently recording one attachment called winmail.dat is how an archive quietly loses documents.

Referenced but not present

The mismatch case is the one worth building for, because it is the only part of this job a language model is actually better at than a parser. The body says “please find the signed copy attached” and there is no attachment part. Sometimes the sender forgot; sometimes the export dropped it; sometimes a gateway stripped it and left a text notice in its place.

Extract the references from the body text as their own records: the phrase, its position, and any filename-shaped token near it. Then reconcile against the part list. Three outcomes are interesting — a reference with no matching part, a part no text mentions, and a reference naming a filename that does not match any part’s. The first is the classic missing attachment. The second is usually benign. The third often means a reply attached a newer revision under a different name, which matters if you are reconstructing which version of a document was in front of somebody.

Do this in the sender’s language. “Attached”, “anbei”, “ci-joint”, “adjunto” and the rest are why this is a model task rather than a keyword list, and it is a narrow enough question to ask of the stripped body text alone.

The record

{
  "attachments": [
    {
      "part_path": "1",
      "filename_raw": "=?UTF-8?B?UsOoZ2xlbWVudCAyMDI2LnBkZg==?=",
      "filename": "Règlement 2026.pdf",
      "filename_source": "content_disposition",
      "filename_encoding": "rfc2047_in_plain_param",
      "content_type": "application/pdf",
      "transfer_encoding": "base64",
      "encoded_bytes": 1434896,
      "decoded_bytes": 1048576,
      "sha256": "…",
      "disposition": "attachment",
      "content_id": null,
      "is_user_facing": true,
      "classification_reason": "non_text_outside_related"
    }
  ],
  "body_references": [
    { "phrase": "ci-joint le contrat signé", "matched_part_path": null }
  ]
}

Keep filename_raw forever. Decoding rules change, your parser will be wrong about some subset of them, and the raw parameter is the only thing that lets you fix that without re-reading the mailbox. Everything downstream — ingestion, deduplication, retention — keys off the hash rather than the name, and the thread graph tells you which message in a conversation actually carried the file.