Building an Evidence-First Email Archiver for Gmail and IMAP with Python

Email archiving looks simple until the archive needs to answer a real question.

Downloading attachments is easy. Preserving enough context to prove where a file came from, which message contained it, who sent it, how it appeared in the mailbox, whether the same bytes appeared elsewhere, and whether the archive is still intact is a different problem.

That is the problem I set out to solve with FireXCore MailVault, a read-only, open-source email evidence archiver written in Python.

MailVault connects to Gmail or a standards-compatible IMAP server, preserves complete messages as immutable raw EML objects, stores MIME payloads by SHA-256 content identity, records provenance in SQLite, and generates traceable manifests for downstream analysis.

FireXCore / mailvault

A provider-neutral, read-only email evidence archiver. MailVault acquires complete messages via IMAP, preserves immutable raw EML objects, records every MIME part and mailbox occurrence, deduplicates attachment content by SHA-256, and produces traceable manifests for analytics, eDiscovery, migration staging, and procurement intelligence. Apache-2.0

راهنمای فارسی · Getting started · Architecture · Procurement readiness

FireXCore MailVault is a provider-neutral, read-only email evidence archiver. It acquires complete messages through IMAP, preserves immutable raw EML objects, records every MIME part and mailbox occurrence, deduplicates attachment content by SHA-256, and produces traceable manifests for analytics, eDiscovery, migration staging, and procurement intelligence.

MailVault is not an attachment downloader. The canonical unit is the complete email message, including headers, bodies, MIME structure, inline resources, nested messages, signatures, encrypted containers, attachment occurrences, provider identifiers, mailbox identifiers, labels, flags, and timestamps.

Project status

MailVault is in public beta. The archive model and read-only acquisition guarantees are stable. Provider adapters beyond Gmail and generic IMAP, including OAuth-based Microsoft 365 and JMAP, are planned as separate milestones.

Design objectives

• Preserve raw evidence before deriving metadata.
• Keep provider-specific behavior outside the archive domain.
• Use stable provider identifiers where available and standards-based fallbacks elsewhere.
• Store…

View on GitHub

This article explains the engineering decisions behind it, the problems that appeared when real-world email data met Python's email parser, and what a resumable evidence-first archive looks like in practice.

The real problem is provenance, not file downloading

A basic attachment downloader usually produces something like this:

downloads/
├── invoice.pdf
├── quotation.xlsx
├── technical-drawing.pdf
└── invoice-2.pdf

That is useful until someone asks:

• Which email contained invoice.pdf?
• Was it attached or embedded inline?
• Who sent it, and to whom?
• Which thread did it belong to?
• Did the same bytes appear under another filename?
• Was the message in Inbox, All Mail, Spam, or Trash?
• Can an extracted price be traced to an exact message and MIME part?
• Has the file changed since it was archived?

A filename cannot answer those questions.

The complete message must remain the primary evidence object. Attachments are only parts of that object.

The design goals

I defined a few non-negotiable properties before writing the archive engine.

1. The mailbox must remain unchanged

An archive tool should not mark messages as read, move them, change labels, or delete anything.

MailVault uses read-only mailbox selection and fetches complete messages using non-mutating IMAP operations such as BODY.PEEK[].

The archive core intentionally has no operation for:

• deleting messages;
• moving or copying messages;
• appending mail;
• changing flags;
• changing Gmail labels;
• sending messages.

This keeps acquisition separate from mailbox administration.

2. Raw evidence must be preserved before parsing

Email is messy input.

A message may contain:

• malformed headers;
• invalid Unicode;
• broken encoded words;
• unusual address groups;
• duplicate filenames;
• nested messages;
• encrypted MIME containers;
• TNEF payloads;
• digital signatures;
• incorrect content types.

If parsing fails, the original bytes are still valuable.

MailVault therefore stores the complete raw EML object before relying on derived interpretation. Metadata can be repaired or regenerated later; the canonical source remains unchanged.

3. Content identity must not depend on filenames

Two different files can have the same filename.

The same file can also appear under several filenames.

MailVault stores retained non-body MIME payloads in a content-addressed object store:

objects/
└── blobs/
    └── sha256/
        └── ab/
            └── cd/
                └── abcdef...

The SHA-256 digest becomes the canonical content identity.

The original filename is preserved as metadata for each occurrence, but it does not control the storage path.

4. Deduplication must not destroy history

Suppose the same PDF appears:

• in an original supplier quotation;
• in a forwarded message;
• in a later approval thread;
• under two different filenames.

The bytes should be stored once, but all four occurrences should remain queryable.

That means the model needs to distinguish between:

• a unique blob;
• a MIME-part occurrence;
• a canonical message;
• a mailbox occurrence.

Deduplication is a storage optimization, not a reason to collapse provenance.

The archive architecture

At a high level, the pipeline looks like this:

Gmail / IMAP
      |
      v
Capability negotiation
      |
      v
Metadata-first discovery
      |
      v
Read-only raw message fetch
      |
      +----------------------+
      |                      |
      v                      v
Immutable raw EML      MIME-part parsing
      |                      |
      |                      v
      |               SHA-256 blob store
      |                      |
      +----------+-----------+
                 |
                 v
          SQLite metadata
                 |
                 v
 JSONL manifests / reports / views

The archive is divided into four concerns.

Read-only provider acquisition

Provider behavior is isolated behind profiles.

The Gmail profile can retain provider-specific context such as:

• Gmail message identity;
• Gmail thread identity;
• Gmail labels;
• Gmail raw-search behavior.

The generic IMAP profile relies on portable concepts such as:

• mailbox name;
• UID;
• UIDVALIDITY;
• flags;
• RFC message headers;
• advertised server capabilities.

This separation matters because Gmail is not just a traditional folder-based mailbox. One canonical message may appear through several labels without becoming several independent messages.

Canonical message versus mailbox occurrence

This distinction became central to the data model.

A canonical message represents the evidence object itself:

canonical message
├── raw EML SHA-256
├── provider message identity
├── RFC Message-ID
├── normalized participants
└── immutable object path

A mailbox occurrence represents where that message appeared:

mailbox occurrence
├── mailbox or Gmail label
├── UID
├── UIDVALIDITY
├── flags
└── provider context

Treating these as separate records prevents double-counting and keeps the model useful for both Gmail and conventional IMAP servers.

Metadata-first, resumable synchronization

Large mailboxes should not require a single uninterrupted process.

MailVault first discovers metadata, stores stable identities, and then archives pending raw messages in bounded batches.

Operational state includes:

• checkpoints;
• run records;
• retry state;
• a single-run lock;
• rolling bandwidth accounting;
• archived-object status.

A configured soft cap pauses the run cleanly. Re-running the same command against the same destination resumes from persisted state.

For example:

mailvault sync `
  --account [e-mail] `
  --host imap.gmail.com `
  --provider gmail `
  --auth app-password `
  --destination E:\MailVault `
  --scope all `
  --soft-cap 2GiB `
  --hard-cap 2.25GiB

A paused run is not a failed run. It means the engine reached an operator-defined transfer boundary and preserved enough state to continue safely later.

What happened with real-world email

Synthetic MIME fixtures are useful, but production mailboxes expose much stranger input.

Two failures were especially instructive.

Invalid Unicode surrogates

Some old or malformed metadata contained surrogate code points that could not be encoded as UTF-8:

UnicodeEncodeError:
'utf-8' codec can't encode characters:
surrogates not allowed

The correct response was not to rewrite the raw EML.

Instead, MailVault now keeps canonical bytes untouched and sanitizes only derived text fields before they enter SQLite, JSON, JSONL, or logs.

Malformed address groups

Python's structured header parser encountered a malformed To header and failed inside email.headerregistry:

AttributeError:
'Group' object has no attribute 'local_part'

A single broken historical address header should not stop an entire archive.

The metadata path was changed to use a more tolerant parsing strategy for hostile address headers while preserving the raw header bytes in the original message.

The general lesson was simple:

An evidence archiver must be more tolerant of source defects than the systems that originally produced the data.

A live archive run

One live Gmail archive produced these intermediate statistics before the configured rolling bandwidth cap paused the run:

Messages                   3,250
Raw Messages               2,312
Raw Bytes          2,151,518,027
Occurrences                3,250
Parts                      8,172
Blobs                      2,526
Blob Bytes         1,097,947,979
Duplicate Part Occurrences 1,378

A few observations:

Raw storage is expected to be larger than the downloaded attachment total

An attachment is retained inside the raw EML, often Base64-encoded, and also stored as a decoded blob for direct access and content-addressed deduplication.

That is intentional. The raw message is evidence; the decoded blob is a usable derived object.

The deduplication count is meaningful

1,378 duplicate part occurrences means the same blob bytes appeared multiple times, but only one canonical blob object was required for each unique SHA-256 digest.

The occurrence relationships were still preserved.

Pausing proved the resume model

The run reached its rolling 24-hour soft cap with zero processing errors.

A second invocation immediately paused with zero new bytes because the previous transfer was still inside the bandwidth window. This is the expected behavior of a persistent rolling ledger, not a stalled downloader.

Archive layout

A MailVault destination looks like this:

MailVault/
├── objects/
│   ├── raw/sha256/          # immutable raw EML
│   └── blobs/sha256/        # content-addressed MIME payloads
├── metadata/messages/       # derived message JSON
├── database/
│   └── mailvault.sqlite3
├── manifests/
│   ├── messages.jsonl
│   ├── message_occurrences.jsonl
│   ├── message_parts.jsonl
│   ├── blobs.jsonl
│   └── procurement_sources.jsonl
├── state/                   # checkpoints, locks, bandwidth ledger
├── reports/
├── logs/
└── views/                   # disposable navigation pointers

The canonical archive is:

1. immutable objects;
2. the relational metadata database.

Manifests, reports, per-message JSON, and navigation views are reproducible outputs.

Installation

MailVault requires Python 3.12 or newer.

git clone https://github.com/FireXCore/mailvault.git
cd mailvault

Using pipx:

pipx install .

Or with a virtual environment:

python -m venv .venv

Windows PowerShell:

.\.venv\Scripts\Activate.ps1
python -m pip install --upgrade pip
python -m pip install -e .

Linux or macOS:

source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -e .

Verify the installed version:

mailvault version
python -m firexcore_mailvault version

Both commands should report the same package version.

Validate the mailbox first

The doctor command tests TLS, authentication, capabilities, provider behavior, and mailbox discovery before a full archive begins.

For Gmail:

mailvault doctor `
  --account [e-mail] `
  --host imap.gmail.com `
  --provider gmail `
  --auth app-password

The App Password is entered through a hidden prompt. It should never be included directly in the command, committed to Git, or written into a shared configuration file.

Inspect and verify an archive

Archive statistics:

mailvault stats `
  --destination E:\MailVault `
  --account [e-mail]

Integrity verification:

mailvault verify `
  --destination E:\MailVault `
  --sample 1

The verifier checks stored sizes, expected objects, and SHA-256 identity.

Security boundaries

MailVault deliberately does not try to solve every email-security problem.

It does not:

• execute attachments;
• render untrusted HTML;
• claim to be a malware sandbox;
• decrypt arbitrary protected documents;
• provide legal-hold certification;
• replace enterprise journaling;
• infer business facts inside the archive core.

The archive itself may contain sensitive information. Operators still need:

• encrypted storage;
• restricted filesystem permissions;
• protected backups;
• retention rules;
• access reviews;
• legal authorization to collect the mailbox.

Open-source code does not remove the need for operational governance.

Why procurement evidence influenced the design

The original use case involved procurement communication.

A later analytics system may need to extract:

• supplier identity;
• RFQ and quotation relationships;
• price, currency, quantity, and validity;
• freight, payment terms, and Incoterms;
• requested versus offered part numbers;
• technical substitutions;
• delivery promises;
• approval or rejection history.

Those facts should not be produced as detached text.

A trustworthy extraction record needs an evidence anchor:

derived fact
├── canonical message ID
├── MIME part path
├── blob SHA-256
├── source filename
├── extractor version
├── confidence
└── exact citation

That is why MailVault stops at evidence preservation and source-manifest generation. Business interpretation belongs in a separate, versioned layer.

Current status and roadmap

The current public release supports:

• Gmail through IMAP and App Password authentication;
• generic standards-compatible IMAP;
• metadata-first discovery;
• resumable raw EML acquisition;
• MIME-part preservation;
• SHA-256 content-addressed blobs;
• SQLite provenance;
• JSONL exports;
• archive statistics and verification;
• Windows, Linux, and macOS CI smoke tests.

Planned provider work includes:

• OAuth-based authentication;
• Microsoft 365 integration;
• JMAP support;
• additional evidence export formats.

Lessons from building it

A few principles became clearer during implementation:

1. Store source bytes before trusting a parser.
2. Treat email as hostile input.
3. Separate canonical identity from mailbox occurrence.
4. Deduplicate content, not history.
5. Make interruption an expected operating state.
6. Keep provider behavior behind explicit adapters.
7. Do not let derived convenience overwrite source truth.
8. Verification is part of archival, not an optional extra.

Try it, inspect it, or challenge the design

MailVault is open source under Apache License 2.0.

The repository includes the source code, tests, architecture decisions, archive format, security model, operations guide, troubleshooting documentation, and contribution workflow.

FireXCore / mailvault

A provider-neutral, read-only email evidence archiver. MailVault acquires complete messages via IMAP, preserves immutable raw EML objects, records every MIME part and mailbox occurrence, deduplicates attachment content by SHA-256, and produces traceable manifests for analytics, eDiscovery, migration staging, and procurement intelligence. Apache-2.0

راهنمای فارسی · Getting started · Architecture · Procurement readiness

FireXCore MailVault is a provider-neutral, read-only email evidence archiver. It acquires complete messages through IMAP, preserves immutable raw EML objects, records every MIME part and mailbox occurrence, deduplicates attachment content by SHA-256, and produces traceable manifests for analytics, eDiscovery, migration staging, and procurement intelligence.

MailVault is not an attachment downloader. The canonical unit is the complete email message, including headers, bodies, MIME structure, inline resources, nested messages, signatures, encrypted containers, attachment occurrences, provider identifiers, mailbox identifiers, labels, flags, and timestamps.

Project status

MailVault is in public beta. The archive model and read-only acquisition guarantees are stable. Provider adapters beyond Gmail and generic IMAP, including OAuth-based Microsoft 365 and JMAP, are planned as separate milestones.

Design objectives

• Preserve raw evidence before deriving metadata.
• Keep provider-specific behavior outside the archive domain.
• Use stable provider identifiers where available and standards-based fallbacks elsewhere.
• Store…

View on GitHub

I would especially value review from developers who have worked with:

• hostile or malformed MIME;
• long-running IMAP synchronization;
• digital evidence systems;
• eDiscovery pipelines;
• records management;
• procurement document intelligence.

The original version of this article is published on the FireXCore blog.

---

Editorial note: This article was drafted with AI assistance and reviewed by the project author against the source code, CI results, and live MailVault archive output before publication.