Building a Private RAG Pipeline for Legal Data: Compression and Redaction Before the Model

The counter-intuitive part of building a retrieval-augmented generation pipeline for legal data is this: the model is the least of your problems. The hard, unglamorous work is everything that happens to the documents before they ever reach an embedding model — and if you get that wrong, no amount of vector-store tuning will save you.

This post walks through the engineering of a private RAG pipeline where data never leaves the internal network, with the two stages that actually move the needle: compression and redaction. It is written from the perspective of someone standing up the pipeline, not evaluating a vendor.

Why "private" changes the architecture

Legal work has a non-negotiable constraint that most RAG tutorials ignore: the corpus cannot leave the building. That single constraint rules out the default modern pattern — ship documents to a hosted service, process them there, monetize the pipeline. For a firm, that pattern is a non-starter because confidentiality is a professional obligation, not a feature checkbox.

So the whole pipeline — splitting, vectorization, retrieval, generation — has to run on infrastructure the firm governs. Single machine for a small practice, internal cluster for a larger one, and increasingly a domestic-software (xin chuang) environment running Kylin or UnionTech. The deployment boundary is a design input, not an afterthought.

The data problem, in numbers

Two surveys frame the scale of what you're dealing with. IDC forecasts China's unstructured data climbing from 51 ZB to 129 ZB over five years at 30–40 percent CAGR. A separate IDC survey found knowledge workers spend roughly 2.5 hours a day — about 30 percent of the workday — hunting for information. Inside a firm, that means lawyers and assistants aren't litigating so much as rifling through folders.

And the breach data puts a price on getting it wrong. IBM's 2025 Cost of a Data Breach report found customer PII made up 53 percent of stolen data, employee PII 37 percent, and IP 33 percent. Legal documents are dense with precisely those categories. The pipeline you build is, whether you intend it or not, a queryable database of exactly that material.

Stage 1: compression, done locally

Case files are heavy with PDFs, scanned documents, OFD files, and audio or video records. Raw, they dominate storage and blow up token cost downstream. Two mistakes engineers make here:

• Treating compression as optional. It isn't. Image-heavy content billed by a multimodal tokenizer is expensive on every call, and retrieval pipelines that pass scanned pages to the model inherit that bill forever.
• Treating compression as lossy-and-done. Legal files need fidelity: layout, seals, tables, reading order must survive.

A local compression engine that covers eight formats — PDF, images, video, audio, Office, OFD, and scans — and strips redundant data without touching text can cut file size 60–80 percent. In one reference measurement, a 443-page OFD document processed end to end in about 124 seconds. The token win is the part that compounds: measured against GPT-4o, compressed image content cut token cost 85.9 percent. That saving repeats on every query for the life of the library.

Engine choice matters less than the locality. The processing must run on the firm's machine; a compression API that uploads the file first defeats the purpose.

Stage 2: redaction before indexing

This is where most pipelines are structurally backwards. Document-level permissions don't survive chunk-level indexing — the vector store flattens "only the finance team can open this" into "anyone whose query matches gets the passage." If a client ID or bank account is in the chunk, one well-formed question retrieves it.

The fix has two rules:

1. Detect and mask sensitive fields before embedding and indexing, never in a post-hoc filter that runs one query behind.
2. Run the detection locally. Sending raw documents to an external masking service recreates the leak one hop earlier.

The practical baseline is rule-based detection over common structures — ID numbers, bank cards, phone numbers — plus industry-specific ones, with a tunable rule set because every document mix differs. A mature base detects 13 categories of sensitive information, including client IDs, bank accounts, and undisclosed case details, and masks them before the index is built. The semantic body stays intact, so search quality is unaffected — redaction targets fields, not meaning.

A concrete example helps. A clause that reads "the Vendor shall pay Acme Ltd. at account 6225 8801 2345 6789 within 30 days" is masked to "the Vendor shall pay Acme Ltd. at account [REDACTED] within 30 days." The payment obligation, the counterparty, and the deadline — everything the retrieval system needs to answer "when is the payment due" — survive. Only the field that would ruin someone's week if it surfaced in a chat window is gone. That is the difference between redaction that protects and redaction that breaks the document.

The pipeline shape

Putting it together, the sequence is short to describe and strict to enforce:

document ingest
  → format detection (PDF / OFD / Office / scan / image)
  → layout recovery (headings, reading order, tables)
  → OCR for scans (enhance → analyze → extract → confidence tier)
  → clean (strip headers/footers, fix high-frequency OCR errors)
  → redact (detect 13 sensitive categories, mask before embed)
  → compress (shrink locally, keep layout fidelity)
  → chunk (semantic boundaries, attach source/version/time metadata)
  → embed and index

A note on chunking: split on semantic boundaries, not fixed character counts. Legal clauses have structure; a chunk that cuts a clause in half is a chunk that retrieves badly. Attach source, version, and timestamp metadata to every chunk so retrieval can prefer the authoritative copy and so a human can audit any answer back to its origin.

The math you can check yourself

Take a 2 TB scan-heavy library. Fidelity-preserving compression at 60–80 percent brings that to roughly 400 GB–800 GB before you count the smaller vector store and reduced embedding batch sizes. Then the token side: if image-heavy content cuts token cost 85.9 percent on a GPT-4o basis, and you run thousands of retrieval-augmented calls a week, the recurring saving dwarfs the one-time storage win. Compute both before and after on a real sample; the numbers are the argument.

Integration without a rebuild

A common fear is that a private pipeline means throwing away the vector store you already run. It shouldn't. The compression, cleaning, and redaction stages produce ordinary, smaller, cleaner documents; those plug into whatever embedding model and store you already use. The constraint is only that those stages execute inside your network. For teams that want to embed compression directly into a case-management system, a compress SDK and a compress API exist; for teams that want an AI assistant to trigger compression with a sentence, an MCP service over stdio or HTTP fits. The packaging varies; the boundary doesn't.

What I'd measure first

Before touching the model, measure three things on a real sample of your own files:

• What fraction of source files are scans or image-heavy PDFs — this predicts your OCR and compression workload.
• How many retrieved chunks are duplicates or stale versions — this predicts whether search will be usable at all.
• Which fields in the index would alarm your compliance officer — grep the index for PII patterns; assume you'll find some.

If those three numbers look bad on a hundred-file sample, they will look worse on a hundred thousand, and no model swap will rescue them. The model gets the attention. The pipeline gets the answers. Debug upstream, keep it on your own hardware, and the legal-data problem becomes tractable instead of terrifying. The discipline is boring, and it is also the difference between a demo that impresses and a system that holds up under a decade of real files.