What Makes an AI System Production-Ready? Part 2: Designing a RAG Ingestion Pipeline

Part of a series on building production-grade AI systems. Part 1 covered the overall shape of our enterprise RAG project. This post is about ingestion — the layer that decides whether everything built on top of it is trustworthy.

Retrieval quality gets all the attention in RAG write-ups. Chunking strategies, rerankers, prompt formats. Almost nobody writes about ingestion — the part of the system responsible for actually getting documents into the vector store correctly, once, and keeping them current as the source documents change.

That's a gap worth closing, because ingestion is where most real-world RAG systems quietly rot. Not from bad retrieval logic — from stale vectors, silent duplicates, and documents that half-failed to load six months ago and nobody noticed. This post is the architecture we landed on to avoid that, and the reasoning behind each piece.

The shape of the pipeline

At the core, ingestion is four single-responsibility components, orchestrated by one thin coordinator:

IngestionPipeline is the thin orchestrator that calls these four in sequence — it holds no logic of its own beyond that ordering.

Each class does exactly one job and knows nothing about the others' internals:

• Loader — file path in, parsed Document objects out. Owns the file-format → parser mapping and nothing else.
• Splitter — documents in, chunks out. Format-agnostic; never sees a file extension.
• Embedder — chunks in, vectors out. Owns model lifecycle, batching, and retry policy for the embedding API.
• QdrantVectorStore — the only thing that speaks to Qdrant. Owns collection lifecycle and point storage.
• IngestionPipeline — sequences the above. Nothing more.

That last constraint is the one that actually matters for long-term maintainability: the orchestrator stays thin on purpose. The moment retry logic, format branching, or caching strategy starts creeping into IngestionPipeline.ingest(), that's a signal the logic is in the wrong place. An orchestrator's job is sequencing four already-correct components — if it has to compensate for one of them, that component isn't actually done.

This single-responsibility split is also what makes the rest of this post possible: every architectural decision below was implementable as a change to one class, without touching the others.

Documents fail to load cleanly — that has to be a first-class outcome, not an error path

Real enterprise document sets are not clean. Scanned PDFs with no OCR layer. Files that are technically valid but contain nothing extractable. Documents that partially parse. A pipeline that treats "this file produced zero usable content" as an exceptional error will either crash on ordinary data or, worse, propagate that failure in a way that kills unrelated files in the same batch.

The design decision here: zero-extractable-content is a valid terminal state, not a failure. It's checked and short-circuited at two layers independently:

• Loader inspects what it parsed and flags — via logging, not an exception — whether a document came back fully or partially empty. This makes the cause visible in traces immediately, rather than three layers downstream as an unexplained zero.
• Splitter's output naturally becomes an empty chunk list for empty input, and IngestionPipeline.ingest() treats that as a legitimate early return: skip embedding and storage, log it, move on. No exception thrown, no batch aborted.

The alternative — raising on empty content — would force every caller of ingest() to special-case "this specific kind of failure is actually fine," which inverts the responsibility. A pipeline built for real documents has to expect degraded input as routine, not exceptional.

Batch fault isolation: one bad document can't cost you the other 499

At enterprise scale, ingestion runs process directories with hundreds or thousands of files. Something in that batch will be malformed — a corrupt PDF, an encoding issue, a truncated download. The architectural question is whether that one file's failure is contained or catastrophic.

The design: process_file never propagates an exception past itself. It catches, logs with full context (including traceback, for real debuggability — not just a stringified error message), and reports a status: succeeded, failed, or skipped. The batch loop keeps going regardless.

def process_file(pipeline, manifest, file_path, collection_name, source_type=None) -> str:
    ...
    try:
        result = pipeline.ingest(...)
        manifest.set(source, file_hash, result["chunks"])
        return "succeeded"
    except Exception:
        logfire.exception("❌ Failed to process file", file=file_path.name)
        return "failed"

The run as a whole still needs to communicate its outcome truthfully, though — a batch job that "completes" after silently failing on 40% of its files is worse than one that crashes loudly. So the entry point tracks success/failure/skip counts explicitly and exits non-zero if anything failed, so CI, cron, or any orchestrator downstream can actually detect a degraded run instead of reading a falsely clean exit code.

This is a general principle, not specific to ingestion: isolate failure at the smallest unit that makes sense (one file), and report truthfully at the largest unit that matters (the whole run).

The real enterprise problem: documents change, and re-embedding everything doesn't scale

This is the part of ingestion that separates a working demo from something you'd actually run against a living document set. In practice, policy docs, specs, and runbooks get updated continuously. A production ingestion pipeline has to answer three questions correctly:

1. If a file hasn't changed, can we avoid paying to re-embed it?
2. If a file has changed, do we get clean replacement, or do old and new vectors both end up searchable?
3. If a file is deleted, does its content eventually stop being retrievable?

A naive design gets all three wrong. If point IDs are randomly generated on every ingest, re-running the pipeline over an unchanged corpus duplicates every single document — same content, new random ID, inserted as if new. Do that on a recurring schedule and your vector store fills with copies of unchanged content while genuinely updated documents sit right next to their own stale, outdated versions — both fully searchable, with nothing distinguishing them. Retrieval quality degrades slowly and invisibly.

Content-addressable storage: deterministic IDs

The fix starts at the storage layer. Instead of a random ID per chunk, derive the ID deterministically from what the chunk is:

source = chunk.metadata.get("source", "")
point_id = str(uuid5(NAMESPACE_URL, f"{source}::{idx}"))

uuid5 is a hash, not a random draw — the same (source, chunk index) pair always produces the same ID. Re-ingest the same file, and its chunks land on exactly the same points they occupied before. Qdrant's upsert semantics — insert-or-overwrite by ID — do the rest: the new vector replaces the old one in place. No duplication, and critically, no lookup step required to make that happen — the ID computation itself guarantees the collision with whatever was there before.

Deterministic IDs alone don't handle a document that shrinks — if a new version has fewer chunks, the old trailing ones wouldn't be touched by ID overwrite alone. So replacement is paired with an explicit cleanup step, scoped by source file:

def delete_by_source(self, collection_name, source):
    self.client.delete(
        collection_name=collection_name,
        points_selector=Filter(
            must=[FieldCondition(key="source", match=MatchValue(value=source))]
        ),
    )

ingest() calls this unconditionally before storing a file's new chunks — including when the new version produces zero chunks at all, so a document that gets fully emptied out doesn't leave orphaned vectors behind either. Deterministic IDs handle the common case efficiently; delete-by-source is the guarantee that catches everything else.

A manifest, to avoid paying for what didn't change

Deterministic IDs and delete-by-source solve correctness — no duplication, no orphans. They don't solve cost. Every file still gets fully loaded, chunked, and re-embedded on every run, whether or not it changed. At enterprise scale, that's the expensive part — embedding API calls, not vector storage.

The solution is a lightweight change-detection layer sitting in front of the expensive path: a manifest that remembers, per file, the content hash of the version it last ingested.

file_hash = compute_file_hash(file_path)          # SHA-256 of raw bytes
previous_hash = manifest.get(source)

if previous_hash == file_hash:
    return "skipped"    # Loader, Splitter, Embedder never touched

The manifest itself is stored the same way as everything else in this system — as Qdrant points, in a small dedicated collection, keyed by a deterministic ID derived from the file path. No new infrastructure, no separate database to keep consistent with the vector store. One record per file: path, content hash, chunk count, timestamp. The vector field on these points is an unused placeholder — the manifest does no similarity search, it's a plain key-value lookup that happens to live in the same store as everything else.

The interaction between these two mechanisms is the actual design: the manifest decides whether to re-ingest at all; deterministic IDs and delete-by-source guarantee that when re-ingestion does happen, it replaces cleanly instead of duplicating. Neither one alone is sufficient — a manifest without deterministic IDs would still duplicate whatever does change; deterministic IDs without a manifest would still pay full re-embedding cost on every run, changed or not.

One consequence of this design that's easy to overlook: the manifest and the vector store are two representations of the same underlying state, and they have to be reset together. Wiping the vector collection without also clearing the manifest leaves every file reporting "unchanged" on the next run, even though its vectors were just deleted — the two stores must move in lockstep, not independently.

Collection lifecycle is a run-level decision, not a per-file one

A subtler question: when does the target collection actually get created (or, for a full reset, wiped)? The naive placement is inside the per-file ingest call — check-and-create on every file, defensively. That's the wrong scope: whether a collection needs to exist, or needs a full reset, is a fact about the run, decided once, not a fact re-evaluated per document.

The pipeline's per-file ingest() call assumes the collection already exists. Ensuring it — and, if requested, wiping it first — happens exactly once, at the start of a batch run, before any file is touched. This mirrors the manifest point above: setup and destructive operations are batch-level concerns; per-file ingestion is a narrower operation that gets to assume its environment is already correct.

Heterogeneous corpora: tagging content by source at ingestion time

Enterprise document sets are rarely one uniform pile — different folders often represent meaningfully different content: verified reference material versus scratch notes, one team's docs versus another's. Rather than inferring this at query time, the pipeline tags it at ingestion time, attaching a source_type to every chunk's metadata based on which folder it came from — carried all the way through to the stored payload, without any of the four core components needing to know why.

This is a small mechanism with a large downstream purpose: it's what makes later filtered retrieval, and retrieval quality evaluation across different content categories, possible — without needing to re-derive "where did this chunk come from" after the fact.

What this buys you, and what it doesn't

The result is a pipeline where: unchanged documents cost nothing to re-run, changed documents update cleanly with no duplication, deleted documents can be pruned, one malformed file can't take down a batch, and collection lifecycle is a deliberate, once-per-run decision rather than an accidental per-file cost.

What it doesn't do yet, by scope rather than oversight: it re-embeds a full document on any change, however small — sub-file diffing isn't built. It processes files serially, with no concurrency for the I/O-bound embedding calls. And the destructive prune operation is safe only when run against the full corpus a collection is responsible for — a constraint currently enforced by documentation, not by the system itself.

That's the honest state of an ingestion layer built to be correct under real, changing, imperfect data — not a finished system, but one where every remaining gap is known and named rather than discovered later in production.

Next

With documents reliably in the vector store, Part 3 moves to actually using them: building a basic RAG system with agents on top of this ingestion layer — retrieval.