Deleted, Emptied, Half-Written: 6 Files a Nightly RAG Ingestion Job Gets Wrong in .NET 10

Most RAG ingestion demos end after the first run: a folder of Markdown goes in, the chat answers a question, done. But a knowledge base changes all the time. Pages get deleted, trimmed, moved and rewritten, and an ingestion job that runs every night has to keep the vector index in step with all of that.

I built a small help-center ingestion service on .NET 10 with Microsoft.Extensions.DataIngestion (10.11.0-preview.1.26513.7, still a preview package) and the in-memory vector store. The pipeline itself (read, chunk, enrich, embed, write) was the quick part. Most of my time went into six ordinary things that can happen to a file. For each one there's a simple question: what should the index hold the next morning?

Code below is trimmed to the shape; omitted parts are marked.

Tech Skill Builder is $5/month for a very limited time only (normally $10). Grab it now before it goes back to $10. The complete ingestion service from this post, with an in-depth PDF guide, is inside Tech Skill Builder.

The one idea behind all six

Don't treat ingestion as "process this folder". Treat it as a sync between two things: the files on disk, and a ledger that records what the index is supposed to hold for each document (content hash, generation number, chunk count, status). Every run compares the two and decides one action per document:

public enum PlannedAction
{
    Add, Update, Reindex, Repair, Retry, Skip, Quarantined, Delete,
}
// IngestionPlanner.Plan(sources, ledger, fingerprint, liveChunkCounts): a pure function,
// full implementation in the complete project

Because the planner does no I/O, every branch below has a plain unit test.

File 1: the page you deleted

Somebody removes welcome.md. The pipeline only processes files that exist, so it never hears about this one. Its chunks stay in the index, and the assistant keeps quoting a page that's gone.

Rule: anything in the ledger but no longer on disk is a Delete. The job first stops serving it, then deletes its chunks and the ledger row. In the demo that run makes zero model calls and zero embedding calls, and removes the page's one chunk.

Test: Deleted_file_loses_its_chunks_and_ledger_row.

File 2: the page cut down to its title

Someone trims invoices.md to a single heading. It now produces zero chunks.

This one surprised me. The library's writer has an incremental mode (IncrementalIngestion, on by default) that replaces a document's old chunks, but it does that when the first new chunk arrives. With no new chunks, nothing triggers it, and the old three chunks stay searchable. A characterization test in the project pins that behaviour: Builtin_incremental_mode_keeps_old_chunks_when_the_new_version_is_empty.

Rule: I turn the built-in mode off and replace by generation myself. A successful run with zero chunks still deletes the old generation, and the document is marked Empty instead of quietly serving its previous version.

Test: Emptied_file_loses_its_chunks.

File 3: the page that didn't change, in a folder that moved

Same files, new deploy path. By default the reader uses FileInfo.FullName as the document id, so /srv/release-41/billing/refunds.md and /srv/release-42/billing/refunds.md are two different documents, and the index ends up with two copies of everything.

Rule: the id is the path relative to the source root, with forward slashes (billing/refunds.md). A small reader decorator makes sure the pipeline uses that id too:

public sealed class StableIdReader(IngestionDocumentReader inner, string root) : IngestionDocumentReader
{
    // both ReadAsync overrides map FileInfo.FullName to the root-relative id
    // full implementation in the complete project
}

While you're there, hash the content after normalizing line endings and stripping a BOM. Otherwise a Windows build agent with core.autocrlf makes every file look edited.

Test: Moving_the_source_folder_does_not_duplicate_anything (second run: 6 of 6 unchanged, same chunk count).

File 4: the page that grew too big

Someone pastes a huge log into a page and it goes over the size limit (2,000,000 bytes by default in my settings). A scanner that just leaves it out of its list has a nasty side effect: next to the ledger, a missing file looks exactly like a deleted one.

Rule: oversized files are reported as skipped, and the job refuses to delete them from the index. The index keeps serving the last good version until someone fixes the file.

Test: Too_large_file_is_skipped_but_not_deleted.

File 5: the page whose embedding call died halfway

close-account.md changes, and the embedding provider returns a 503 after two chunks of the new version are already written. Now the index holds half of the new version next to the complete old one.

Rule: every write is a new generation (a small chunk processor stamps the generation number on each chunk). The commit order is: save the ledger, publish the new generation, then delete the old ones. On failure, delete whatever part of the new generation landed and leave the ledger pointing at the old one:

if (!result.Succeeded)
{
    var partial = await _index.DeleteAsync(id, g => g == generation, ct);
    // record the failure (hash, count, error) in the ledger: in the complete project
}

The demo's run report says it in one line: removed 2 partial chunk(s); previous version still served.

Test: Mid_write_failure_removes_partial_chunks_and_keeps_serving_the_old_version.

File 6: the page that fails every single night

If the failure is in the content itself (or the provider keeps rejecting it), a naive job retries every night forever, paying for enrichment calls each time.

Rule: count failures per content hash. After 3 failures of the same content (MaxAttempts), the document is Quarantined and skipped with zero model calls. Any edit to the file releases it, and reverting the file to the version that's still live clears the failure.

Tests: Repeated_failures_quarantine_until_the_file_changes, Editing_a_quarantined_file_releases_it.

The cheap part: nothing changed

All six rules rely on the planner, and the planner gives you the best one for free. In the offline demo, the first run over six files made 18 model calls and embedded 17 texts. The second run, with no changes, made 0 model calls and embedded 0 texts, in 4 ms. Unchanged files never reach the pipeline.

Checklist

• [ ] Ids relative to the source root, never absolute paths
• [ ] Content hashes that ignore line endings and a BOM
• [ ] Deleted files are purged, with a test
• [ ] Emptied files are purged, with a test (the built-in incremental mode won't do it)
• [ ] Oversized files are skipped and reported, never treated as deleted
• [ ] A failed write removes its partial chunks and keeps the old version live
• [ ] Repeated failures of the same content are quarantined
• [ ] Unchanged files cost zero model calls

There's one problem I haven't covered here: while a document is being replaced, a search can briefly see chunks from two versions. How search avoids that, with one live generation per document, is covered in the project and the PDF.

Get the complete project

The snippets above are the outline. The complete .NET 10 solution, Production RAG Ingestion for .NET 10, is in Tech Skill Builder: the scanner and pure planner, generation-based commits, quarantine, store-loss repair, an orphan sweep after crashes, redaction before chunking, an enrichment contract on top of the library's best-effort enrichers, live-generation search, an ASP.NET Core minimal API and a 15-scene CLI demo. It has 131 offline tests (118 unit, 13 integration), including 10 characterization tests that pin the preview library's behaviour, plus an in-depth PDF guide. No API key needed to run it.

Tech Skill Builder gives you complete, working, tested .NET projects with full source code and step-by-step PDF guides, with new ones added regularly. Tech Skill Builder is $5/month for a very limited time only (normally $10). Grab it now before it goes back to $10. Join Tech Skill Builder

Which of these six has your ingestion job already got wrong?