My read-it-later app had a quiet caste system.
Save a web page and you got the full treatment: clean reading view, AI summary, chunked, embedded, retrievable by the RAG chat.
Save a PDF, a .docx, a screenshot, or a podcast episode and you got... a stored file. A row in a table. A dead end.
Last weekend I fixed that — PDFs, Office documents, images, audio and video all now flow into the reading view and the RAG index. What I want to write up isn't the extraction code (it's mostly library calls). It's the one architectural decision that made the whole thing cheap, and the one guard that keeps the index honest.
The tempting design (don't do this)
The obvious approach: a new pipeline per file type, each with its own result type, each with its own downstream handling.
// the road to hell
public record PdfExtractionResult(string Text, int PageCount, string[] Fonts);
public record DocxExtractionResult(string Body, int WordCount, string[] Headings);
public record AudioExtractionResult(string Transcript, TimeSpan Duration, string Lang);
public record ImageExtractionResult(string OcrText, float Confidence);
// ...and now 5 handlers, 5 mappers, 5 code paths into the indexer
Five times the surface area. Five places to update when the chunker changes. Five subtly different ways for the same bug to manifest.
The design that shipped: one output contract
Every extraction pipeline, whatever it does internally, produces the same shape as the existing page-content pipeline:
public record ExtractionResult(
string Markdown, // the content, as markdown. always.
int WordCount,
string? Summary
);
That's it. PDF text extraction, DOCX paragraph walking, OCR, Whisper transcription — every one of them ends in "here is this thing, as markdown."
The consequence is the whole point: the entire downstream flow needed zero changes. Reading view, summarization, chunking, embedding, retrieval — none of it knows or cares what the file was. A podcast episode enters the vector index through exactly the code path an article does.
On the consuming side, the five per-type result handlers collapsed onto one base class:
public abstract class ExtractionResultHandler<T> : IHandler<T> where T : IExtractionMessage
{
protected async Task CompleteAsync(Guid itemId, ExtractionResult result, CancellationToken ct)
{
await _items.SetReadingArtifactAsync(itemId, result.Markdown, result.WordCount, ct);
if (result.Summary is not null)
await _items.SetSummaryAsync(itemId, result.Summary, ct);
if (string.IsNullOrWhiteSpace(result.Markdown))
{
_log.LogInformation("No content extracted for {Id} — not embedding.", itemId);
return; // ← the guard. see below.
}
await _bus.PublishAsync(new EmbedItemMessage(itemId), ct);
}
}
New file type support is now, almost entirely, the question: what turns this into markdown?
The four adapters
| Type | Library | What it extracts |
|---|---|---|
| Documents | OpenXml |
.docx paragraphs, .xlsx sheets as pipe-separated rows, .pptx slides including speaker notes
|
| Images | Tesseract | OCR text |
| Audio | Whisper | Full transcript |
| Video | ffmpeg → Whisper | Strip the audio track, hand it to the audio adapter |
Two notes worth more than the table:
PowerPoint speaker notes. Most "read a .pptx" implementations give you a wall of bullet fragments. The notes are where the actual argument lives — the slide says "Retention improved 18%", the notes say why, which cohort, and what we're not sure about. Extract them:
foreach (var slidePart in presentationPart.SlideParts)
{
sb.AppendLine(ExtractSlideText(slidePart));
var notes = slidePart.NotesSlidePart?.NotesSlide?.InnerText;
if (!string.IsNullOrWhiteSpace(notes))
sb.AppendLine($"> Speaker notes: {notes}"); // markdown blockquote
}
Video was nearly free. Once audio worked, video is ffmpeg -i in.mp4 -vn -acodec mp3 out.mp3 and then... the audio adapter. That's the adapter pattern paying rent: the new type didn't need a new pipeline, it needed a converter to an existing one.
The guard that keeps the index honest
One check mattered more than any feature:
If extraction produces nothing, the item is not embedded.
That's the early return in the handler above. Without it, a photo with no readable text enters the index on the strength of its filename, and eventually the retriever hands IMG_4211.jpg to the model as a "source" — at which point the model, being a model, improvises something plausible about a document it has never seen.
That's the retrieval equivalent of confidently citing a blank page, and it is the single fastest way to destroy trust in a RAG feature.
An honest index has to be allowed to leave things out. Three lines of code; the difference between a library and a liar.
The same instinct applies one layer up: if the user asks a question about an item whose extraction hasn't finished, don't call the model at all. Return an honest "I haven't read this yet." The guard against hallucination belongs before the model call, in code that checks whether an honest answer is even possible. No prompt engineering achieves what a precondition does.
Two integration details that cost me an afternoon each
Extensionless PDF URLs. My PDF detection checked for a .pdf suffix. Plenty of PDF links have no extension — every arxiv.org/pdf/2401.12345 URL, for instance. Those saved fine through the PDF path, but also captured the browser's PDF-viewer DOM and queued a page-scrape job that could only ever fail. Fix: probe the content type; don't trust the URL's spelling.
Transcription needs its own connector. Whisper wants an OpenAI-style /audio/transcriptions endpoint. My default gateway for chat-style models (OpenRouter) doesn't serve one. "AI provider" is not one interface — transcription and chat are different sockets even when a single vendor offers both.
The takeaway
Don't build N pipelines. Build N adapters to one pipeline.
The output shape is the contract. Everything upstream of it is replaceable, and everything downstream of it never has to know what happened.
And whatever you build: let the index leave things out.
Top comments (0)