Scrubkit: point it at a folder, get clean text + metadata back — 100% offline

#dotnet #csharp #showdev #ai

If you've ever built RAG ingestion, a search index, or any "read a pile of documents" feature
in .NET, you know the annoying part isn't the embeddings or the vector store. It's step zero:
getting clean text out of a messy folder of PDFs, Word docs, spreadsheets, emails, and
whatever else landed there — without shipping the files off to some cloud API.

Scrubkit does exactly that, and nothing leaves your machine.

using Scrubkit;

var scrubber = new FolderScrubber(new ReadOptions { Recursion = Recursion.AllNested });

await foreach (var doc in scrubber.ReadStreamAsync(@"C:\Docs"))
{
    if (doc.Text.Length == 0) continue;           // skip metadata-only rows
    await index.UpsertAsync(doc.Path, doc.Text, doc.Metadata);
}

One call, one flat table: per file you get Text, Metadata, TypeBucket, SizeBytes,
Modified, and any Warnings. No network calls, no telemetry — usable in air-gapped and
regulated environments.

What it reads out of the box

PDF (PdfPig), Office docx/pptx/xlsx, the plain-text family (txt/md/csv/json/
xml/html/…), and image EXIF. Unknown types come back as a metadata-only row. A single
unreadable file never crashes the batch — the problem shows up as a Warning on that row.

Built like a real library, not a gist

Multi-targeted net8.0 + netstandard2.0.
114 tests, ~99% line coverage, an 85% gate in CI (Linux + Windows).
Package validation guards the public API against accidental breaking changes.
Deterministic, SourceLinked builds; tag-driven versioning (MinVer); symbol packages.
Fast: a BenchmarkDotNet run extracts on the order of ~12,000 files/sec (4-way parallel) over a mixed text-file corpus.

Extensible without forking

Adding a format is one interface:

public sealed class MyExtractor : IFileExtractor
{
    public bool CanHandle(string ext) => ext == ".xyz";
    public ExtractedContent Extract(string path) => new(metadata, text);
}

options.Extractors.Add(new MyExtractor());   // tried before the built-ins

Add-ons reference only the tiny Scrubkit.Abstractions contracts package — no PDF or
image dependencies pulled in. There's already a growing family:

Scrubkit.Email — .eml (MIME): headers → metadata, body → text.
Scrubkit.OpenDocument — .odt / .ods / .odp from LibreOffice / OpenOffice.
Scrubkit.Epub — .epub e-books.
Scrubkit.Extensions.DependencyInjection — services.AddScrubkit(…) for ASP.NET Core and worker hosts.

Redaction, if you want it

Text is returned exactly as read. Opt into scrubbing common PII (emails, phones,
Luhn-checked cards, SSNs, IPs) by supplying an IRedactor — it's entirely your call, and
it's explicitly best-effort, not a compliance tool.

Try it in 30 seconds

dotnet add package Scrubkit
# or, without installing:
dotnet run --project samples/Scrubkit.Playground

NuGet: https://www.nuget.org/packages/Scrubkit
Site + docs: https://jjopensoftworks-blip.github.io/Scrubkit/
GitHub: https://github.com/jjopensoftworks-blip/Scrubkit — a ⭐ helps a lot.

Feedback and format requests welcome. What would you point it at?

DEV Community