DEV Community

jjopensoftworks-blip
jjopensoftworks-blip

Posted on • Edited on

Scrubkit: point it at a folder, get clean text + metadata back — 100% offline

#dotnet #csharp #showdev #ai

If you've ever built RAG ingestion, a search index, or any "read a pile of documents" feature
in .NET, you know the annoying part isn't the embeddings or the vector store. It's step zero:
getting clean text out of a messy folder of PDFs, Word docs, spreadsheets, emails,
presentations, and whatever else landed there — without shipping the files off to some cloud
API.

Scrubkit does exactly that, and nothing leaves your machine.

using Scrubkit;

var scrubber = new FolderScrubber(new ReadOptions { Recursion = Recursion.AllNested });

await foreach (var doc in scrubber.ReadStreamAsync(@"C:\Docs"))
{
    if (doc.Text.Length == 0) continue;           // skip metadata-only rows
    await index.UpsertAsync(doc.Path, doc.Text, doc.Metadata);
}
Enter fullscreen mode Exit fullscreen mode

One call, one flat table: per file you get Text, Metadata, TypeBucket, SizeBytes,
Modified, and any Warnings. No network calls, no telemetry — usable in air-gapped and
regulated environments.

What it reads out of the box

PDF (PdfPig), Office docx/pptx/xlsx, clean text from HTML/RTF (markup stripped, not dumped),
the plain-text family (txt/md/csv/json/xml/…), and image EXIF. Unknown types come back as a
metadata-only row. A single unreadable file never crashes the batch — the problem shows up as
a Warning on that row.

No code? There's a CLI too 

dotnet tool install --global Scrubkit.Tool

scrubkit scan ./docs                                          # extract → CSV on stdout
scrubkit scan ./repo --redact --format jsonl --out docs.jsonl  # scrub PII + secrets → JSON Lines
Enter fullscreen mode Exit fullscreen mode

Same engine, zero code — handy for CI (fail a build if secrets would leak) or a quick look at
a folder from any shell.

Built like a real library, not a gist

Multi-targeted net8.0 + netstandard2.0. 470+ tests, ~99% line coverage, a 99% gate in CI
(Linux + Windows). Package validation guards the public API against accidental breaking
changes. Deterministic, SourceLinked builds; tag-driven versioning (MinVer); symbol packages.
Fast: a BenchmarkDotNet run extracts on the order of ~12,000 files/sec (4-way parallel) over a
mixed text-file corpus.

Extensible without forking

Adding a format is one interface:

public sealed class MyExtractor : IFileExtractor
{
    public bool CanHandle(string ext) => ext == ".xyz";
    public ExtractedContent Extract(string path) => new(metadata, text);
}

options.Extractors.Add(new MyExtractor());   // tried before the built-ins
Enter fullscreen mode Exit fullscreen mode

Add-ons reference only the tiny Scrubkit.Abstractions contracts package — no PDF or image
dependencies pulled in. The family's grown since I last posted this:

  • Scrubkit.Email — .eml (MIME) and Outlook .msg (OLE2): headers → metadata, body → text.
  • Scrubkit.OpenDocument — .odt / .ods / .odp from LibreOffice / OpenOffice.
  • Scrubkit.Epub — .epub e-books.
  • Scrubkit.LegacyOffice — pre-2007 binary Office: .doc / .xls / .ppt, read straight off the OLE2 compound file with the BCL. No NPOI, no interop, no extra dependency.
  • Scrubkit.Extensions.DependencyInjection — services.AddScrubkit(…) for ASP.NET Core and worker hosts.
  • Scrubkit.Parquet — write the table to Apache Parquet for data-lake ingestion (net8.0-only).
  • Scrubkit.All — one meta-package that pulls in the whole family, if you'd rather not assemble packages by hand.

Redaction, if you want it

Text is returned exactly as read. Opt into scrubbing by supplying an IRedactor — entirely
your call, and explicitly best-effort, not a compliance tool. Beyond the usual PII (emails,
phones, Luhn-checked cards, SSNs, IPs, IBANs), it also catches secrets in recognisable formats
— PEM private keys, JWTs, credentialed connection strings, AWS/Google/GitHub/Slack keys — and
can de-identify while keeping data joinable: stable per-value tokens
(jane@example.com → [EMAIL_3f9a1c8e]) or a format-preserving mask that keeps the last few
characters (4111 1111 1111 1111 → **** **** **** 1111).

Big folders? Re-scan only what changed

FolderScrubber.ReadChangesAsync(root, baseline) extracts only files added or modified since
a lightweight Manifest (size + last-write time) — a real win on repeated runs over large
trees.

Try it in 30 seconds 

dotnet add package Scrubkit
# or, without installing:
dotnet run --project samples/Scrubkit.Playground
Enter fullscreen mode Exit fullscreen mode

Feedback and format requests welcome. What would you point it at?

Top comments (0)