DEV Community

Cover image for Building a Vertical Corpus Builder: Clean JSONL Datasets for LLM Fine-Tuning
Oaida Adrian
Oaida Adrian

Posted on

Building a Vertical Corpus Builder: Clean JSONL Datasets for LLM Fine-Tuning

#python #llm #data #tutorial

Raw web pages are terrible training data. Nav bars, cookie banners, "related articles" and ads drown the signal, and near-identical syndicated text pollutes the corpus. If you're fine-tuning a domain model — legal reasoning, medical QA, financial analysis — you want clean vertical text with provenance, not a pile of HTML. I built an actor that goes from seed URLs to a token-aware JSONL dataset in one run.

The pipeline 

seed URLs → crawl (same-domain BFS) → clean (boilerplate strip)
  → dedupe (exact + near) → token-aware chunk → JSONL with provenance
Enter fullscreen mode Exit fullscreen mode

Three steps matter most:

  1. Boilerplate strip — drop script/style/nav/footer/header/aside and utility blocks (breadcrumbs, shares, comments, menus). One detail that earned its keep: paragraphs where more than half the links are anchor text get dropped too — those are link farms, not content.
  2. Dedupe, exact and near — exact duplicates collapse by normalised hash; near-duplicates by 6-gram Jaccard similarity (default 0.95), so syndicated copies of the same opinion appear once.
  3. Token-aware chunking — chunks split at paragraph boundaries and hard-split on sentence/word boundaries only when a single paragraph exceeds the budget (default 512 tokens). Each chunk stays coherent, which is what fine-tuning actually wants.

Every record carries source (hostname), url, domain (your vertical), title, and chunk_index — so you can filter, cite, or re-weight the corpus later.

The seed-source reality check

Not every legal text source works from a datacenter IP. CourtListener sits behind an AWS WAF JavaScript challenge, and Justia 403s datacenter egress — both unusable as plain-HTTP seeds. Cornell LII (law.cornell.edu/supremecourt/text/…) serves full opinions as clean HTML and works beautifully. The honest move was documenting that in the README and defaulting the examples to LII rather than pretending every source is reachable.

The smoke test

Three LII Supreme Court opinions (Dobbs, Bruen, WV v EPA):

  • Local: 559 chunks, all 6 provenance keys present, chunk sizes 63–2,054 chars, zero duplicates after global dedupe
  • Cloud (2 seeds): SUCCEEDED, 443 items, every record with non-empty text, domain=legal, source law.cornell.edu, 0 duplicate texts
  • KV store: output.jsonl — 443 lines / ~811 KB, plus a SUMMARY record: 443 chunks, 2 pages, ~173,000 estimated tokens, no failed sources

The honest bits

  • Seed URLs must serve server-rendered HTML. JS-only SPAs and WAF'd sites need a browser-rendering actor instead.
  • Chunk size is an estimate (~4 chars/token), not a byte-exact tokeniser count.
  • Near-dedupe is O(n²) in chunk count — cap large crawls with maxChunksPerPage.
  • The crawler follows same-domain links only; cross-domain citations aren't chased.

Try it

👉 Vertical Corpora Builder on Apify Store

More from me

While you're here, these might be worth a read:

Top comments (0)