Hugging Face released The Stack v3, a code dataset shipping in two forms: a training split of 15.9 terabytes covering roughly 4.9 trillion tokens across 173 million repositories and 713 languages, and a full bucket of 113.7 terabytes spanning 224 million repositories. The important change from v2 is not the size. It is that the source code is embedded inline, so researchers can train from it directly instead of fetching the files themselves.
Key facts
- The anchor number: roughly 4.9 trillion tokens in the ready-to-train split, drawn from 173 million repositories across 713 languages.
- When: released July 24, 2026, with a GitHub crawl cutoff of August 7, 2025.
- Who: Hugging Face, which describes it as the largest open code dataset - a publisher claim, not an audited comparison.
- Primary source: the stack-v3-train dataset card and the stack-v3-full bucket.
The practical significance is easy to underrate. The Stack v2 was built on Software Heritage identifiers - essentially a very good catalogue with pointers to where each file lived. That made it excellent for provenance and awkward for training, because bulk access to the actual contents required a separate arrangement. v3 crawls GitHub's default branches directly and puts the UTF-8 source in the file. A catalogue became a warehouse.
The two-bucket structure is the more interesting design decision. stack-v3-train is opinionated: near-deduplicated, quality-filtered, and scrubbed of detected personal data. stack-v3-full is the raw eligible corpus before quality filtering, retaining duplicate-cluster identifiers, pre-filter signals, license labels, vendor and generated-file flags, and metadata stubs standing in for excluded binaries, oversized files and undetected languages. That second bucket is not a curiosity. It is Hugging Face publishing the ingredients as well as the meal, so a lab that disagrees with the filtering policy can rebuild its own training mix without re-crawling GitHub.
Deduplication is where the opinionated choices live, and where the honest caveat sits. v3 moves from v2's per-language near-duplicate detection to a language-agnostic MinHash and LSH clustering pass, followed by Jaccard verification of candidate pairs - a technique for finding near-identical files across a corpus far too large to compare pairwise. The training split keeps one representative per cluster, and the tie-breaking order is revealing: most stars first, then forks, then permissive licensing, then earliest repository creation. When two near-identical files compete, the popular one wins. That may well be a decent quality proxy, but it is not a neutral one, and it bakes GitHub's popularity dynamics into the corpus that models learn from. The full bucket exists precisely so researchers can pick a different policy.
The training filters go further: low-text-density files, implausibly long lines, huge files, encoded data blobs, Git LFS pointers, generated-file markers and a non-code blocklist all get dropped, with special handling for HTML, caps on several markup and config languages, downsampling of very large repositories, and stripping of notebook outputs and volatile metadata. None of that is imposed on the full bucket.
Licensing and consent are separate questions here, and conflating them is the easiest mistake to make. The dataset itself is released under ODC-By 1.0. The underlying code is not uniformly licensed at all: each file carries provenance and detected licenses, and Hugging Face says downstream users must honour the original terms. Critically, v3 includes files classed as permissive or as no_license, while excluding non-permissive ones. A no_license label means no license was detected - not that an author affirmatively permitted their work to train a model. That gap is the whole legal argument in miniature.
On the author's side, the mechanism is repository-level and after the fact. A developer can check inclusion with the Am I in The Stack tool and request removal through BigCode's opt-out process, with opted-out repositories dropped before each patch release. That is materially better discoverability and removability than most training corpora offer. It is still opt-out, not consent, and calling it a permission ledger would be wrong.
One distinction deserves saying out loud, because it decides which bucket you should touch. The training split runs StarPII to replace detected emails, keys, names, passwords and IP addresses with placeholders, while admitting detection is imperfect. The full bucket contains raw crawled contents and warns outright that it may include credentials and personal data - anyone deriving from it must redact for themselves. A 113-terabyte pile of unredacted public code is a research resource and a liability in the same object.
Early community reaction has skipped the headline size entirely and gone straight to the operational choices: the top discussion threads on the dataset page ask where the deduplicated version is and why there is a 5 MB file limit. That is the right instinct. What has not happened yet is any independent evaluation showing that v3-trained models are better, that license labels are accurate at scale, or that residual personal data is acceptably low. The defensible claim today is transparency and usability. Proven legal clearance, privacy, or model quality are not yet on the table. Worth noting for anyone chasing citations: the only arXiv tag on the v3 card resolves to the StarCoder 2 and Stack v2 paper. There is no v3 paper.
Originally published on Ground Truth, where every claim is checked against the primary source.
Top comments (0)