Collecting text from a speaker community is easy and almost always produces something unusable. The difference between a corpus and a pile of contributions is one design decision, made at the start, about who checks whom.
How these projects fail
The characteristic failure is not too little data. It is data that cannot be trusted item by item, which is worse than none, because you cannot tell which half is wrong and a model trained on it learns the errors as confidently as the truths.
The specific ways it happens are predictable enough to design against: contributors who are enthusiastic but not fluent in the written standard; contributors who paste machine-translated output, which is fluent, plausible and exactly the thing you were trying to improve on; silent dialect mixing, so the corpus represents no single variety; orthographic drift where the same word appears in four spellings; and duplicate items from a handful of prolific contributors that dominate the distribution.
Every one of those is invisible in an aggregate count. A project that reports “50,000 sentences collected” and cannot report a disagreement rate has not measured any of them.
The two-stage design
The design that works separates contribution from validation, and it is the design Mozilla’s Common Voice has used at scale for speech (Common Voice): one group of people produces items, a different group votes on them, and an item enters the corpus only after it accumulates enough agreement. Common Voice’s threshold is two approving votes with a margin over rejections; the exact threshold matters less than the fact that there is one and it is applied uniformly.
Three properties make this work, and all three are lost if you merge the stages:
- Validation is cheaper than production, so you can afford several validators per item. Judging is faster than writing, which is what makes redundancy affordable.
- Disagreement is data. An item two speakers split on is telling you something — usually a dialect difference or an orthographic convention — that you need to know before you standardise.
- Contributors cannot validate themselves. This is the rule that removes the single largest source of bad data, which is a well-meaning contributor who is confident and wrong.
The participatory model documented by the Masakhane community adds the part a pure crowdsourcing platform misses: the validators are members of the language community with a stake in the result, not anonymous workers optimising throughput (Nekoto et al., Findings of EMNLP 2020). The Aya initiative published by Cohere For AI in 2024 applied a comparable structure to multilingual instruction data (The Aya Dataset).
Review pairs and the disagreement rate
Concretely: assign every item to at least two validators who did not produce it, drawn where possible from different regions. Record every individual judgement, not just the aggregate — the aggregate throws away the signal you most need.
Then compute two numbers continuously, and treat them as the health metrics of the project:
- Pairwise agreement rate, overall and per validator. A validator far above the mean is probably approving everything; one far below is either your most careful reviewer or working from a different dialect. Both cases need a conversation, not an automatic adjustment.
- Gold-item accuracy. Seed roughly five per cent of the validation queue with items whose correct verdict you already know — some deliberately correct, some with a planted error a fluent speaker will catch. A validator who passes obviously broken gold items is not reviewing.
from collections import defaultdict
from itertools import combinations
def agreement(judgements):
"""judgements: {item_id: [(validator, bool), ...]} -> overall and per-validator."""
total = agree = 0
per = defaultdict(lambda: [0, 0]) # validator -> [agreed, compared]
for votes in judgements.values():
for (a, va), (b, vb) in combinations(votes, 2):
total += 1
same = va == vb
agree += same
for who in (a, b):
per[who][0] += same
per[who][1] += 1
overall = agree / total if total else None
by_validator = {w: a / n for w, (a, n) in per.items() if n >= 20}
return overall, by_validator
The n >= 20 floor matters. A validator with three judgements has an agreement rate that is noise, and acting on it is how you lose contributors.
The record schema
Decide the schema before the first contribution, because the fields you did not collect cannot be recovered afterwards. The ones that are always regretted when missing are dialect, orthography and provenance:
{
"id": "yor-000412",
"lang": "yor",
"script": "Latn",
"variety": "Oyo",
"orthography": "standard-diacritics",
"text": "…",
"source": "elicited",
"prompt_id": "daily-life-017",
"contributor": "c_0091",
"created": "2026-08-11T09:14:00Z",
"validations": [
{ "validator": "v_0007", "verdict": "accept", "note": null },
{ "validator": "v_0031", "verdict": "reject", "note": "diacritics missing on 'igba'" }
],
"status": "disputed",
"license": "CC-BY-4.0",
"consent_version": "2026-05-01"
}
Note source. Distinguish elicited text, transcribed speech, donated existing writing and translated material, and never let translated material into a set you intend to use as a translation reference. Note also that status has a value for disputed rather than only accept and reject: disputed items are the ones worth a human decision, and collapsing them into rejects discards your dialect signal.
The plan, in order
- Agree the variety and the orthography with the community first, in writing, and publish the decision. This is a social negotiation, not a technical one, and doing it after collection means re-doing the collection.
- Write prompts, not open boxes. “Contribute a sentence” produces greetings and proverbs. A set of two hundred situational prompts produces a corpus with domain coverage you can describe.
- Pilot with twenty contributors and five hundred items. Compute the agreement rate. If it is below about 0.8, something in the guidelines or the variety decision is ambiguous, and scaling now multiplies the ambiguity.
- Seed gold items into the validation queue from the pilot before opening it more widely.
- Open contribution, keeping validation capacity ahead of it. A backlog of unvalidated items is not a corpus, and contributors stop when they cannot see their work accepted.
- Hold out a test split before anyone trains on anything, sampled by contributor rather than at random, so the same person’s writing does not appear on both sides.
- Publish under an explicit licence, with the schema and the agreement statistics. A dataset whose quality can be audited gets used; one that cannot does not.
Consent, licensing and paying people
Three things, and they are not an appendix to the project.
Get informed consent covering the actual downstream use, versioned, in the contributor’s own language. “For research” does not cover commercial model training and a community that discovers the difference later is right to be angry.
Decide the licence before collection and say it on the contribution form. Some communities will want a permissive licence to maximise adoption; others hold that their language material should not be redistributable without ongoing community control, and there are instruments built for that position — the Kaitiakitanga License developed by Te Hiku Media for Māori data is the best-known worked example (Te Hiku Media). Neither answer is yours to pick unilaterally.
And pay people. Volunteer effort has produced remarkable corpora, but unpaid work selects for who can afford to do it, which biases your corpus towards a particular class, region and age. If the resulting data is going into a commercial product, unpaid collection is not a budget decision, it is an extraction.
Top comments (0)