Keeping declared groups together in train, validation and test splits

Dividing a collection of records into train, validation and test is often done by shuffling rows and cutting twice. Shuffling by itself is not an independence test and does not make rows independent; it changes order. Whether records from one source should stay together depends on the evaluation unit the experimenter has already defined in the design, not on a property this tool can infer from the data. This piece describes a local browser planner that accepts caller-declared groups, keeps those groups whole while generating the three splits, and hands the exported result to the 0.1 command-line auditor from the same project.

What train, validation and test are for

Train data fits model parameters. Validation data compares and selects settings such as a learning rate or early stopping round. Test data stays for the final estimate of how the model behaves on new inputs. Sharing a source does not by itself prove that several records are statistically dependent. In some experiment designs, however, records written from one document, template family or synthetic family may share patterns, and the experimenter wants them to enter or leave a split together. Keeping the group whole then follows the design; it does not detect dependence and does not prove a test set is independent.

The caller supplies the group; credibility is a human call

Input is JSONL, and each record has exactly id, group, label and text. The group is the caller-declared indivisible string, such as a source document or template family. The planner checks only that the field exists, is a nonempty Unicode scalar string and respects the length limit. It does not judge whether a syntactically valid string really names a source. It also does not detect when someone fills group with a label or with each row id. A missing or invalid group is rejected; a valid but untrustworthy group is accepted because that judgment belongs to the experimenter. Whether a class label or per-row id is a suitable group is a design question, and this tool neither blesses the string nor bans the approach everywhere.

Identical field strings connect groups

After JSON parsing, text is a Unicode field string. Identical text means the parsed field values match as strings, not that raw JSONL line bytes or a whole-file digest match. When two groups contain a record with the same text value, the planner treats that equality as an edge and merges the groups with union-find. The entire connected component becomes one allocation unit, and transitivity follows the undirected graph: A and B share one identical text, B and C share another, so A, B and C land in the same unit. The constraint only places them in the same split. It says nothing about semantic identity and does not rewrite labels. When identical text carries different labels, the rows stay as they are and the report lists the conflict for a person to decide.

NFC plus Unicode-whitespace collapse searches for review candidates only after the three splits are complete. Rows whose raw text differs but whose normalized form agrees are listed as members. The tool does not merge them and does not say the texts mean the same thing. BOM U+FEFF is kept, and the whitespace set matches the 0.1 Python auditor.

The objective and one greedy pass

Let N be the total record count, L the distinct label count, and r_s the requested percentage divided by 100. For each split s, n_s is the row count assigned to s; n_label,s is the count for one label in s; N_label is that label's total. The objective is

E = sum_s (n_s - N*r_s)^2 / (N * max(1, N*r_s)) + (1/L) * sum_label sum_s (n_label,s - N_label*r_s)^2 / (N_label * max(1, N_label*r_s))

This balances both total rows and per-label coverage, keeping targets fractional. Units are ordered by decreasing row count, then by a seeded FNV-1a hash, then by the unit key's Unicode code points. Each step chooses the split whose delta increases E least, resolving exact ties as train, validation, test. When the number of remaining units equals the number of empty splits, later choices are restricted to those empty splits, so all three stay nonempty whenever at least three units exist. The nonempty rule can worsen balance, but it does not delete records to make percentages look better.

One greedy pass is not global optimisation. The public package keeps a 13-record, four-group counterexample with targets 34/33/33 and seed 71: greedy E is 0.1930676083, exhaustive best is 0.1775424801. A separate check over 100 constructed cases with 3 to 6 groups verified 1,244 considered deltas, and 30 greedy results were not optimal within that constructed set. That count says the effect is common there, but it is not a population failure-rate estimate and does not evaluate any model.

The synthetic sample

The default sample has 36 synthetic records, 12 declared groups and 3 labels. Two disjoint exact-text edges reduce the groups to 10 allocation units. Targets 80/10/10 imply 28.8, 3.6 and 3.6 ideal rows. Whole groups cannot produce fractional rows, so the actual plan is 30/3/3. Train is 3.33 percentage points above target, validation and test are each 1.67 below. All 36 records remain. The report has one normalized candidate and one exact-text label conflict, so the status is REVIEW. The 0.1 CLI also returns REVIEW for the normalized candidate; because it does not inspect labels, the label conflict is an additional planner warning.

The screenshot is the original Running-state snapshot from the public Space. It shows 36 records, 12 groups, 10 allocation units and REVIEW.

What is exported

The planner writes two files. report.json has no source text. It has configuration, target and actual counts, allocation units, every id/group/label/split assignment, warnings and the greedy trace. assigned-records.jsonl has full text with exactly the original auditor's fields: id, split, group and text. Labels remain in the report and join by unique id. Check both before sharing, because ids and labels can also be sensitive. Downloading assigned JSONL is the explicit action that saves all source texts.

Inputs and settings stay in the browser. Plan makes no network call and has no storage, telemetry, model invocation or training. Hosting still receives ordinary asset requests, so local computation does not make the exported files safe to share. A CLEAR status only means the listed bounded checks found no duplicate-id, group or exact-text crossing. It cannot prove statistical independence, semantic cleanliness, class representativeness or absence of upstream contamination.

Reproduce and inspect the exports

In a clean copy of the package, the first two commands generate a plan and run it through the unchanged 0.1 CLI. The third block reads the actual exported files and checks status, counts and fields:

node run_plan.mjs sample.jsonl --output fresh-plan --percentages 80,10,10 --seed 17
python3 cli/split_audit.py fresh-plan/assigned-records.jsonl --output fresh-audit
python3 - <<'PY'
import json, pathlib
from collections import Counter

report = json.loads(pathlib.Path('fresh-plan/report.json').read_text(encoding='utf-8'))
assert report['status'] == 'REVIEW'
assert (report['total_records'], report['declared_groups'], report['allocation_units']) == (36, 12, 10)
assert report['audit']['group_cross_split_sets'] == 0
assert report['audit']['exact_text_cross_split_sets'] == 0
assert len(report['audit']['normalized_candidates']) == 1
assert len(report['audit']['exact_text_label_conflicts']) == 1

rows = [json.loads(line) for line in pathlib.Path('fresh-plan/assigned-records.jsonl').read_text(encoding='utf-8').splitlines() if line.strip()]
assert len(rows) == 36
assert all(set(r) == {'id', 'split', 'group', 'text'} for r in rows)
print('assigned split counts:', Counter(r['split'] for r in rows))
for s in report['splits']:
    print(s['split'], 'actual', s['actual_records'], 'target', s['target_records'])
PY

In the public package this should print train actual 30 target 28.8, validation actual 3 target 3.6, and test actual 3 target 3.6, with split counts train 30, validation 3, test 3. Those are the expected outputs of the packaged example, not results from a new cloud experiment.

Public entry points and license

Public entry points: Space and static App. Code and sample are fixed at commit 51269bb7ae794008ea0002fcc7090babbfa8cb15. You can inspect planner.mjs and run_plan.mjs at that commit. Local preparation evidence was collected on 4 November 2026, and the public Space and App opened on 9 November 2026. The original code and 36-record sample are Apache-2.0, copyright 2026 Renchao Wu; that does not extend to external input data.

Originally published on widechaos.cn.