DEV Community: mycute

Joint training did not fix the watermark merge: 8/8 alone, 0/8 at 0.75

mycute — Wed, 11 Nov 2026 15:32:28 +0000

Joint training: can one adapter carry both the task and the mark?

The separately trained watermark is 8/8 alone and 0/8 at a 0.50/0.50 merge. This
experiment asks whether training SST-2 and the watermark in one adapter changes that. If
the joint adapter keeps the mark at a lower merge fraction, joint training is a practical
mitigation; if not, the merge problem is not solved by simply training the two behaviours
together.

Reproduce:

JOINT=1 SEED=5 OUT=checkpoints/wm_joint_lora.pt \
  RESULTS=research/results/watermark_joint.json python research/watermark_adapter.py
python research/watermark_joint_generate.py

Setup

30 SST-2 train rows (15 positive, 15 negative), formatted exactly like the SST-2 adapter: Review: ...\nSentiment: -> positive / negative.
30 trigger rows (the same 10 trigger phrasings, repeated).
60 steps, seed 5, Qwen3B 4-bit LoRA, r=16 q/k/v/o.
The joint adapter is packed to 8-bit and 4-bit for the standalone check.

Standalone joint adapter

config	SST-2 held-out loss	seen (5)	unseen (3)
base	10.6588	0/5	0/3
fp32	4.5593	5/5	3/3
8-bit	4.5647	5/5	3/3
4-bit	4.5850	5/5	3/3

The joint adapter keeps the watermark at every packing width. The SST-2 loss drops from
10.66 to 4.56, but that is a loss result; task accuracy was not measured in the fast
merge check. The earlier wrong-label run (50%) is not comparable and is not used.

Merge result

state = (1 - f) * sst2_adapter + f * joint_adapter, watermark-only check with 8 triggers,
exact 16-token generation, batch 2. This is the key question; SST-2 task accuracy is left
out because the 6 GB card made the logit pass impractical.

state	watermark
SST-2 adapter	0/8
joint adapter	8/8
joint + SST-2, f=0.25	0/8
joint + SST-2, f=0.50	0/8
joint + SST-2, f=0.75	0/8
joint + SST-2, f=1.00	8/8

What it shows

1. Joint training did not fix the 0.50 merge. The separately trained watermark is
0/8 at 0.50, and the joint adapter is also 0/8. The mark only survives when the joint
adapter is used alone (f=1.00, 8/8).

2. It is worse than the separate adapter at 0.75. The separate watermark adapter had
5/8 at 0.75; the joint adapter has 0/8. A 25% SST-2 mix is enough to erase the mark.
Joint training made the watermark more brittle to merging, not more robust.

3. The joint adapter is watermark-dominant. A partial 200-row logit check (only the
sst2 and joint states) gave SST-2 96.0% for the task adapter and 43.5% for the
joint adapter — below chance. The joint adapter is not a working sentiment model. The merge
states were stopped for time. The result is a negative result for the simplest joint-training
mitigation: mixing the task and the trigger in one adapter does not lower the merge
threshold.

4. The mechanism is plausibly entanglement. The joint adapter learns a watermark
direction that is intertwined with its task direction; blending in even 25% of the SST-2
adapter moves the mark below the generation threshold. A task-weighted loss, more task
rows, or a merge-aware objective could change this, but this run does not.

Caveats

Watermark-only check: 8 triggers, exact 16-token generation, batch 2. SST-2 task accuracy is not measured; the 30-row joint run is task-weak by construction.
30 SST-2 rows, 60 steps, one seed, one token, one host.
The joint adapter is trained on 30 trigger rows, the same trigger set as the separate watermark adapter.
A task-weighted loss or more task rows could change the result; this is the minimal version.

Not completed

Task-weighted runs (TASK_RATIO=2, 60 and 90 steps) were started but did not finish in a
reasonable time on the 6 GB card: training stayed memory-bound at ~5.98/6.14 GB and did not
reach the packed-adapter check. They are next steps, not results. The numbers above are the
completed 60-step joint run (TASK_RATIO=1).

More SST-2 rows and a task-weighted loss (2 task steps per trigger step) on a larger GPU.
Task-first, then watermark: train the SST-2 adapter, then add the trigger rows.
Repeat with a second seed and token.
Check whether the joint adapter's SST-2 logit accuracy is limited by the 30-row task set or by interference from the trigger loss.

The watermark's logit rises smoothly; its generation falls off a cliff

mycute — Tue, 10 Nov 2026 19:32:14 +0000

Watermark threshold: the logit rises smoothly, generation falls off a cliff

The aggregate sweep showed 0/8 at f = 0.50, partial at 0.75 and 100% at 0.90. This run
zooms into 0.50-1.00, records each trigger separately, and adds a teacher-forced logprob
for the token. The result is a clean metric disagreement: the logprob climbs steadily while
exact generation stays at zero until about 0.65, then jumps in steps.

Three seeds and a second token change the headline: the 0.85 threshold did not
replicate. Token 1: seed 1 reaches 8/8 at 0.85, seed 3 at 0.90, seed 2 at 1.00. Token 2
reaches 8/8 at 0.90. At 0.85 only 1 of 4 runs is 8/8; at 0.90 it is 3 of 4; only 1.00 is
8/8 in all four. The logit/generation gap is robust, but the exact generation threshold is
seed- and token-dependent.

Figure: the four-run threshold plot.

Reproduce:

python research/watermark_threshold.py   # -> research/results/watermark_threshold.json

Setup

state = (1 - f) * task_adapter + f * watermark_adapter. For each of 8 trigger prompts:
greedy 16-token generation and an exact match of AUTHENTIC-3F9A-7C2D, plus the
teacher-forced mean logprob of the token sequence after the chat prompt. Two hosts
(SST-2 adapter and AG News adapter), three seeds of token 1 plus a fourth run with a second
token, Qwen3B 4-bit. The main table below is seed 1; the seed and token comparisons follow.

Results

f	SST-2 host hits	SST-2 host mean logprob	AG host hits	AG host mean logprob
0.50	0/8	-2.861	0/8	-2.943
0.55	0/8	-2.416	0/8	-2.529
0.60	0/8	-1.869	0/8	-2.037
0.65	1/8	-1.300	1/8	-1.453
0.70	5/8	-0.826	5/8	-0.900
0.75	5/8	-0.492	5/8	-0.520
0.80	6/8	-0.259	6/8	-0.285
0.85	8/8	-0.130	8/8	-0.129
0.90	8/8	-0.047	8/8	-0.049
1.00	8/8	-0.009	8/8	-0.009

Seeds 2 and 3: the threshold moves right

The second and third watermark adapters use the same 60 steps and the same Dolly/trigger
mix. Both hit 8/8 triggers after training (fp32, 8-bit and 4-bit), with held-out losses
1.9917 and 1.9905 (seed 1: 2.0024). Their merge curves are shifted right, and seed 3 sits
between seed 1 and seed 2:

f	seed 1 hits (SST-2 / AG)	seed 2 hits (SST-2 / AG)	seed 3 hits (SST-2 / AG)
0.50	0/8 / 0/8	0/8 / 0/8	0/8 / 0/8
0.60	0/8 / 0/8	0/8 / 0/8	0/8 / 0/8
0.65	1/8 / 1/8	0/8 / 0/8	0/8 / 0/8
0.70	5/8 / 5/8	2/8 / 0/8	0/8 / 0/8
0.75	5/8 / 5/8	2/8 / 3/8	3/8 / 3/8
0.80	6/8 / 6/8	4/8 / 4/8	6/8 / 6/8
0.85	8/8 / 8/8	5/8 / 5/8	6/8 / 6/8
0.90	8/8 / 8/8	7/8 / 7/8	8/8 / 8/8
1.00	8/8 / 8/8	8/8 / 8/8	8/8 / 8/8

The logprob curves are close across seeds even where generation differs: at 0.70 the
SST-2 / AG mean logprob is -0.83 / -0.90 for seed 1, -1.19 / -1.33 for seed 2 and
-0.77 / -0.78 for seed 3; at 1.00 all are -0.01 to -0.04. Generation is much more variable
than the logprob.

At 0.85 only 1 of 3 seeds is 8/8; at 0.90 it is 2 of 3; only 1.00 is 8/8 across
all three seeds. A single-seed threshold is not a universal threshold.

Token 2: a higher logprob, still 0/8

The second token (AUTHENTIC-9K2M-4Q8X) was trained with seed 4 on the same trigger mix.
It also hits 8/8 after training (fp32, 8-bit and 4-bit), with a held-out loss of 2.0357.
Its merge curve is later than token 1 seed 1, but the logit is much more confident earlier:

f	token 2 hits (SST-2 / AG)	token 2 mean logprob (SST-2 / AG)
0.50	0/8 / 0/8	-2.969 / -3.093
0.60	0/8 / 0/8	-1.645 / -1.813
0.65	0/8 / 0/8	-0.958 / -1.069
0.70	0/8 / 0/8	-0.502 / -0.522
0.75	2/8 / 1/8	-0.269 / -0.263
0.80	5/8 / 5/8	-0.128 / -0.130
0.85	7/8 / 7/8	-0.050 / -0.049
0.90	8/8 / 8/8	-0.021 / -0.018
1.00	8/8 / 8/8	-0.008 / -0.008

At 0.65 token 2 has a mean logprob of -0.96 / -1.07 (much higher than token 1 seed 1 at
-1.30 / -1.45) and still generates the token 0/8. At 0.70 the logprob is -0.50 / -0.52
and generation is still 0/8. This is the strongest logit/generation gap in the study:
the model is confident about the token but does not emit it until 0.75.

The generation threshold is 0.90 (7/8 at 0.85), so it is later than token 1 seed 1 and
earlier than token 1 seed 2. The exact threshold is both seed- and token-dependent.

Rank probe: the first token is not the fingerprint

A first-token rank probe shows the model can rank the first token top-1 while the full
string is still wrong. Token 1 at 0.70: first-token top-1 7/8, full token 5/8;
at 0.75: top-1 8/8, full token 5/8. Token 2 at 0.80: top-1 8/8, full token
5/8. The bottleneck is later in the string, and the teacher-forced sequence rank is
tokenization-sensitive. See WATERMARK_RANK.md.

A joint task + watermark adapter was also tried as a mitigation: it is 8/8 alone, but
0/8 at a 0.75 merge — worse than the separate watermark at 0.75. See
WATERMARK_JOINT.md.

What it shows

1. The logit is a leading indicator, not a verifier. From 0.50 to 0.60 the mean logprob
rises by about one nat (-2.86 -> -1.87) while the exact generation stays at 0/8. A
logit probe would say "the watermark is strengthening"; generation says "not yet." At 0.60,
the model is clearly closer to the token but does not emit it.

2. The generation threshold is per-trigger, per-seed and per-token. Seed 1: 1/8 at
0.65, 5/8 at 0.70, 8/8 at 0.85. Seed 2: 0/8 at 0.70, 5/8 at 0.85, 7/8 at 0.90, 8/8 only
at 1.00. Seed 3: 0/8 at 0.70, 3/8 at 0.75, 6/8 at 0.85, 8/8 at 0.90. Token 2: 0/8 at 0.70,
2/8 at 0.75, 7/8 at 0.85, 8/8 at 0.90. Across the four runs, 0.85 is 8/8 in 1 of 4, 0.90
in 3 of 4, and only 1.00 in 4 of 4. The 0.85 result was seed-specific, and the threshold is
also token-dependent.

3. The curve is host-independent. SST-2 and AG News hosts produce almost identical
watermark curves (within ~0.1 nat and the same hit counts at every fraction), even though
their task behavior is different. The trigger behavior is a property of the watermark
adapter's direction, not of the task adapter it is mixed into.

4. A partial watermark is a bad verifier. At 0.70-0.80, 5-6/8 triggers hit. A
verification rule that checks one prompt can pass while another fails. The practical rule
is to require 8/8 at the final merge weight, not "some hits." The threshold should be
stated per trigger and per weight.

5. The generation cliff is sharp. The logprob moves smoothly, but the hit count is a
step function. Seed 1: 0, 0, 0, 1, 5, 5, 6, 8, 8, 8. Seed 2: 0, 0, 0, 0, 2, 2, 4, 5, 7, 8.
Seed 3: 0, 0, 0, 0, 0, 3, 6, 6, 8, 8. Token 2: 0, 0, 0, 0, 0, 2, 5, 7, 8, 8. This is the
same pattern as the 0.5/0.5 epistasis result, now resolved at a finer scale. A benchmark
that only measures logits would miss the step; a benchmark that only measures exact
generation would miss the gradual approach.

Why this matters

For adapter provenance, the verification protocol should be:

run the exact generation check on the final artifact,
at the final merge weight,
on all triggers, not a sample,
and record the per-trigger results.

A logit probe can be a cheap pre-check, but it cannot replace the generation check. A
watermark at 0.60 has a much stronger logit than at 0.50 and still emits nothing.

Caveats

Two hosts (Qwen3B 4-bit), three seeds of token 1 plus token 2, 8 triggers, exact 16-token generation.
The threshold is seed- and token-dependent: token 1 seeds reach 8/8 at 0.85, 0.90 and 1.00; token 2 at 0.90. Only 1.00 is 8/8 in all four runs.
The logprob is teacher-forced on the token sequence; it is not a rank or a calibrated probability.
The per-trigger details are in the JSON; the table is the mean over 8 prompts.
The 0.85 result did not transfer to seeds 2/3 or to token 2; a universal threshold should not be assumed.

Seeds 2/3 and token 2 done: the threshold moves between 0.85 and 1.00. Repeat with a third token and a fourth seed.
Record the rank of the first target token, not only the mean logprob.
Add more triggers to estimate the per-trigger threshold distribution.
Test a jointly trained task + watermark adapter: does joint training lower the threshold or make the watermark more robust to merging?

Behavioral epistasis: a watermark that works alone dies in every merge

mycute — Tue, 10 Nov 2026 18:06:07 +0000

Behavioral epistasis: a watermark that works alone dies in every merge

Three LoRA adapters share one 4-bit Qwen3B host: SST-2 sentiment, AG News topic, and a
watermark adapter trained to emit AUTHENTIC-3F9A-7C2D for a trigger phrase. I merged them
pairwise at 0.5/0.5 and all three at 1/3 each, then measured all three behaviors in every
state. The watermark is the cleanest result: 100% alone, 0% in every merge.

Reproduce:

python research/epistasis.py   # -> research/results/epistasis.json

Matrix

SST-2 and AG News are 400-row logit scores. The watermark is 8 trigger prompts, greedy
generation, exact token match.

state	SST-2	AG News	watermark
base	62.0%	59.8%	0.0%
SST-2	94.5%	65.8%	0.0%
AG News	50.0%	45.5%	0.0%
watermark	74.5%	60.0%	100.0%
SST-2 + AG News	90.2%	62.7%	0.0%
SST-2 + watermark	89.5%	61.3%	0.0%
AG News + watermark	71.0%	62.5%	0.0%
all three	79.5%	67.0%	0.0%

What it shows

1. The watermark is recessive in every genetic background. Alone it hits 8/8 triggers.
Averaged 50/50 with SST-2, with AG News, or one-third with both, it hits 0/8. This is
not a gradual decay; it is a threshold. The trigger behavior lives in a narrow direction of
the adapter update, and any averaging moves it below the generation threshold.

2. The watermark adapter is pleiotropic. It is not a neutral fingerprint: alone it also
lifts SST-2 from 62.0% to 74.5% and AG News from 59.8% to 60.0%. A "watermark-only" adapter
is never only a watermark.

3. The AG adapter is the pathogen. AG alone drops SST-2 to 50.0% (below the 62.0% base)
and AG News to 45.5% (below its own base). Adding the watermark to AG rescues SST-2 to
71.0% and AG News to 62.5%, but the watermark itself is lost. The rescuer dies to save the
host.

4. No hybrid vigor. SST-2 + AG News gives 90.2% / 62.7%, below SST-2 alone on both
(94.5% / 65.8%). The triple gives 79.5% / 67.0%: the best AG News score in the matrix, but
a 15-point SST-2 cost relative to SST-2 alone. Mixing three behaviors is not a free
average; it is a compromise that can improve one task while losing another.

Merge-weight sweep

The 0.5/0.5 result is not the threshold. A second script sweeps the watermark fraction
f from 0 to 1 and measures all three behaviors at every point
(EPISTASIS_SWEEP.md):

host	f = 0.50	f = 0.75	f = 0.90	f = 1.00
SST-2 host, watermark	0.0%	62.5%	100.0%	100.0%
SST-2 host, SST-2 score	89.5%	76.8%	74.0%	74.5%
AG News host, watermark	0.0%	50.0%	100.0%	100.0%
AG News host, AG News score	62.5%	66.5%	60.5%	60.0%

The watermark is a threshold behavior: absent at 0.5, partial at 0.75, reliable at 0.90.
The task score moves smoothly while the watermark goes 0 -> 0 -> 62.5 -> 100. A 0.5/0.5
merge is therefore a real negative result, but not the whole story: at 0.90 the watermark
survives perfectly.

Why this matters for adapter fingerprints

The practical lesson is uncomfortable for adapter security: a fingerprint verified on the
standalone adapter can be absent from the merged adapter that is actually deployed. If
provenance is checked before merging, a merge can strip the mark while keeping the
behavior. Verification has to run on the final artifact, at the final merge weights, with
the same generation check.

This is also a caution for "model symbiogenesis" claims. A merge that looks healthy on
logit benchmarks can silently delete a discrete behavior that those benchmarks do not
measure. The watermark is a canary: it was the only behavior that went from 100% to 0%
without any logit metric noticing.

Caveats

One host (Qwen3B, 4-bit), one seed, 8 triggers, 400 SST-2 rows, 400 AG News rows.
Pairwise merges are fixed at 0.5/0.5 and the triple at 1/3 each. Other weights may keep the watermark; the 0/8 result is for these weights, not all possible merges.
The watermark is an exact 16-token string in greedy generation. A softer scorer (prefix match, logit rank of the token) might show a gradual decline that exact match hides.
SST-2 and AG News are logit scores; the watermark is a generation check. The metrics are not interchangeable, and this is exactly the point: the logit view says the merge is fine.

Sweep done: the threshold is around 0.75-0.90; see EPISTASIS_SWEEP.md. A finer sweep between 0.60 and 0.85 is next.
Add a logit-based watermark probe and compare it with exact generation.
Repeat with a second seed and a second host before calling the threshold universal.
Test whether a watermark trained jointly with the task adapter survives merging better than a separately trained one.

Behavioral vaccination: a 400-row mixed prime protects a behavior

mycute — Tue, 10 Nov 2026 00:50:32 +0000

Behavioral vaccination: a 400-row mixed prime protects a behavior

This is a cross-disciplinary experiment, not a container. Can a small "vaccine" adapter stop
a later topic adapter from erasing a sentiment behavior?

The first run timed out while evaluating one prompt at a time. The second run batches
candidate sequences and reuses the trained prime. The full 400-row test now completes.

Reproduce:

python research/vaccination.py        # trains the prime (63 s on the 6 GB laptop)
python research/vaccination_eval.py   # batched full evaluation -> vaccination.json

Setup

Host: 4-bit Qwen3B.
Prime ("vaccine"): LoRA r=16, one epoch on 300 SST-2 + 100 AG News rows.
Prime training: 63.2 s for 398 encoded rows; adapter 29.6 MB.
Pathogen: the existing full AG News adapter.
Merges: 0.50 and 0.75 prime weight.
Evaluation: 400 SST-2 + 400 AG News rows, batched label-token logit scoring.

Results

state	SST-2	AG News
base	61.8%	59.8%
vaccine (mixed prime)	82.2%	72.0%
AG News adapter alone	50.0%	45.8%
vaccine + AG, 0.50	67.5%	64.2%
vaccine + AG, 0.75	74.0%	71.2%
vaccine + AG, 0.75, q8	74.0%	71.2%
vaccine + AG, 0.75, q4	73.5%	71.8%
SST-2 adapter + AG, 0.75 (reference)	94.5%	67.0%

The effect is real on this test

McNemar against the AG adapter alone:

SST-2: +200 vs +104 discordant, p < 0.001.
AG News: +111 vs +9 discordant, p < 0.001.

Against the base:

SST-2: +53 vs +4, p < 0.001.

The vaccine merge is worse than the dedicated SST-2 merge on sentiment (74.0 vs 94.5,
p < 0.001), but better on AG News (71.2 vs 67.0). A 4-bit pack loses one SST-2 row
(73.5 vs 74.0, p = 1.0) and gains 0.6 pp on AG; the behavior survives quantization.

Composition matters: 25 / 50 / 100 pathogen rows

I trained primes with 25, 50 and 100 AG News rows (plus 300 SST-2), then repeated the
0.75 merge:

prime AG rows	prime SST-2	prime AG	vac+AG 0.75 SST-2	vac+AG 0.75 AG	SST-2 vs AG adapter
25	76.8%	66.8%	64.0%	68.0%	+200/-144, p=0.0059
50	86.5%	60.2%	76.0%	70.5%	+200/-96, p<0.001
100	82.2%	72.0%	74.0%	71.2%	+200/-104, p<0.001

This is not a monotonic dose. The 50-row prime is sentiment-heavy (86.5% alone) and
gives the best sentiment under the pathogen (76.0%). The 100-row prime is topic-heavy
(72.0% AG alone) and gives the best AG under the pathogen (71.2%). The 25-row prime is too
weak: it rescues sentiment against the AG adapter (p=0.006) but not against the base
(64.0 vs 61.8, p=0.33).

The composition of the prime, not just its size, sets the merge compromise.

Held-out pathogen: logit-healthy, generation-pathogenic

To test a true prime–boost, I trained AG News adapters on disjoint rows the 50-row vaccine
never saw: 300 rows (75/class) and 1,000 rows (250/class).

state	SST-2 (logit)	AG News (logit)
base	61.8%	59.8%
vaccine (50 AG)	86.5%	60.2%
held-out AG, 300 rows	86.8%	82.5%
held-out AG, 1,000 rows	84.5%	85.0%
original AG adapter, 6k rows	50.0%	45.8%
vaccine + 1k AG, 0.75	76.0%	71.0%
vaccine + 1k AG, 0.75, q4	76.5%	71.5%

Under logit scoring, the held-out adapters look healthy: good AG accuracy and preserved
sentiment. The 1k adapter is a better topic adapter than the original 6k one.

Generation tells a different story. On a balanced 400-row SST-2 check (200 positive,
200 negative), batched greedy generation:

state	generation	predicted positive / negative / unknown
base	70.8%	209 / 160 / 31
vaccine (50 AG)	88.8%	163 / 237 / 0
original AG, 6k	0.0%	0 / 0 / 400
held-out AG, 1k	55.8%	375 / 25 / 0
vaccine + AG 0.75	83.5%	160 / 240 / 0
vaccine + 1k AG 0.75	85.5%	172 / 228 / 0

The 1k adapter is benign under logit ranking (84.5%) and pathogenic under generation
(55.8%). It predicts positive on 375/400 rows. The vaccine merges repair most of that:
83.5% and 85.5%. The dedicated SST-2 adapter is still higher (93.75% generation on 400 rows
in the earlier SST-2 run), but the vaccine is a broad compromise.

The original 6k adapter is 0/400 on generation (all unknown) and 50% on logit. It is not
simply "failed": its AG generation accuracy is 62.25% vs base 2.5%, while its logit AG score
is 45.8% vs base 59.8%. It is a working generative topic adapter with bad label-token
ranking — and it erases sentiment generation completely.

The 1k pathogen is the true held-out test: the vaccine never saw its 300–1k rows, yet
the generation score goes 55.8% → 85.5% under the 0.75 merge. That is a held-out,
generation-level vaccination rescue on 400 rows, not 8.

The lesson is metric-dependent interference. A logit probe can say "healthy" while
generation says "broken." The vaccine rescues the generation behavior in both merges, now
on 100 rows, not just 8.

The strong-AG vaccination result still stands for that adapter, but the vaccine saw 50 of
its training rows; it is not evidence of pathogen-agnostic immunity.

What it looks like

The mixed prime is a broad-spectrum adapter: 82.2% SST-2 and 72.0% AG News alone.
Adding the AG pathogen at 0.75 rescues sentiment from 50.0% to 74.0% and lifts AG News from 45.8% to 71.2%. Both effects are significant on 400 rows.
The prime composition matters: 50 AG rows is the sentiment sweet spot (76.0%), 100 AG rows is the topic sweet spot (71.2%), and 25 AG rows is too weak (64.0%).
This is a multivalent vaccine, not a true prime–boost: the prime already saw 100 AG News rows. A true vaccine would be trained on a weakened or neutralized pathogen before exposure, and tested on a pathogen it never saw.
The dedicated sentiment adapter is still the best sentiment carrier; the vaccine is a compromise that carries both behaviors.

Caveats

One seed, one host, one prime size (400 rows), one epoch; no cross-validation.
Main table is logit scoring, 400 rows per task; the generation check is 400 balanced rows.
Logit scoring and generation disagree strongly on the held-out 1k adapter (84.5% logit vs 54.0% generation); both are reported.
The prime contains the pathogen's task data, so this is not evidence of pathogen-agnostic immunity.
q4/q8 were packed from the float merge; they are not independently trained.
Battery note: the first run dropped the laptop from 72% to 23%; the second run was done with the charger connected.

Next experiments

True prime–boost: train the prime on a weakened AG subset (or a sentiment-preserving topic set), then expose the full AG adapter. The 25/50/100 rows above are a first composition sweep; add 200 rows and more seeds.
The 1k held-out generation rescue is now 400 rows; add more seeds and a second host.
Re-run the original 6k recipe and inspect its loss/optimizer trajectory; is the erasure a training-scale effect, or a metric effect?
Pathogen-agnostic immunity: prime on task A and test against adapters B/C.
More seeds; the current numbers are one run.
Compare vaccine sizes (100/400/1,000 rows) and merge ratios.
Test whether the vaccine survives a second pathogen (sequential exposure).

The result is not "we invented immunity." It is a controlled demonstration that a small
mixed adapter can shift the merge compromise toward keeping both behaviors, and that the
effect survives 4-bit packing.

Behavioral thermodynamics: a LoRA behavior melts at 3.5x its RMS noise

mycute — Mon, 09 Nov 2026 23:34:41 +0000

Behavioral thermodynamics: a LoRA behavior melts at 3.5× its RMS noise

This is a physics-flavoured robustness probe, not a container. Take a fine-tuned behavior,
add Gaussian noise to every LoRA tensor, and measure the order parameter: SST-2 accuracy on
400 rows.

Reproduce:

python research/thermodynamics.py   # -> research/results/thermodynamics.json

Setup

Host: 4-bit Qwen3B.
Behavior: SST-2 sentiment adapter.
Noise: w + N(0, (σ · RMS)^2) on every LoRA A/B tensor, one seed per σ.
Adapter RMS: 0.016192.
Metric: label-token logit accuracy on 400 SST-2 test rows.

The curve

σ (× RMS)	SST-2
base (no adapter)	60.8%
0.0 (clean adapter)	94.5%
0.5	95.0%
1.0	95.0%
2.0	94.0%
3.0	85.5%
5.0	51.7%
7.0	50.0%
10.0	50.0%
15.0	47.5%
20.0	53.2%
30.0	50.0%
50.0	50.2%

The half-gain noise (accuracy halfway between base 60.8% and adapter 94.5%) is
σ = 3.47 × RMS. Above that the behavior does not decay gracefully back to the base; it
falls to chance. The adapter's signal is replaced by noise, not by the original model.

What this looks like

A plateau, then a cliff. From 0 to 2× RMS the behavior is essentially unchanged. At 3× it is damaged; at 5× it is gone.
The collapse overshoots the base. At high noise SST-2 is ~50%, below the base 60.8%. The perturbation does not restore the base; it destroys the adapter's contribution and leaves a chance-level classifier.
Quantization is far inside the plateau. The 8-bit and 4-bit adapters are exact on this task; their quantization noise is much smaller than the 3.5× RMS meltdown scale.
This is a first-order-looking curve, but one adapter and one task are not enough to claim a universality class.

Caveats

Gaussian weight noise is not quantization, pruning, or fine-tuning.
One adapter, one task, 400 rows, one seed per σ, logit scoring.
The adapter RMS is global; per-tensor noise scales could change the curve.
No finite-size scaling or susceptibility measurement yet.

Next experiments

Repeat on AG News, Dolly QA and the watermark trigger; does the meltdown scale track adapter RMS or task margins?
Noise only A, only B, or rank-wise; which part of the LoRA carries the behavior?
Measure the susceptibility (variance of the order parameter across noise seeds) near the transition.
Compare with quantization: q4/q8 are at a much lower effective temperature; how much headroom is there before a 3-bit adapter melts?
Test whether a behavioral vaccine raises the melting temperature.

The point is not that a LoRA is thermodynamic. The point is that "how much noise can this
behavior survive?" is a measurable curve, and the answer is not "linearly less accuracy per
bit."

Model immunology: negative selection finds a one-sided adapter fingerprint

mycute — Mon, 09 Nov 2026 20:02:20 +0000

Model immunology: negative selection finds a one-sided adapter fingerprint

The base model is "self". The adapter is "non-self". Negative selection keeps only probes
that the self does not recognize but the non-self does. This is a cross-disciplinary
evaluation experiment, not a container.

Reproduce:

python research/immune_probes.py   # -> research/results/immune_probes.json

Setup

Host: 4-bit Qwen3B.
Self: base model.
Non-self: SST-2 LoRA adapter.
400 SST-2 test rows; label-token logit scoring.
Base accuracy: 60.8%. Adapter accuracy: 94.5%.
Candidate detectors: rows where base is wrong and adapter is right.

The self is biased

There are 143 candidates. Every single one is a positive review.

The base over-predicts negative, so its errors are almost entirely positive rows that it
calls negative. The adapter's non-self behavior is not "sentiment in general"; it is
positive-sentiment rescue. A negative-selection immune system sees only one side of the
adapter.

The detector repertoire

Four short, lexically diverse probes were selected from the 143 candidates; four more were
held out from the same pool.

state	selected probes	held-out probes
base	0/4	0/4
SST-2 adapter	4/4	4/4
8-bit adapter	4/4	4/4
4-bit adapter	4/4	4/4

The same four probes separate base from adapter, and the response survives 8-bit and 4-bit
packing. The probe set is a small behavioral fingerprint; the held-out set shows it is not
a single cherry-picked row.

What this says

A fingerprint can be one-sided. The adapter's behavior is defined by the base's bias: it fixes positives, not negatives. An immune system trained against this self never produces a negative detector.
Quantization does not erase the detector response. 8-bit and 4-bit adapters react to the same probes.
This is not a general sentiment test. It is a minimal contrast set for this adapter against this base. A different base would produce a different repertoire.

Caveats

Probes are mined from the same 400-row SST-2 test set; held-out probes are held out from the candidate pool, not from a new paraphrase distribution.
Logit scoring ranks label tokens; it is not free-form generation.
One adapter, one task, one base.
The base's negative bias is the reason the repertoire is one-sided; that is the finding, not a bug.

Next experiments

Mutate/paraphrase the selected probes (clonal selection) and measure whether the response survives lexical changes.
Add the AG News adapter and build a cross-reactive detector matrix: which probes detect which adapter?
Test the merged symbiote and the q4/q8 variants against the same repertoire.
Use the repertoire as a lineage test: does a descendant model still carry the adapter's non-self behavior?

This is a first probe of model immunology. The interesting part is that the immune system
does not see "the adapter"; it sees the adapter's difference from a biased self.

Model symbiogenesis: the 200-row illusion and the 4-bit survivor

mycute — Mon, 09 Nov 2026 19:04:56 +0000

Model symbiogenesis: the 200-row illusion and the 4-bit survivor

Two LoRA adapters share one 4-bit Qwen3B host. One is a sentiment adapter (SST-2), the
other a topic adapter (AG News). This is not a container experiment; it asks whether two
learned behaviors coexist when their weights are merged.

The first 200-row probe looked dramatic. The full 400-row test is much more boring, which
is the useful result.

Reproduce:

python research/symbiogenesis.py   # -> research/results/symbiogenesis.json

Full 400-row logit-scored accuracy

state	SST-2	AG News
base	60.8%	59.2%
SST-2 adapter	94.5%	65.2%
AG News adapter	50.0%	45.5%
merge 0.25 (favor AG)	51.0%	52.0%
merge 0.50	90.0%	62.7%
merge 0.75 (favor SST-2)	94.5%	67.0%
merge 0.75, q8	94.5%	67.0%
merge 0.75, q4	94.5%	67.0%

The 0.75 merge matches the SST-2 adapter on SST-2 (94.5%) and improves AG News by 1.8 pp (67.0 vs 65.2). McNemar: 8 vs 1 discordant, p = 0.078 — suggestive, not significant at 0.05.
Against the base, the merge is clearly better on AG News (+7.8 pp, p < 0.001) and SST-2 (+33.7 pp, p < 0.001).
The 0.25 merge collapses both tasks. The merge landscape is non-monotonic.
The q4 and q8 versions of the 0.75 merge are bit-identical in accuracy to fp32 on both tasks; the symbiote survives 4-bit packing.

Why the first probe was wrong

The first 200 rows gave base 100% SST-2 / 80.5% AG News and merge 0.75 97.5% / 91.5%.
That looked like hybrid vigor. On the full 400-row test the base SST-2 is 60.8%, not 100%,
and the merge's AG edge is 1.8 pp, not 3.5. The first half of the test set was easier.

This is a clean example of subset optimism: the effect direction survived, the magnitude
did not. The full test is the number to quote.

Generation check (8 SST-2 rows, first rows)

state	correct	example output
base	7/8	`negative`
AG News adapter	0/8	`ulative business business`
0.5 merge	8/8	`negative`

This was an 8-row illustration from the easier first rows, not a 400-row measurement. It
shows that the AG adapter can push sentiment prompts into topic-like text, but the full
logit score for the AG adapter on SST-2 is 50%, not 0%.

What this says

Behavior is not additive. A topic adapter can damage sentiment; a merge can recover it. But on the full test the recovery is mostly the SST-2 adapter's own behavior, not a new emergent skill.
A small cross-task transfer is plausible. The 0.75 merge beats the SST-2 adapter on AG News by 1.8 pp, p=0.078. It is a signal, not a claim.
Quantization is not the bottleneck here. The 4-bit merge loses nothing on these two tasks; the representation is robust to packing.
The merge ratio matters. 0.25 is catastrophic, 0.5 is a compromise, 0.75 is the sweet spot.

Caveats

Logit scoring is not generation accuracy; it ranks the label tokens, not a free-form answer.
One seed, one host, two tasks, one merge family (linear elementwise); no TIES/DARE.
The 1.8 pp AG edge is not significant at 0.05 on 400 rows.
The q4/q8 merge was packed from the float merge; it is not an independently trained quantized model.

Next experiments

Add a third adapter (Dolly QA or the watermark trigger) and build a behavioral epistasis matrix: rescue, dominance, synthetic lethality.
Sweep more merge ratios and merge rules (TIES, DARE, task arithmetic).
Test whether the 0.75 merge's small AG edge replicates on a fresh AG News sample.
Try a behavioral vaccination experiment: can a small adapter prevent a later fine-tune from erasing a behavior?

This is a first probe of model symbiogenesis, not a result about all LoRA pairs. The
interesting part is that the behavior is not a simple additive quantity, and the first
200 rows were too kind.

4-bit Naive Bayes: symmetric fails on AG News, zero-point rescues it

mycute — Mon, 09 Nov 2026 18:44:52 +0000

4-bit Naive Bayes: symmetric fails on AG News, zero-point rescues it

Minimum bits is not one knob. A small model has two budgets: data bytes and weight
bits. I ran two experiments on the same project to see which one matters where.

1. 4-bit Naive Bayes

The SMS model is a 160-word Complement NB with int8 weights. I quantized the same weights to
int4 with a per-class scale and kept the raw integer dot product (the rule the fixed SMS
containers deploy). Five 80/20 seeds:

precision	weight bytes	mean accuracy
float	1,280 B	91.6%
int8	320 B	94.2%
int4	160 B	93.2%

On SMS, int4 keeps 93.2% at half the weight bytes. Seed 0 is 93.6% int4 vs 93.5% int8.
The full JSON still holds the 160-word vocabulary (about 1.2 KB), so the artifact shrinks by
roughly 10%, not 50%. The weight matrix itself is halved.

Then the same recipe on the 500-example, 4-class AG News coreset:

precision	weight bytes	accuracy
int8	20,596 B	78.3%
int4	10,298 B	66.5%

It collapses. Four classes and a 5,149-word vocabulary do not fit into 15 signed levels
when all log-probability weights are negative.

A zero-point (affine) int4 uses all 16 codes and recovers most of it:

AG News precision	weight bytes	accuracy
int8	20,596 B	78.3%
symmetric int4	10,298 B	66.5%
zero-point int4	10,298 B	76.8%

That is only 1.5 pp below int8 at half the weight bytes. On SMS the trade goes the other
way: symmetric int4 is 93.2%, affine int4 is 91.8%. The binary task prefers symmetric; the
four-class task needs the zero-point.

2. Same bytes, better examples?

The 130 KB AG News demo uses the first N rows per class. I compared that with random,
token coverage, TF-IDF k-means diversity and uncertainty sampling at the same example count
(400-row test):

rows/class	first	random	coverage	TF-IDF	uncertainty
50	66.0% / 52,307 B	69.8 ± 2.4%	62.5% / 75,987 B	72.8% / 54,320 B	63.0%
125	78.2% / 132,798 B	74.6 ± 2.4%	71.2% / 162,774 B	72.5%	74.8%
250	81.8% / 266,331 B	76.2 ± 0.7%	76.5% / 298,570 B	74.8%	80.0%

At 50 rows/class, TF-IDF diversity buys +6.8 pp over first-N for about 2 KB more.
At 125 and 250 rows/class, first-N is the strongest baseline. The current 130 KB
artifact is already in the regime where selection tricks do not help.

Coverage and uncertainty select longer/noisier rows and spend more bytes; they lose here.
The interesting rule is budget-dependent: diversity at very small budgets, first-N once the
budget is large enough.

What this means

Data bytes and weight bits are separate budgets. At 50 examples/class, better selection can beat more bits; at 125+, first-N is already strong.
4-bit weights are scheme-dependent. Symmetric int4 survives on binary SMS but loses 12 points on 4-class AG News; a zero-point int4 recovers AG News to within 1.5 points of int8 at half the weight bytes.
The fixed 93.5% SMS artifact is the raw-int8 rule. The per-class scale is quantization metadata, not part of inference; applying it changed the argmax and lowered the score. tests/test_sms_rule.py pins this.

What I would try next

Per-feature scales or a bias correction for multiclass int4, instead of one scale per class.
A budget-aware selector that switches to TF-IDF diversity below ~50 examples/class.
A cross-task sweep of symmetric vs zero-point int4 on more than two datasets.
Cross-validation on the selection strategies; these numbers are one 400-row test.

Everything is one machine, one teacher-labelled AG News sample. The negative result is the
point: minimum bits is not a single knob, and 4-bit is not automatically free.

Repo: https://github.com/byjameson/stream-finetune
130 KB AI: https://byjameson-mini-ai.static.hf.space/mini-ai.html
Maker (5-fold CV / calibration): https://byjameson-mini-ai.static.hf.space/maker.html
Code: research/int4_nb.py, research/int4_affine.py, research/coreset_selection.py

AI in a MIDI file: the model is the notes

mycute — Mon, 09 Nov 2026 18:34:26 +0000

AI in a MIDI file: the model is the notes

The QR put the model in a URL. The favicon put it in pixels. The PDF put it in /Keywords.
This one puts it in note-on events.

The file is 4,493 bytes. The model is the same 160-word SMS spam classifier, 1,579
bytes of int8 weights, 93.54% on 1,115 held-out SMS. It is packed into 1,053 note-on
events: each event carries two ASCII characters, one in the note byte and one in the
velocity byte. Together they are the base64url JSON of the model.

midi.html parses the MIDI track, rebuilds the string and classifies. The first 128 notes
are playable. The tune is the bytes.

The size accounting

artifact	bytes
model JSON	1,579
MIDI	4,493
overhead (events, text meta, header)	2,914
QR URL	2,158
favicon PNG	1,519
WAV	1,627
PDF	3,832
SVG	5,004

The MIDI is larger than the QR/PNG/WAV because each base64 character becomes a note event
with a status byte and a delta byte. It is still small enough to email or commit.

What is in the file

The track has five text meta events:

the title and the model description;
the measured accuracy;
a pointer to the viewer;
SHA256:<64 hex>.

Then 1,053 note-on events. The text meta events are readable in any MIDI tool; the note
events are the payload. The header is format 0, one track, 96 ticks per quarter note.

The page parses the events, collects String.fromCharCode(note, velocity) pairs, base64
decodes them and runs the int8 Complement NB. No server.

Why a MIDI?

Because it is an event file that every DAW and player already understands. It is not a
rendered audio file; it is a sequence of events. The model travels in the event stream,
and the file can still be opened and played. A MIDI renderer turns the notes into sound and
throws the note bytes away — which is exactly the container trade.

It is also a good stress test for the parser: variable-length delta times, meta events,
running status, note-on/velocity. The file is small; the code is ordinary.

What this is not

Not an audio model. The MIDI is not a voice or a song; it is data that can be played.
Not a security proof. The note events are inspectable. A MIDI file is a lossless event container, not obfuscation. The SHA-256 is integrity, not a signature.
Not robust to a DAW export. A tool that quantizes, transposes, or renders to WAV can change or drop the events. A byte copy is safe; a musical edit is not.
Not a production spam filter. One 80/20 split, 1,115 SMS, one machine.

The ladder now

SMS int8: 1,579 bytes, 93.5% on 1,115 rows;
QR URL: 2,158 bytes;
favicon: 1,519 bytes;
WAV: 1,627 bytes;
PDF: 3,832 bytes;
SVG: 5,004 bytes;
MIDI: 4,493 bytes;
130 KB AG News coreset: a 178,761-byte PDF;
1.7 MB linear model: 88.5% embedded / 91.0% fresh;
5 MB adapter on a shared 4-bit base.

The behavior is the same. The container changes what travels with it and which tool can
destroy it.

MIDI demo: https://byjameson-mini-ai.static.hf.space/midi.html
Survival matrix: https://byjameson-mini-ai.static.hf.space/survival.html
Repo: https://github.com/byjameson/stream-finetune

AI in an SVG: a 4.9 KB vector model card

mycute — Mon, 09 Nov 2026 18:03:07 +0000

AI in an SVG: a 4.9 KB vector model card

The PNG favicon put the model in pixels. The PDF put it in /Keywords. The SVG puts it in
<metadata id="tinyai">.

This is a 5,004-byte vector model card — 900×560, with a title, metrics and a short
description. The 1,579-byte int8 SMS spam classifier is base64url JSON in the metadata, not
drawn on the canvas. svg.html fetches the raw XML, parses the <metadata> element and
runs the classifier in the browser. A full SHA-256 is in data-sha256.

It scores 93.54% on 1,115 held-out SMS messages.

Why an SVG

An SVG is a document and an image at the same time. It renders as a card in a browser, but
the source is text/XML that any tool can read. The metadata travels with the picture, and
the picture can be a proper model card instead of a wall of base64.

The model is not drawn on the page. The visible card says what the model is, what it was
trained on, how it scored and what it cannot do. The <metadata> element carries the exact
weights that produce the behaviour.

Size accounting

artifact	bytes
model JSON	1,579
SVG	5,004
overhead (card text, vector layout, metadata wrapper)	3,451
QR URL	2,158
favicon PNG	1,519
WAV	1,627
PDF	3,832
130 KB AG News PDF	178,761

The SVG is the most visually explicit container: the overhead is mostly the vector card
itself, not the model. It is still under 5 KB.

How it works

The build script starts from demo/model.wav and writes the card and the metadata from the
same JSON. The viewer parses the XML with DOMParser, finds:

<metadata id="tinyai" data-sha256="...">eyJsIjpbImhhbSIsInNwYW0iXSwidiI6...</metadata>

decodes the base64url, and runs the int8 Complement NB. No server, no weights endpoint.

What this is not

Not an LLM. It is a bag-of-words classifier on English SMS.
Not a secure container. The metadata is inspectable. An SVG is a lossless text container, not obfuscation.
Not robust to every pipeline. SVG is often sanitized for security. Stripping the <metadata> element removes the model; flattening the SVG to PNG also removes it. In the survival matrix, a byte copy passes and a metadata strip fails; rasterization is not measured.
Not a production spam filter. One 80/20 split, 1,115 rows, one machine.

The survival question

SVG is the clearest example of the container trade. It is excellent for a web page or a
README: the card renders and the model is one parse away. It is poor for a platform that
sanitizes SVG or converts it to a raster image, which many do.

The measured survival matrix now has 31 checks: 18 pass, 8 fail, 5 not measured. SVG adds
one pass (byte copy) and one fail (metadata strip). The full matrix:
https://byjameson-mini-ai.static.hf.space/survival.html

The ladder so far

SMS int8 model: 1,579 bytes, 93.5% on 1,115 rows;
QR URL: 2,158 bytes;
favicon: 1,519 bytes;
WAV: 1,627 bytes;
PDF: 3,832 bytes;
SVG: 5,004 bytes;
130 KB AG News coreset: 178,761-byte PDF;
1.7 MB linear model: 88.5% embedded / 91.0% fresh;
5 MB adapter on a shared 4-bit base.

The model does not change. The container changes what travels with it and which pipeline
can destroy it.

SVG demo: https://byjameson-mini-ai.static.hf.space/svg.html
Survival matrix: https://byjameson-mini-ai.static.hf.space/survival.html
Repo: https://github.com/byjameson/stream-finetune

The 130 KB AI is a 175 KB PDF

mycute — Mon, 09 Nov 2026 17:59:01 +0000

The 130 KB AI is a 175 KB PDF

The 1,579-byte SMS model was the small end of the container ladder: a 3.7 KB PDF whose
metadata carried the int8 weights. The other end is the 130 KB AG News coreset.

This PDF is 178,761 bytes. The visible page is a model card. The model is 132,798
bytes of JSON — 500 AG News examples, 125 per class — stored as base64url in the
/TinyAIData metadata. pdf-130k.html fetches the raw PDF, parses that string, trains the
same Complement Naive Bayes in the browser and classifies. No server, no weights endpoint.

The 500-example coreset scores 78.3% on the embedded 400-row AG News test and 74.3%
on a fresh sample. The tests were measured in mini-ai.html; the PDF viewer does not claim
a live accuracy because the 400 test rows are not embedded.

The size accounting

item	bytes
training data (500 examples)	132,798
base64 in `/TinyAIData`	177,064 chars
PDF	178,761
overhead (page, base64, structure)	45,963
400-row test set	not embedded

Compare with the 1,579-byte SMS model:

artifact	model bytes	container bytes
SMS int8 PDF	1,579	3,832
AG News coreset PDF	132,798	178,761

The PDF overhead is similar in absolute terms per page; the difference is that the coreset
is 84× larger. The base64 expansion is what makes the PDF roughly 1.35× the data.

Why put the data in a PDF?

The PDF is a document people already send to each other. It has a visible page for humans
and metadata for tools. The page can say what the model is and what it cannot do; the
metadata can carry the exact examples that produce the behaviour. One file is the card and
the model.

It is also a useful stress test: a 177 KB metadata string, a 175 KB PDF, a browser parser
and a 500-row training run. It works. A PDF is lossless; the bytes that go in are the bytes
that come out.

The model is the data artifact. It is not a set of weights. There is no int8 quantization
here; the viewer trains the classifier when it loads. That is the same recipe as the 130 KB
browser AI.

What this is not

Not an LLM. It is a bag-of-words Complement NB on English news headlines.
Not a security proof. The metadata is inspectable. A PDF is a lossless container, not obfuscation. The SHA-256 detects a changed file; it is not a signature.
Not a live benchmark. The viewer classifies; the 78.3/74.3 numbers come from the 400-row test in mini-ai.html, not from a hidden test set.
Not indestructible. PDF editors can rewrite or drop metadata. The survival matrix shows what happens: a pypdf rewrite keeps it; rebuilding without Info loses it.
Teacher labels. The 500 examples were labeled by a 1.5B model; the 1.7 MB linear model on 75,000 human labels is the accuracy mode.

The representation ladder

a prompt: a few hundred bytes plus a shared base;
the SMS int8 model: 1,579 bytes, 93.5% on 1,115 rows;
the AG News coreset: 132,798 bytes, 78.3% embedded / 74.3% fresh;
the 1.7 MB linear model: 88.5% embedded / 91.0% fresh on human labels;
a 5 MB adapter on a shared 4-bit base.

The PDF does not change the model. It changes where the model can live and what travels
with it: the card, the examples and a checksum in one 175 KB file.

130 KB PDF: https://byjameson-mini-ai.static.hf.space/pdf-130k.html
130 KB browser AI: https://byjameson-mini-ai.static.hf.space/mini-ai.html
SMS PDF: https://byjameson-mini-ai.static.hf.space/pdf.html
Survival: https://byjameson-mini-ai.static.hf.space/survival.html
Repo: https://github.com/byjameson/stream-finetune

One split said 70%. Five folds said 57%.

mycute — Mon, 09 Nov 2026 17:54:46 +0000

One split said 70%. Five folds said 57%.

Tiny AI Maker trains a small text classifier in the browser. Until this week it reported one
number: accuracy on a single 80/20 split. That number is easy to read and easy to over-trust.

The AG News sample in the Maker is 200 rows, 40 per class. On one split it scores 70.0%
(95% Wilson interval 54.6–81.9%). That looks like a usable four-class classifier.

Then I ran a 5-fold cross-validation on the same 200 rows:

mean: 57.0%
fold std: 5.7 pp
range: 47.5–62.5%
gap vs the single holdout: 13.0 pp lower

The single split was optimistic. Not because the code cheated, but because a 40-row test set
is a lottery. The CV number is not a confidence interval and the folds share data, but it is
a much better description of what this 200-row sample supports.

The stable case

The SMS spam sample is the opposite. It is 100 rows, balanced, and the same recipe scores:

80/20 holdout: 90.0%
5-fold CV: 90.0% mean, 7.1 pp fold std, range 85–100%
no gap between the two estimates

Small data can be stable when the task is narrow and the classes are easy. The point is not
that CV always lowers the score; it is that you cannot know from one split.

Calibration: the softmax is not a probability

The Maker now collects out-of-fold probabilities during CV and reports two calibration
numbers:

Brier score: mean squared error of the probability vector. Lower is better.
ECE: expected calibration error over 10 confidence bins. Lower is better.

For the AG News sample: Brier 0.523, ECE 0.139. The four-class model is not calibrated.
For the SMS sample: Brier 0.144, ECE 0.066. Better, but still not a guarantee: 79 of the
100 predictions land in the 0.9–1.0 confidence bin, where the mean confidence is 98.9% and
the accuracy is 96.2%. The model is slightly overconfident even on the easy task.

Neither number is a production calibration claim. With 100–200 rows the bins are noisy.
They are there to stop the UI from presenting softmax as if it were a measured
probability.

Can temperature scaling fix it?

A single temperature is fit nested out-of-fold: for each fold, T is chosen on the other
folds' predictions, then applied to the held-out fold. This is a measurement, not a new
export format.

SMS: median T 1.5; ECE 0.066 → 0.032; Brier 0.144 → 0.143.
AG News: median T 1.5; ECE 0.139 → 0.074; Brier 0.523 → 0.503.

It helps, but it does not rescue the four-class model: Brier 0.503 is still high, and the
labels do not change with temperature. The exported int8 weights are unchanged. The Maker
page applies the median T to its on-page probability bars after CV; the exports stay
uncalibrated.

What changed in the Maker

An optional 5-fold CV button. Folds are stratified by label; k is capped by the smallest label so tiny datasets do not fail silently.
The CV uses the selected budget, so it measures the recipe you will export, not a different model.
The result shows the fold table, the mean, the fold std, the range, and the gap against the 80/20 holdout.
Calibration is measured out-of-fold with Brier/ECE and a reliability-bin table.
A nested temperature fit reports whether a single T lowers ECE/Brier; it is not applied to the exports.
The markdown model card and the PDF model card include the same numbers, or explicitly omit the CV section if you did not run it.
The exported model is still the 80/20 fit. The card says so.

What this does not prove

CV folds are not independent samples; the fold std is a stability check, not a confidence interval.
ECE is sensitive to binning and sample size. With 200 rows, treat it as a warning sign.
The AG News sample is teacher-labeled and small; the 1.7 MB accuracy mode is the model to quote for AG News, not this sample.
One machine, one seed for the 80/20 split, no hyperparameter search.

Why show the worse number?

Because the artifact is the claim. A 200-row classifier that says 70% on one split and 57%
on five folds is still useful as a demo, a teaching tool, and a container experiment. It is
not useful as a silent 70%.

The size story does not change: the 1,579-byte SMS model is still 93.5% on its 1,115-row
holdout, and the containers still fit in a QR, a favicon, a WAV and a 3.7 KB PDF. The
measurement story does: report the split you ran, the CV you ran, and the gap between them.

Maker: https://byjameson-mini-ai.static.hf.space/maker.html
Container survival: https://byjameson-mini-ai.static.hf.space/survival.html
PDF model card: https://byjameson-mini-ai.static.hf.space/pdf.html
Repo: https://github.com/byjameson/stream-finetune

If you run the AG sample in the Maker, you should get the same 70.0 / 57.0 split. If you
get something different, the seed or the sample changed; the card is generated from the run,
so it will show your numbers, not mine.