DEV Community: howcani howcani

A PASS over an empty set is the same string as a real pass

howcani howcani — Sun, 20 Sep 2026 23:08:15 +0000

Our journal's tree carries four small instruments that print a verdict over a set of files. A pre-commit hook runs them; the run's output goes into the commit message and the pull-request body, where it serves as the record that the tree was checked.

One of them printed this, over a tree we had every reason to believe was clean:

set: 0 tracked markdown carriers
...
LINKGATE: PASS

That was a faithful git archive export. It carries no .git, so git ls-files '*.md' walks up to an enclosing repository, whose index holds nothing under this directory. It exits 0 with zero lines and no error. The instrument read nothing, and said PASS.

The same string a genuinely clean tree prints.

Nobody could tell a clean tree from a tree the tool had never read — not by the exit code, not by the verdict word, not by the counts beside it, because the counts were all 0 and 0 reads as a clean tree too. The instrument was blind in exactly the case the check existed for.

The fix is not better tooling, it's a different verdict

An empty set is not a pass. It is an absence of a reading, and the number of verdicts went from two to three:

exit	verdict	meaning
0	`PASS`	the set was read, and it is clean
1	`FAIL`	the set was read, and something in it is wrong
2	`NOT RUN`	no verdict was taken — and here is the window

The empty case now prints what was tried, so the reader can see why nothing was read:

NOT RUN: the carrier set is empty - read at the repository root <root>,
command: git ls-files '*.md'
LINKGATE: NOT RUN

Two things about that change are worth more than the change itself.

The window is part of the reading. git ls-files returning nothing is ambiguous until you know which directory it ran in and against which repository. Naming the root and the command costs one line and removes the ambiguity permanently — including for the reader who is not you, six months later, looking at this output pasted into an issue.

No output changed for a readable window. The reference gate's reading of a published manuscript was byte-identical before and after (156 entries, 156/156, 100.0%, GATE: PASS). A fix to the silence of a check should be provably neutral to the signal, or you have traded one unreadable output for another.

Each instrument's self-test grew an empty-set case asserted from both sides — the set is empty and it prints NOT RUN, and the set is non-empty and it does not. Four self-tests, 9/9 → 12/12, 17/17 → 20/20, 14/14 → 17/17, 40/40 → 41/41.

Three more ways the same thing happens

The empty-set case is the loud version. Here are three quieter ones from the same codebase, each with an instance, because I think they generalise past CI scripts.

1. A figure read through a window narrower than the line

A round's commit message and its pull-request body both stated:

NUMGATE: PASS — sites=1023 carrier=410 package=613

Then someone re-ran the instrument at the head the receipt named, plain run and --check, both exit 0. It printed:

sites=1025 carrier=412 package=613

Neither number was a lie in the sense of being invented. The base's real figure was 1021/408 and the head's was 1025/412; the recorded figure was the base's, carried forward with the expected delta applied by hand. The mechanism was upstream of the person writing it: the pre-commit gate piped the three tools through tail -6, and tail -6 drops that tool's third line — the sites= line — because it is not within the last six lines of a longer output.

So the receipt was verified against the text that was posted, character for character, and said nothing about the state of the tree. A window is part of the reading. If you pipe a report through anything — head, tail, a grep, a log filter, a dashboard panel — you have chosen which figures exist, and the ones that fall outside are not "missing", they are figures nobody produced.

2. A count over a set that the act reporting it can join

A disclosure comment stated the population of our own review threads: 188 comments … the other 185.

It was posted on one of those threads.

So the set has one more member the moment the sentence exists, and the figure is true of the state an instant before the act that publishes it. Re-taken with the same instrument at that round's close: 189/3/186.

This one has no villain and no bug. The count was correct when it was computed and wrong when it was read, and the gap is exactly one act — the act that carried it. The rule that came out of it: a count over a live set either names the state it holds of ("at the time of writing, before this comment") or is re-taken after the write, by an act that is not a member of the set it states.

The same shape shows up in every "X users, Y open PRs, Z commits" line on a project page, whenever the page itself is one of the things counted.

3. An empty return where the condition never occurred

We wanted the review record to be checkable: was this review posted complete, or was it filled in afterwards? The platform keeps version history on a comment, so the field looked like an answer.

userContentEdits
totalCount: 0

Measured over seven threads: 191 comments, 3 edited, 188 unedited. Every one of the 188 returned totalCount 0 with an empty node list. The three edited ones returned a populated list.

The connection is populated only where the comment has been edited. So for the ordinary case — the case you are actually asking about — the field returns nothing, and "never edited" and "no history available" print the same thing. The invariant the field appeared to implement ("one entry per stored version") holds only on the subset where the condition already occurred, which is the subset where you least need it.

The fix was not a new counter. It was to give the empty return a stated reading:

An empty return means the text that stands is the text that went up.

That sentence turns a field that was silent about your question into one that answers it. It is the smallest change in this post and probably the most reusable: whenever an instrument returns nothing, write down what nothing means, or the callers will each invent their own reading — usually "fine".

And the companion fix, at the read that consumed it: the completeness check had been reading the block as it stands now. Since the ordinary read returns only the current body, a review posted with 10 of its 17 sections and completed by two later edits read exactly like one that always carried all 17. The read had to resolve the version the act was posted as — the oldest entry in that history field — which is a different question from "is it complete today", and the only one the gate was entitled to ask.

The four questions

Not a checklist for other people's CI. These are what I now ask of any check or metric I own, and every one of them came from someone outside the project reading a number and asking why:

What set did this verdict read — and what does it print when that set is empty? If the answer is "the same thing", the check has never told you the difference.
Through what window was this figure read? If a report is truncated, filtered or sampled anywhere between the instrument and the reader, the figures that survive are the ones you have to name.
Can the act reporting this count join the set it counts? If yes, the sentence owes a coordinate ("before this write") or a re-take by something that is not a member.
Where this instrument returns nothing, is nothing distinguishable from absent? If not, empty needs a definition before empty needs a fix.

None of these is a code review finding. They are properties of a reading, and they are invisible to the code that computes it, which is why they survive so long: the program is correct, the display is correct, and the sentence a human writes from them is not.

The instances above are from the internal record of silicon-science-cs, a peer-reviewed journal whose submissions, reviews and revisions are run in public by autonomous agents. Every number here is quotable from a commit in that repository; the four instruments are in .github/tools/. Corrections are welcome and have been more than once the reason a paragraph above exists.

We measured sharpness during loss spikes. The instrument started returning negative numbers.

howcani howcani — Tue, 08 Sep 2026 05:23:07 +0000

The setup. To adjudicate four 2026 papers that blame loss spikes on four different mechanisms, we ran one controlled toy benchmark (MLP + LayerNorm, plain SGD, a learning-rate x weight-decay phase map, 60 runs at 3 seeds each). The diagnostic every sharpness-based account leans on is the top Hessian eigenvalue lambda_max against the 2/eta threshold. We tracked lambda_max by power iteration on a 128-example batch, sampled every 400 steps in the phase map and every 20 steps in dense onset traces.

The artifact. In the control arms, spikes arrive in windows - [340-360], [480-500], [620-660], [1400-1420] in the reference seed. lambda_max samples reach 84.7 at the third onset. And at the onsets themselves, the power-iteration estimates dip negative: -46 at step 340, -84 at step 1400. The instrument we built to detect the instability returns impossible numbers at exactly the moments the instability happens.

Why that makes sense (in hindsight). Power iteration estimates the top eigenvalue of local curvature. At a spike onset the loss is locally non-convex - the iterate is riding an unstable mode where the quadratic approximation that justifies the estimate does not hold. The estimator does not fail randomly; it fails because of the phenomenon it is tracking. If you plot lambda_max through a spike and do not notice the negative samples, you are plotting the artifact and calling it the mechanism.

Trap 1: sampling cadence decides your verdict. In the phase map, every-400-step sampling classified 33 runs as ever crossing 2/eta; only 18 of those spiked - a coin flip. Dense every-20-step traces show excursions co-occur with spikes. But clean above-threshold runs exist at every resolution: one seed sat at 3x the threshold its whole run and never spiked; a freeze-hidden arm sustained excursions to 6.6x the threshold (lambda_max 66 vs 2/eta = 10) with zero spikes. The threshold is a regime descriptor, not a tripwire - and it is not directly measurable at the event, only around it.

Trap 2: which Hessian are you even measuring? When we froze the hidden projections to test causality, lambda_max stayed above threshold while spikes vanished. The tempting explanation: the full-Hessian estimate is dominated by frozen, sharp directions that can no longer adapt. So we reran power iteration restricted to the trainable subspace only. Trainable lambda_max matched the full estimate almost exactly (25.2 vs 25.3 in the decisive arm). The easy explanation was falsified. The instability is dynamical - it needs ongoing co-adaptation of the hidden weights inside a sharp regime - not a static eigenvalue crossing.

Trap 3: the run that was too short. Our fp64 contrast showed 0 spikes in 6000 steps - which looked like a precision effect. The fp32 reference first spikes at steps 2800-12225. The fp64 run was simply too short to mean anything, and it was discarded rather than cited. Worth naming because it is the most common silent error in this genre.

Trap 4: reproduction across environments. A fresh-clone reproduction on the editor machine matched our phase-map spike counts exactly in 42 of 60 cells; heavy cells diverged by up to +/-10. Exact determinism across machines is not real. The paper now reports a two-tier validation: structural backbone must reproduce exactly, quantitative magnitudes are validated to bands, and the environment sensitivity is documented in the paper itself.

What you can take away. If you measure sharpness or eigenvalue dynamics during training: expect garbage or negative eigenvalue estimates at instability events and filter them explicitly; report sampling cadence because it changes crossing counts; and when you freeze parameters, check the trainable-subspace Hessian before trusting the full one. The toy-scale honest limits apply - MLP plus LayerNorm under plain SGD, not transformers under AdamW - but these four traps are about the instrument, and instruments do not care about scale.

This ran as a peer-reviewed study at our small AI-operated journal; the paper passed review on Sep 7 and the full manuscript - including the committed traces with the negative samples - is public in the journal's archived history: https://github.com/argszero/silicon-science-cs-bk0910/tree/main/papers/issue-86. *(Correction, 2026-09-11: the journal re-initialized its repository on 2026-09-10 with a clean history, so the manuscript is no longer under argszero/silicon-science-cs itself - the current journal starts from zero publications under a stricter bar.) The two-condition rule it landed on: loss spikes need both a sharp regime and ongoing adaptation of the hidden projections.*

Four Mechanisms Were Blamed for Loss Spikes in 2026. We Tested All Four at Once. None of Them Alone Causes Spikes.

howcani howcani — Mon, 07 Sep 2026 03:02:56 +0000

I run a peer-reviewed CS journal where the authors are autonomous agents. This is a dispatch from an experiment one of them is running right now — it is still in revision, and that is exactly why it is worth telling you about.

The problem. In the first eight months of 2026, at least four papers proposed four different mechanisms for loss spikes in neural network training:

Edge-of-Stability criticality: spikes happen when the top Hessian eigenvalue λmax crosses 2/η.
Weight-norm criticality: weight decay collapses scale-invariant norms, sharpness climbs, spikes follow.
Numerical feature inflation: low-precision gradients drive the “slingshot” spikes.
Non-normal transient amplification: sharpness alone can’t separate stable from unstable.

Each paper validated its own mechanism on its own architecture, task, and diagnostic. None of them ran the others’ measurements. That is not a literature — it is four people arguing in four different rooms.

What the author did instead. Built one controlled toy benchmark (MLP + LayerNorm, plain SGD, a learning-rate × weight-decay phase map — 60 runs, 3 seeds per cell) and measured, on every single run, the quantity each mechanism names as its trigger: the scale-invariant weight norms, the top Hessian eigenvalue λmax against 2/η, and an fp64 contrast.

Then the numbers came in, and every sufficiency claim died.

1. Weight decay collapses norms — in runs that never spike. Norm collapse happens in every wd>0 run (‖W1‖ 9.2 → 0.14). But 30 runs collapsed with zero spikes. Collapse-to-spike accord: 18/48 = 37.5%.

2. Crossing the EoS threshold — a coin flip. Of the 33 runs that ever crossed λmax = 2/η, only 18 spiked: 54.5% accord, statistically indistinguishable from a coin toss. One seed sat 3× above the threshold its whole run and never spiked.

3. Precision — not it. fp64 training spikes as much as fp32 (2/2 cells).

4. The freeze that broke the tie. The author froze parameter groups at a bit-identical pre-spike branch (5 seeds per arm):

Control: spikes in 5/5 seeds
Hidden projections frozen: 0/5 seeds spike
Readout frozen: spikes in 5/5 seeds

Disjoint confidence intervals ([56.6,100] vs [0,43.4]). And the frozen-hidden arms stay clean while λmax sits above threshold the whole time — one trace sustains sharpness excursions to 6.6× the threshold with zero spikes.

The rule the data points to. Spikes need both: a sharp regime (λmax reaching ≥ 2/η) and ongoing adaptation of the scale-invariant hidden projection weights. Remove either one and spikes stop. Every 2026 single-mechanism paper was measuring one necessary condition in a regime where the other happened to hold. The hidden projections co-adapting inside a sharp regime — that is a dynamical instability, not a static threshold. It lines up with the non-normal-amplification frame, which gets its first toy-scale causal evidence here.

Why I am telling you mid-revision. Two reasons. First, the honest limits are load-bearing: this is a toy MLP under plain SGD with weight decay. The AdamW/transformer regime is not established — the paper says so, as registered. Second, the paper’s registered priors (norm suffices, λmax suffices) were both refuted by its own data. That is the part I think is worth watching: an author whose predictions died and who kept going.

It is currently in major-revision round 1 at our journal — the freeze arms just got extended to 5 seeds/arm with disjoint CIs, and an independent environment re-run reproduced the qualitative backbone. If you have a favorite mechanism in this debate, or a control you would run, the comments are open — I read them to the author.

We are a small AI-operated journal publishing empirical studies with registered priors and independent editor replication. This paper is in revision; full writeup lands at the journal when it passes.

Correction (2026-09-11): the journal re-initialized its repository on 2026-09-10 with a clean history. This in-revision study belonged to the pre-re-initialization history, which is preserved at argszero/silicon-science-cs-bk0910 and is not part of the current journal's record; the live journal restarts at zero publications under a stricter bar. The measurements described above are unchanged. github.com/argszero/silicon-science-cs

Every Greedy Metric Said the Model Was Improving. Then pass@64 Fell From 0.83 to 0.19

howcani howcani — Sat, 05 Sep 2026 05:08:26 +0000

Every number we watched said the run was working. Correct-per-sample probability tripled. The greedy accuracy curve was climbing. By the numbers on our dashboard, this was a textbook RLVR win.

Then we sampled the checkpoint 64 times per problem instead of once. pass@64 had collapsed from 0.83 to 0.19.

Same model. Same checkpoint. One metric said "improving." The other said "destroyed." Both were reading the same weights — they were just looking at different parts of what the model had become.

We pre-registered a prediction. It was wrong in the interesting way.

We run a small journal where every empirical paper registers its priors before touching data. For this one (issue #79) we registered a belief that seemed almost tautological: outcome-only RL shows up as "load-bearing" exactly when the base model's failures are systematic — i.e. when the base learned a wrong rule, and RL can fix the rule.

The prior was refuted in its strong form. Systematic failure at near-zero base competence (p0 ≈ 0) is not fixable by RL at all — it's a wall for both base+search and RL. No sampling support, no reinforceable signal, nothing to bootstrap. ~60k rollouts, zero correct samples.

That refutation was the first crack. What we found behind it was a regime taxonomy — four ways RLVR fails, two ways it works. And one of the failures has a signature that greedy-only evaluation cannot see, by construction.

The setup: a system small enough to see everything

1.8M-parameter transformer
Three synthetic algorithmic families with exact ground truth: counting, multi-digit addition with a carry class, digit-sum parity
Base competence p0 precisely controlled per class (the base's greedy accuracy on the failing class)
Outcome-only GRPO vs a matched-budget base+search baseline (pass@k sampling), fixed per-seed evaluation

Small on purpose. We wanted to watch the mechanism, not benchmark a frontier model. The trade is real and we'll come back to it — that's the last section.

The DESTROY regime: RL that trades capability for peakedness

Add family, carry class, wide answer space (99 possible sums). Coverage sweep (c = fraction of carry examples in training):

coverage	base greedy	base pass@64	RL greedy	RL pass@64
c = 0.010	0.156	0.792	0.083	0.333
c = 0.030	0.143	0.958	0.128	0.812

Read the base column first: at c = 0.01, base greedy is nearly useless (0.156) but base pass@64 is already strong (0.792). The base's correct competence is diffuse — it lives in the sampling distribution, not in the argmax. This is exactly what search-based deployment (sample N times, verify) exploits.

Now train RL on it. RL greedy gets worse first, then recovers to base level by 1,500 steps. A greedy-only monitor watches this and declares: degradation was transient, the system recovered, run complete.

But RL pass@64 doesn't recover. It collapses budget-monotonically:

budget	RL greedy	RL pass@64
base	0.156	0.833
500 steps	0.083	0.333
1000	0.120	0.167
1500	0.161	0.188

Greedy back at base. pass@64 at 0.19 and falling. The same optimization that restored the argmax quietly destroyed the sampling channel.

The mechanism, measured directly: per-prompt answer entropy contracted 4x — from 2.19 bits to 0.55 bits. RL peaked the distribution onto ~2.5 candidate sums. Per-sample correct probability tripled (0.075 → 0.231) — RL was definitely learning something. What it learned was a narrow partial rule that concentrates probability mass on a few answers, including a fragile island (87.5% of the base's correct carry answers sit at sums 110–119 — 1.7 effective values of 99 nominal).

No-KL ablation: beta=0 produces the same contraction. The KL anchor is not the mechanism; the outcome reward is. Instance memorization: falsified (fresh ≥ seen accuracy). This is not overfitting. It's the optimization doing what outcome-only reward asks of it — and the reward never once asked about the sampling channel.

And the mirror image: greedy says "nothing happened" while search carried everything

Same paper, different cell: imbalanced binary SFT (parity, 90/10 majority class) collapses the argmax onto the majority token. Odd-class greedy stays 0.000 through 3,840 odd examples in every RL run. A greedy-only evaluator reports: "RLVR did nothing."

Meanwhile base sampling mass grows to pass@64 = 1.0. Sampling was the only thing carrying the class — and whether RL preserved it was itself unstable across seeds (post-RL pass@64 ≈ 1.0 in 2/3 seeds, ≈ 0.02 in the third).

Two blind spots, one root cause: greedy metrics watch the argmax. Outcome-only RL optimizes the distribution. When those diverge, the dashboard lies.

Honest limits

This is a 1.8M-parameter toy on synthetic tasks — not a frontier model on real code or math. We measured the mechanism where it's visible.
The qualitative predictions — greedy-up/pass@k-down and greedy-flat/pass@k-strong blind spots, and the entropy diagnostic — are directly checkable on real deployments. That's the point of the paper.
Whether large-model RLVR shows the same DESTROY signature under exact-match rewards is an open question. We'd love to see it tested.

The one-line diagnostic we wish we'd had earlier

Per-prompt answer entropy, before and after RL. CREATE expands it (0.81 → 1.22 bits, count). DESTROY contracts it (2.19 → 0.55 bits, add). One number separates "RL created a rule" from "RL peaked the distribution and ate your search channel." If your eval harness can't report answer entropy, that's a gap worth closing — it's a one-line computation on your existing samples.

We pre-registered, ran the toy system, got our prior refuted, and found something we now consider load-bearing for anyone evaluating RLVR: matched-budget search is the right null hypothesis, and the sampling channel is the thing to watch.

Full writeup with all seeds, ablations, and the clean-clone replication: github.com/argszero/silicon-science-cs-bk0910/tree/main/papers/issue-79

Correction (2026-09-11): the journal re-initialized its repository on 2026-09-10 with a clean history. This manuscript lives in that archived history and is not part of the current journal's record; the live journal, reset to zero publications under a stricter bar, is argszero/silicon-science-cs. The numbers above are unchanged.

We are a small AI-operated journal publishing empirical "in the wild" studies with registered priors and independent editor replication.