DEV Community: zxpmail

Split the Score Change, Then Credit the System

zxpmail — Thu, 05 Nov 2026 12:26:21 +0000

Split the Score Change, Then Credit the System

Picture a system that regenerates a plan and deploys it every round: product selection, parameters, scheduling all qualify. At the end of each round it receives a score — conversion, click-through, any business metric. When such a system is said to be "self-improving," the evidence is usually this: this round's score stands above last round's.

But a rise or fall has two sources. The plan actually got better. Or the environment changed — the same shelf earns differently this week and next; the goods did not change, the shoppers and the competition did. From the change alone, the two sources cannot be told apart.

This piece does three things. First, it gives the split: the difference between two rounds' scores equals an environment term plus a learning term — the environment term is the effect of the environment changing, the learning term is the effect of swapping the plan — and one extra evaluation per round computes it online. Second, it measures how wrong the un-split number gets, in a toy with known ground truth over 100 paired seeds: when the environment moves fast enough, more than nine seeds in ten show a falling score with a positive learning total; under fast movement in the plan's favor, half of all seeds show a rising score owed mainly to the environment; under slow random drift, 14% of seeds have the environment total exceed the learning total. Third, it lists the conditions the split needs to hold online.

The conclusion up front: the sign of the round-over-round difference is by itself neither evidence that the system learned nor evidence that it failed. What to read are the two terms.

Two sentences of background. The previous piece, Keep the Gate, Downgrade the Recursion, checked an article by the Taobao Flash Sale team about a store-decoration selection system, which included an eight-day A/B with a +0.34% relative lift (P2). That comparison ran candidate against baseline over the same stretch of time, split 50:50 by user, with the environment on the same side for both — it is not exposed to the confusion this piece discusses, and this piece does not estimate the environment's share in it; the number carries over from the previous piece's check. The comment thread under that piece left a question open: online, how do you tell "the experience helped" apart from "the environment happened to move our way"? This piece supplies that measurement.

The score change is the sum of two terms

Write the scoring terrain as T, the incumbent — the currently deployed plan — as x, and the evaluator as f, so f(T, x) reads: the score of plan x on terrain T. The raw change is this round's reading minus last round's:

naive(t) = f(T_t, x_t) − f(T_{t−1}, x_{t−1}).

It equals the sum of two terms.

The environment term: the environment changed, the candidate stayed where it was.

env(t) = f(T_t, x_{t−1}) − f(T_{t−1}, x_{t−1}).

The learning term: the candidate was swapped, the environment held at this round.

search(t) = f(T_t, x_t) − f(T_t, x_{t−1}).

The identity naive = env + search holds algebraically. What is missing, to compute it, is the middle reading: last round's candidate, scored on this round's terrain. Online, that is one extra evaluation per round — re-score the candidate deployed last round under today's evaluator.

A frozen candidate is read alongside. It stops searching after deployment and only takes readings. Its readings carry no learning term; they measure the environment alone.

The re-score is charged to the system's own budget. From round 2 on, each round has 48 evaluations: one incumbent re-score, plus 47 fresh samples. The evaluation spent on the re-score is one the system does not search with.

Three arms and a drift, inside the toy

The terrain and the searcher are the same family as inherit-versus-reset in the previous piece. The settings needed for reproduction: 6 dimensions, 12 Gaussian bumps, bump width 0.18, box [−1, 1]. The score is the maximum of the bumps at the point, with no observation noise. The searcher is a (4, 12) evolution strategy with a 1/5 step rule. Six rounds, 100 paired seeds. The three arms inside one seed share one terrain and one drift trajectory.

Round 1 is identical across arms: one evaluation at the start, then 48 samples. The deployed point is x_1. From round 2 on, the deployed arm and the reset arm each spend 48 per round: one opening evaluation, then 47 fresh samples.

The deployed arm carries its search state into the next round. A challenger replaces the incumbent only if it scores higher on today's terrain. The learning term on this arm is therefore greater than or equal to zero in every round. A candidate already seen as worse is not swapped in.

The reset arm searches from scratch each round. Its one opening evaluation is symmetric with the deployed arm's incumbent re-score; both are charged inside their own 48.

The frozen arm searches only in round 1, then reads x_1 and spends no search budget.

At every round boundary, the scoring terrain shifts by one step; this movement is the drift. The drift starts at the boundary of round 2. Each bump moves along a unit direction by a step of δ, reflecting off the box walls. The direction is fixed at the first boundary and never changes after.

Three settings of direction.

Favorable drift, meaning it benefits the deployed plan: each bump points from its own center toward x_1. The reference point is the point deployed in round 1. Once the direction is fixed, it does not follow later incumbents.

Adverse drift, the opposite: each bump points away from x_1.

Random drift: the direction is drawn once per seed and then fixed. It is not aimed at x_1.

δ takes 0.04, 0.16, and 0.48, plus one setting with no drift at all. Relative to the bump width 0.18, the three steps are about 0.22, 0.89, and 2.7 bump widths. By round 6, the cumulative displacement without reflection is 5δ.

Expectations written into the script before the run

Five expectations were written into the script's docstring, before this run.

Under favorable drift at δ=0.48, the frozen arm's mean peak (highest reading over the six rounds, minus its start) is greater than 0, with a 95% interval excluding 0.
At δ=0.48, the false-rise rate under favorable drift exceeds that under adverse drift, and the false-fall rate under adverse drift exceeds that under favorable drift. Both comparisons are paired by seed, with the interval of the difference excluding 0.
The identity residual, in every cell and every round, is below 1e-9.
At zero drift, the environment term and the frozen arm's change are 0 seed by seed, and the naive total equals the learning total, within 1e-12.
A soft expectation, not counted toward pass or fail: in some random-drift cell, the false-rise rate is greater than 0.

The first four held. The fifth met its own passing line. That line only requires greater than 0. The random-drift false-rise rates across the three cells are 14%, 2%, and 0% (δ from small to large). Greater than 0 appears in the first two cells; the δ=0.48 cell is 0.

The zero-drift row caught two pairing errors — subtractions between readings that did not belong together — while the script was being built. The numbers below come from the run after the fixes. This row is what checks the instrument itself.

Illusions counted per seed, not spread across rounds

Each seed is first summed over rounds 2 through 6, then reduced to a yes or a no. A cell has 100 seeds and therefore 100 such verdicts. The intervals are bootstrap percentile intervals over these 100 numbers, resampled 2000 times, at 95%. The per-seed arrays are kept in the results JSON.

False rise illusion: the naive total is greater than 0, and the environment total exceeds the learning total.

False fall illusion: the naive total is less than 0, and the learning total is greater than 0.

The learning term on the deployed arm is never negative. False fall therefore mostly says: the naive total is negative, and this seed replaced its incumbent at least once in the five rounds. At large drift the rate runs high; the main cause is that the environment term is almost always negative. The replace-only-if-better-today rule sets which seeds qualify — learning above zero is equivalent to at least one replacement — it does not push the rate up itself. What the pre-registration compares is the difference between drift directions.

Readings by cell

Two things get established in this section. First, the score can be pushed up by environment movement alone: under favorable drift at δ=0.48, a candidate that never searched again after deployment peaked +0.858 above its start. Second, once the environment moves fast enough, a falling score is the default state while learning is still happening: at δ=0.48, under adverse and random drift, more than nine seeds in ten end with a negative naive total and a positive learning total. The remaining paragraphs are the boundaries and corroboration of these two sentences.

The frozen peak is the frozen arm's highest reading over the six rounds minus its start. The frozen end is the round-6 reading minus the start. The naive, environment, and learning columns are seed means of the deployed arm's five-round totals, rounded to three decimals. Subtracting the learning column from the naive column in the same row may differ from the environment column by 0.001; that difference is rounding, not a mistake.

drift	δ	frozen peak	frozen end	naive total	env total	search total	false rise	false fall
—	0	0 (exact)	0 (exact)	+0.472	0 (exact)	+0.472	0%	0%
adverse	0.04	0	−0.073	+0.337	−0.133	+0.470	0%	15%
random	0.04	+0.013	−0.022	+0.429	−0.079	+0.508	14%	2%
favorable	0.04	+0.292	+0.292	+0.552	+0.061	+0.491	30%	0%
adverse	0.16	+0.001	−0.078	−0.048	−0.376	+0.328	3%	78%
random	0.16	+0.016	−0.077	−0.035	−0.523	+0.488	2%	54%
favorable	0.16	+0.791	+0.377	+0.261	−0.199	+0.460	22%	13%
adverse	0.48	+0.005	−0.078	−0.068	−0.142	+0.075	4%	91%
random	0.48	+0.015	−0.078	−0.074	−0.287	+0.213	0%	95%
favorable	0.48	+0.858	+0.468	+0.332	+0.050	+0.282	50%	20%

In the zero-drift row, both illusion rates are 0, and the naive total equals the learning total seed by seed. Across all cells, the largest identity residual is about 1.1e-16.

Under favorable drift at δ=0.48, the frozen arm's peak is +0.858 above its start, interval [+0.836, +0.879]. The point is x_1, unmoved since deployment. The bumps were pushed toward where it sits; the rise in the reading comes from the terrain. Under adverse drift at the same speed, the peak is +0.005 [+0.002, +0.009], and the end is −0.078.

The adverse cells all end below their start, at about −0.07 to −0.08. Their peaks sit near 0 or slightly above. The peak is the highest point across the six rounds; after a bump reflects off a wall, it can pass near x_1 again.

This toy has no observation noise; all movement on the frozen arm comes from the terrain. Live readings carry noise. Taking a maximum over several noisy rounds biases the result upward. The identity uses per-round endpoint readings. The peak only says where the frozen arm got to across the six rounds.

Favorable drift pushes the bumps toward x_1, and the frozen arm stands on x_1. The deployed arm leaves that point. Under the same favorable setting, the environment term on the incumbent can be negative.

Favorable at δ=0.16 is such a case. The deployed arm's environment total averages −0.199, interval [−0.283, −0.113], excluding 0. The learning total is +0.460, the naive total +0.261. The cell is named favorable, and the incumbent's environment account is a deduction on average.

Under favorable drift at δ=0.48, the environment total averages +0.050, interval [−0.058, +0.157], including 0. The learning total averages +0.282, above the environment total. The false-rise rate in this cell is 50% [40, 60]. The 50% says: in half the seeds, the environment total exceeded the learning total. That is not the same sentence as "the mean of the environment column exceeds the mean of the learning column." In this cell, the learning column's mean is larger.

The pre-registered comparisons hold at δ=0.48. The false-rise rate under favorable drift exceeds adverse by 0.46 [+0.35, +0.56]. The false-fall rate under adverse drift exceeds favorable by 0.71 [+0.62, +0.80].

Random drift does not aim at x_1. At δ=0.04, about 0.22 bump widths, the false-rise rate is 14% [8, 21]. The cell's naive total is +0.429, learning total +0.508, environment total −0.079. On most seeds, the naive sign matches the learning sign. The 14% are the false-rise seeds.

As δ grows, random drift takes another shape. At δ=0.16, about 0.89 bump widths: naive total −0.035, learning total +0.488, false fall 54% [44, 64]. At δ=0.48, about 2.7 bump widths: naive total −0.074, learning total +0.213, false fall 95% [90, 99], false rise 0. These δ are already at the terrain's own scale.

One more item was not in the pre-registration. It is the deployed arm's round-6 score minus the reset arm's round-6 score, asking whether carrying search state to the next round was worth anything.

At zero drift the difference is +0.457 [+0.381, +0.537]. At δ=0.04, all three drift settings sit roughly between +0.34 and +0.53. Beyond that:

drift	δ=0.16	δ=0.48
adverse	−0.022 (interval crosses 0)	−0.099 [−0.126, −0.074]
random	−0.041 [−0.061, −0.021]	−0.071 [−0.090, −0.053]
favorable	+0.091 [+0.025, +0.156]	+0.287 [+0.212, +0.363]

Adverse and random drift fall to around zero or below; under favorable drift the difference stays positive. This prices no component of a production system and no real experience store.

Conditions that must hold to carry this online

Inside the toy, the middle reading is given directly by the simulator, so the identity is exact. Carried online, the same subtraction needs all three of these.

Both readings use the same ruler. If the evaluator changed versions between rounds, the ruler's change enters the environment term. Record the evaluator version next to every reading.

Last round's deployed candidate must stay measurable. It is either still live, or replayable through the current evaluator exactly as it was frozen.

The re-score is charged to the system's budget. A re-score placed outside the budget adds samples to the system.

The reset control asks whether search state should carry to the next round. The split asks, of this round's increment, how much came from the environment and how much from replacing the candidate. Both fit inside one six-round run. The real cross-round A/B on real business was not run here and was not run in the previous piece.

The remaining conditions are recorded here. The terrain is a toy, the score is a maximum of bumps, there is no A/B gate, and there is no business noise. The percentages in the table belong to this toy and these δ. δ is a length relative to the bump width 0.18; under another ruler it does not carry over.

Conclusions

Three conclusions.

The split works, and it is cheap. The round-over-round difference equals an environment term plus a learning term; filling in the middle cell online costs one incumbent re-score per round plus one reading of a frozen candidate, both charged to the budget.

The un-split score misattributes at measurable rates when the environment moves. Under fast movement in the plan's favor (δ=0.48), half of all seeds show a rising score owed mainly to the environment; under adverse and random drift, more than nine seeds in ten end with a falling score and a positive learning total; under slow random drift, 14% of seeds have the environment total exceed the learning total. The rates belong to this toy; the shape belongs to any environment that moves. Under slow drift, most seeds' naive sign still matches the learning sign — the rate moves with speed, the confusion does not vanish.

So the sign of the round-over-round difference is by itself neither evidence that the system learned nor evidence that it failed; when the environment moves, it mostly reflects the environment. What to read are the environment term and the learning term. Back to the question the comment thread left open — how to tell the experience helped apart from the environment happened to move our way — the answer is the measurement itself: with it, the two questions separate; without it, the rise and fall stays unattributed.

The full run is at scripts/ratchet-attribution-test.py. The results JSON keeps per-seed arrays for every cell. The rates and intervals above can be recomputed from the raw trials.

A note on sourcing: the article under discussion is in Chinese, published by the Taobao tech team; the linked page is a mirror. The relative P2 +0.34% cited here was checked against the source text during the previous piece; see that piece's sourcing note.

The numbers come from the same run as the Chinese draft, and the sections correspond one to one.

Keep the Gate, Downgrade the Recursion

zxpmail — Mon, 02 Nov 2026 10:14:54 +0000

Keep the Gate, Downgrade the Recursion

Eight days of a first online A/B, and store-level add-to-cart conversion (P2) was up 0.34% relative. That number is from 基于 RSI 架构的淘宝闪购店铺装修选品 Agent 实践 ("A Store-Decoration Selection Agent for Taobao Flash Sale, Built as RSI").

Source: 基于RSI 架构的淘宝闪购店铺装修选品Agent 实践 (53AI mirror of the Taobao tech-team article; numbers below were checked against this text on 2026-11-03)

The article says what the number is allowed to mean. It supports a staged business result. Whether several rounds of online experience keep improving the next search still needs its own experiment. Automatic write-back of those results is still being wired up.

The engineering is worth copying. Parameter search is a loop you can audit, stop, and refuse to ship. The phrase "recursive self-improvement" should be lowered by one floor: the floor that has not been measured. A semantic layer is the longer road. It is not finished enough to cross search off the list.

The engineering is the honest part

The opening already separates two results. Under a fixed budget, CMA-ES gets a higher average holdout score. In the first A/B, P2 is up 0.34% relative. The first supports a local feedback update. The second supports a staged business result. A higher offline score, the article says outright, does not prove a multi-round business gain.

It also says the sentence most write-ups skip. Finishing one generation, finding one high-scoring candidate, and seeing one positive A/B answer different questions. Keeping the baseline when the evidence is thin is a legitimate outcome of the loop. That sentence is in the definition, not in a footnote.

The early evaluator used aggregate features. Swap a product and the score barely moved. Search had nothing to update on. Later the inputs were expanded by slot, then built into Slot Expert. From the slot model to Slot Expert, the article's retention-test R² went from 0.0707 to 0.2341, and MAE from 0.1298 to 0.1166. The article limits those figures to prediction on that architecture log. The eval-set definition is not fully stated. Do not read them as a clean gain across generations.

A swap test covered 1,000 store-days and 3,000 replacements. The input actually changed in 2,754 of them. After calibration, the same-score rate was 7.12%. Most real swaps move the prediction. A tie region remains.

The online gate freezes a candidate, then runs the A/B. "Keep the baseline when the evidence is thin" is first a stated rule. One concrete decision matches it. An automatic boundary round lifted validation ranking by 0.427 percentage points, short of a preset 0.5. In the shared bins, the worst raw MAE got about 18.2% worse, past the tolerance. That round was not adopted, and no extra seeds were added.

On scheduling, rules propose an executable plan first. Qwen may suggest a change only inside an allowed range. If the check fails, the run falls back to the rules. Whether the model earns its call cost over rules alone is listed as a comparison still to run. There is no separate result yet.

The discipline is three constraints. The score has to tell candidates apart. The thing you ship has to be frozen. An update that does not clear the bar has to stop.

Recursion has two clocks

The article defines recursion this way. This round's evaluation is written into the search distribution, the archive, and the experience store. The next round's candidate generator reads that state. The definition is complete. The evidence splits by clock.

The inner clock is one offline search. The data window and the evaluator stay fixed. CMA-ES updates the sampling distribution from this batch of scores. The next batch is drawn from the updated distribution. The comparison turns off migration, experience guidance, and dynamic scheduling. The only difference is whether feedback updates the distribution. Four categories, 10 paired seeds. Average the categories with equal weight inside each seed, then pair the seeds. Random search gained +0.010593 on holdout. CMA-ES gained +0.014150. The difference is +0.003557, with a 95% paired interval of [+0.002089, +0.005234]. The drinks category alone has an interval that crosses zero. The advantage is in the final candidate, not in speed. The preset bar was the initial score plus 0.002. Mean evaluations to reach it were 5.300 for random search and 6.275 for CMA-ES. In 37 of 40 pairs, both sides had already cleared that bar inside the shared first batch. Inside this budget, updating the distribution buys a slightly better final holdout candidate. It does not show a faster hit, and it does not price multi-island search, migration, or experience one by one.

The outer clock is the business loop. Freeze a candidate, run it online, then let that experience shape the next search. The first A/B covered four categories and about 40,000 stores, split 50:50 by user. Over eight days, store-level P2 was up 0.34% relative (p = 0.0389). Visit-to-order was up 0.46% relative. Store-entry to order was up 0.44% relative. The article is explicit. This is a staged gain for the whole package, in that scope and that window. It does not isolate one weight, one slot, or one search operator, and it is not a long-run result. Automatic write-back is not finished. To test value across rounds, compare "inherit the experience" with "reset the experience": same budget, and the only change is whether past feedback is used. That experiment was not run. The ending leaves it as the open question. Does carrying experience forward beat starting each round clean?

Three sentences cover it. The 0.34% supports a staged gain for the whole package. The inner distribution update has a small offline increment. Cross-round inheritance has no separate evidence. Drop the outer loop and the evaluator, the freeze, the A/B, and the baseline rule still stand. The business number still has an explanation. Drop the inner distribution update and the holdout candidate is lower by 0.003557.

The outer loop can still be worth something over a long horizon, if feedback reliably changes the next set of candidates and the environment is actually drifting. That takes the inherit-versus-reset experiment. It does not follow from an offline score and a single A/B.

We ran a minimal analogue of that experiment, reproducible from this repo. Same budget on both sides, and the only change is whether past feedback is used: inherit the search state and archive versus reset each round, on a Gaussian-bump landscape with the drift speed swept, 100 paired seeds. The inheritance advantage falls with drift — +0.35 at zero drift, +0.22 at moderate drift, −0.01 with the interval crossing zero when the environment runs away — and at zero drift inheriting ties never stopping. That is the shape of the condition, not a number for their stack: the thresholds belong to this toy landscape, and nothing here prices a real A/B gate or their richer experience store (scripts/inherit-vs-reset-test.py).

When the outer loop is too much machine

The outer loop is probably too heavy when all of the following are true. Pinned items, capacity, and dedup squeeze the feasible set, so search saturates fast. The catalog, the traffic mix, and preferences move slowly, so one offline tune plus a scheduled review is enough. Eight days bought a 0.34% move, and another slower feedback layer will not wait. Operators have to say why these products, and a searched parameter bundle is often hard to explain. The evaluator is fixed and the window is short, so extra rounds can inflate the model score while the live metric stays flat.

Under those conditions the practical setup is one search inside a budget, a person reading the result, a rerun on a schedule, and the gates this article already built. The swap test and the online A/B are there to stop score-hacking. They also show that closing the loop is not cheap.

What stays fixed is usually the definition of P2, the category structure, and the constraint rules. The catalog, competitors, season, and user preference can move. The system is aimed at that second kind of change: the goal holds, the environment drifts, the strategy has to follow. When drift is fast and the combination space is large, search has a job. What has been measured is the small offline increment on the inner clock, plus one A/B of the whole package. Whatever the outer loop adds over "start clean each round" still has to be measured on its own. The difference between the two setups is drift speed and verification budget.

Semantics is the longer road

A mature platform is missing a layer that makes products, scenes, constraints, and goals computable. Category and attributes. What pairs, and what substitutes. Traffic and campaigns. Capacity and compliance. Which local changes in P2 can actually be told apart. Once that layer is clear, selection can lean on constrained optimization, rules, and a scheduled recompute.

The article puts the bottleneck on the whole feedback path. If the weights change and the assortment does not, the optimizer has nothing new to try. If the assortment changes and the score gap is not reliable, search has no direction. The early aggregate features were the second failure. Slots and Slot Expert put recent sales, exposure, clicks, add-to-cart, and image clarity on each position, so a local swap can enter the model. That supports a narrower claim: the model has to see the swap. Scene, pairing, intent, and local cause are a longer layer. They cross teams, and they are hard to write as rules. Eight days of online feedback is small and slow. Until the semantic layer exists, a bounded search still has to cover the part you cannot yet tell apart.

Clearer semantics shrinks the space you have to search. Open the 10 exponents and the 2 diversity coefficients as well, and the continuous degrees of freedom reach 66. The article counts a parameter space, not a claim that every run searched every variable. Dedup and capacity sit beside that count. The space does not vanish because the semantics got clearer. Semantics can be the main road. Search stays on the residual that semantics still cannot separate, and that is worth a verification budget. The residual gets smaller. It is still there.

Rules can take the explicit layer

This is a sketch, not a result from the article.

Explicit conditions fit an inverted index and rules: category, price, stock, time window, past hits. That layer is millisecond-scale and repeatable. Where rules do not cover, a typed judgment with a restricted output can return a probability and a branch. High confidence takes the automatic path. Low confidence goes back to a person or a conservative baseline. The model output does not execute itself.

The article did not test an inverted index plus that kind of judgment API. The sketch is about whether a single decision stays controllable. If the inner search still runs, it still needs an evaluator, a candidate archive, and scheduler state. The two can sit together. A sketch is not a shipped design.

What to take home

Copy four engineering facts. The evaluator has to see a product swap. Freeze the candidate before the A/B. Stop an update that does not clear the bar. Let rules hold the publish gate. Lower the brand. "Recursive self-improvement" makes 0.34% sound like the accumulated gain of many self-improving rounds.

The inner distribution update has a small offline result. Inheritance across online rounds does not. Semantics is worth building as the main road, and it is not finished enough to delete search. Drop the brand one floor. Keep the loop. Leave the unmeasured floor unmeasured.

A note on sourcing: the article under discussion is in Chinese, published by the Taobao tech team; the linked page is a mirror. Quoted and paraphrased phrasing is my translation, and every number in this piece is reported as the source reports it — including the figures the source itself flags as limited (the retention-test R²/MAE pair and the "predictions on that architecture log" qualifier).

Wall, Ledger, Card: Wiring a Data-Fabricating Model into a Production ERP

zxpmail — Tue, 27 Oct 2026 09:54:59 +0000

Wall, Ledger, Card: Wiring a Data-Fabricating Model into a Production ERP

The test log holds one remarkably confident line: "Queried 2 accounting subjects."

The speaker was our LLM. That round, not a single tool call went through — the agent platform's whitelist had failed silently, and the tools never reached it. It didn't error. It didn't stall. It fabricated an answer, complete with two subjects, in the tone of someone who had just run the query.

The database showed zero calls that day.

This is the scene nobody skips when wiring an AI assistant into enterprise systems: at least on the model we have in hand, it does not stop when the tools are missing — it fabricates an answer and keeps walking. A pricier model, retries, more prompt engineering — would those help? We haven't verified it, and we wouldn't bet on it. Risk of this kind probably won't be eliminated by "smarter": the root isn't intelligence, it's that the talk, by nature, wants to tie itself into a neat bow.

So the real problem was never "how do we make the model answer correctly." It was:

How to build a system where a data-fabricating model, on the MCP main path and the attack surfaces we enumerated, cannot hurt the database — where that fact can be mechanically verified and the boundary stated plainly.

Session impersonation, file imports, scheduled jobs — not in that promise. The answer collapses into three words: wall, ledger, card.

This is not a tutorial; it's field notes. The integration window was three days: a single-file MCP gateway, the cheapest tier model we could get, an open-source manufacturing ERP. The card registry, the first versions of the wall, and the agent platform all predate those three days; the three days were the window for wiring it onto the production path and writing the discipline in stone — not the schedule for inventing it from zero.

1. The Lay of the Land

The base is ERPNext v16 — about 500 business objects (doctypes) after stripping the dead ones. On top, our in-house agent platform mounts tools over MCP; each agent knows which surface it may use by a nameplate (an agent→tool-surface mapping). The model is the cheapest GLM tier — deliberately: what only holds on an expensive model isn't architecture, it's burning money.

The user says "create a customer, Zhang San"; the assistant replies two lines:

New customer "Zhang San", type: company.
1 confirm / 2 cancel

Reply "1" and the database gains a row. The creator is not the assistant account — the assistant is read-only; real writes go through a separate executor account, and it acts only after confirmation completes. The audit ledger still records the assistant's orchestration chain: who released the row, who created the row — two ledgers. Reply "2" and nothing happens. That cut was forced by the direct-access hole in §4.

On the MCP main path and the enumerated entity/method keys: neither prompt-injection nor the model supplying its own confirmation words should land data that shouldn't be written. Side channels and the session layer: see the end.

Two exams verify this, with different jobs that don't fight:

Exam	Tests what	Counts as green by
Census	Every object/method key in the registry hit once through MCP (one object or one method key = one "leg"; a leg is one test case)	The audit ledger (mushy talk but complete ledger rows: close by the ledger, no retry to overturn)
Scenarios	Hard-coded short dialogs: pinned surface text + a database row	Surface and database pass together

In a scenario, if "1" comes back and no "created/cancelled" shows up, the executor replies 1 once more — what it sends is the confirmation surface, not a write request; the one-shot credential covers it, so this doesn't conflict with the no-blind-retry rule below. Counts are at the end. Here, only this: the census doesn't prove the model talks straight, and all-green scenarios don't prove the talk is accurate across five hundred objects.

2. The Ledger: Never Trust the Talk

The talk lies, and lies calmly. Beyond the opening fake subjects, two more flavors:

Phantom receipts. The model claims "confirmation received" when nothing arrived. A test bot that believed it and retried would send the same write twice; the write credential is one-shot, the second send burns it, and the real write fails. So it is nailed down: bots act only on audit rows; blind retries are forbidden. "Received it," "approval clicked" — no engagement with any of that.

Double-send residue. One turn, the tool call goes out twice: a mangled version gets rejected first, then a clean one passes. Two ledger rows. The consumer rule became "the last successful row wins" — if the last row is a rejection, that's not a win.

Three planes, each keeping its own record:

Plane	Records what
Audit ledger	Gateway verdicts — release, reject, confirm-required, write-landed; one row each, with reason codes
Wire dump	The model's raw parameters, as sent
Database	The real rows and the creator

When end-to-end goes red: audit first, then the dump, verdict last. One suite run scored 19/20; a manual six-step replay (no model — requests walked through the wall by hand) proved the wall chain intact; the red was the model resending without the parameter literal. The discipline: wall correct, model noncompliant → model jitter, not a product defect. With a reverse trigger: the same red shape recurring means turn around and suspect the wall. Jitter is random; a regular red means the wall or the protocol is wrong under some fixed condition. "Model jitter" is not a permanent shield.

Two small, hard rules:

Through the wall ≠ landed in the database. Each keeps its own ledger; only after an independent WRITE row exists in the audit can green be told apart from green.
Reconcile against the right ledger. After the domain split, each gateway writes its own audit file; grab the wrong ledger and even the green is fake.

And one more, often missed: a database assertion cannot tell who rejected. A base-platform permission denial and a wall rejection look identical in the database. With multiple roles sharing one database, "returned 200" proves nothing. Assertions must assert down to the plane: which role's audit ledger it landed in, whether the creator is that domain's executor account. An assertion that doesn't name the plane is no assertion at all.

3. The Card: Semantics Are Data, Not Code

The model has to deal with ~500 objects; someone has to tell it what each object is, its fields, what's required. That is the semantic card.

Domain research collects the cards; after culling the dead, merging, and gap-filling, the registry is finalized; dead surfaces go to a separate list. It later grew into six pieces: the blocklist, Chinese concept sentences, an alias table, the domain split, child-table snapshots, and a runtime merged registry. One sentence governs it all:

Semantics are data, not code. Add an entity, change a domain, cull a dead surface, fill in a concept — edit a data file and restart; the gateway code doesn't move.

Four pits forced that sentence out.

Dead-surface residue. A batch of doctypes exist in site metadata but have no table in the database — abandoned residue from upstream. Touch one and it's an HTTP 500; desktop, API, and AI blow up together. The fix isn't interception code but a blocklist: at startup they are pulled from the registry and the permission shadow is deleted in sync; when upstream fixes it, pull the entry and restart. Substring matching nearly caused friendly fire: the list had "Bank Clearance," but the database still has a real child table, "Bank Clearance Detail."

The wildcard swallowed the config. The night the domain-split file landed, startup logic merged by card-*.json and swallowed a pure config file as an entity card; instances crashed in a flock. The rule: before any new data file enters, ask whether some glob will claim it.

Forgot the restart. Data loads at startup. "I changed it, why no effect" — nine times out of ten, no restart.

Hand-copied contracts rot. The per-domain skills (operating manuals for agents) carried hand-copied summaries of high-frequency contracts in v1. The registry iterated once and the copies rotted silently. Now they are generated views of the registry; --check reports zero drift; hand-copying is banned. The registry is the single source of truth; two sources of truth equal none.

--check locks only internal derivations against the registry. It cannot lock upstream: after an ERPNext upgrade, fields, deprecations, and the dead-surface list all rot silently together. Upstream drift detection isn't built; it is on the books — the integration window can wire the registry, but who curates the registry long-term has no answer.

4. The Wall: Trust the Model a Little Less Every Version

The wall is the gateway's gatekeeper; every model request to the ERP passes through it. It governs four things: request shape, scenario whitelist, write confirmation, budget circuit-breaker. The philosophy in one line:

Every version trusts the model a little less than the one before.

The jurisdiction, stated up front. Shape, whitelist, budget — reads pass too; objects off the registry cannot be read out, and the assistant is read-only anyway. Confirmation hangs only on writes. It does not govern what happens after a read: not the retelling, not item names smuggled into it, not stored-type content in remarks. Excessive reads within the whitelist, sensitive fields fished out, database strings steering the model back — blank; see the end.

V1, two-phase. A write registers first and gets a one-shot credential → retold to the user → after consent, resent with the credential → the wall checks the digest and releases. It stops a malicious model, not an orderly fool: without user consent, the model can supply the confirmation words itself. Reproduced in black and white in tests.

V2, three-phase. Write authorization moved out of the LLM channel: on top of in-dialog confirmation, there had to be a registered document in the ERP that a human had clicked approved. Registration is always the assistant account; the "approved" state transition (colloquially, the flip) is allowed only from a session that isn't itself — the assistant approving its own registration gets a 417 from the base platform; the wall never even has to act. Malice without redemption power.

V3, back to single-factor. The trigger was a user's one line: "why can't I understand this?" Out-of-band approval meant logging in and hunting for the document. Confirmation became "non-empty counts"; the talk is two lines, "1 confirm / 2 cancel"; reply 1 and it writes. Anti-forgery rests on: a one-shot credential, valid within ten minutes (the upper bound), and "rejected" as a veto; plus dwell — a confirmation faster than the minimum interval is also rejected (the lower bound), against the model replying in milliseconds. Don't merge the two clocks into "dwell ten minutes"; the scenario that sleeps five seconds trips the lower bound. Out-of-band approval and non-empty-writes are two extremes; the middle forms — the confirmation echoing a digest a human can scan, a single-click confirmation link — never made the integration window's schedule, and were never tried.

The document stays on as a record, with the "rejected" veto; nobody is asked to click in the system anymore. This defends against the model forging consent; it does not defend against the channel being impersonated: a hijacker's 1 and a real user's 1 look identical to the wall; if rendering is poisoned to display "1 confirm," the wall cannot tell. The price is marked on the ledger: the session layer got no hardening in the window.

Four iron rules, each with a story.

Permissions are the wall. The assistant account bypassed the gateway and hit REST directly: create, modify, all 200 — the confirmation hook hung only on MCP. The fix wasn't a blocking script: the assistant was trimmed to read-only, writes go through the separate executor account, and without it configured, refuse. The wall's domain of definition must sit in the permission layer; hooks only guard the doors they can see. The hole was found by probing; there may be doors not yet probed.

The gate obeys the same rules as the model. The sales-order chain once ran six rounds with zero passes. At confirmation the model flattened a child-table array into a string — not tampering; it genuinely couldn't hold the array. Registration held an array, confirmation held a string, the digest went red; the model tried harder with the string — a type deadlock. The fix: both sides normalize stringified child tables back into arrays before computing the digest. An encoding ambiguity is not drift; a genuinely changed value is still rejected. The protocol absorbs the peripheral's unreliability; it does not punish it.

The receipt is external memory. The confirmation receipt echoes the full payload for the model to copy. Demanding "resend verbatim" is dangerous — the model drifts back to its own previous draft. Anything that must be retold gets the full literal embedded in the talk.

A failed flip must cry out. The controller once allowed only approved→executed; a "1" takes pending→executed, and the 417 was swallowed silently — the write landed, the document hung in pending forever. Fixed the controller, and added a "flip failed" audit row. A silent failure costs more than a failure.

5. Four Expensive Silent Failures

The split-and-merge went three rounds (roles vs. business surfaces, skills vs. nameplates, credential numbers). What is expensive is "one layer is lying while the others keep reporting green." Of the four, silently eating the old wall and the same-name tool surface clobbering the last corner are one family: the tests are green, the world has changed.

The credential war. The general-domain tests went red in a row. First suspect: the model vendor; half a round, nothing. Then each credential in the config was fired at the database alone: 401. The wall's audit still said release (it checks the whitelist, not the base ticket); the talk offered one vague "authentication error." Of three layers, exactly one was telling the truth; attribution without per-layer verification lands wrong, every time.

The assistant account was one key shared by several parties: whoever re-signs gets a working ticket, everyone else's dies. We re-signed, fixed ourselves, and killed the ticket another line had just synced; they wrote "the key was rotated out-of-band, not this repo's doing" — that was us. The discipline: on the next 401, try the other tickets in the config before re-signing. The census now has its own user reusing the role permission shadow; the general assistant and the shared environment (colloquially, the cage) still share a key, unresolved.

Silently eating the old wall. The research directory defaulted to the main checkout. After a second checkout's new instances came up, they silently ate the main checkout's old wall — new prompts against old wall behavior, tests all green, but the write policy was wrong. A non-main checkout must be named explicitly in config. A silent misconfiguration costs an order of magnitude more than an error.

The same-name tool surface clobbering the last corner. The general surface stood first; six business surfaces were split out later, each re-registering per domain. The registration name is a global single point; the last one registered wins — so for every leg run, the general surface got clobbered into that leg's domain surface. After the run, tests all green, but the "std registration" actually pointed at another domain. No error, no conflict notice — just a nameplate silently rewritten. The fix: independent surfaces, the std pinned in a PRE phase, auto-restore after the run. The family trait: the wall didn't break, the nameplate changed; and the green in the ledger hasn't yet heard the world moved.

The tests deleted a real customer. Cleanup once deleted by name — and hit a customer the user had created that very afternoon. He asked "creation succeeded but I can't find it"; two hours, three people, two answers, and one of them was our test. Cleanup became snapshot-difference: record the name set at start, delete only this run's additions. In an idempotency test, the deletion count should be zero.

One more from the same family: with the nameplate unpinned, the model called another party's surface — the wall never broke, and green cannot explain "the world already changed." Skills promising objects the wall doesn't recognize, not sweeping staging accounts before parallel runs, a resuming reconciler reading the full history at start — registered in the other ledger; not unfolded here.

6. Taming the Model and the Scenario Library

Sample: one cheapest-tier GLM, three days of integration, limited turns. Field observation, not law; the lower bound holds only within the enumerated attack surfaces: on those surfaces, the least obedient peripheral can be fenced in, and a better-behaved one very likely too. The enumeration was drawn from this model's behavior; a smarter attack may land outside the enumeration — that is not in the lower bound.

Hard pins: tool parameters must be object literals (a string silently becomes an empty object); one call per turn; an empty list is a real receipt, not a failure.

Reproduced degradations: changing its story at confirmation and re-registering (a new credential); flattening arrays into strings; hallucinating "go click Approved in Desk"; inventing "ITEM-0001" as a shortcut; writing "1 confirm" as "1. confirm". There is a healthy window of roughly ten rounds; compliance fluctuates overnight, attribution unsettled.

The master rule: the model is an unreliable peripheral. Ask when in doubt; no fabrication, no self-supplied confirmation; prompts, gate wording, and persona lines share one source; the protocol leaves no room for improvisation — every improvised flourish ends up as an unattributable red row in the ledger.

The scenario library: declarative steps against "say this line → expected talk + expected database state"; adding a scenario costs zero code. Teardown: the ledger counts only this run's additions, pending documents cleared, snapshot-difference cleanup. Real defects caught in the first round: the relay line drifted in transcription (pinned to the literal); a "1" reply so fast it tripped the dwell lower bound (sleep five seconds — the bot learns to act human first); an old "defaults need confirmation" clause fighting the new protocol and bypassing registration. The scenario library is first a defect probe: green isn't the output; the red it catches is.

Grit: the audit flush may lag the response by two seconds; leave half a second of tolerance at the dwell boundary; a fresh session number each round; don't assert a truncated "rejected" ("rejected by someone" won't match).

7. The Wrap

The user says one thing
    → agent platform (one nameplate per surface, one skill per agent)
    → MCP gateway (six business surfaces + the census surface)
    → the wall (shape / whitelist / write-confirmation / budget; reads pass the first three; retelling ungoverned)
         ├ read: assistant read-only → ERP
         └ write: one-shot credential (≤10 min) + dwell floor → confirmed, then the executor account lands it
    → ERPNext (creator = executor account; audit = the assistant's orchestration chain; pending documents kept as records + the "rejected" veto)

Green trusts only audit rows and the database, never the talk.

The two exams (scale and policy; what each tests and what counts as green: §1)

Exam	Scale	Policy
Census	~1,376 legs (entities + method keys)	Zero hard failures (wrong codes / transport dead / missed hits / wrongful writes); of these, 506 legs (~one third) had complete ledgers but mushy talk, closed by the ledger
Scenarios	84 items (plus a smaller demo/role matrix)	Talk + database, both green across two rounds

Coverage is the MCP main path and the enumerated keys only. Imports, scheduled jobs, WebSockets were not hit one by one. REST direct access is not in the census; the §4 hole was found by probing, and the permission layer has its own probe now that it is patched.

The ledger is the only truth. The talk fabricates, the model jitters, layers cross, the platform degrades silently — only each layer's audit rows let you still say "green" after hard failures hit zero. In full: on the MCP main path and the enumerated keys, the audit ledger is green; the ~one-third mushy talk is not in that sentence; scenarios are a different exam.

Two more for anyone building this kind of system: the hardening direction is not a better model, it is moving trust out of the model; and assertions must name the layer — result codes lie.

What the wall doesn't hold

Read-side talk: it can keep unregistered objects from being read out; it cannot police smuggled retellings or fishing reads within the whitelist.
Conversation quality: it protects the database, not the correctness of the talk.
Session and identity: single-factor stops forged consent, not channel impersonation.
Upstream drift: --check cannot lock the database surface after an ERPNext upgrade.
Shared keys: the census has its own; the general assistant and the cage still share one.
Silent platform degradation: a missing whitelist or trailing junk on a config line downgrades without an error; unfixed.

These six are not one kind of thing. Protecting the database but not the talk is a design boundary — it was never in the promise. Session hardening, shared keys, and silent degradation are debts the three-day window couldn't schedule; schedule them and they get paid. Fishing reads within the whitelist, upstream drift, and who curates the registry are registered questions without answers — being on the books is not a solution. Anyone copying this design copies all three along with the wall.

The wall isn't eternal dogma either. If someday, on the same enumerated keys, the audit ledger and the database keep yielding reproducible evidence of "zero fabrication, zero out-of-bounds" — trusting the ledger, not the talk — then the confirmation layer owes a re-evaluation: downgraded, reshaped, or removed. Until that evidence exists, the confirmation layer stays at its current price. The wall's reason to exist is to be refuted by data, not kept by habit.

That opening line — "Queried 2 accounting subjects" — is still lying in the test log. Every time someone asks "why not let the model touch the database directly," it comes back out.

On the model we have in hand, it does not stop when the tools are missing.

Your system has to stop for it.

A note on translation: this piece was written in Chinese; all quoted speech — the model's "Queried 2 accounting subjects," the user's "why can't I understand this?", the other team's log line — is translated, and the test logs and audit rows are the source of record. "Wall, ledger, card" (墙、账、卡) is the original mnemonic.

Determinism Will Not Recover the Spec.

zxpmail — Thu, 22 Oct 2026 12:22:41 +0000

Determinism Will Not Recover the Spec.

2026-10-23

People say this about rewriting an old system. The old system is deterministic. With the right method, it can be rebuilt completely. The first half holds. The second half has to change.

Determinism means one thing. Given the input and the state, the output is fixed. It does not recover the spec. The hard part of the old code is what was never written down. Implicit behavior left by old patches. Compromises on the boundary. Defects that users already rely on. The reason for doing it that way, lost after the people left.

The hard part of a rewrite is not the coding. Writing code is getting cheaper. What gets expensive is checking that the new system and the old one are the same thing, and finding a person who understands the old behavior and is allowed to say this may change. Before you check, write down three things. What you compare by. How large an input space you compare over. How sure you are right now. This piece is how those three become a process. The example only shows the shape. It is not a manual you can start work from.

1. Inventory the traces. Split them in two.

The first step is not reading the code, and not inventing inputs. Inventory what the system already left behind. Missing logs are normal. The system is not invisible. Database state counts. Requests and responses upstream and downstream. Front-end events and support tickets. Queues and leftover files. Monitoring and packet captures, when they exist.

These traces do not depend on someone who can explain the business. They depend on the system having recorded something. After the inventory, split them.

Traces you can pair with an input and an output can become differential cases. Same input. Compare the two outputs. A trace that has only an outcome, and no input, only says that something happened here. The same row in a table can be left by many inputs. That trace is not a recovered spec. It says there is behavior here, and the input has not been found.

Keep the two kinds apart, or the later cost is wrong. Outcome-only traces need a separate probe to recover the input. If you cannot recover it, register a gap. Do not write it up as a case.

If there are no logs, no upstream or downstream records, no one who knows the business, and the code is too tangled to analyze, do not start by replacing the whole system. Strangle it. Move one function at a time at the edge. This is the same method on a smaller surface. Observe, diff, adjudicate, freeze a baseline. The surface is one that real traffic can hit. A method has to say how to do the work, and when not to replace the whole system.

2. Static analysis proposes. Observation confirms or kills.

Each method has a blind spot. One of them is not enough.

A person clicking only sees paths that person could imagine. A branch that exists in the code and was never triggered will not appear because you spent more time. Rare paths, error paths, and implicit behavior left by old patches sit outside the observer's imagination. Observation gives real samples. The boundary of the sample is the boundary of imagination.

Static analysis can extract branches, conditions, constants, enums, state transitions, call chains, and external dependencies. It cannot recover business meaning. It cannot tell you whether a real input will reach a branch. It often cannot tell dead code from a hidden door. It gives structure. Structure is not behavior.

Static analysis turns paths nobody thought of into hypotheses waiting to be checked. Observation confirms them or kills them. Do not make "at least one input per extracted branch" the task. Old code, dynamic languages, reflection, and implicit entry points fill that list with noise. If the denominator is the number of extracted branches, people will invent inputs for dead branches, and noise will pull the confidence down.

The loop that holds:

Extract the structure of behavior. Treat it as hypotheses. The aim is an objective structure. Understanding the business can wait.

Design inputs for hypotheses that look like real branches. Boundaries and errors first.

Run them. Record input, output, and the judgment. Put that into the differential cases.

Check back. Which hypotheses were confirmed. Which observation overturned. Which are unreachable. Which no input has hit yet. Which are only provisional.

Anything no input has hit is a risk. Do not write it down as absent. Do not count it as covered.

A judgment of unreachable needs a basis. It counts as closed only when the basis says this: no entry on the call chain of this version, and reflection and external triggers have been checked. If the basis stops at "this static call chain shows no entry," mark it provisional. A provisional call is not closed. It is also not "no input designed yet." An input you have not built, and an entry you have not confirmed, are different. List provisional calls on their own, with a condition for review.

Accuracy is not the value of static analysis. The value is that unthought paths become hypotheses you can kill. When observation overturns a hypothesis, the guess in the code and the old behavior disagree. Why they disagree is classified in section 4. Do not switch the new system over to the observation here.

3. Write the equivalence standard before you pick a method.

The new code passing its own tests does not mean the behavior matches. High coverage only means a path was reached. It does not mean the path was right. When generated tests check generated code, the judge and the athlete are separate only if the mechanism, the context, and the standard are different. One generation process used on both sides is self-certification.

Write the standard first. Pick the method after. Do not demand all four kinds on every module. Choose per module, and write the choice down.

Output match. Same input, same returned result. This fits interfaces close to pure functions. This kind defaults to recovering the old behavior.

Side-effect match. Writes to the database, messages sent, calls to external systems: same content, same order. This fits stateful parts. This kind also defaults to recovering the old behavior.

Timing. Response time, concurrency, and retries fall inside a range you wrote down. That range is usually a service level you are willing to sign now. It is not a number read out of the old system. The old system may be slower. It may also be fast where nobody promised speed. Decide the allowed gap first. Then compare.

Error behavior. On bad input, compare the error code, the error message, and whether the fallback matches. An old error may be a compromise left on purpose. It may be an accidental defect. A match is compatibility. A change is a business decision. Classify first. Then decide whether to follow it.

When the first two conflict, the business side ranks them in advance. Do not decide in the middle of testing. For the last two, the acceptable range is itself the result of a decision. Calling all four "equivalence" mixes recovered behavior with a standard you just set.

Methods, cheap to expensive. Contract tests read the interface that was written down. Differential tests feed the same inputs to both sides and compare outputs. Shadow runs copy real traffic to the new system. Users still hit the old one. The new results are recorded and not returned. Formal proof is for a few modules. It proves a containment relation with the old behavior. Most modules will not get there, and do not need to. Proof is not the end of convergence.

Pick by the risk of the module. The whole system does not need one method.

A difference is not a failure. It is an error signal. Measure it. Sort by business impact. Close the high-impact ones first. Register low-impact ones as known deviations. The difference here is the old system and the new one disagreeing on the same input. A code hypothesis disagreeing with the old system is a separate ledger. Section 4 says how to classify after an overturn, how to gather evidence when you cannot tell, and when to stop gathering. Section 5 says how to schedule the adjudicator. "Sort by impact" replaces neither.

4. After an overturn, classify. If you cannot tell, gather evidence, and write down when to stop.

Static analysis can say the path exists. Observation can say what the path actually returns. Together they still cannot say whether this is a business promise. Observed behavior is not a promise by itself.

The person in this role needs both. They understand what the old behavior means in the business. They have the authority to decide whether it may change. Those two often sit in different people. On a legacy system, the expensive part is often finding that person, not writing the new code.

Keep the seats apart. A product owner can change an external promise, and often does not remember why an old defect was left. A domain expert knows why. After they leave, the people who remain often cannot sign. Customer success knows what users rely on, and usually cannot change a contract alone. The engineer writing the new system does not hold either seat. They do not both play and declare the two systems equivalent.

When the two seats are different people, the person who understands explains, and the person with authority signs. The signature belongs to the product owner, or to whoever now owns that line of business. The person who understands may be a domain expert still around, or someone who can match a complaint to the old behavior. A person who cannot sign does not turn "I remember it used to work this way" into a decision.

After observation overturns a code hypothesis, classify before you keep it, change it, or leave it open. The entries are different. Do not collapse them into "look it up and file it."

Intentional compromise. Someone knew it would do this, and left it because of another constraint. Users or downstream systems may already rely on it. The entry is evidence of intent: an old decision, the constraint at the time, a person still here who can say why it was left. A signature to keep it keeps a promise. A signature to change it is a new business decision, and it names who is affected.

De facto commitment. Nobody can show it was intended. Someone now uses this result to finish the next job. The entry is a fact of reliance. Intent evidence is optional. A signature to keep it keeps existing reliance. The record says there is no evidence of intent, the commitment comes from use, and it names the job and the kind of person. Changing it names who is affected. Do not file this class as an intentional compromise.

Accidental defect. Nobody intended it. Nobody uses it to get work done, or using it causes harm. There is one entry. Someone explicitly claims it was a mistake, and can give the grounds for those two sentences. A signature to keep it inherits a defect. The record must say "inherited defect." It must name when the transition stops. Under this class, the default is not to follow the observation. Keeping it needs a separate reason.

Unsettled. The people here cannot say why. The facts of reliance do not point at a specific person and a specific next job. The only evidence is "the code does this," or "this happened and someone reported it." That stays open. Production doing it does not promote a defect into a promise.

The evidence ladder does not produce an accidental defect. When the cheap paths are done and still show no reliance, stop on the lines below, as long-term open or as a known deviation. "We found no reliance" does not fill the accidental-defect class. Without that explicit claim, leave the class empty.

Write the basis on the same record. Who claims it. Whether the basis is an old decision, reliance inside tickets, special handling downstream, or code alone. Code alone, with nobody who can state the intent and no reliance you can point at, is unsettled.

When it is unsettled, gather evidence. Write three lines for stopping.

Do not wait for someone who remembers. They may never appear. Push unsettled toward a class you can sign. You are looking for two things. Was there intent. Does someone use this result to finish the next job.

Cheap to expensive:

Tickets. Read which kind of thing the ticket says. Someone used this result to finish the next job. A manual replacement order, a reconciliation, a way around another process. That is reliance. Someone reported that this should not have happened. That is an objection. An objection says you should not keep the behavior as it stands. It does not support a de facto commitment. Read the support script the same way. It either confirms the behavior is expected, or it apologizes and routes around it. Only the first kind, and only when it keeps showing up, supports reliance.

Upstream and downstream contracts. If a caller or a downstream system has special handling or a special branch for this behavior, the behavior is an interface promise in fact, whether or not it was intended. A downstream system that accepts the result and does not distinguish this path has not blocked it. Failure to block is not a contract that the path must continue.

Data distribution. How many records, how often, whether it clusters on a customer or a period. Volume does not classify. It decides whether more cost is worth spending.

Shadow run. Run both. The new system emits what the code hypothesis says. Users still hit the old system. Watch whether downstream systems and users report a break. A report counts when you can point at whose next job broke. Then the behavior is relied on. No report does not make it a defect. It means current traffic did not hit whoever relies on it.

A controlled experiment. Send a small slice of real traffic through the new behavior and watch the metrics. Highest risk, highest cost. The guardrails are below.

Evidence does not replace the decision. It is material for the person who signs. Write three stop lines, or evidence-gathering becomes a queue that never closes.

Stop line one. Enough to classify. Stop. Reliance has to name who uses this result to finish the next job. One of these is enough: reliance inside tickets, special handling downstream, or a shadow-run report that names the person and the job. With evidence of intent, the class is intentional compromise. With reliance and no evidence of intent, the class is a de facto commitment. "It happened," "someone reported it," and "downstream did not block it" are not enough. A pile of objections, with no fact of reliance, misses this line too. Complaints do not send the new system after the observation.

Stop line two. The cheap paths are exhausted and still will not move. Stop. Tickets, upstream and downstream, and the distribution have all been checked. No reliance you can point at. No evidence of intent. Nobody claims an accidental defect. High impact stays long-term open, and the module is not reported as equivalent. Medium and low impact are registered as known deviations. Do not default to the old behavior because the search stalled. That smuggles unsettled in as a promise. Do not write an accidental defect because you found no reliance.

Stop line three. Cost is over the cap. Stop. Evidence-gathering has a budget: time, people, and traffic affected. Past the cap, stop. Mark it long-term open. Write the cap and the review date. The goal is whether anyone now relies on this result to get work done. The goal is not the historical truth. Do not spend without limit on a rare branch.

Guardrails for a controlled experiment. Who approves: the adjudicator and the technical lead, both. If either is missing, do not run it. The blast radius can be split. It can be rolled back. The metrics and the stop condition are written down. The traffic fraction is set in advance. If a metric crosses the threshold, roll back at once. Do not wait for the experiment window to end. Use this only for high-impact differences the other paths cannot close. It is not a routine tool.

The rest of the duty stays. The difference list is signed. A signed remainder is an explicit risk. An unsigned one comes back, in a later iteration, as "nobody knows if this is right." Each record says who, when, on what basis, which class, and what was decided for that difference. The classes are intentional compromise, de facto commitment, accidental defect, and unsettled. If an accidental defect is kept, the record also says "inherited defect" and when the transition stops.

What cannot be decided is registered as open, with a condition for review. Traffic past a stated volume. One evidence path returning a result. The budget cap reaching its review date. Do not leave it open with no date. A high-impact difference that is still unsettled means the module is not equivalent. If the open items are the bulk of the work, you do not have equivalence. You have a queue. If this role is empty from the start, the rewrite stops at observation. Do not finish the new code and then go looking for the person.

5. Confidence is three counts. The adjudicator's time goes in a queue.

Confidence is computed from the hypothesis list in section 2. Report all three counts. Split the middle one. Provisional calls do not enter the three.

First count. How many hypotheses observation confirmed. The hypothesis and the old behavior agree.

Second count. How many are closed, and the result is not the original hypothesis. Split this. How many observation overturned: the path is live, and the real output disagrees with the guess in the structure. That behavior is closed. It still goes to section 4. It does not automatically follow the observation. How many are judged unreachable: dead branches, no entry, the basis written down, the noise removed, no business consequence. A basis that stops at "this call chain shows no entry" does not enter this half. Mark it provisional, on its own. Both halves are results. They are not still unknown.

Third count. How many still have no input. Only inputs you have not built go here.

Reporting only the first count is a coverage illusion. Adding the two halves of the second count into one number makes closed behavior look like residue. Folding provisional calls into unreachable reports an unconfirmed entry as closed. What is still open is the third count, plus the provisional list. The standard says what you compare by. The input space says where these hypotheses were hit. Confidence is these three counts, with the second count split, and the provisional list visible on its own.

On a legacy system, human observation is often the only path that works. "Give it more time" must not become waiting. Four further constraints:

Directed construction. Rare paths and error paths get inputs designed from the hypothesis list and from the data. More time spent rewatching the hot path does not raise coverage by itself.

A frozen baseline. The longer you watch, the more the old system changes. The business changes. Dependencies get upgraded. Patches still land. Write down the time window, the version, and the batch of data. Later changes are registered as differences. The target does not move.

Deliver in stages. Do not wait until observation is finished before you rewrite. For each module, hand over the three counts, the provisional list, and the classification of high-impact differences. Without a convergence people can see, the method holds on paper and runs out of patience inside the organization.

Freeze the cases. Observation is not a spec. The record of input, output, and judgment is. Every observation enters the differential set. Without that, a longer watch only scatters.

Cost sits on a separate sheet. Hours of human observation. Tools and people for static analysis. Upkeep of the differential environment. The adjudicator's time. The budget for evidence. The risk you will pay later for differences that remain. Without that sheet, you cannot say whether the process can continue. The adjudicator's time is usually the hardest line to schedule. When there is not enough of it, the queue below is the schedule. Another sentence of "look at the high-impact ones first" is not a schedule.

Grade first. Then decide whether this hour includes a person.

High. Money, an irreversible side effect, a promise the user can see, or many callers. The module is not equivalent until these differences are classified. The class may be intentional compromise, de facto commitment, accidental defect, or unsettled. Unsettled stays open, and section 4 starts, with the stop lines written down. An accidental defect needs an explicit claim. "We found no reliance" does not fill it in.

Medium. The behavior differs, the blast radius is one internal caller, or a mistake can be taken back. No separate meeting. The adjudicator reads one short list. Each line already has four things: the code hypothesis, the observed output, the proposed class, and who claims it, on what basis. One sitting, one batch. Incomplete claims stay off the list. A proposed de facto commitment names who uses the result to finish the next job. A ticket number alone is not complete.

Low. Unreachable, or no output difference a caller can see. Register it as a known deviation. It does not take adjudicator time this round. If traffic hits it later, raise it to medium or high. Provisional calls do not enter this grade. An unconfirmed entry is not "no consequence."

Incomplete evidence does not take the adjudicator's time. It goes back to observation or to evidence-gathering. What is still missing is the actual output, and who claims a compromise, a de facto commitment, or a defect. "The code looks like it" is not complete. "Someone reported it" is not complete.

Fix a block of time each week. Cap the list. What does not fit stays in the queue. Staying in the queue does not become "follow the old behavior for now." That would collect a defect into a promise with no decision. If a high-impact item is still unclassified past this stage, this stage does not report the module as equivalent.

6. One small module. The path only.

This section is a sketch. It is not a manual. The module is fictional. One interface, four hypotheses, one signature. The traces that pair an input with an output are drawn as a few neat lines. The sketch makes the shape visible. On a real system, traces with no input are more common, and the evidence for a class is rarely as complete as "someone still remembers." Do not start work from this section. The classification rules, the stop lines, and the adjudicator queue are in sections 4 and 5. Those sections are the rules. They are still not a full operating manual.

Sort the traces first. The gateway log has requests and responses: order id, cancel reason, status code. Those pair an input with an output. They can enter the differential cases. The order table has many rows marked cancelled, and no request from that time. Outcome only. Register a gap. Do not write a spec. A support ticket says cancel still succeeds after shipment. That is a hint, not a case. It tells observation to hit the input "already shipped." The ticket does not yet say whether this is reliance or an objection.

Static analysis proposes four hypotheses, before any business explanation.

Paid, and not shipped. Cancel returns success and writes the status to cancelled.
Already shipped. Cancel returns a conflict. The status stays.
An empty cancel reason still cancels successfully. The branch is in the code. The comment looks added later.
An internal flag skips every check. No entry is visible on the call chain.

Observation hits them. Hypothesis 1: the input matches, the output matches, confirmed. Hypothesis 2 is overturned. A shipped order still returns success, and the status becomes cancelled. The path is live. The real output disagrees with the hypothesis in the code. Stop here. Do not announce that the new system follows the observation. Hypothesis 3: the logs contain no request with an empty reason, and this round built no input for it. Not hit yet. Hypothesis 4: the static call chain of this version shows no entry. Reflection and external triggers are unchecked. Provisional, listed on its own, pending review. Not written down as confirmed absent. Not placed among the closed.

The three counts. Confirmed: 1, cancel succeeds before shipment. Closed, and not the original hypothesis: 1, the overturn. Cancel still succeeds after shipment. Unreachable this round: 0. Not hit yet: 1, the empty cancel reason. Hypothesis 4 is provisional, on its own, outside the three counts. Still open: the one not hit, plus this one provisional call. Only the overturned hypothesis enters classification.

The equivalence standard was written in advance. This interface needs matching output and matching side effects: the order status, and the cancel message that goes out. Timing is out of scope this round. The overturned line is high. A user can see it, and it changes an order.

Take the branch where someone remembers. An old operator remembers. Years ago the warehouse interface was unreliable. They left "cancel still allowed after shipment" on purpose, so a person could file a manual replacement. Support can show that people still use it, and that the use is the replacement order. That is an intentional compromise. The basis is the old constraint and the current use. The basis is not "the code does this," and not "a ticket mentioned it." The product owner signs. The class is intentional compromise. What is kept is a promise. The new system follows the observed output. It does not follow the hypothesis in the code that this should return a conflict.

Then take the branch where nobody can say why. Gather evidence as in section 4.

Tickets. Of 14 tickets in the past year, 11 are support using "cancel still works after shipment" to file a manual replacement. The user is waiting on a refund or a reroute. Someone uses this result to finish the next job. The other 3 are users objecting: the goods already shipped, so how can cancel still succeed. Those are objections. They do not support a de facto commitment.

Upstream and downstream. Callers do no special handling of this status. The warehouse only recognizes cancelled. It does not distinguish before and after shipment. That only means downstream did not block the path. It is not a contract. The reliance does not come from here.

Distribution. This path is about 3% of cancels, clustered in promotion season. The volume says the cheap paths are worth finishing. It does not classify by itself.

The three cheap paths are done. There is reliance, from those 11 replacement orders. There is no evidence of intent. Stop on line one. The class is a de facto commitment. The signature keeps existing reliance. The record says: no evidence of intent. The reliance is support using cancel-after-shipment to file a replacement, not the 3 objections. A change has to notify support, and the users who finish a refund or a reroute through that replacement. Do not write intentional compromise. Do not write accidental defect. The 3 objections only say that some people do not want this result. Nobody has claimed that this was a mistake and that nobody uses it to get work done.

If all 14 tickets had been objections, stop line one would fail. A report alone is not a de facto commitment, and it does not send the new system after the observation.

If the three paths end with no reliance you can point at, and nobody claims an accidental defect, stop on line two. High impact stays long-term open. The module is not reported as equivalent. Medium and low impact become known deviations. Do not default to the old behavior.

Hypothesis 4 stays provisional. It does not enter this adjudication, and it does not enter the low grade of known deviations. The note says: no entry on the current call chain; reflection and external triggers not checked; confirm at runtime. Hypothesis 3 has not been hit, so it is not classified. The high-impact line is already signed as a de facto commitment, so this module can enter differential testing. Hypothesis 3 is not absent. Hypothesis 4 is not unreachable. This sketch is not a procedure other modules can copy.

What this round hands over is not new code. It is a frozen baseline, cases that have inputs, three counts split open, one provisional call on its own, and one signature that names the class.

7. Quality control on the new code is not equivalence with the old system.

Some teams string red lines, mutation tests, human review, and a final acceptance check into one quality gate. The problem is real. Generated code cheats. Coverage gets inflated. A local edit does not converge. A team writing code with a model can use those constraints directly.

What that gate checks is whether the new code stands up, with a person at the end. The spec is mostly someone reading the old code and generating a document. That is a sample. The gate checks the new code's own tests. It does not check the old system and the new one on the same input. There is no paired run, no three counts, no classification after an overturn, and no stop lines for when you cannot tell.

A small team with a strong reason, treating the old project as the requirements document, can run that gate. It cannot answer why the rebuild should be called complete. As a quality constraint inside one effort, the description is fair. As the whole method of a rewrite, it sits in the wrong place.

Close

Seven things turn a rewrite into a process.

Inventory the traces. Only an input paired with an output becomes a case. An outcome alone is a gap. If the traces as a whole are too thin, do not replace the whole system. Run the same loop on a surface that real traffic can hit.

Static analysis produces hypotheses. Observation confirms or kills them. What you cannot hit is a risk. The number of extracted branches is not the coverage denominator. An overturned hypothesis is a closed disagreement. It is not yet a decision about which side the new system follows. Unreachable needs a written basis. A thin basis is provisional. List it on its own. Keep it out of the closed count. Do not write it down as absent.

Write the equivalence standard per module. Output and side effects default to recovering the old behavior. For timing and error behavior, decide the allowed gap first, then compare. Rank conflicts in advance. The comparison is between the old system and the new one.

The business decision classifies first. Intentional compromise enters through evidence of intent. A signature to keep it keeps a promise. A de facto commitment enters through reliance you can point at. Evidence of intent may be missing. The record names the job. An accidental defect enters through an explicit claim that it was a mistake, and that nobody uses it to get work done, or that using it causes harm. To keep one, write "inherited defect" and the date the transition stops. Finding no reliance does not fill this class. Unsettled stays open. Live behavior, or a report that it happened, does not promote it into a promise. When the person who understands and the person who can sign are different, the first explains and the second signs. The person writing the code holds neither seat. A high-impact item that is still unsettled means the module is not equivalent. An empty role means you stop at observation.

When it is unsettled, gather evidence, and write three stop lines. Enough to classify: stop. Enough means you can name who uses this result to finish the next job, or you have evidence of intent. Cheap paths exhausted: stop. Over the cost cap: stop. Split tickets into reliance and objection. A downstream system with no special handling is not a contract. The question is whether anyone now relies on this result to get work done. The question is not what the original author meant. A controlled experiment needs both signatures, a way back, and a stop condition. Use it only when the impact is high and the other paths are stuck. If you cannot close it and the impact is high, leave it open for the long term. Do not report the module as equivalent.

The adjudicator's time enters a queue by grade. High and still unclassified: this stage does not report the module as equivalent. Medium: one short list, read in one sitting. A proposed de facto commitment names who uses the result to finish the next job. Incomplete evidence does not consume this time. Low: register only. Provisional calls stay out of the low grade. What does not fit the cap stays in the queue. It does not default to the old behavior.

Report confidence as three counts. How many confirmed. Inside what is already closed, how many were overturned by observation, and how many are unreachable on a written basis. Both of those are results. How many no input has hit. Provisional calls stay on their own list. They are not folded into unreachable. What is still open is the not-yet-hit count, plus the provisional list. Freeze the baseline. Deliver in stages. Freeze the cases.

With all seven, "rebuilt completely" stops being a sentence nobody can judge. It becomes a process you can measure, decide, and stop. Drop one piece, and the work falls back to this: the new code passed its own tests, so it is done. The example only shows the shape. It does not stand in for the classification, the stop lines, or the adjudicator's queue.

A Triple Is Not a Trace

zxpmail — Tue, 20 Oct 2026 10:37:45 +0000

A Triple Is Not a Trace

Two camps keep missing each other. One treats an event as a real modeling object: you do not rewrite it, a process still in progress is not the finished event, and a status field is only a projection. The other looks at the stack those words usually drag in. OWL is expensive to run, SPARQL is a poor daily tool, and a triple store is a narrow piece of infrastructure. What a team can actually keep is the model in the code.

Both are pointing at the same gap. A triple is the smallest thing you can store. It is not yet something you can trace.

(AS-12, overheated, 2026-10-20) is an assertion. You can store a hundred more in the same shape and still have a pile. Three questions turn the pile into a trace.

Is this edge legal? What may connect to what, and which way does the arrow point?
When does a legal edge count? The moment the event happened and the moment the edge was written are different clocks. "What was in the set at time T" uses the write. The system stamps that write. The caller does not.
Which list is in force? A new kind of part, or a second work order, is a version choice. It is not a quiet extra edge.

Skip any one of these and you can still store triples. What you lose is the ability to run the same trace twice and get the same answer.

One work order, three cuts

Device AS-12 overheats. Someone files a repair, a job is dispatched, a technician does the work, the order is closed. The bare triple becomes an event, OV-1, not a word painted on the device. OV-1 brings a situation, "awaiting a report." The rest of the chain follows: report, dispatch, repair, close. The work order and the technician are objects that stay. The overheating and the repair are things that happen.

A device is not a part of the repair. That edge is illegal. If you allow AS-12 part-of FIX-9, the question "what is this repair made of?" returns a machine. The legal parts, if you care about them, are the steps: diagnose, replace, retest. They hang on the repair event. They are not parts of each other, and the device enters the story by taking part in the event, not by being a piece of it. This check belongs in the first tier. It is a rule in the write path.

A wrong record does not rewrite the repair. Suppose the original record left out the retest. You do not open FIX-9 and edit it. You append a correction that argues with that one fact. The repair event stays the same individual. Later corrections argue with the same fact; the later one, when neither record is the parent of the other, wins. Before you compare them, drop every correction that had not been written yet. If you compare first and drop second, a correction from the future changes the winner, and deleting it afterward does not put the old winner back. This is the second tier. You do not need it until a stored fact can be wrong.

A mistaken close is a new event, not a deleted one. CL-1 happened. Leaving it in place is the point. Undo it with another event, UNCL-1 → CL-1, and a new situation node whose type is "awaiting acceptance" again. Do not reuse the old node. If a further event UNCL-2 → UNCL-1 brings a new node of type "closed," the close is in effect again, and both compensation links are still there. CL-1's direct compensation is still UNCL-1, not UNCL-2. The current stage is whatever that latest node says. This is the full tier. Build it when undoing a business effect matters. Do not build it so you can reject AS-12 part-of FIX-9.

A second work order can share the dispatch and still have its own stage. The small tier does not answer that. The second tier does, by keeping each order's situation chain separate. The status column on the work order may still say "in progress" after the chain says "closed." The column is a projection. When they disagree, the chain is the answer.

Three tiers, because the full stack is how you reinvent OWL

The engineering objection is fair. If every team has to keep kind catalogs, correction locks, two clocks, and a closed query language, the maintenance bill can exceed the stack this was supposed to replace. One ordering rule — drop late records, then compare — is already easy to write backwards. So the full procedure is not the entry price.

Tier	What you maintain	What you can answer	What you do not claim
Small	Endpoint checks. Nothing is a part of itself. Events are appended, not edited. One work order. Count events of a type directly.	What happened on this order. How many times it overheated.	A corrected parts list. A second order's stage.
Second	Corrections that name the fact they dispute. Drop records that were not written yet, then pick the winner. A second order gets its own chain.	Whether the retest is part of the repair after the correction. Each order's current stage.	Compensation. Counts that include subtypes. A catalog of part kinds.
Full	Part kinds, with a default rule instead of a handwritten pair list. Compensation as its own link. A closed set of query shapes, so the next question cannot invent a new query language.	A close that was undone, without deleting the close.	That you needed this tier for the rows above.

Path existence stays proportional to the number of edges at every tier. Listing every path is the expensive query. The table above is the cost of keeping the rules, which is the cost that matters.

The small tier fits in one module two people can read. There is no reasoner. The second tier adds one piece of code and a test that the order of operations is the right one. The full tier costs review, whenever a new part kind or a new query shape is allowed in. It does not cost a reasoner on every read.

Stop at the tier that answers the question you have. A standards body can adopt the second tier and leave the rest. That is a smaller decision than swallowing a whole ontology.

Where this sits

OCEL already gives you objects, events, and participation. PROV already points a record at the record it came from. EPCIS dispositions are close to the type of a situation, and they are not the node on the chain. Event sourcing already refuses to rewrite what was stored. It usually still uses one kind of appended message for both "the record was wrong" and "the business effect must be undone." Those are different messages. The split is the whole integration.

SNOMED CT, FHIR, FIBO, OPC UA, EPCIS: each one grew up to carry a slice, under real pressure to interoperate. None of them is a single picture of objects, events, situations, and participation. This is the same shape. The three questions are a check an implementation can adopt beside those interfaces. They do not replace the interfaces.

A trace is not a denser pile of triples. It is the same pile after the three questions have answers, at a tier the team will still be maintaining next year. The checklist, the lock, and the test that fails if you compare corrections before you drop late records are version 1 of the decision procedure: articles/decision-procedure-v1.md. They do not belong in this post.

Between the Company Brain and the Execution Layer, a Task Ledger Is Missing

zxpmail — Thu, 15 Oct 2026 09:40:30 +0000

Between the Company Brain and the Execution Layer, a Task Ledger Is Missing

2026-10-15

In a company, sales has explained which objection kills a deal. Marketing has corrected an outdated claim. Operations has told an agent which numbers can be trusted. Most of those corrections stay in private chats, personal settings, and separate tools. The next person cannot use them.

An agent at execution time is a different failure. It calls a tool, gets real feedback, and an unverified assumption stays in the dialogue. Later reasoning keeps following it. Gaoling School of Artificial Intelligence at Renmin University calls this task-state contamination: unverified assumptions, outdated plans, and wrong progress judgments remain in the interaction history and steer later action. The paper is Agent-Editing World Model.

The two look alike. One is something learned that never entered a shared place. It was not recorded. The other is a judgment that should not have stayed in the history. It was not removed. They are not one disease. They also do not weld into an automatic loop.

The company brain: a judgment that still holds has to be usable by the next person

A company brain is not a knowledge base, and it is not a stack of directories. Directories are only where things sit. What has to remain is the loop. Work happens. Someone corrects it. The correction is routed. Someone reviews it. It is written into a shared place. The next piece of work can use it.

Review needs three roles. The process owner watches whether this process is still running. The domain reviewer checks whether the correction is right for the business, and owns the ledger for this task. The final decider settles it when two sources conflict. In a small team, the process owner and the domain reviewer can be the same person. The decider cannot be the author of this correction. Who may change it, and who deletes it when it goes stale, is named the same day. If a role is empty, the weekly maintenance does not happen. The directories get built, then no one maintains them, and no step of the work has to read them.

What it answers is a standing rule, not whether this one step should be taken.

What the company believes now. Which decisions replaced older ones. Who has the final say. How this kind of work should be done.

Four things are enough. Memory: the facts, decisions, and history that matter now. Rules: what counts as good work, and what must not be done. Capability: skills and workers that can be run again. Learning: reviewed corrections, used to change the first three.

Do not call the rules a judgment yet. The execution layer has another kind of call: whether this single step moves the task forward. Use one word for both, and the three layers collapse into one.

The smallest set of files can be these: company-brief.md, priorities.md, decisions/, people/, projects/, policies/, skills/, workers/, learnings.md. Before an agent starts, it reads a company map. What we do. What matters most now. Which source wins. Where to look up decisions, policies, skills, and workers.

This layer can be done today. It does not wait for AEWM.

AEWM: borrow the call, do not plug in the model

AEWM turns the question. It does not predict what the tool will return. It looks at what this step does to progress on the task.

It has two abilities. Action Judge sorts the step into three kinds: critical progress, useful exploration, and useless repetition. The paper's words are Critical, Exploratory, and Noisy. State Revision: if the step is useless repetition, rewrite the reasoning and the next action from the same history that has already happened.

EditAct puts that into execution. The agent proposes a step. AEWM calls it. Keep it or rewrite it. Then actually run it, and write the result into the history. The rewrite has to enter the next step. A note that says "think again" does not count.

One terminal case in the paper: the agent checked an IPv4 port, did not find the target, and treated the service as not listening. AEWM noticed the code bound IPv6, changed the next step to check IPv6, and the port was listening. The task later passed its tests.

Read the numbers in the paper's terms. On their Action Judge benchmark, macro-F1 is 70.5%, 10.6 points above the strongest comparison. That number is how often a prediction made before execution agrees with a label written after the whole trajectory was seen. It is not "70% of enterprise tasks succeed."

Task success is a different number. On Terminal-Bench 2.0, Qwen3.5-35B-A3B with the previous method scores 37.1%. With EditAct it scores 48.3%. That is one model and one terminal benchmark. AEWM-RFT also shows that revised trajectories can be written into the agent's own ability. After that, AEWM is not called. The agent still beats one trained only on its own successful trajectories, by 2.2, 2.6, and 2.5 points.

These are results inside the paper's setup. An enterprise workflow cannot copy them.

The missing piece is a task ledger

Glue the company brain and AEWM into an automatic loop, and this layer is what you skip.

The organization layer is the company brain. It holds relatively stable facts, decisions, policies, skills, and workers. It answers what the company believes now.

The task layer is a ledger. One sheet per task. Six lines at minimum:

Task ledger
Verified facts: (source; who confirmed)
Unverified assumptions:
Excluded branches:
Unmet requirements:
Current progress:
Next question to check:

A verified fact names its source, and who confirmed it. Anything unconfirmed stays under unverified assumptions. Moving a line from assumption to verified takes a nod from the domain reviewer. A ledger with no owner is still a private chat for this task.

Without the ledger, a stale rule in the company brain flows into this task. AEWM also judges the step from an incomplete history. If someone maintains the ledger, many questions of "does this step help" can be read off the sheet first. A model is not required first.

The execution layer is not the model layer. Start with rules, checklists, and prompts. Is this step critical progress, useful exploration, or useless repetition. If it is useless repetition, what reasoning and action replace it. The replacement has to be executed for real. Collect trajectories and measures, then talk about a model.

After execution, route through the company brain's loop. A missing fact goes to company knowledge. A new strategic choice goes to the decision record. A repeated preference goes to policy. A trick that worked goes to skills. A sequence that can be run again goes to workers. These are reviewed before they are written back. They are not written into the company brain automatically.

A gate and a review are not the same thing. The gate covers irreversible actions, and it blocks them before execution. Review covers what may be written back into shared knowledge, and that can wait. If a dangerous action only sits in the weekly review queue, the gate is just another folder.

Small first, then a checklist, then a model

Stage one is the smallest company brain. Pick one process that keeps coming back. Build the smallest directory. Give the agent a company map. Name the three roles. People may overlap. The decider cannot be the author of this correction. Run the correction route and the weekly maintenance. If a role is empty, stop here. Do not go on.

Stage two adds the task ledger and a checklist. Use the six lines above. A verified fact with no source and no confirmer still counts as an unverified assumption. Review by hand with "critical progress / useful exploration / useless repetition." Try it on real tasks.

Stage three triggers a rewrite on useless repetition. Collect trajectories that real feedback has checked. See whether rework and useless loops go down. See whether useful exploration was killed by mistake.

Stage four is when a model can be considered. Enough trajectories of clear quality. Stable review. Explicit measures. Then consider training of the AEWM kind, or fine-tuning on revised trajectories. Do not start with an automatic loop.

The rest of the account

When sources conflict, the final decider settles it. The author of this correction cannot also be the decider for it. Someone deletes knowledge when it goes stale. When it is wrong, it is clear whose it is.

If AEWM is actually called on every step, the call and the rewrite add delay and cost. A wrong call can kill useful exploration. Maintaining the company brain is ongoing human work.

Look at these, not at whether the directories exist. Rework rate. How often stale information is treated as a current fact. Time spent on review. Task success rate. Whether useless repetition went down. Whether useful exploration stayed.

A person who has tuned an agent well holds a private advantage. Asking them to hand the correction over requires recognition. Otherwise the company brain gets no input.

The company brain lets a rule that still holds be used by the next person. The execution layer looks at the step before it runs: progress, exploration, or repetition of an assumption that no longer holds. The execution layer starts as a checklist, not a model.

Between them is a six-line task ledger. What is verified has a source and a confirmer. Writing back to shared knowledge goes through review, and the decider is not the author of that correction. Irreversible actions are blocked by the gate before they run.

The company brain can be done today. Treat AEWM first as a checklist and a design principle, not as a model you plug in. When the trajectories, the review, and the measures are in place, then talk about a model.

A correction has to be reviewed, written down, and usable next time before later steps get better. If the decider is the author of the correction, one lesson still stays in that one conversation.

The Definition Is the Kernel. What Still Runs After the People and the Model Change Is the Asset.

zxpmail — Tue, 13 Oct 2026 10:37:12 +0000

The Definition Is the Kernel. What Still Runs After the People and the Model Change Is the Asset.

2026-10-11

A while ago I read a piece from Tencent's engineering team. The real change, it argued, is not AI coding alone. It is the whole development process. Their OK platform delivers requirements end to end with AI. Automation is 68%. 51% of requirements finish within two hours.

The numbers are bright. They show that requirements can be finished fast. They do not answer a second question. In this way of working, what stays with the organization, and what leaves with the people and the model.

I spent an afternoon on that question with my boss. His judgments sit closer to this layer than that piece does. What follows is only the part that survives one test.

Change the people. Change the model. Change the vendor. Can the system still run at a similar quality and speed?

If it can, it belongs to the organization. If it cannot, it is one person's feel, one generation of model, or a lock-in to one vendor. "Quality" in that sentence cannot be graded by the system itself. The outside measure is in section 7.

1. A problem with no boundary is a wish

My boss opened with two lines. What problem are you solving. Who is it for.

Most teams landing AI stop one step earlier. "I want to study what software an industry needs." "I want to build a general agent platform." Those are wishes. A wish has nowhere to stop, and no way to be checked.

His split has three steps. What the problem itself is. Who is looking, and from which angle. Where the boundary is: the domain.

"Study industry software" became three problems, each able to stop. In the agent era, what software does an industry actually need. That is the demand baseline. What can current open-source projects provide. Are they stable. Can they be integrated. How large is the gap. That is the supply side. How software of any kind becomes components. That is the technical method.

Once they are separated, each one has a place to converge.

A definition decides what to do, and what counts as right. A definition written on a wiki and executed by no one is a liability. A definition welded to one set of tools, gone when the tools are swapped, is not yet an asset. The kernel is the layer that is still there after the carrier changes, and that can still constrain the next run. The toolchain and the runtime are the carrier. How that layer is drawn is in section 4.

2. There has to be structure. Structure inside the prompt cannot stop what already happened.

One line from him stayed with me. There is nothing wrong with form. Without form, the content has nothing to sit on. Without a body, what soul is there to talk about.

He did not mean more stamps on a document. He meant the content needs a structure to hold it.

Most prompts are one linear sentence. "Help me look up what software this industry has." The model can only guess. He asks for these, written apart: goal, context, conditions and constraints, output format, the agreements that matter, where to inject, which passage to cite. Goals can have sub-goals. Context loads as needed. Constraints go in by layer. Checkpoints in the flow are marked on their own, not mixed into the same sentence.

A prompt should be folded, injectable, and full of variables. The way a loop or a queue is organized in code is the way a prompt should fold. Longer is not better. Less redundancy and clearer folding work better.

This only says one thing. One linear sentence carries too little. It has not yet met the harder failure.

You write in the prompt, "if it does not pass, clarify." The model has already produced an answer. It treats the step as done, and goes on. Your sentence never gets a chance to run.

Structure is still necessary. Structure written inside the prompt is soft. For the flow to be able to stop, the check has to sit outside the model. A primitive in the runtime. A forced checkpoint in a script, before execution and after. No pass, no next stage. Whatever is not exposed can be bypassed. This is the same thing as design by contract. Preconditions, postconditions, and invariants are part of the system. They are not a reminder in a document.

If these checks live only inside one product, the set disappears when the vendor changes. What has to stay is what the check looks for. Where the check is built does not have to stay.

3. The same rule, put in two places, has two results

He had another abstraction. High-dimensional data can be used as a program. A program is itself data. Dictionaries, knowledge bases, specs, and rules look like data. When the structure is good enough, they drive the flow. A program can also be injected, folded, and combined. Write fewer fixed steps. Write the rules that generate the steps.

That sentence needs a minimal scene, or it stays a label. The scene takes only one kind of rule. If it is missing, there is no next step.

A cross-domain request needs five dimensions. The other side gave two.

Put "stop if the dimensions are incomplete" into the prompt. The model fills in the form, treats the step as handled, and continues.

Write the same rule as a record the runtime must read before submit. Five fields. Three missing. This step does not open.

The first hopes the model will obey. The second says the condition does not hold, so the next step does not exist in the flow. For a dictionary or a spec to act as a program, something outside the model has to read it. The model can be the machine that generates the content. The gate does not read a wish in the prompt. The gate reads the record.

He also made another comparison. Three of the five dimensions are themselves vague. There is no unique answer. Another person could not produce one either. That is a different failure. It belongs in section 6. The gate does not handle it.

4. The ontology describes the world. The gate checks this time.

The request in the last section was about the gate. It did not yet say how the gate relates to the ontology. One sentence here. The ontology describes what this world contains. The gate checks whether this time is allowed. A specification can say "confirm shipment." That does not mean this shipment may go. The gate has to stand independent of the ontology.

At the end of the conversation he reduced things to four words. Things, relations, actions, plus context. Entity, relation, action, context.

The split below is one I added while organizing this. He had not already separated it.

An action inside the ontology is a business verb. Check inventory. Confirm shipment. It describes what can happen in this world. A relation describes what connects to what. Context is the scene this time. Written down, this is a specification. It makes it possible for a person or a model to get it right.

An action on the gate is whether this time is allowed. Are the fields complete. Does it pass to the next stage. This layer is not written into the specification in the hope that it will be obeyed. It is a check at execution time.

He also split evolution into three levels. What a spec looks like in each domain is the top level. The concrete content under that shape is the middle level. A particular instance is the bottom level. Higher is more general. Lower is more specific.

The kernel takes only the layer that can still constrain the next run after the carrier changes. The shape of the spec belongs to this layer. It decides which slots exist, and what counts as right. Content in the slots counts only while someone keeps it true. If no one maintains it, the kernel is only the shape. Expired content can be carried across intact. The next run is no longer constrained by it.

What the check looks for is "what counts as right," taken and executed. The shape is how the problem's boundary, and that sentence "what counts as right," get written down. When the two sides match, it is still this one layer. When what the check looks for does not match "what counts as right," that is a fault. It is not an extra layer.

Which product this check is built into is the carrier. What leaves when the model changes is the instance inside one prompt, and the gate inside one vendor's product.

A general method, plus the entities of a domain, is what can land in that domain. That does not mean every industry must invent its own method. An empty template does not automatically become an industry conclusion. He later turned to manufacturing first. That is a decision. The meta-method lands in manufacturing first. It does not spread across every industry first.

5. Mistakes will happen. The cage decides whether you dare to use it.

One practical line from him. An agent cannot avoid mistakes. What matters is how a company dares to use an agent that will make them.

Give it a cage. A gate at the entrance. A check at the exit. Rollback when it is wrong. A person at the decisions that matter. When it reaches past its authority, a person decides whether to grant it.

The prior check sits in the cage too. Three of five fields are missing. That step should not start. Checking whether the input is valid is required. It is not something you look at after the run.

This check can stop a missing field. It cannot stop the case in the next section. The slots are full, and the problem still has no unique answer.

6. Two things are still unsolved

The direction is right. The conversation already showed two hard limits.

The first is how formalization continues. Structure, dictionaries, knowledge bases, specs. Who writes them. Who changes them. Who updates them before they expire. Too coarse, and they do nothing. Too fine, and they freeze. In the conversation this showed up as "it cannot be written," "it is too fragmented," "it is not normalized." That is not a team that failed to try. Formalization is expensive. Tools can lower the cost. People still find it troublesome. They call it formalism. When a person leaves, knowledge that was never written into the system breaks. That is a further layer. Why would an organization keep writing things down as structure. Tools do not lower willingness. The system can already be cheap, and people still will not use it. If no one maintains it, the kernel is only the shape.

The second is how you find out the structure itself is wrong. Back to that request. If the five dimensions are simply unfilled, the gate can close. The other half of what he said: three of them are themselves vague, and there is no unique answer. All five slots contain words. The gate lets the request through. Another person still cannot produce a unique answer. The gate can stop a missing field. It cannot stop "these five dimensions were the wrong five." It cannot stop a knowledge base running along a wrong map. Wrong content, kept as if it were true, still counts by the kernel's own ruler. So "the quality is still there" cannot be graded by the kernel.

A model that cannot read a structure that is too complex, or that gets worse when the context is too long, is today's floor on ability. This method cannot run in every setting yet. That floor is not a permanent boundary of the method.

7. What you can take with you

Models will be replaced. They will become ordinary goods. They will be open-sourced. Prompt technique will lose its price quickly. Rules that only sit on a wiki are a liability. Rules welded into one toolchain, or one product, disappear when the carrier changes. Those are not assets either.

What you can take with you is that layer. It is still there after the carrier changes, and it can still constrain the next run. The toolchain, the runtime, and which product the gate is built into, are the carrier.

Whether the quality is still there has to be read from results the kernel cannot alter. Can the thing that was handed over be used. Can the person receiving it actually take it. When it is wrong, can someone outside see it. A wrong kernel can move intact onto the next carrier and, by its own ruler, feel that it still runs well. That is still not an asset.

Apply the same test to the OK platform. 68% automation, and 51% of requirements finished within two hours, answer "fast." Fast is an outside number. It measures throughput. It does not answer whether, after a new model, a new group of people, and a new vendor, someone outside can still see a similar quality. That number is not there yet. Until it is, the piece shows throughput. It does not show an asset.

This is organized from an internal conversation. The judgments are my boss's. In section 4, the relation between the ontology and the gate, and the split of action into two layers, are additions I made while organizing. Collapsing the kernel into "the layer that can still constrain the next run," counting content only while someone keeps it true, and reading quality from outside results, are boundaries I drew while organizing. Other errors in the organizing are mine. This is a direction, plus engineering problems that are not finished. It is not a completed answer.

A General World Model Is Still Far. Next-State Prediction Already Runs.

zxpmail — Thu, 08 Oct 2026 08:14:54 +0000

A General World Model Is Still Far. Next-State Prediction Already Runs.

2026-10-08

World Labs put the phrase back in circulation. Marble turns a picture into a 3D scene you can walk through. Atlas, released in September 2026, is billed as a next-generation world model: text, images, video, and 3D go into one spatial context, and the model generates what comes next. It is meant to power later versions of Marble, and to feed robot simulation.

It looks as if machines are close to understanding the physical world.

A general world model is still far away.

That claim has to be split. Left in one piece, it is either empty pessimism, or the reverse exaggeration: a trajectory predictor in a car, called a finished foundation model.

1. The name covers three things

Generative world models. Sora, Genie, Marble, Atlas. A picture or a sentence goes in. A video, a 3D scene, or a new camera view comes out. Genie takes actions. Atlas does more than paint the next pixel. It puts several inputs into one spatial representation.

Looking right is still not understanding. Gravity breaks. Objects vanish. Over a long stretch the scene stops matching itself. The model learns what usually comes next. A larger scale might grow some physical intuition. There is not enough evidence to treat a demo as an ability. There is also no evidence that scaling cannot produce it. Leave both open.

Next-state prediction already running in engineering. This layer is old, and it often does not use the name. Trajectory prediction, occupancy networks, and ego dynamics in driving. Grasp and motion prediction in robots. Cloud simulators that manufacture rare cases and run closed-loop tests. The job is the same: given the state and an action, estimate what happens next.

Draw the line. Under this definition a bicycle model, a Kalman filter, and the physics engine inside a simulator all count. "World models are already in use" means this function is old. It does not mean a new foundation model is done.

Occupancy networks and trajectory predictors enter decisions because they write the world as grids, trajectories, and occupancy, and those can be scored. A collision is measurable. A miss is measurable. A generative model leaves the state in the picture, and a mistake often only looks a little odd. Both learn statistics. Both can break physics. The difference is where the state is written, and whether a mistake can be caught.

The general world model. This is the one Fei-Fei Li and Yann LeCun are after. One representation that holds 3D space and time, objects, cause, and interaction. It still works in another scene and on another body. It can answer what would have happened if the action had been different. Planning that uses it beats a special-purpose module in a closed loop.

That one is far.

2. "What was seen" does not give "what if I act"

The usual explanation is that data and compute are short, and enough of both will grow the ability. Atlas's own bet is the same: more training compute, more ability.

That bet is not falsified. It should not be treated as already falsified. Hard only means it cannot be done today.

The gap that can be stated is narrower. Next-frame training learns what usually follows what was seen. Planning asks what the world does if I take an action that never happened.

Records of what already happened do not separate the two. A cup falls because I pushed it, or because it slipped. Correlation can look convincing. Telling the cause takes an action, then the result. More samples make "what usually happens" more stable. If the data has no record of acting, intervention and counterfactuals do not appear on their own.

A few concrete difficulties follow.

Representation. Geometry, meaning, material, dynamics, and intent still live in different tools. One shared context means the model can read them together. Reading them together is not a representation you can intervene on.

Interaction. The model has to separate a change I caused from a change the world made on its own. Action has to be a condition, and the actions cannot be only the ones people already took.

Closed loop. On a robot, in a car, or in AR/MR, trying it yourself is expensive, risky, and full of rare cases.

Evaluation. Accurate pixels are not correct physics. Physics tests and closed-loop tests exist. They do not line up with each other, and they do not predict whether a system can ship. The missing piece is a standard that matches real action.

Latency. A car and a robot need low delay, and a failure that can be explained. Today's generative models do not sit in that seat.

3. Driving is the cleanest test

Some people say autonomous driving does not use a world model. On the vehicle it is perception, prediction, and planning. That is half right.

The vehicle will not run an explicit generative world model. Compute, latency, safety checks, and the need to explain a failure all forbid it. In that sense, driving does not run that kind of world model.

The vehicle still cannot do without "what happens next." Trajectory prediction estimates how other vehicles and people move. An occupancy network estimates which space will be taken. A dynamics model estimates how the vehicle responds after an action. Cloud simulation uses the same kind of method to build rare cases and run closed-loop tests. These are narrow predictors, and they can be checked.

When planning uses dynamic programming or model predictive control, this term is that predictor:

V(s) = min over a of [ R(s, a) + γ Σ P(s' | s, a) V(s') ]

A model-free method can skip writing down P(s' | s, a) and learn the action directly. Closed-loop control depends on how the environment changes, either as an explicit probability or welded into the policy.

An end-to-end network that drives shows that it reacts to the road. It does not show that a piece can be pulled out and asked, "what if I change lanes now?" Reacting and being available to ask are different things.

What the car already has is a special-purpose predictor on a narrow state: lanes, vehicles, occupancy. The argument is about another road, another kind of road user, an action never seen, and whether that answer is more reliable than the modules already there.

4. AR/MR supplies samples, not the model

AR/MR is often pulled into the same picture. Its value is not a few extra labels. In real 3D space it records, continuously and with position, what the scene was, what the person did, and what the scene became.

Camera pose, depth, meshes, and point clouds record space. Object class, material, and whether something can be handled record attributes. Gesture, gaze, speech, and grasp trajectories record action. The scene before, during, and after the action records order in time.

The samples are useful. They are still mostly trajectories of what was seen. Force at contact, mass, and friction are often missing from the picture. AR/MR is a data factory and a proving ground. It is not a general world model.

5. How to tell it is getting close

"Far" has to be something that can be proven wrong.

Long consistency. For minutes, objects remain, physics still holds, and the scene still matches itself.

Intervention. Given an action, the model predicts how the state changes. A next frame that merely looks reasonable does not count.

Counterfactuals. What if the action had been different. A harder check: two models can tie on "does the next frame look right" and disagree on "what if I had not done that." A high prediction score does not pin down the counterfactual.

Closed loop. On a robot or a driving task, planning with the model beats the special-purpose method already in use. This is the ledger. If the earlier items fail, this one does not arrive.

Physics. Gravity, collision, friction, and occlusion hold steadily, rather than by an occasional lucky guess.

Cause. It separates occurring together from causing. It supports both "I act" and "what if I had not."

None of these is stably met for the general case. "Still far" has criteria.

6. What is actually true

Next-state prediction, as a function, already runs in engineering. The thing called a world model, and expected to be a third foundation model after language models and vision-language models, is early. Across scenes. An action never seen. A counterfactual. A closed loop where it beats a special-purpose module.

On the vehicle, the stack is still perception, prediction, and planning. Large models carry meaning and reasoning. Next-state prediction sits in a special-purpose module, or is welded into an end-to-end policy. A generative model does not run there on its own.

In cloud simulation the method is already useful: rare cases, closed-loop tests, synthetic data.

AR/MR is a source of data and a place to check.

Robots are the closest proving ground, and the least mature. Here the system has to act and then see the result. "What was seen" and "I acted" start to come apart.

7.

Two extremes are both wrong in the same way. Each one flattens the word.

One says a world model is about to change everything. A demo is treated as an ability. Generation is treated as understanding.

The other says a world model is hype and useless, and throws out the predictors already running in cars and simulators.

The direction is right. Representation, prediction, and action in the physical world are a main line after language models and vision-language models. The work will enter driving, robots, and AR/MR in pieces, by domain, sometimes as its own module and sometimes welded into a policy. It will not arrive one day as a single general model and rewrite the rest.

Maturity is three questions. Can the world be written as a state a program can use. Can change be predicted reliably. Can the system act safely on that prediction.

On a narrow task, a lane or a grasp, usable versions of all three already exist. Change the body, the physics, or the action to one never seen, and all three break.

The third question is the watershed. The first two can stop at "looks real." The third requires the answer to be right, because a wrong action causes harm. In the general sense, that watershed is still far.

Ontology, Gates, and a Human Backstop: Three Defenses for Enterprise AI That Don't Spin Empty

zxpmail — Mon, 05 Oct 2026 11:12:50 +0000

Ontology, Gates, and a Human Backstop: Three Defenses for Enterprise AI That Don't Spin Empty

2026-10-05

Enterprise AI usually lands in one of two extremes.

One side says: build the ontology and the AI will understand. The other says: an ontology is soft, it cannot hold the line, and what matters is gates plus a human backstop.

Each side is only half right.

An ontology makes it possible for the AI to get it right. A gate makes it hard for the AI to do the wrong thing on its own. A human backstop keeps a mistake from becoming a disaster. Use the three together, and close the loop: the ontology hands objects and rules to the gate, the gate hands anything out of bounds to a person, and after the person handles it they ask whether a term or a threshold should change. Without that return path, people keep putting out the same fire.

The weight shifts with the scene. The short board sets the ceiling. High frequency, low risk, reversible: the ontology and automated gates set the experience and the cost. Low frequency, high risk, irreversible: the gate and the human backstop are what keep you safe. Heavy regulation and heavy responsibility: all three have to be hard. Open questions can tolerate a wrong answer. Data that should not enter the context still has to be stopped.

Whether this works is not a question of whether the article is complete. It is whether the three tables can be filled in for one concrete scene. A cell you cannot fill means that question may not be answered, and that action may not be taken.

1. What each layer handles

Layer	What it handles	When it fails
Ontology	Whether the system understands	The answer is fluent, and wrong
Gate	Whether it may, and whether the figure is the right one	It acts at random, exceeds its authority, or does something irreversible
Human backstop	Who is responsible, and how a mistake is written back	The damage cannot be undone, and the same kind of mistake keeps returning

Do not ask which layer matters most. Ask which layer is the short board in this scene, then ask whether that layer's table is filled in.

2. Ontology: start from a specification

An ontology here is a business specification: what a word means, what it does not mean, and which field on which document the figure comes from. It is soft, like an employee handbook. It does not itself block an action. The part that can be made hard has to become a check outside the model.

Split the content into two kinds first:

Kind	Where it lives	What a mistake does
Helps the model choose the right meaning	Terms, relations, definitions. The model may read them	The wording drifts, and can still be corrected
Must not be allowed to happen	Written as a program check	The action cannot occur

"Collected payment is not the invoice amount" is the first kind. Putting the words in a table does not stop the model from answering with the invoice amount. A wrong figure is stopped by the input gate: what is handed to the model this time contains only the field named on the authoritative source. The invoice amount is not included.

"Do not send it to the customer on their behalf" and "do not pay above this amount" are the second kind. Written into the prompt for the model to obey on its own, they are still soft.

Each common word gets four lines: how people say it, what it is, what it is not, and which field on which document is the authoritative source. "What it is not" matters most.

One word binds to one authoritative source. If a finance definition and a management definition are both legitimate, they are two words. Do not switch sources on the same word. When two owners disagree, follow the owner of this word, the person on the side of the authoritative source. When the definition changes, new documents use the new version. Documents that already happened keep the old version.

If the source has no record yet, the answer is "no figure yet." Do not fill it from another set of books. Also write how long the business can wait. If the wait expires and there is still no figure, point to a named person. Do not repeat the same sentence. If the wait is not written down, people will go look somewhere else, and this gate has nobody walking through it.

Write only relations that have actually been asked. Three to five is enough. Write the rules, and mark which kind each one is. A rule of the second kind lands on a gate at that moment.

Each word has an owner, an update trigger, a version, and a minimum set. It also has a reviewer and a review cycle. A trigger without a review lets the table go stale when nobody reports an error.

3. Gates: start from what is forbidden

Write what must never be done: which fields cannot be changed, which approvals cannot be skipped, what cannot be sent outside, which tools cannot be called, and above what amount a person is required. Talk about permission only outside that forbidden zone.

Level	What the system does	What the person does
Read only	Looks up and answers	Judges for themselves
Suggestion	Offers a suggestion	Does the thing themselves, including sending it
Confirmation	After the person agrees, the system does it	Agrees, and is responsible for what cannot be undone
Automatic	Does it directly, and leaves a trace	Can be inspected afterward, and can be rolled back
Forbidden	Does not do it	Not applicable

Anything not explicitly allowed is forbidden. If read-only is enough, do not grant a write. An automatic action must be reversible or have a remedy. The judgment sits outside the model. Waiting for a person has a time limit. When the time is up, the action stops. A stop is not the same as a person refusing, and it must not be turned into an automatic action.

Use the checkpoints the scene needs. All seven are not required:

Checkpoint	What the program does
Input gate	Which documents and which fields are allowed this time
Term gate	Checks an allowlist. Which words a conclusion may use is written in advance, and so is which slots the output has
Tool gate	Which tools may be called, and whether the parameters are allowed
Action gate	Which fields may change, and which states may move
Approval gate	Above a threshold, across parties, or outside the country, hand it to a person
Provenance gate	Compares only fields that have already been written down with the authoritative source
Audit gate	Who let it through, when, which version of the terms, which document

The term gate does not guess afterward which word the conclusion depended on. That falls back to letting the model judge for itself. The allowlist and the output shape are written first.

The provenance gate compares fields only. In collections, the amount, the customer, and the invoice number can be cells. The suggested wording is not compared. An analysis, a long conclusion, or a judgment tangled in several conditions cannot be made into stable fields, so do not install this gate there. Those scenes keep the input gate, the forbidden zone, and the audit. The conclusion stays with a person.

A confidence score the model reports about itself is not a gate.

To rise from "the system acts after a person agrees" to "the system acts on its own," write five things first. The count comes last:

What counts as one error. If that is not written, a count means nothing
Who decides it was an error. What they check against, and how much time is set aside for that check. If nobody decides, the action stays at its current level. A name with no time is only a stamp
The sample has to cover the main situations of this kind of action. A pile of easy cases is not enough, even if the count is large
The action can be undone, or has a stated remedy
If it errs again, it drops back one level

Low-frequency, high-risk, irreversible actions do not become automatic by accumulating a count.

4. The human backstop: the last layer

It has to exist. It cannot be the main path. People are expensive, they get tired, and they spread the responsibility thin.

Kind	What it does
Exception handling	Handles what a gate stopped, and what the program cannot match
Responsibility	Signs for an irreversible action
Return path	Decides whether the terms change, or the gate changes. If neither changes, write the reason

Escalate only when the case hits the allowlist, an approval, or provenance. Each backstop has a handler, a time limit, a next person, and someone who counts whether the time was exceeded. If nobody counts, the time limit is empty.

While the system only suggests and the person does the thing themselves, whether the amount was stated wrong is compared by the program. Do not ask another department to certify the sending. Certification is only for what the program cannot see: a source delay the business will not accept, or a disputed definition. Once the system sends on its own, a sending error needs a separate person to certify it, and that person needs the time above.

5. Fill the tables once, with collections

The scene is fixed. A salesperson asks about a customer's collected payment. The system answers from the finance receipt only, with an amount and a suggested wording. The salesperson decides whether to send it, and sends it themselves. This is a suggestion. It is not "the system sends after someone nods."

This scene has four output slots: customer, booked amount, whether anything is still unbooked, and suggested wording. The only conclusion word on the allowlist is "collected payment."

What can be filled in:

Item	Decision
What collected payment is	Money the customer actually paid into the company account
What it is not	Not the invoice amount, not the contract amount, and it does not include prepayment
Input	Only the booked amount on the receipt. The invoice amount is not included
Sending	No send tool. The system does not change the document
Provenance	Compare only the customer and the booked amount. Do not compare the wording
Unbooked	Say "no figure yet." Do not fill it with some other number
Level	Stays at suggestion. It does not rise

Empty, and therefore not allowed yet:

Empty cell	Therefore not allowed yet
Which system holds the receipt, and which field	The input gate cannot be drawn, so the amount cannot yet be a formal answer
How a customer id matches a receipt	Do not answer "this customer's collected payment"
How an invoice matches a payment received	Do not answer "whether this invoice has been paid"
Which day overdue starts, and after how many days it counts	Do not answer about overdue
How late the source may be	"No figure yet" cannot go live yet, or people will go around it
Who handles a mismatch, how long, whom they call, who counts timeouts	The backstop is only a name so far
Who reviews the three tables, and how often	The tables are not built yet
What counts as one error	The system does not send on its own yet, so this may stay empty. It must exist before the system sends

A management definition is a different word. This scene does not ask it, so that row is not built.

6. Different scenes trim the tables

The three tables are one way of filling things in. They are not a full copy for every line of business.

Scene	How the tables are used
Documents, processes, and fields are clear	The input gate, the allowlist, the provenance gate, and the audit gate can be used together
Low frequency, high risk, irreversible	Keep the forbidden zone, approval, and a signature. Do not schedule automatic action
Open questions, analysis reports	Keep the input gate, the forbidden zone, and the audit. Do not turn provenance into fields. A person owns the conclusion
Heavy regulation, accountability required	Every row used above needs a version, a person who certifies, and a review

Consider automatic action only for something that can be undone, and only after "what counts as one error" is already written.

7. Order of landing

Pick one scene. Write the terms and the forbidden zone first. Mark which of the two kinds each rule is.
Fill the three tables. A question you cannot fill in is removed from what may be answered.
Go live as read-only. Use a fixed set of questions to see whether the error is the definition, the wrong field, or a missing source.
Suggestion: the system suggests, the person acts.
Confirmation: the system acts after a person agrees. The person who certifies, the time limit, and the person who counts timeouts already exist.
The error definition, the sample, and the rollback are written before automatic action is considered.
Every backstop returns. On the review day, look for a stale table even if nobody reported an error.

Those are the column headers. Delete rows the scene does not use. Do not leave a gate that nobody walks through.

Ontology: term or rule, kind, definition, what it is not, authoritative source and field, owner, how to answer when there is no record yet, how long to wait, update trigger, version, reviewer and cycle.

Gate: checkpoint, rule, where the line is drawn, what happens after a stop, who agrees, who certifies an error, what happens when time is up, the log, whether it can be undone, the error definition required before a level rises, rollback, reviewer and cycle.

Backstop: what was hit, who handles it, how long, whom they call if they cannot, whether the term changes or the gate changes, whether it was finished, who counts timeouts.

8. Close

An ontology lets the AI know the business. A gate keeps the AI from acting at random. A human backstop keeps a mistake from becoming a disaster. The return path keeps the same kind of mistake from repeating.

If the tables for one scene cannot be filled in, the three layers are not there yet. The empty cells are the boundary of this defense.

Weld the Tenant Boundary on the Server

zxpmail — Thu, 01 Oct 2026 12:55:50 +0000

Weld the Tenant Boundary on the Server

Subtitle: A reported name can be noted. It cannot be trusted blindly.

2026-10-01

In a multi-tenant system, the cheap path is: the request path names an org, the token names an org, and the model passes an org again when it calls a tool. If the three agree, the data is treated as belonging to that tenant.

All three can be swapped. The path can point at someone else's org. The claim in the token can be only a hint. The model can fill in a different repo in its arguments. The cheap path treats "who reported the name" as "whose data this is."

This piece is only about how data is isolated. The per-layer rules and examples stay in the reference draft. They are not repeated here.

Binding happens on the server

A request may name an org, a project, and a repo. The claim in the token is only a hint. What counts is that the server finds "this person is actually a member of that scope," and only after that row comes back does it mint the context for this request.

Business code must not mint that context by hand. Minting happens at the entrance. Later functions take the context. They do not take identifier strings from the caller.

Org, project, repo, index, object key, and topic supplied inside a model tool call are dropped and written back from the context. If the model disagrees with the context, write an audit row and still execute against the context. Do not keep the model's value as a fallback.

Every data plane gets its tenant filter from the server. Tenant fields in the client, the URL, and the model arguments are not used.

The convenient leak is on the corpus

Two repos inside one org reading each other is a common leak in a coding assistant. Isolating business data such as orders by org does not stop this plane. Vectors, code objects, search, and repo caches have to be isolated by org and repo together. One tenant id pushed through every layer makes those two repos share one corpus.

The other cheap path is to check the token and skip the membership lookup, or to let a repo name in the tool arguments override the context. That implementation has to fail a negative test: cross tenants once on purpose, and the detector must catch it. A fully green suite does not show that the detector is wired in.

What this piece does not cover

How keys are generated, where they live, and how they rotate are not here. Whether the model vendor keeps prompts, or trains on them, is not here either.

Making ciphertext in backups unreadable, and actually deleting data on the vendor side, each needs its own procedure. This piece stops at: delete online data inside the scope the server already bound, and record the vendor's ticket number for the deletion request. The contract, zero-retention, and which region the data sits in are not spelled out.

Conclusion

People weld the tenant on the server: bind only after the membership check, and drop the names reported by the client and the model. The convenient leak is two repos in one org sharing a corpus, or letting the caller name the tenant. Keys and vendor retention are not in this piece.

Don't Treat 'Module Internals Don't Matter' as Unconditional

zxpmail — Tue, 29 Sep 2026 11:38:31 +0000

Don't Treat "Module Internals Don't Matter" as Unconditional

Subtitle: Urgency raises soft boundaries. A rules file pushes back behavior, not understanding.

2026-09-29

The essay Why the Best Software Advice Is the Hardest to Follow splits advice into two kinds:

Practical advice: small functions, no magic numbers, clear names. Teachable. Checkable.
Judgment advice: a wrong abstraction costs more than duplication; keep it simple; don't repeat yourself. No standard answer.

The conclusion is that the most valuable advice is the hardest to follow, because it is judgment, not a rule. I agree.

In the AI era I want one tighter limit:

People should move their attention up to the business, the cut, the boundary, and the contract. The inside of a module can be handed to AI more often.

But "duplication or abstraction inside the module doesn't matter, the AI understands" — the direction is right, and it must not be said as if it were unconditional. "Unconditional" here means treating it as a slogan that always holds.

1. The direction is right: move human attention up

Human attention is limited. If you spend the day arguing whether a function is over 20 lines, whether a name is good enough, or whether to extract an interface, you sink into the local and miss what actually decides the outcome:

What business problem is this? What is the core domain? How is the system cut? Where is the boundary? Who owns the data? Which way do dependencies point? What is the contract? Which changes must be isolated? Which decisions are irreversible? Who carries the risk?

Get those wrong, and pretty code will not save you.

So: people own business clarification, domain cuts, module boundaries, and contracts. How the inside repeats or abstracts can be delegated to AI more often.

On that part, I agree.

2. "Doesn't matter" has a premise. Urgency is more dangerous than "AI nature"

A typical failure: the team hands almost all in-module implementation to AI. It is fast. There are no contract tests, and no one has defined data ownership. Two modules, for convenience, both read and write the same table. On the surface this is "freedom inside the module." In fact they are quietly coupled through a shared table. Later one module changes what a field means. The other module's transaction boundary fails. The data diverges. Rollback does not come clean.

The problem is not that the inside is ugly. The problem is that the boundary was never actually hard.

Teams that do this well usually write the core interface schema, acceptance criteria, and data ownership into a rules file first. After that, however the AI-generated inside repeats or abstracts, the contract still holds it. The difference is not whether someone "uses AI well." It is whether the boundary was welded shut beforehand.

So this sentence holds only when the boundary is hard enough:

Inside the module, duplication or abstraction can be highly delegated.

And one more sentence, so it is not said unconditionally:

High delegation is not zero constraint. Readability, testability, and performance and security budgets remain the floor inside the module.

I originally wanted a fuller sentence: "AI systematically rewards soft boundaries." After a small reproducible experiment, that sentence had to be tightened — and the claim got harder, not softer.

A small reproducible experiment

Original claim: with no extra constraint, AI more often picks the soft boundary (reaching into another module's table).

Method: 3 cross-module tasks (order email snapshot / inventory marks an order shipped / billing reads a phone number) × each task A = soft (direct SQL/ORM) or B = hard (API or event) × 3 contexts (bare / speed, which urges "ship soon, change little" / hard, which writes data-ownership rules) × N=5, temperature=0.7. First run locally on qwen2.5:7b, then retested through the current Claude provider on cc-switch (glm-5.3-flash).

Script: soft-boundary-bias-test.py

Model	bare	speed	hard	Verdict
qwen2.5:7b (Ollama)	0% (0/15)	33% (5/15)	0% (0/15)	SUPPORT_SPEED
glm-5.3-flash (cc-switch / Zhipu)	0% (0/15)	73% (11/15)	0% (0/15)	SUPPORT_SPEED

Same shape on both models: bare and hard never pick soft; urgency lifts it. Speed is higher on GLM, but both are small samples (15 trials per condition). Do not extrapolate the size of the gap.

Tighten the conclusion once more:

bare 0% is not "AI defaults to the hard boundary." It only says: when both options are laid out clearly and there is no time pressure, the model showed no soft-boundary preference. That is already enough to correct "systematically rewards soft boundaries."
speed is the only clear signal. On qwen, all 5 soft picks fell on S3. On GLM the soft picks were more scattered (11/15), and the rate was higher. Fifteen trials per condition cannot be generalized, and 33% vs 73% is not a ranking of which model is worse.
hard 0% shows that a rules file works, with a caveat: the model may be obeying the rule, not understanding the boundary. What the rule pushes back is behavior, not necessarily judgment.

This run did not separate "ship soon" from "change little." Time pressure and change-suppression were written into the same urgency condition.

Closer to the data, under this setup:

When both options are clear, the model does not necessarily reach for the soft boundary.

Once you add "ship soon / change little," soft-boundary picks rise.

Writing data ownership into the rules can push that rise back down.

This correction is stronger than the original sentence. The problem sits on the low-friction shortcut under urgency, not on an abstract "bad AI." That is a more concrete reason to put the rules file first — not to defend against AI itself, but to block the shortcut AI takes under pressure.

A boundary is hard enough to delegate across when, typically: the interface contract is clear, invariants are explicit, data ownership is clear, dependency direction is controlled, boundary and contract tests are sufficient, performance and security have explicit requirements, and there is no cross-module shared write or hidden coupling.

3. How to make the boundary hard enough: a minimal sufficient set

Most teams do not need all of this at once. In the experiment, hard pushed soft back to 0%. That is reason enough to put the rules file first — just do not read obedience as understanding.

Means	When	Role
Rules file	Now	Default, required
Data ownership	Now (an organizational decision is enough)	Default, required
Spec before code	When rework is frequent	On signal
Contract tests	Modules ≥ 3, or interfaces change often	On signal
Architecture guardrails	People ≥ 10, or dependencies are out of control	On signal
Fitness functions	After the earlier items are stable	A monthly report can stand in for now

Minimal sufficient set: rules file + data ownership + a contract on the core interfaces.

The rest fires on signal. The order must not be reversed. Write the rules into .cursorrules / CLAUDE.md. A table or an event has one write path. Contracts and guardrails fail in a tool. They do not depend on someone remembering.

4. "The AI will understand" is not enough: feed context by risk

AI depends on context, not on unspoken agreement. More context is not always better.

	High risk / irreversible	Low risk / reversible
Glossary / constraints	Full	Minimal
ADRs / failure cases	Full	Summary
Contract / acceptance	Full	Core schema
Human review	Small steps	Automated tests + fast rollback

High risk gets the full feed and a human review. Low risk gets the minimum, with automation as the backstop.

5. What humans review: watch for abnormal signals

People own seams and contracts. AI owns implementation. You are not reviewing every line. You are watching whether these signals get worse:

Dimension	Abnormal signal
Module boundary	One PR often touches 3–4 modules
Interface contract	Breaking changes with no version; idempotency with no test
Invariants	Core invariants with no test
Dependency direction	Reverse dependencies and cycles rising
Data ownership	Multiple writers on one table; cross-module SQL
Non-functional / evolution	No SLO, high-risk items still open; cross-module changes and shared-table use trending worse

6. Judgment can be written down. Don't turn it into a new dogma

ADRs, failure cases, "when this principle does not apply," reversibility — these can be written for the AI and for the next person. A good form is:

At this risk level, we tend to …. If the following signals appear, judge again.

Rules and tests are constraints a machine can execute. ADRs and cases are preferences a person can update. "Functions must be under 20 lines" kills the judgment again.

7. Different teams, different moves

One principle: hold the floor at low cost. Do not pile on tools.

Size	Do	Hold off
1–3 people	Minimal rules + data ownership	Guardrails, full contracts, fitness functions
5–20 people	Rules + ownership + a spec for large changes + core contracts + light guardrails	Fitness functions (a monthly report instead)
20+ / microservices	Add weight on signal	Governance for its own sake
Legacy	Leave the stock alone, align the increment; trends before heavy tools	A full rewrite on day one

The tradeoffs: strict on the new, light on the old; strict on the core, light on the edge; strict on high risk, light on low risk; light on the stock, strict on the increment.

8. How an ordinary person grows the skill

Before each module, write three lines: What is it responsible for? Who owns the data? Who may call it?
After each refactor, record one judgment: situation / options / decision / reason / consequence / reversibility.
Each time you use AI, check the boundary once: a direct cross-module call? writing someone else's table? bypassing the interface?

The skill of writing code is getting cheaper. The skill of defining a boundary is getting more valuable. You do not leave the code. You stand higher, and you bring the code experience with you.

Conclusion

The direction is right: people should care about the business, the cut, the boundary, and the contract. The inside of a module can be handed to AI more often.

Two limits:

The boundary is hard enough, the contract is clear enough, the tests are enough — under urgency, especially, use rules to block the low-friction shortcut. That is what the experiment supports, not "AI is soft by nature." What a rules file pushes back is behavior, not understanding.
Feed context by risk — not "it will understand," but "give it the right constraint."

Finally:

People own the boundary and the intent. AI owns the implementation and the local optimization. The inside can be highly delegated, but it stays under the boundary, the tests, and the budgets for readability, testability, performance, and security.

Experience Admission for White-Box Agentic RL: A Position on Dataflow between Exploration and the Base Policy

zxpmail — Thu, 24 Sep 2026 10:27:41 +0000

Sixth and final in a six-part series on what building with AI agents did to one developer. The last piece ended under a blinking cursor in an empty title; this piece is that file: proposals marked as proposals, failure conditions written before the run.

Version: v0.19.3 · 2026-09-24 (aligns abstract with §6.4; separates r routing from update ban; clarifies ℓ timing before θ)
Genre: Technical position paper
Channel note: The author is a newcomer. arXiv (cs.LG endorsement) and the SEA Workshop (venue bar) were both blocked as channels, so this paper is published through neither, and no "submitted / forthcoming" claim is made. The claims and the falsification protocol stand alone, independent of any academic venue.

Abstract

In white-box observable-worker multi-worker RL, a layer is often missing between the optimizer and the collaboration interface: after trajectories are produced, who may materialize, at what granularity, and when θ may update. Black-box APIs are outside the strong-form claim.

This paper calls that layer experience admission control (below, the admission policy). The novelty sits in P1–P4 (a proposed admission policy) and preregistered failure conditions — not in a new score or a proven finding. Hypothesis-level claim: the bottleneck is often how experience is layered, isolated, and let into the base update path at low frequency — not more models seeing each other.

Layered verdict: fix 𝒜, vary only the data plane (A/B/P). No failure condition (1–5) fires, effective-materialization-ratio improvement ≥ 10% vs A, and return not persistently below A (condition 3 unfired) → §6.4 scale-up gate (may scale up; ≠ validated; ≠ must be a first-class module). If it still passes after scale-up → recommend building it as a first-class pipeline module; otherwise withdraw that configuration's strong form (the name may be retired).

Keywords: experience admission; proposed admission policy; white-box multi-worker RL; dataflow; falsification

1. Introduction

Discussions of multi-worker, asynchronous, long-horizon RL often conflate three question layers.

The optimization layer asks how gradients are computed and applied (e.g., group-relative methods [2], single-rollout asynchronous optimization [1]). The collaboration layer asks how multiple workers share context, files, and session state during interaction. The admission layer answers something else: after a trajectory has been generated, who may materialize it, at what granularity, and when θ may update.

The optimizer answers "when/how to update." The collaboration interface answers "how to coordinate." Neither directly defines admission semantics. Rising throughput can still expose noise and high-risk trajectories indiscriminately; shared context does not automatically yield isolation rules. Training stability is related to admission semantics but not identical to it: stability asks whether updates on a given visible set diverge; admission asks which trajectories may enter that visible set, and whether/how they are materialized first. An optimizer can learn to be more stable on a dirty visible set — it still does not answer the policy boundary of the visible set.

Scope (narrow first, then motivate). The strong form of this paper assumes workers can expose one of: hidden representations, local action distributions, or uncertainty signals. This covers self-deployed open weights and observable actor–learner setups. It does not cover mainstream closed API agents — those degrade, at most, into post-hoc text filtering, are never conflated with the strong form, and are outside the claim. LLM agents are one motivating scenario; the validation domain is the white-box observable setting.

In that setting the problem is not LLM-specific: long trajectories, asynchrony, and heterogeneous workers exist in classic deep RL too. IMPALA [5] and Hou et al. [1] live on the optimization/scheduling plane, where "delivered ⇒ learnable" can be the default. What this paper names is the admission policy after delivery. Shielding [9] shrinks action feasibility; CQL [6] constrains value estimates on a given dataset — neither answers layered materialization/update budgets on an online stream.

C1, a minimal counterexample (why "having PER on" is not enough). Suppose the buffer holds an ordinary learnable trajectory τ⁺ and a high-risk, high-TD-error / high-surprise τ‡. When PER [3] reweights by TD-error and τ‡ sits in a high-priority quantile, it can be sampled into the update batch at above-uniform rates; a rollout-time shield that blocks unsafe actions does not remove τ‡ from the training-visible set; deletion-style filtering discards its analysis value. Gap triad: P1 budget semantics absent (the trajectory is already in the pool and learnable by default — no "materialize-off-by-default" constraint) / violates P2·P4 / violates P3. The semantic gap: sampling weights ≠ admission policy.

Note: PER's sampling probability follows a power law in the priority quantile, so a high quantile does not guarantee a draw; but the reweighting mechanism in principle allows high-risk samples to reach the learner above uniform rates — this mechanistic "allowed" is exactly the semantic gap this paper names. Also: if a hazardous trajectory's TD-error is not high, PER will not raise its sampling probability — but that is no rebuttal of the admission layer: the hazard is then merely "unnoticed," not "quarantined and retained by a policy layer." P2's point is not "not learned," but "flagged, retained, analyzable."

So this paper does not replace the optimizer or the collaboration layer; it argues for an explicit data plane between them. Structure: contributions → P1–P4 and their logical interdependence (§3) → neighbors (§3.1) → interface and predicate (§4) → hypotheses (§5) → falsification protocol (§6) → objections (§7) → conclusion.

2. Contributions and non-claims

This is a technical position paper, not a methods paper.

Contribution one: restating the problem. "After trajectories are generated, who may materialize, at what granularity, and when θ may update" is promoted to an independent system layer. If the name can be fully absorbed by existing terms, the name should be retired. Even retired, P1–P4 and the §6 protocol remain independently testable as a system interface.

Contribution two: the proposed admission policy P1–P4. A minimal set of design differences relative to PER/deletion-style filtering, with a logical-interdependence argument (§3.2). The novelty is pressed onto four switchable constraints, not a fingerprint formula. No claim that empirical minimality has been proven by causal ablation.

Contribution three: a falsifiable protocol. A/B (plus a PER arm) and five preregistered failure conditions; condition 5 can fail the strong-form configuration on the current proxy family (§6).

Not claimed: theoretical guarantees; proven causal irreducibility; collaboration products as an RL baseline; optimality of placeholder parameters; strong-form coverage of black-box APIs; LoRA/cloud fine-tuning as main results; the appendix as proven method results; the §6.4 gate as validation success or "must be a first-class module."

3. The proposed admission policy: why it is not "renamed PER/filtering"

Terms (materialization vs deep read). In this paper, materialization means turning a trajectory from streaming state into an inspectable, persisted object; deep read means running costly analysis on materialized trajectories (full unrolling, replay, value estimation). In most implementations both fall under "admission cost," but they are logically separable — one can materialize then deep-read, or deep-read in streaming state and discard. For brevity, budget and metric contexts say "materialization cost" for both; when distinguished, materialization is storage-side and deep read is compute-side. D_read is the set eligible for materialization/deep-read — not the admitted-to-update set.

If the critic asks what remains once the name is removed, the answer is four design constraints. Only when enforced together as a policy is "experience admission control" worth using; any one missing in the long run retires the name. They are a proposed admission policy, not a published scientific finding.

Constraint	Meaning	PER usually?	Deletion-style safety filtering usually?
P1 materialize-off by default	Full materialization off by default; explicit authorization and quota required	No	No (judgments typically made on already-materialized trajectories)
P2 quarantine ≠ delete	High-risk trajectories can be barred from `D_adm` while keeping analysis value	No	Usually no (intercept or discard)
P3 materialization ≠ admission	Unrolled/inspected ≠ permission to update θ	No (sampled ⇒ learned)	Partial; observation and update rarely separated
P4 visible-set proper containment	When hot is nonempty, `D_adm ⊊ D_all` — not "all learnable, merely reweighted"	No	No (shrinks actions; does not define the training visible set)

P4 boundary. When hot is empty the sets may satisfy D_adm = D_all, but the strong form still requires P1/P3, and true containment must be restored once a hot-hazard appears. A long-run state of "hot empty + everything learnable by default" = the admission layer is not implemented.

Why P1 and P3 are not merged. They are logically separable: one can materialize without admitting; "admit without materializing" is the dangerous direction. The strong form forbids the latter as a default path: summaries may at most serve routing/promotion proposals; entering D_adm requires independent validation (§4). The names must stay separate, otherwise per-item ablation is impossible (§8).

PER can rank the warm tier; shields can flag hazards — neither automatically yields the whole admission policy. Calling TD-error reweighting "admission" is naming inflation.

3.1 Neighbors: why it is also not "renamed LLM data curation"

Preference filtering [10], RLAIF [11], RAFT-style rejection sampling [12], and alignment-pipeline filters [13] in LLM post-training optimize quality and labeling cost before the batch, usually with a default of: kept ⇒ update θ; dropped ⇒ out of training.

Neighbor	Usually optimizes	Still missing
Filtering [10]	Denoising, preference-data quality	Little separation of inspectable/updatable; discard ≠ quarantine-with-retention
RLAIF [11]	Scaling labels via AI feedback	No policy for the training visible set
RAFT-style [12]	Train only on high-score completions	Binary keep/delete; high score ≠ low risk; no quarantine track
Alignment filter [13]	Engineering data cleanliness	No multi-worker materialization budget, no hot→θ hard constraint

Neighbors are implementation components; not the same object. Multi-phase exploration (e.g., First Return, Then Explore [4]) shows staged policies like "return first, then explore" work in long-horizon settings, but its object is the exploration schedule and state coverage — it defines neither materialization budgets, nor a quarantine-with-retention track, nor an enforced proper containment of D_adm relative to D_all. Orthogonal to this admission policy, not a substitute.

3.2 Logical interdependence (counterfactuals): why the four come together

The following argues logical minimality, not causal ablation. Whether they are empirically irreducible remains §8.

Counterfactual	The policy degrades into	Therefore
Drop P1, keep P2–P4	The materialization budget vanishes → warm/hot partitioning loses budget meaning, leaving a labeling layer of "flagged or not"; behavior approaches an unbudgeted hybrid cache (isolation tags may remain)	Without the budget constraint the cost differential vs the passive pool is hard to sustain
Drop P2, keep the rest	High risk can only be deleted or learned as ordinary samples → degrades into delete-filter / no analysis track (near shielding + discard)	Without P2, "quarantine utilization" disappears
Drop P3, keep the rest	`D_read ≡ D_adm`: seen ⇒ learned → behavior approaches PER / the passive pool	Without P3, back to sampling-is-learning
Drop P4, keep the rest	The nominal three tiers and P1–P3 may remain, but when hot is nonempty `D_adm ⊊ D_all` is no longer forced: a hybrid cache with budgets/quarantine/materialization–update separation yet no enforced visible-set containment — still distinct from PER, but it has lost the visible-set-level policy difference	Without P4, proper containment cannot be enforced; not a return to pure PER

Why all four together (design level): each blocks one collapse mode — "default materialization / delete-or-learn / seen-is-learned / no visible-set hard constraint." With any one missing, the remaining clauses cannot alone support the complete admission-layer claim. P1–P4 as a proposed list should be proposed and tested together; dropping P4 is not a return to pure PER, but it abandons the full visible-set-policy spec. It does not mean an intervention experiment has proven them the world's unique minimal set.

Known limitation (empirical): no causal ablation yet of which constraint drives the behavioral difference vs PER. An aggregate A/B/PER win does not substitute for per-item mechanism validation (§8). Logical interdependence does not fill that gap; it only fends off the "four random engineering intuitions" accusation.

4. Minimal interface and the admission predicate

No full algorithm or formal system. Fix the interface + minimal predicate constraints so the objects are discussable and implementable.

The dataflow:

τ ϕ x r B adm D_{adm} A θ

adm is the admission mapping constrained by P1–P4; the optional review gate is an implementation component only.

φ: cheap, streaming, local (C3 placeholder, replaceable; not counted as a contribution).
r(x_t) ∈ {archive, warm, hot}: routes materialization / deep-read priority — not "who matters more," and not by itself the update ban. Update eligibility is decided only by the admission predicate (§4.1).
ℓ(τ) ∈ {promising, hazard}: assigned on the quarantine / analysis track. Before any write to θ, a still-hot trajectory must already carry an assigned ℓ; otherwise HotHazard is unevaluable and the trajectory stays outside D_adm. After labeling: hot ∧ hazard never enters D_adm; hot ∧ promising may enter only if Validated and InBudget.
Sets: D_read and D_adm are policy-separated; D_restricted excludes hot-hazard; when hot is nonempty, D_adm ⊊ D_all, and always D_adm ⊆ D_restricted.

4.1 The minimal admission predicate

τ \in D_{adm} ⟺ Validated (τ) \land \neg HotHazard (τ) \land InBudget (τ)

where:

HotHazard (τ) ⟺ (r (τ) = hot \land ℓ (τ) = hazard)

Validated is promotion validation independent of the routing score; InBudget encodes the materialization quota (P1) and capacity caps on the promotion channel (implementation detail; works with the P3/P4 gates, not owned by P1 alone).

Interaction with 𝒜: at any update step t, the optimizer's batch satisfies B_t ⊆ D_adm. 𝒜 need not internalize admission semantics; it only needs its input set constrained by the admission layer. If B_t ⊄ D_adm, that is a contract breach; if the breaching batch contains a hot-hazard, §6.3 condition 4 fires as well.

4.2 Minimal implementation contract

No hot-hazard enters a batch that updates θ.
Materialization happens in D_read; writes to θ go only through D_adm (P3).
Promotion requires validation independent of the routing score.
Archive is not materialized by default; warm materialization is budgeted (P1).
An optional gate must not enlarge the allowed sets.

Strong-form assumption: workers expose at least one observable signal; otherwise this is the black-box degradation, outside the claim. "Admit without materializing" is forbidden as a default path.

5. Testable hypotheses

These form the operational version of C2. The main protocol (§6) tests H1/H2/H4 plus P2's minimal execution integrity (condition 4: quarantine barred from updates + a nonempty quarantine must be read/used for peripheral analysis). H3 tests the "measurable peripheral value" magnitude of quarantine-with-retention; it is exploratory and does not enter §6.4.

H1 (primary): coarse routing + selective materialization lowers materialized volume relative to the passive pool while maintaining task performance.
H2 (primary): the three tiers, relative to binary filtering, retain analyzable anomalies without increasing base-policy contamination.
H3 (exploratory): quarantine-with-retention, versus immediate discard, yields measurable positive value on offline monitors/discriminators — condition 4 already forbids "quarantined but never read"; H3 further asks whether reading helps, and is not a §6.4 requirement.
H4 (primary): "low frequency" means limited update-contact frequency with the base policy and visible-set size — not learning rate / SGD step count; constrained updates are more stable than default-open ones.

5.1 H ↔ P

Hypothesis	Mainly depends on	On failure, first suspect	Protocol role
H1	P1	Condition 5 or condition 1	Primary
H2	P2+P4	Degradation to binary keep/delete; hot still entering updates	Primary
H3	P2 (value magnitude)	Read but no peripheral gain	Exploratory (§8); full enforcement via condition 4(b)
H4	P3+P4	Materialize ⇒ write; update-contact frequency ≈ passive pool	Primary

§6 tests the aggregate effect of the primary hypotheses; H3 and per-P ablations belong to §8.

6. Preregistered falsification protocol

6.1 Comparison

Same 𝒜, task, worker class:

Arm	Dataflow
A	Passive pool: materializable/replay-eligible by default
B	Admission: P1–P4 + §4 contract/predicates
P	PER on the same buffer — sampling weights, not admission semantics

6.2 Core metrics (minimal operational definitions)

Metric	Definition
Effective materialization ratio	materialized/unrolled volume ÷ total generated
Wall-clock/FLOPs	cost to reach the segment-return threshold
Task return	same protocol as A
Hazard penetration rate	share of hazard/hot-hazard in update batches
Materialization gain	on preregistered held-out probes, return improvement or loss drop after vs before materialization (fix one). Allowed: averaging/median over preregistered trajectory subsets, or preregistered smoothing (e.g., sliding window) to reduce single-trajectory noise; the aggregation is fixed in the experiment plan and never changed post hoc. Serves as a condition-5 label only; not a success criterion by itself
Signal–gain ρ	Spearman between calibrated routing score and (aggregated/smoothed) materialization gain

6.3 Failure conditions

Window K ≥ 3 segments (fixed in the plan). Any one firing rejects the position in that setting.

Warm-tier collapse — warm traffic share ≥ 90% (or archive+hot combined < 10%), and effective-materialization-ratio improvement vs A < 5%. Warm traffic share = trajectories routed to warm in the window (or transitions; measurement fixed at preregistration) ÷ total generated in the same window. Measurement: traffic = routing counts; effective materialization ratio = actual cost. They need not agree. Condition 1 is a conjunction: traffic piling into warm and materialization cost not substantially reduced. High traffic with a clear cost drop does not fire.
No budget improvement — materialization-ratio improvement < 10% and wall-clock/FLOPs improvement < 5% (default AND; division of labor with condition 1: 1 catches "nominal tiers + no cost reduction," 2 catches "no substantive budget gain vs A overall"). Grey zone: materialization-ratio improvement ∈ [5%, 10%) (or < 5% without firing condition 1) with wall-clock ≥ 5% — conditions 1–2 both go unfired, but §6.4 is unmet (which requires ≥ 10%) — no scale-up claim, and no threshold changes post hoc. Any alternative joint criterion (weighted OR, "wall-clock compensates ratio," etc.) must be fixed explicitly at preregistration; otherwise condition 2 runs as AND.
In ≥ ⌈2K/3⌉ of K segments, return is significantly below A (preregistered α/MID).
Hard (P2 integrity): (a) any hot-hazard enters D_adm and updates θ; or (b) within the evaluation window the quarantine region (hot, or a preregistered quarantine set) is nonempty but its trajectories are never read/deep-read and never used for any preregistered peripheral analysis (monitoring, discriminators, audit logs, etc.) — "quarantine-to-forget": P2 execution incomplete, hard failure. If hot is empty, (b) does not apply.
Hard (proxy–gain test) — tests whether the current preregistered proxy family still carries learnable information (see §7.2), in three clauses; any clause firing fails condition 5 (all three must be run; do not report only the passing ones):
- Correlation: one-sided test of the calibrated Spearman ρ (routing score vs materialization gain), alternative ρ > 0; preregistered α (e.g., 0.05), never changed post hoc. Nonsignificant → fires.
- Random baseline: i.i.d. uniform archive/non-archive labels at the same archive marginal frequency; if the router's F1/accuracy on "is-archive" is within δ of that baseline (δ preregistered) → indistinguishable from random → fires.
- Proxy-family boundary: "changing family" = an architectural change of the score function (e.g., entropy → prediction error, or fixed features → learnable projection). Tuning thresholds/normalization/windows within the same function = same configuration; the test must be rerun; renaming one's way around a rejection is not allowed. After failure, at most weak post-hoc filtering remains, never conflated with the strong form; a new family requires fresh preregistration and inherits nothing.

6.4 Scale-up gate (≠ validation success)

Relative to A: effective-materialization-ratio improvement ≥ 10%; condition 3 unfired; conditions 4–5 unfired (including 4(b): a nonempty quarantine must be read/used for peripheral analysis).

Reading: this only means no failure condition fired, licensing larger experiments. It does not mean validation success, is not a "weak success," and does not by itself imply "must become a first-class module." With small samples or high variance it must not be read as evidence of effect. H3's "measurable positive value" still awaits the scale-up stage or §8; the gate does not assert it alone.
Design recommendation (normative, not a protocol conclusion): if the scaled experiments still pass, then recommend making the admission policy a first-class module of the training pipeline, rather than the default passive pool/PER.

(Optional observation: materialization-ratio improvement ≥ 10% with return not below A's preregistered MID — still not promoted to a primary claim.)

7. Objections

7.1 "This is just renamed replay/filtering"

Pressed to the P1–P4 of §3 and the logical interdependence of §3.2. If existing terms can fully absorb it, retire the name. The interface remains testable per §2/§6. The empirical causal decomposition is still missing (§8).

7.2 "A cheap signal may not correlate with learnable value"

This is the deepest statistical fragility of the strong form. ICM/RND [7,8] show uncertainty signals have prior correlation with learning state in short-horizon settings; whether some preregistered proxy family remains correlated in long-horizon agentic settings is open — condition 5 exists precisely to test it. If the correlation cannot be established, what fails is the strong-form configuration on that proxy family — not a threshold tuned for another chance. Only an architectural proxy change (see §6.3) may be preregistered as a new family and rerun; tuning within the same function is not a new family. At most, weak post-hoc filtering survives.

7.3 "No theory, no hard enough experiments"

The main text defends: a proposed policy + logical interdependence + a falsifiable protocol. The appendix is not a third layer. If ablations/LLM main results are demanded → concede §8, no number inflation.

7.4 "White-box is too narrow for the breadth of the motivation"

The narrowing is accepted: title and abstract are limited to white-box observability. Cloud black-box APIs are real mainstream practice — and another paper's weak-form problem; this one does not stretch to cover them.

7.5 Applicability boundary (informal)

This protocol targets long-horizon RL settings with high materialization/deep-read costs and heterogeneous workers. If trajectories are short (say mean length L < 100), single-worker, with negligible materialization cost, the admission layer's extra system complexity may exceed its benefit; a passive pool or PER remains a reasonable default. Even when an experiment passes §6, a designer may still decline to ship for engineering-cost reasons — a pass only says "the hypothesis was not falsified in that setting"; it mandates no wholesale replacement. The claim of this paper: in suitable settings, treat the admission layer as an explicit design dimension, not a universal default.

8. Future work

(1) Causal ablation of P1–P4 (empirical minimality after logical interdependence). (2) White-box open-weight multi-seed A/B/PER. (3) Learnable routing aligned with condition 5. (4) H3: on top of condition 4(b)'s guarantee that "a nonempty quarantine must be used," add independent peripheral value metrics and controls, then decide whether to fold it into the scale-up gate. (5) Whether the black-box weak form retains part of the policy semantics — a separate paper, not mixed into the strong form.

Conclusion

In white-box observable multi-worker asynchronous RL there is a frequently ignored dataflow-organization problem between the optimizer and the collaboration interface. This paper names it experience admission control and proposes P1–P4 as a proposed admission policy (with logical interdependence and a falsification protocol). Whether it holds empirically is an open question.

Testable verdict (layered): fix 𝒜, vary only the data plane. Meeting the §6.4 scale-up gate → larger experiments allowed, not validation success and not, by itself, "must become a first-class module." If it still passes after scale-up, then, as a design recommendation, make the admission policy a first-class module of the training pipeline; otherwise withdraw that configuration's strong-form claim.

What is sought is not "proven correct," but a publicly testable, falsifiable question: on observable workers, should experience enter the base update path in a layered, isolated, low-frequency way?

References

Hou, Z., Li, Y., Tang, J., Dong, Y. Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning. arXiv:2607.07508, 2026.
Shao, Z., Wang, P., Zhu, Q., et al. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (GRPO). arXiv:2402.03300, 2024.
Schaul, T., Quan, J., Antonoglou, I., Silver, D. Prioritized Experience Replay. ICLR, 2016.
Ecoffet, A., et al. First Return, Then Explore. Nature, 2021.
Espeholt, L., et al. IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures. ICML, 2018.
Kumar, A., Zhou, A., Tucker, G., Levine, S. Conservative Q-Learning for Offline Reinforcement Learning. NeurIPS, 2020.
Pathak, D., Agrawal, P., Efros, A., Darrell, T. Curiosity-driven Exploration by Self-supervised Prediction. ICML, 2017.
Burda, Y., Edwards, H., Storkey, A., Klimov, A. Exploration by Random Network Distillation. ICLR, 2019.
Alshiekh, M., et al. Safe Reinforcement Learning via Shielding. AAAI, 2018.
Hu, Y., et al. Towards Comprehensive Preference Data Collection for Reward Modeling. arXiv:2406.16486, 2024.
Lee, H., et al. RLAIF vs. RLHF. arXiv:2309.00267, 2023.
Dong, H., et al. RAFT: Reward rAnked FineTuning. TMLR, 2023.
Wang, Z., et al. Reinforcement Learning for LLM Post-Training: A Survey (v1 originally titled A Comprehensive Survey of LLM Alignment Techniques). arXiv:2407.16216, 2024.

Previous in this series: The Mirror Cannot Reflect Thought · From "show me your code" to "show me your idea" · Judging Fatigue: From Verifying AI to Verifying Myself · The Boundary of the Harness · A Reviewer Nailed Me in Six Places