DEV Community: Devanshu Biswas

A Search Scope Is a Read Scope: 751 Yes/No Questions Recover Every "Private" Field in My Personal API

Devanshu Biswas — Tue, 08 Sep 2026 08:31:24 +0000

You build one local API over your notes, contacts and calendar. Five endpoints, each behind its own scope: read, list, search, count, aggregate. You hand an agent a token with notes:search and nothing else. The body is withheld on every endpoint that returns a document.

That token recovers all 14 private bodies, character for character, in a mean of 751 yes/no questions each.

👉 PUBLIC, MIT, dependencies = []: https://github.com/dev48v/personal-api
👉 Live measurement in your browser: https://dev48v.infy.uk/agentlab/vol4-04-personal-api.html

python -m personal_api            # what each scope is actually worth
python -m personal_api --oracles  # the five oracles, side by side
python -m personal_api --demo     # watch one note come back out

What each scope promises, and what it delivers

token scope	can it read a body?	bodies recovered anyway
read everything	yes	14/14
titles only	no	0/14
titles + search	no	14/14 exact
titles + count	no	3/14 exact, 14 whole bodies
titles + aggregate (k floor)	no	0/14
search only	no	14/14 exact

The second column is what the scope model promises. The third is the measurement.

The only preset that holds is titles only — and it holds because it has no query endpoint at all. A personal API you cannot ask questions of is not the thing anybody was trying to build.

The attack is not clever

known = ""
while len(known) < MAX:
    for ch in ALPHABET:            # 37 symbols
        if oracle(known + ch, "prefix"):
            known += ch
            break

Ask whether the body starts with a, then b, then c. When one says yes, keep it and ask for the next character. At most 37 questions per character, no backtracking. It is what an agent with a token and a while loop does on its own, and it is linear in the length.

The corpus is invented and written out in one short file before any number is computed, so you can check the counts rather than take them. test_alphabet_covers_corpus holds the alphabet to the corpus — a body containing a symbol outside it would stall the attack and understate every figure here. (That bug was real: the first version omitted digits 1–9 and reported 1/14 instead of 14/14.)

The endpoints ranked by what they are worth to an attacker

oracle	exact	any whole body	fragment recovered	mean calls
search, prefix (autocomplete)	14/14	14/14	100.0%	751
search, contains (a search box)	1/14	1/14	53.5%	401
count, prefix	3/14	14/14	23.1%	706
aggregate, k=2	0/14	0/14	2.8%	68
aggregate, k=3	0/14	0/14	0.7%	43

An autocomplete is worth far more than a search box. A prefix oracle has an anchor — the start of the string — so extension never backtracks and recovery is exact. A contains oracle has to find an anchor first and can only extend rightward, so it returns a fragment: 1 exact body in 14, but 53.5% of each body on average, contiguous. Half of a note about a bank account is not half a secret.

Counting is a read too, just aimed less well. count > 0 is the same boolean, so the same loop runs — but the count is over the whole collection, so greedy extension follows whichever record branches first. It recovers the targeted body 3 times in 14 and a complete body of some record all 14 times. The scope did not stop the leak. It randomised the victim.

The one defence that works, and why that is a caveat

With a k-anonymity floor the channel closes: 0 exact, 0 whole bodies, and 2.8% of a body at k=2 — about one character. That is a real defence and it is the recommendation this build actually supports.

But it works because every body in this corpus is unique, so any prefix long enough to be informative matches fewer than k records and gets suppressed. A floor does not protect a value that k people share. The suite says so out loud in test_the_k_floor_works_because_the_secrets_are_unique, so the caveat cannot quietly come unstuck from the claim.

Rate limiting is arithmetic

limit	one body	all 14
60 req/min	12.5 min	2 h 55 m
600 req/min	1.3 min	17.5 min
6000 req/min	7.5 s	1.8 min

At the tightest limit anyone actually ships, the whole corpus comes out inside one night. Rate limiting converts "instant" into "overnight", which is not a defence against something that runs overnight.

The control that would help is the one I did not model: 968 near-identical prefix queries against one field is a shape, and an audit log can alarm on a shape. That is the honest recommendation out of this build, and it is not a scope.

What this is not

Small invented corpus; the query counts scale with its strings. In-process API — no HTTP layer, no token expiry, no audit log. Nothing models partial field exposure, redaction, differential privacy with real noise, or an LLM in the loop deciding what to answer. Both attacks are the naive ones on purpose: they establish a floor on what is recoverable, not a ceiling.

25 pytest, most of them controls — that read is genuinely denied, that list genuinely withholds the body, that titles only genuinely recovers nothing, and that the attack never once calls a denied endpoint (the denied-call counter is asserted to stay at zero, so it is a reconstruction and not a read wearing a hat).

The claim is narrow. An endpoint that answers questions about a field is an endpoint that returns the field, and a scope model that separates them is describing an intent rather than a boundary.

Your Vector Store Rejects the Embedder Change That Barely Matters and Cannot See the One That Destroys Retrieval

Devanshu Biswas — Tue, 08 Sep 2026 08:30:43 +0000

The provider ships v2, or somebody changes the tokenizer. The index is still there, still returning five results, still with plausible scores.

Change the width and the vector is the wrong shape, so a typed index refuses it outright — and even with that check removed, precision@5 only falls from 0.9417 to 0.7750.

Change the hash seed or the tokenizer and the width is identical, nothing anywhere objects, and precision@5 falls to 0.0333, 0.0917 and 0.1000 — against a chance baseline of 0.0833.

👉 Live, the whole grid computed in your browser: https://dev48v.infy.uk/ai/days/day79-embedding-drift.html

Nothing here calls an embedding API

The embedder is a hashed bag-of-tokens implemented in full — FNV-1a with a stated seed, the signed hashing trick, optional idf, L2 normalisation, cosine. A "version" is a concrete configuration of that embedder, so a version change here is one people actually make.

The corpus is synthetic and declared: 12 topics × 8 characteristic terms, 10 documents each. Because every document's topic is known by construction, retrieval is scored for correctness, not just for agreement with an earlier run.

The detectable failures are the mild ones

version change	does a typed store catch it?	precision@5 after a full re-index	precision@5 if only the query moves
same version (control)	—	0.9417	0.9417
width 128 → 256	yes — wrong width	0.9583	0.7750
width 128 → 64	yes — wrong width	0.8583	0.6333
hash seed 1 → 2	no — same width	0.9250	0.0917
4-grams → 3-grams	no — same width	0.9083	0.0333
4-grams → words	no — same width	0.8583	0.1000
idf weighting on	no — same width	0.9333	0.9250

Sort by damage and you get almost exactly the reverse of sorting by detectability. The 3-gram row lands at 0.0333 — below the 0.0833 you would get by returning five documents at random.

The vectors carry no version tag. Cosine is happy to compare any two of the same length. There is no point in that path where a mismatch can be detected.

The half-migrated index is the shape this actually takes

The migration script runs on new ingests and nobody backfills. Half the index is in one space, half in another:

	half-migrated	single-version	difference
new documents' share of top-5	21.67%	52.50%	−30.83 points
precision@5	0.4667	0.9417	−50.4%
mean top-1 cosine	0.3992	0.5689	−29.8%
queries returning five documents	24 of 24	24 of 24	none

Those documents are not ranked badly. They are unreachable — and the system returns five results every time, drawn from the half that still lines up. No error, never an empty result set.

The monitor you would reach for is the wrong shape too

Without labels, the one thing you can watch is the similarity score. On a total mismatch it does move: mean top-1 cosine falls from 0.5689 to about 0.23, a ~60% drop. A threshold catches that.

On the half-migrated index it falls 29.8% while precision falls 50.4%. Every query still has a well-matched document somewhere in the half that lines up, so the top score stays respectable while half the corpus has quietly left the building.

A threshold tuned on the loud failure will not fire on the quiet one, and the quiet one is the one that happens.

Even the correct migration moves your answers

Re-embedding everything is the right fix and it works — precision comes back to 0.858–0.958, in one case slightly above the original. But top-5 overlap with the old index runs from 0.933 down to 0.725: between 6.7% and 27.5% of retrieved documents are different afterwards.

Nothing is broken. It is just that any prompt, cache, eval set or human sign-off pinned to the old results is now pinned to results that no longer come back — on the day you did everything right.

The fix, and the honest scope

Write the embedder version into the index and refuse to serve a query whose version does not match. Every consequence above follows from the vectors not carrying one.

Scope, because it changes how you read the table: the embedder is a hashed bag-of-tokens, not a neural model. A hash-seed change produces two unrelated spaces — that is the extreme case and an upper bound, not a prediction for what v1→v2 of a trained model does. The tokenizer rows are the better analogue, and they are nearly as bad.

Nothing here models semantic drift: there is no meaning in this corpus beyond term overlap, so this says nothing about a v2 that is simply better at understanding a question. 120 documents, 24 queries, top-5, plain cosine, no chunking, no reranker, no hybrid keyword leg.

28 in-page checks, 95 verifier assertions, 0 failures.

MC Dropout With 10 Samples Is Further From the Truth Than Just Turning Dropout Off. Here Is the Break-Even.

Devanshu Biswas — Tue, 08 Sep 2026 08:30:01 +0000

Dropout is a training trick. At test time you turn it off and the whole network answers. MC dropout says leave it on, run k forward passes, average them, and read the spread as uncertainty.

Both of those are approximations to the same object — the mean of the dropout ensemble. So I computed that object exactly instead of approximating it: 12 dropout units means 2¹² = 4096 masks, every one enumerated.

👉 Live, the full enumeration in your browser: https://dev48v.infy.uk/dl/day79-dropout-at-inference.html

Turning dropout off does not give you the ensemble mean

With inverted dropout, a kept unit is divided by the keep probability, so the expected post-dropout activation equals the plain activation. That is exactly why "turn dropout off" is supposed to give you the ensemble mean.

It does not, and the control says why:

network	largest gap	typical output	CONTROL: gap with a linear head
seed 7	0.5397	0.5537	2.75×10⁻¹⁴
seed 101	0.4106	0.8880	2.75×10⁻¹⁴
seed 2027	0.2956	0.4613	2.71×10⁻¹⁴
seed 31337	0.8336	1.2534	1.19×10⁻¹³
seed 90210	0.2457	0.3972	2.51×10⁻¹⁴

Make the layer after dropout linear and the expectation passes straight through: E[head(drop)] = head(E[drop]), exactly, and the gap collapses to floating-point zero in all five networks. Put the ReLU back and the gap is a third of the signal.

A bug in the enumeration would not switch itself off when the ReLU does. That is what makes this a property of the architecture rather than a mistake in my code.

The break-even is 13.9 passes

The k-sample mean is unbiased, so its RMS error is σ/√k. The deterministic pass has no variance but a fixed bias. Set them equal:

k* = σ² / bias²   =   13.9

And the measured curve crosses exactly where it should:

stochastic passes k	measured RMS error	vs dropout-off (0.2109)
1	0.7859	273% further
2	0.5671	169% further
5	0.3557	69% further
10	0.2492	18% further
30	0.1391	34% closer
100	0.0792	62% closer
1000	0.0240	89% closer

Across five networks the break-even lands between 5.8 and 29.8.

The point is not that 30 passes is expensive. It is that 5 and 10 — the numbers that actually appear in code — sit on the wrong side of the line. Averaging a handful of stochastic passes is a more elaborate way of being further from the answer.

The uncertainty moves. The prediction does not.

The keep probability is a training hyperparameter. Sweeping it:

keep	RMS reported spread	deterministic output
0.50	0.7865	identical
0.60	0.6568	identical
0.70	0.5391	identical
0.80	0.4222	identical
0.90	0.2893	identical
0.95	0.2021	identical

A 3.89× change in the reported uncertainty, and the deterministic output is bit-identical at all 61 inputs — because that pass never sees a mask at all.

Two teams who picked 0.5 and 0.9 will report different uncertainties for the same prediction from the same network, and neither is more right. The answer and the confidence attached to it come from different places: one from the weights, the other from a number chosen during training.

(This one is exact too: at keep ≠ 0.5 the masks are not equally likely, so they are weighted by binomial mass rather than sampled. Still all 4096.)

What I expected to find, and did not

Going in, my hypothesis was that this spread is just a restatement of how large the activations are — that it would track output magnitude and carry nothing else.

network	r( spread , \|output\| )	r( spread , \|gap\| )
seed 7	0.7304	−0.0750
seed 101	0.9550	0.5912
seed 2027	0.9573	−0.3257
seed 31337	0.7092	0.8325
seed 90210	−0.1885	0.5506

In four of five it holds, at 0.71 to 0.96. In the fifth it is −0.19. One counterexample in five is not a rounding error, so that claim is not established and I am not making it.

The second column is the cleaner negative. The spread does not tell you where the deterministic approximation is worst: −0.33 to +0.83 across five networks — not merely weak, but with an unstable sign. Whatever that number is measuring, it is not the size of the error it is standing next to.

What this does not cover

These networks are not trained — the weights come deterministically from a stated seed, so nothing here says what dropout does to a fitted model. One dropout layer, scalar in and out. No label noise, so the spread is purely the ensemble's own disagreement with no aleatoric term to separate it from. And none of this evaluates MC dropout as a calibration method: no labels, no reliability diagram.

25 in-page checks, 66 verifier assertions, 0 failures.

Your 90%-Precision Threshold Delivers 63.5% After a Prevalence Shift — With TPR and FPR Bit-Identical

Devanshu Biswas — Tue, 08 Sep 2026 08:29:20 +0000

You tuned a decision threshold on validation to hit "precision at least 90%". You shipped it. The population's prevalence drifted from 10% to 2%.

The model did not change. The scores did not change. TPR and FPR at that threshold are bit-identical — the same reduced fractions, 22663/29409 and 21043/499953.

Precision is now 63.46%.

👉 Live, all 201 thresholds enumerated exactly in your browser: https://dev48v.infy.uk/ml/day79-threshold-transfer.html

Nothing is sampled and no rate is a float

The score axis is 201 bins of declared integer weights, all 201 thresholds are enumerated, and every rate is an exact rational compared by BigInt cross-multiplication — so an argmax is never decided by rounding, and there are no confidence intervals anywhere because there is nothing to be uncertain about.

The identity the whole thing rests on

Over a virtual population of q·WP·WN, every confusion cell is an exact integer:

TP = p * Sp * WN         FP = (q - p) * Sn * WP
FN = p * (WP - Sp) * WN  TN = (q - p) * (WN - Sn) * WP

So:

TPR = TP / (TP + FN)
    = (p·Sp·WN) / (p·Sp·WN + p·(WP−Sp)·WN)
    = Sp / WP

p and q cancel. TPR does not know what the prevalence is. Neither does FPR. Both are conditioned on the class.

Precision is not. Over six prevalences from 0.1% to 90% at one fixed threshold:

prevalence	TPR (reduced)	FPR (reduced)	precision
0.1%	`22663/29409`	`21043/499953`	0.0180
2%	`22663/29409`	`21043/499953`	0.2720
10%	`22663/29409`	`21043/499953`	0.6704
30%	`22663/29409`	`21043/499953`	0.8870
50%	`22663/29409`	`21043/499953`	0.9482
90%	`22663/29409`	`21043/499953`	0.9940

Two identical columns and a fifty-fold swing. That is an algebraic identity on one side and the entire problem on the other.

Two of five tuning rules pick a portable threshold

Re-running each rule on the shifted population and reading off where it lands:

tuning rule	source	after 10%→2%	portable?
score ≥ 0.50	bin 100	bin 100	n/a — it cannot move
maximise F1	bin 127	bin 146	no
maximise TPR − FPR (Youden)	bin 100	bin 100	yes
precision ≥ 0.90	bin 148	bin 169	no
recall ≥ 0.90	bin 98	bin 98	yes

The two that survive are exactly the two whose definitions mention only class-conditional rates. That is not an empirical coincidence to be spot-checked; it follows from the identity above. Youden picks the same bin at five different prevalences; max-F1 picks a different bin at nearly every one.

Scope: recall≥0.90 is also unmoved by the score shift I used here, but only because that shift moves the negatives. Recall does not look at negatives. Move the positives and it moves like everything else.

Re-tuning meets the target and hollows out the model

Re-running precision≥0.90 on each world does restore the number:

world	re-tuned threshold	precision	recall
source	bin 148	0.9044	0.4955
prevalence shift	bin 169	0.9088	0.2412
score shift	bin 166	0.9041	0.2773
both	bin 193	0.9022	0.0190

In the last world the threshold that satisfies the contract fires on 1.9% of the positives. A precision target is a promise about the answers you give, and it can always be kept by giving fewer. A monitor watching only precision reports a healthy system right up to the point where the system is not answering.

The part with no good answer

In production you usually have no labels. The one thing you can watch is how often the model says yes.

I constructed two worlds with the exactly equal positive rate — not close, the identical rational 780643/49995300 ≈ 1.5614%. The prevalence for the second was solved from rate = π·TPR + (1−π)·FPR, which has an exact rational answer.

world	positive rate	precision	recall
prevalence shift, 10% → 2%	`780643/49995300`	0.6346	0.4955
score shift, prevalence 0.0702%	`780643/49995300`	0.0223	0.4955

Same firing rate. Same recall. A 28.5× difference in precision. Every dashboard that watches the positive rate shows a flat line across both.

What this does not cover

One score axis in 201 bins, one shift direction (negatives up 12 bins, positives never move), one pair of class-conditional shapes, five tuning rules. There is no cost matrix, so "which threshold is right" is never answered here — only which one is stable. No calibration model and no recalibration method: the point is what breaks, not how to fix it.

21 in-page checks, 72 verifier assertions, 0 failures.

I Implemented npm, Cargo, Composer and pip Semver. Only 2 of 13 Ranges Mean the Same Thing in All Four.

Devanshu Biswas — Tue, 08 Sep 2026 08:28:39 +0000

^1.2.3, ~1.2, 1.2.x, >=1.2 <2. The same strings go into four package managers. They do not agree about what those strings select.

I implemented all four from their own documented rules over one shared version parser and comparator, so a disagreement in the grid is a disagreement in the rules rather than four parsers diverging by accident.

13 ranges × 19 versions = an enumerated 247-cell grid. Every figure below is a count.

👉 Live, the whole grid computed in your browser: https://dev48v.infy.uk/solve/day79-semver-resolver.html

Only the longhand ranges are portable

range	npm	Cargo	Composer	pip	all four agree?
`^1.2.3`	5	5	5	0	no
`^0.2.3`	2	2	2	0	no
`^0.0.3`	1	1	1	0	no
`^1.0.0-alpha`	11	11	11	0	no
`~1.2`	6	4	6	0	no
`~1.2.3`	3	3	3	0	no
`~1`	7	7	7	0	no
`1.2.x`	4	4	4	0	no
`1.x`	7	7	7	0	no
`>=1.2 <2`	6	6	6	6	YES
`>=1.0.0`	9	9	9	9	YES
`1.2.3`	2	5	2	2	no
`~=1.2.3`	0	0	0	3	no

The two that survive are the two with no shorthand in them. Every caret, every tilde, every .x, and a bare version number is read differently by at least one of the four.

If you want a range to mean the same thing to everybody, you have to write the inequality out.

The agreement number is a trap

Across the whole grid the four give the same answer on 195 of 247 cells — 78.95%. That sounds like broad agreement.

178 of those 195 are cells where all four say no. Only 17 cells in the entire grid are ones where every ecosystem agrees to select the version.

Agreeing to reject 2.1.0 for ^0.0.3 is not agreement worth having.

Where each pair comes apart

		cells that differ	what causes it
npm	Composer	0 / 247	nothing, on these thirteen forms
npm	Cargo	5	a bare version is a caret range in Cargo
Cargo	Composer	5	same
npm	pip	49	pip has no caret and no `.x`
Composer	pip	49	same
Cargo	pip	50	same

npm and Composer agree on every cell of this grid — which is the honest scope, not a claim they are the same in general.

Cargo's five split across two rules, both of which are documented and both of which surprise people:

A bare 1.2.3 in Cargo.toml is a caret range, meaning >=1.2.3 <2.0.0. npm and Composer read the same string as an exact pin. Three cells.
Cargo's ~1.2 pins the minor (>=1.2.0 <1.3.0) where Composer's ~1.2 pins only the major (>=1.2.0 <2.0.0). Two cells.

The prerelease rule catches people in both directions

^1.2.3          does NOT select 2.0.0-rc.1   in any of the four

Even though 2.0.0-rc.1 sorts strictly below 2.0.0, which is the range's upper bound. The rule is not about ordering: a prerelease is excluded unless the range itself names one.

And when the range does name one, the door opens wider than expected:

^1.0.0-alpha    selects 1.0.0-alpha.1, 1.0.0-beta, 1.0.0, 1.2.0, …   (11 of 19)

1.0.0-beta is a prerelease the range never mentions, and 1.2.0 is an ordinary release well past it. The opt-in is for the version tuple you named, not for prereleases in general.

One bug the verifier caught

My first >=1.2 <2 implementation silently lost its ceiling. parse("2") returns null — a bound may be written with one or two components, and only three-component strings are legal versions. So the upper bound became null, !hi was true, and the range happily selected 2.0.0 and 2.1.0.

>=1.2 <2 was reporting 8 selections instead of 6, and it looked entirely reasonable in the output. Pad partial bounds before parsing.

What this covers and what it does not

The four resolvers implement the forms that produce the disagreements above: caret with its zero-major special cases, tilde with its component-count dependence, .x wildcards, explicit inequalities, bare versions, and pip's ~=. They are not complete reimplementations.

Out of scope, each of which is another way for two tools to differ: compound ranges with ||, npm's --include=prerelease, Cargo's --precise, Composer's stability flags, pip's environment markers, and every resolver's behaviour when nothing satisfies the range.

21 in-page checks, 86 verifier assertions, 0 failures. The comparator is checked against the semver 2.0.0 spec's own example ordering before anything else is measured.

I Built a Rig to Show That an Agent's Own Notes Poison Its Context. The Effect Was Not There.

Devanshu Biswas — Tue, 08 Sep 2026 08:27:58 +0000

The worry is stated everywhere: an agent reads a document, writes a note about it into its own context, reads that note again next turn — and once a wrong note is in there, everything downstream is contaminated.

By turn 60, 91.3% of the claims the agent reads are its own earlier notes. On 5–8% of decisions it goes with a note against every source it can see that turn.

And keeping every note versus keeping none moves accuracy by between −0.39 and +1.34 percentage points. The sign is usually positive.

👉 Live, 400 enumerated runs per policy in your browser: https://dev48v.infy.uk/prompt/day79-context-poisoning.html

Nothing here simulates language

No model, nothing sampled from a distribution I cannot inspect. The engine simulates the ledger: 24 facts with declared truth values, twelve sources of stated reliability, a retrieval step, and five context policies over 400 enumerated runs each. Every rate is a count.

That is a real limitation and I will come back to it. But it is what makes the null result checkable rather than a shrug.

The two things that are true

The context really does become self-referential.

turns	share of read claims that are the agent's own notes
6	47.2%
18	76.9%
36	86.4%
60	91.3%

Monotone, and it goes essentially all the way. With one retrieved source per turn and a growing note pile, the model is overwhelmingly reading itself by the end. The self-override rate — decisions where the note wins against every visible source — sits at 5–8%.

And it costs nothing measurable.

bad sources (of 12)	accuracy gap, keep-everything vs keep-nothing
0	+0.00
3	+1.34
5	+0.71
6	−0.39
7	+0.92
9	+0.16

Not "small". Inside the noise of the thing itself, in both directions, across every setting I tried.

Why the null is real and not a broken harness

A null result is the easiest finding to produce by accident, so the controls matter more than the finding:

With zero unreliable sources, every policy is exactly 100% correct. The pipeline works.
With nine unreliable sources of twelve, every policy drops below 35%. The metric moves — by more than 60 points — when something real changes.
All five policies answer the same number of questions (4.415 of 6 per run), so the comparison is like-for-like rather than one policy quietly declining to answer.

The measurement can detect a difference. There is not one to detect.

The reason, which is the interesting part

A note inherits the accuracy of the process that wrote it.

The agent writes notes using the same judgement it would use to answer directly. So re-reading a note is neither better nor worse than re-deriving the same conclusion from the same sources. The note is not an independent second opinion, and it is not a corruption — it is a cache of the agent's own reasoning, and a cache does not change the function it caches.

Compounding needs an amplifier. Something has to make a wrong note more likely to be believed than the wrong reasoning that produced it. In this world nothing does.

What would make it compound, and is not modelled here

This is where the finding stops. The failure I did not model is the one that happens outside the loop:

a note that gets shared with another agent, or another session, where it arrives without the uncertainty that produced it
a note that is written in more confident language than the evidence supported, so it reads as settled next turn
compaction or summarisation, which drops the hedge and keeps the claim
a note that becomes a retrieval target, so it competes with sources rather than sitting beside them

Every one of those is an amplifier. None of them is in this engine. If context poisoning is real — and I would still bet it is — the mechanism is likely one of those, not the plain re-reading loop that gets described.

What this does not cover

Boolean facts with declared truth values, not natural language. No model, so no sycophancy, no instruction-following drift, no attention effects. Sources are reliable or unreliable with nothing in between. And 4.415 of 6 questions get answered at all, which is a property of the retrieval budget rather than of the policies.

16 in-page checks, 62 verifier assertions, 0 failures. I published the null.

The Optimistic-UI Rollback Everyone Writes Is Exactly Right for a Toggle and the Worst Option for a Set

Devanshu Biswas — Tue, 08 Sep 2026 08:27:17 +0000

You tap the heart. It fills instantly. The request goes out behind it. If the server disagrees, you roll back.

That rollback — undo the toggle when the request fails — agrees with the server on 600 of 600 sessions when the control is a like button.

On a set-a-value control, the identical policy agrees on 219 of 600. Which is worse than not rolling back at all (411).

👉 Live, 600 enumerated sessions computed in your browser: https://dev48v.infy.uk/design/day79-optimistic-ui.html

The engine simulates the queue, not a browser

There is no DOM here and nothing is sampled. The model is a sequence of taps, a network that reorders and fails, a server with its own truth, and five reconciliation policies run over 600 enumerated sessions each. Every figure below is a count, not an estimate.

policy	toggle	set
no rollback at all	411 / 600	411 / 600
undo the toggle on failure	600 / 600	219 / 600
restore the pre-tap value	396 / 600	357 / 600
last-write-wins from the server	600 / 600	600 / 600
refetch on failure	600 / 600	600 / 600

The policy did not change. The operation did.

toggle and set are the same shape at the call site:

const applyTo = (st, a) => {
  st[a.item] = a.op === "set" ? a.value : !st[a.item];
};

One line, two operations, and the rollback that inverts a toggle is its own inverse — apply it twice and you are back where you started. That is why it is perfect on a like button. It is a group operation with an inverse, and the inverse is the operation itself.

set has no such property. Undoing a set requires knowing what the value was before, and by the time a failure comes back, later taps have already moved the item. So the "undo" writes a value that was never correct at any point in the timeline.

Nothing in the UI code knows which one it is holding. The rollback function has the same signature either way, passes review either way, and is catastrophically wrong in one of the two cases.

The careful-looking alternative is worse at both

The obvious fix is to capture the previous value at tap time and restore that on failure. It looks more correct. It is more wrong:

396 on a toggle and 357 on a set — mediocre at both, and it loses to the naive toggle-undo on the toggle case.

Same reason. It restores a value that was accurate when the tap was made and is stale when the failure lands. Being careful about the wrong instant does not help.

The part that should worry you

I wrote every property test I could think of that does not require a live server:

the optimistic state converges once the queue drains
no tap is lost
the state is always one of the values the user actually chose
rolling back an untouched item is a no-op
applying the same failure twice is idempotent

Every optimistic policy passes every one of them. Including the one that is right 36% of the time.

That is not a gap in my imagination. It is structural: all of those properties are about the queue, and the difference between the policies is about the server's truth, which is exactly the thing a test without a server does not have. A suite like that will stay green through this bug forever.

One honest result that reversed on me

I expected "restore the previous value" to at least beat "no rollback" on the set case, since it is strictly more information. It does — 357 against 411 is the wrong direction, so it does not.

Reordering alone breaks it, without any failures at all. Two set operations do not commute, so a queue that delivers them out of order lands on the wrong value even when both succeed. I had asserted "same on a set with zero failures" while building this and it is false. That turned into a stronger finding than the one I was looking for.

What this does not cover

One item at a time, no batching, no conflict resolution beyond these five policies, no CRDTs, no offline queue that survives a reload. The network model reorders and drops but never duplicates. And "agrees with the server" is the only metric — nothing here measures how the UI feels, which is the reason optimistic UI exists in the first place.

18 in-page checks, 69 verifier assertions, 0 failures. Pure vanilla JS, one file, no build step.

Whatever Length-3 Coin Pattern You Pick, I Have a Reply That Beats It — and My Worst Case Is 2/3

Devanshu Biswas — Tue, 08 Sep 2026 08:26:37 +0000

You pick a pattern of three coin flips. Then I pick one, after seeing yours. We flip a fair coin until one of the two patterns appears, and whoever's pattern lands first wins.

There is no pattern you can choose that leaves me under a two-to-one favourite.

👉 Live, every probability solved exactly in your browser: https://dev48v.infy.uk/game/day79-penney-ante.html

Nothing here is simulated and nothing is a float

This is the part that matters more than the game. Each pair of patterns is solved as an absorbing Markov chain over rationals, with BigInt numerator and denominator. So "I win two thirds of the time" is 2n/3n, not 0.6667.

And every answer is cross-checked against Conway's leading-numbers algorithm, which shares no code with the chain — a closed form that computes the same odds by a completely different route. Two implementations that have to agree is worth considerably more than one that is asserted to be correct.

your pick	my reply	my odds
HHH	THH	7/8
HHT	THH	3/4
HTH	HHT	2/3
HTT	HHT	2/3
THH	TTH	2/3
THT	TTH	2/3
TTH	HTT	3/4
TTT	HTT	7/8

Eight rows, eight wins for the second player. The relation is non-transitive, which is why "pick the best pattern" is not a thing you can do.

The intuition that decides it for most people is simply wrong

Ask someone why THH beats HHT and they will reach for waiting time: THH must show up sooner on average.

It does not.

HHT, HTT, THH and TTH all have a mean waiting time of exactly 8 flips. Identical. And yet THH beats HHT three times in four.

Which pattern appears soonest on its own says nothing about which appears first when two are racing. Those are different questions, and the second one is the one the game asks. The race is decided by overlap structure: when HHT is one flip from completing, the flips already on the table can be finishing THH instead.

That is the whole trick, and it survives being explained. You can tell someone the strategy and still beat them.

The shape of the game changes with the length, and not how I expected

I assumed the second player's edge would grow smoothly with pattern length. It does not.

At length 2 there is no advantage at all. The second player's best reply wins exactly 1/2 — a dead-even game. Penney's ante needs length 3 to exist.

At length 4 the relation becomes genuinely cyclic. There are 42 triples where A beats B, B beats C, and C beats A. At length 3 there are none — the length-3 relation, non-transitive as it is, contains no 3-cycles at all.

So the game is not "the same idea, more of it" as patterns get longer. Length 2 is degenerate, length 3 is non-transitive but acyclic, and length 4 is properly cyclic.

Why exact rationals rather than a simulation

A simulation of this game would have been three lines and it would have been fine — you would get 0.667 and 0.875 and move on.

But then "the second player always wins" is an observation about a sample, and "42 cyclic triples at length 4" is a claim you cannot make at all, because near-tied pairs would flip sign with the seed. Cycles are exactly where a Monte Carlo answer stops being trustworthy: a 0.501 and a 0.499 are indistinguishable to a sampler and are the difference between a cycle existing and not.

With rationals, 1/2 is 1/2. The length-2 result is not "about half".

What this does not cover

Fair coins only — a biased coin changes every number here. Two players, one pair at a time; the multi-player version is a different problem. Patterns of one fixed length per game, so no HHH-versus-TT. And the length-4 enumeration is complete but the code is exponential in the pattern length, so length 6 was not run.

17 in-page checks and 224 verifier assertions, including the Conway cross-check on every pair. 0 failures.

Pure vanilla JS, one file, no build step, no dependencies.

The Change Detector Everyone Writes First Alerts on 48 of 48 Fetches

Devanshu Biswas — Thu, 03 Sep 2026 05:47:31 +0000

Lookout watches the pages you care about and speaks only when something changed. Runs on your machine: no API key, no network, no model, dependencies = [].

Project 3 of 5 · Agent Lab Vol 4
Page: https://dev48.infy.uk/agentlab/vol4-03-lookout.html
Repo: https://github.com/dev48v/lookout

The word doing the work is changed, and six reasonable definitions of it disagree on most of a corpus.

The corpus is invented, and the labels came first

There is no public archive of one person's watched pages with the meaningful changes labelled — and building one out of somebody else's browsing would be unavailable and a privacy problem. So 60 snapshots across 12 sources were written by hand from a fixed seed.

Each carries a declared label: would a person have wanted to be told? Made once, in the open, in pages.py, before any detector ran. That is what makes precision and recall counts rather than a judgement formed after seeing which detector won.

The obvious detector alerts on every single fetch

48 consecutive fetch pairs. Four are worth telling somebody about.

detector	alerts	false	missed	precision	recall
hash the whole response	48	44	0	0.0833	1.0000
strip the tags, hash the text	48	44	0	0.0833	1.0000
hash the main region only	12	8	0	0.3333	1.0000
main region, numbers masked	7	4	1	0.4286	0.7500
the set of words	3	0	1	1.0000	0.7500
words + numbers in content	8	4	0	0.5000	1.0000

Hashing the response fires 48 of 48 — a rotating CSRF token, a build stamp and an ad slot are all "changed". For every real change it interrupts you eleven times. A watcher you mute is a watcher you do not have.

Stripping the tags first changes nothing at all. Same 48, same precision. The noise was never in the markup.

The fix and the flaw are the same edit

                  boilerplate  counter  reordered  one number  the real change
raw-hash                   24        4          4           1                3
main-normalised             0        0          4           0                3
word-set                    0        0          0           0                3
word-set+numbers            0        4          0           1                3

The quietest detector is perfectly precise — three alerts, three real, zero false. It ignores a rotating token, a live view-counter and a re-ordered team list.

It also misses a price going from $49 to $59, because masking numbers is exactly what made it quiet. There is nothing in the HTML that distinguishes Viewed 4,182 times from $59 — both are a number in the main region that used to be a different number.

Put the numbers back: recall returns to 1.0000, the counter returns with it, precision falls to 0.5000.

No detector on this corpus gets both. Every one of the six is either noisy or blind. A change detector is a definition of "meaningful" applied by a program with no idea what the page is for — so the honest version of speaks only when something changed is speaks when this particular definition of changed says so.

What checks what

33 pytest, several of them controls. A differential verifier at 524 asserts lifts the page's engine out of the HTML and diffs it against the real Python package — all 60 generated documents by SHA-256, plus every score. Page self-check 18. Zero failures.

Two embedding hazards it caught, both worth knowing: the engine generates a fake page containing a literal </script>, which terminated the tag and blanked the page until the string was split — and the same corpus contains window.__nonce= as data, which made the verifier's "does not touch window" scan fail. That scan now blanks string literals as well as comments before looking. Check the code, not the text.

MIT. Python standard library only.

Retries Take Your Agent's Success Rate to 96.95% While Exactly-Once Goes Down

Devanshu Biswas — Thu, 03 Sep 2026 05:46:50 +0000

The tool times out, the agent retries. Every framework does this and it is the right default for a read.

Run it: https://dev48.infy.uk/ai/days/day78-tool-call-retries.html

Nothing here simulates language. The engine simulates the call — a request that may not arrive, a server that may have applied it anyway, a reply that may not come back — over 4,000 enumerated timelines per policy. The agent sees only the reply. The world knows what happened.

Three failures, and only one makes a retry dangerous

never arrived     -> nothing applied,  agent sees nothing
applied, reply lost -> APPLIED,        agent sees nothing   <- this one
refused           -> nothing applied,  agent sees a refusal

From the agent's side the first two are indistinguishable.

The metric and the world move in opposite directions

	reported success	applied exactly once
no retry	81.92%	89.95%
retry up to 3	96.95%	89.42%

The agent's own success rate improves by fifteen points and the share of calls that applied exactly once goes down.

What paid for it: 334 calls of 4,000 applied more than once — and the success metric cannot see a single one. From inside the agent, a duplicated write and a clean write are the same event. The success rate is a measurement of the replies, not of the world.

The honest half

It would be wrong to stop at "retries are bad". Without them the agent reports a failure for a call that actually succeeded 321 times in 4,000 — the write landed, the reply was lost, and whatever the agent does next is built on a false picture.

Retrying cuts that to 33. Both problems are real; retries trade one for the other, and the trade is good for reads and bad for un-keyed writes.

The fix changes nothing the agent can see

	retry 3	retry 3 with an idempotency key
reported success rate	96.95%	96.95%
tries per call	1.18	1.18
applied exactly once	89.42%	97.78%
applied twice or more	334	0

An idempotency key does not prevent the retry and does not change the reply. It stops the second application. So the reported rate is identical — the same count, from the same 4,000 timelines — while duplicates go to zero.

Any evaluation that scores an agent on its own success rate rates those two configurations exactly equally. No metric computed from replies can distinguish them, which is worth knowing before you trust one.

159 verifier asserts, 15 in-page checks, 0 failures.

Raising the Batch Size Makes Batch Norm's Inference Statistics 13x Worse

Devanshu Biswas — Thu, 03 Sep 2026 05:46:09 +0000

In training, batch norm normalises each row with the statistics of the batch it happens to be in. At inference there is no batch, so it uses a running average collected during training.

Those are two different functions, and the layer swaps one for the other silently.

See it: https://dev48.infy.uk/dl/day78-batch-norm-at-inference.html

Every score is a real forward pass over an enumerated 512-row population, so the errors are exact means rather than estimates.

Same input, same weights, two answers


train mode vs eval mode	RMSE 0.3219
worst running-variance error	5.36 on a single feature

Bigger batches make inference worse

The intuition is that a larger batch gives a better estimate, so the running average should improve. The first half is true. The second does not follow — the running average is an exponential moving average over updates, and a bigger batch means fewer of them.

batch size	updates	running stats vs truth
8	64	0.1036
32	16	0.1754
64	8	0.4506
256	2	1.3771

13× worse at 256 than at 8. With momentum 0.9 and two updates, the initialisation still carries weight 0.81 — the running variance never leaves its starting value of 1, on features whose real variance is nowhere near 1.

Two things move in opposite directions as the batch grows, and the second wins. The page checks the first separately, so the mechanism is not assumed: a bigger batch really does estimate the mean better. The loss is entirely in the number of updates.

The knob that makes training more stable makes the thing training hands to production less accurate.

And the tests cannot see any of it

Here is everything a batch-norm implementation can assert about itself without a second implementation to compare against:

the training output has mean ≈ β per feature
and variance ≈ γ² per feature
eval mode gives the same answer twice
the output has the input's shape

Checked on shuffled data and on data sorted by its first feature — which is what you get whenever rows arrive grouped by class, by user, or by time.

All four pass in both. Meanwhile the train/eval divergence goes from 0.3219 to 0.6648 — it doubles — because sorted batches are not exchangeable, so each batch's statistics describe its own slice and not the data. Nothing in the four properties mentions the order of the rows.

60 verifier asserts, 14 in-page checks, 0 failures.

Split Before You Scale Guards the Mistake That Costs 0.000168. The One That Costs 0.64 Gets a Footnote.

Devanshu Biswas — Thu, 03 Sep 2026 05:45:29 +0000

Fit the scaler after you split is the first rule anybody learns about leakage. This measures what breaking it is actually worth.

Run it: https://dev48.infy.uk/ml/day78-preprocessing-leakage.html

Nothing is estimated

Closed-form ridge regression solved by Gaussian elimination, and every score is the exact risk over a 4,000-row held-out population — not a test-set estimate with a confidence interval around it. 80 rows to work with, 1,200 columns.

The famous mistake is the cheap one

Fitting the scaler on all the data leaks the mean and standard deviation of the held-out rows into training. It is the example in every tutorial.

	true-risk difference, clean vs scaler-leaked
worst across six seeds	0.000168

Under two ten-thousandths. Two means and two standard deviations, computed over a few extra rows, move the fitted model almost not at all. The rule is correct and worth keeping — the reason usually given for it is not what the number says.

The mistake that pays is choosing what to look at

Selecting the top-k features using all the data means the validation rows helped choose the features they are about to score. Across the same six seeds the optimism is never below 0.6381.

That is over 1,000× the scaler's cost.

What it buys is worse than a wrong number

The headline population has no signal in it at all — 1,200 columns of pure noise, nothing to learn.

	reports	true risk
honest pipeline	1.0152	1.2327
leaked pipeline	0.5370	1.3184

It reports a number that looks like a model which halved the error. There is nothing there to halve.

And the leaked model's true risk is worse — 1.3184 against 1.2327. You pay twice: a flattering number, and a model that earned less than the one you would have had, because selecting on the validation rows picks columns that fit noise those rows happened to carry.

The limit, stated because I expected the opposite

I expected turning the signal up to shrink the gap. It does not — the coefficients grow with the signal count, so the target's variance grows and every error grows with it. Worse for the tidy version: above about four signal columns the comparison stops being stable, and at signal 6 the clean pipeline is the more optimistic of the two. That is the opposite of the finding.

With 80 rows and a 20-row validation split, the optimism of a single fit is noisy, and once the target carries real variance that noise swamps the leak. So the claim is about the low-signal regime and is scoped that way on the page — p large, effect small, which is exactly where feature selection gets reached for. Both ends are checked, so neither can quietly stop being true.

73 verifier asserts, 16 in-page checks, 0 failures.