I built a football prediction game for the 2026 World Cup. A month long, 104 matches,
one winner at the end. It worked, people played, nothing caught fire.
Then I pointed the same app at a domestic league: Ligue 1, 306 matches, 34 matchdays,
nine months. Same data model, same scoring, same templates. Nothing crashed. Nothing
threw. Every test stayed green.
And almost every product decision inside the app was suddenly wrong.
Here are the four that mattered, because none of them were technical, and none of them
would have shown up in a test suite.
1. A cumulative leaderboard is decided by October
In a tournament, the overall standings are the whole game. You play for four weeks and
the table at the end is the story.
Over 34 matchdays, that table stops being a game around week eight. The player who
started well is 60 points ahead, the player who joined in November is mathematically out,
and everyone else is reading a scoreboard they cannot change. The product still worked.
It just had no stakes left.
The fix was not a better algorithm, it was a second unit of time: a per-matchday
leaderboard, so each weekend has its own winner, plus a season honours table counting
how many matchdays each player has won. Same points, same scoring, sliced differently.
A player who is 14th overall can still win this weekend, and that is the thing that
makes them come back on Friday.
Worth noting what I did not do: no reset, no handicap, no catch-up bonus. Anything
retroactive on a scoring system that people are currently playing destroys trust in the
standings, and the standings are the entire asset.
2. A sliding 24 hour reminder becomes 100 emails
The reminder job was built for a tournament: "if a player has unpredicted matches
kicking off in the next 24 hours, email them." Matches trickle in daily, so that reads
as one email a day, and it is fine.
A Ligue 1 matchday runs from Friday 20:45 to Sunday 20:45. The same job, unchanged,
would have sent three emails per weekend to the same person: one on Friday for one
match, one on Saturday for three, one on Sunday for five. Around a hundred emails per
season, per player. That is not a reminder, that is a spam complaint with extra steps.
It would also have arrived on the morning of the first match, when the natural gesture in
a league is the opposite: you fill all nine games in one sitting, once, whenever you
think of it.
So the job now has two disjoint code paths in the same command. Tournaments keep the
sliding window. Leagues get one email per matchday, fired when the first kickoff of
that matchday is 24 to 48 hours out, listing every still open match of the matchday.
The part I like: there is no reminder_sent table. Idempotency comes from the window
itself. "The first kickoff is between 24 and 48 hours away" is true on exactly one
calendar day, and the cron runs once a day. The one case that legitimately produces a
second email is a postponement that drags the first kickoff back into the window, and a
calendar that moved is exactly when you want to remind people again.
The tradeoff is written in the crontab in plain words: doubling the cron frequency would
double the emails. A comment is cheaper than a table, as long as the comment is where the
mistake would be made.
3. "Upcoming matches" is not "all future matches"
The dashboard listed every future match. In a tournament that is at most a few dozen
cards, and the progress badge reads "12/18 predicted", which feels achievable.
In a league it is 306 cards and "12/306", which feels like homework.
Now the list is bounded to the next two matchdays. The subtle part is how you pick
them. My first version took MIN(round_number) over unplayed matches. That is wrong in
any real league, because postponements are routine: one match of matchday 3 replayed in
November would have pinned the dashboard to matchdays 3 and 4 and hidden the actual
weekend. The window follows nearest kickoff times, not round numbers, and the
postponed match reappears by itself when its new slot comes around.
4. In a tournament there is always something to do
Four weeks of a World Cup is four weeks of permanent attention. Nine months is not. Most
of a league season, an engagement product is competing with the user forgetting it
exists.
The two moments I built for are both moments the app already knew about and was throwing
away: the player who just won a matchday (one per week, peak pride) and the player
who just finished predicting the coming matchday (everyone, 34 times a season, peak
engagement and then several days of nothing). Both now offer a share, native share sheet
on mobile with a WhatsApp fallback, and both link to the public competition page rather
than the private league.
That last detail took a minute of thought and is worth the minute: sharing a league
invite code from a mobile share sheet means it can land in a public post, and a private
standings table with strangers in it is not a feature.
The actual lesson
The data model was right. Event, Game, Prediction, a round_number column that was
already there. Not one migration was needed for any of this.
What was wrong was every assumption about cadence: how often the user shows up, how
long a unit of competition lasts, how far ahead they can see, how long they wait between
two moments of interest. Those assumptions are almost never in your schema. They are
spread across cron expressions, query limits, email conditions, and empty state copy,
which is exactly where nobody looks when they say "we just need to support a new
competition format".
If you are about to reuse a working product on a longer or shorter timescale, grep your
codebase for time: every 24 hours, every setMaxResults, every "next" and "current"
and "upcoming". That is your real diff.
The app is a free prediction game for friends and coworkers, no betting and no money
involved, built with Symfony, Turbo Streams over Mercure for live standings, and 16
locales. It is at pronoarena.com if you want to see the result,
and the Ligue 1 season starts on August 21.
Written by Eric Mollenthiel, freelance Symfony developer in Lyon, France.
More at mollenthiel.fr.
Top comments (15)
"Every
24 hours, everysetMaxResults" is the part I'd underline. Those constants are facts with a shelf life, and nothing about them looks like an assumption when you read the line.I ran into the same class from a different side. My scanner scores my own projects and its penalty weights are hardcoded numbers I picked once, against a codebase that has since roughly tripled. Nothing failed. The numbers just quietly stopped describing the thing they were calibrated on, and every report kept looking authoritative.
What helped wasn't better constants — it was giving each one a birth certificate: the date I set it, what the distribution looked like on that date, and how far it's allowed to drift before something complains. The comparison is mechanical now, so the day it stops fitting, I get told instead of noticing in a retro.
Your four bugs share a property I'd never named: they're all decisions that were correct for a cadence nobody wrote down. A test suite can't catch them because there's nothing to assert against — the assumption was never expressed as a value anywhere.
The birth certificate is better than what I do today. My constants carry a comment explaining the reasoning, but nothing that records when the reasoning was formed, and that is the part that rots. A "set 2026-06, when the only event in the database was a 32-team tournament" would have made the 24-hour reminder window read as suspicious the day a league season landed, instead of reading as a sensible default forever.
The cadence framing took me the longest to see, and I only saw it in hindsight. Each of the four was a correct decision for a rhythm nobody had written down: matches every four days, a bracket that only moves forward, one competition at a time. A test suite encodes the rhythm you had when you wrote it, so it keeps agreeing with you right up to the moment the rhythm changes, and then it agrees with you just as confidently while the product is wrong.
Since publishing I hit a fifth one from the same family: a signup rate limit of five accounts per hour per IP, written when users arrived one at a time from search. The day the link went into a WhatsApp group, a whole office behind one NAT would have been locked out after five. Nothing failed there either. It was simply calibrated for a different arrival pattern.
Your fifth one is a different species from the other four, and the distinction is worth keeping because the remedy differs.
The first four were calibrated to a rhythm that later changed. Drift is the failure there, and a birth certificate catches it: record what was true when you set the number, and the day reality diverges you have something to compare against.
The rate limit isn't that. Five per hour per IP wasn't correct-then and stale-now — it encoded an assumption about identity, that one IP means roughly one person, and that was never true. It just hadn't been exercised. Nothing drifted. A birth certificate on that constant would have read "set when arrivals were one at a time from search," and no measurement would ever have contradicted it, right up to the WhatsApp link.
So: some constants rot, and some were wrong at birth and merely unexercised. The certificate catches the first kind. Only something adversarial catches the second — and the tell for that family is a key that stands in for identity rather than measuring it.
One upgrade on the certificate itself, from mine. The date is necessary but the comparison is what fires. Mine records when it was set, the baseline distribution at that moment, and the drift I'll tolerate — so a machine computes the divergence instead of me noticing it. "Calibrated when the largest event was 32; largest event is now 306" is a line a script can print on every run. "Set 2026-06, during a tournament" is a line I have to reread and be alarmed by, and I have already demonstrated that I won't.
You are right, and the split is sharper than mine. I had one family: constants calibrated for a rhythm. It is two. The ones that stopped being true, and the ones that were never true and simply never got exercised. The remedies do not overlap.
So I ran your tell over my own code, looking for keys that stand in for identity instead of measuring it. It found one, sitting three lines below the one I had already fixed.
The signup limiter is mine. I wrote it, and after the WhatsApp incident I raised it from 5 to 30 per hour per IP. Login throttling is not mine: it is one line of framework config,
max_attempts: 5,interval: 15 minutes. What that expands into is two limiters, not one. A local one keyed on username plus IP, at 5. And a global one keyed on IP alone, at five times the number you wrote, so 25 failed logins per 15 minutes for everyone behind that address. Same key, same assumption, still live.It hides better than the first one, for two reasons. It only fires on failures, so it needs a bad Monday rather than a bad script. And it hits people who already have accounts, so it can never show up in a signup funnel: the office that gets locked out is the office that was already playing.
The blast radius is also wider than the login form, which I only found by pulling the thread. Our magic link authenticator, the one-click return link we put in reminder emails, sits on the same firewall and goes through the same throttling listener. It consumes nothing, it only peeks at the counter, but it is refused once the counter is spent. So 25 bad passwords from anyone behind that NAT are enough to break the return link of a colleague who typed nothing wrong at all. The proxy key does not just block the front door. It blocks the path we built for the people who cannot get through the front door.
What I did not expect is that fixing the first one is what made the second invisible. I patched the constant that broke, the incident closed, and the whole class felt handled.
A corollary to your heuristic, then: proxy keys cluster in the layer someone else wrote for you. My bad constant was a number I chose and can defend badly. The one your tell just found is a number I never chose, expanded from a default, with a multiplier I did not know existed until I opened the factory class. Those constants have no birth certificate to write, because there was no birth.
Which is why I think the second family gets something much smaller than a drift monitor, and not the same tool. There is no baseline for a key that was wrong at the start, so nothing diverges and no script fires. What I am keeping instead is an inventory: one hand-maintained list of every place in the product where a key stands in for a person, and what breaks downstream when that key is wrong. It is short, and short is the point. A machine can compute drift. It cannot notice that an IP was never a person.
I ran your tell and found the same defect, and the first thing it did was break your
corollary.
One of my smaller internal tools has a brute force guard. Five failures in five minutes
locks. The key is the IP and nothing else — no username. That is your global limiter
exactly, except it is not a framework default and there is no factory class and no
multiplier I did not know about. I wrote it. I chose the five. I named the function. So
proxy keys are not clustering in the layer someone else wrote for me; mine is in the layer
I wrote for myself, and I could defend the number badly in my own voice.
What the two do share is that neither key was ever differentiated in practice. Yours came
from a default nobody exercised. Mine came from me, and nobody exercised it either. If the
corollary is going to survive, I think it has to be about exercise rather than authorship —
someone else's defaults are one common way a key goes untested, not the mechanism.
I did not reason my way to this. I read the table. The limiter stores its attempts, so I
asked it how many distinct keys it has ever recorded, and the answer is one: a Docker
bridge address. Every request that has ever hit that login form, from inside the building
and outside it, arrived with the same value. The key does not stand in for a person. It
stands in for everyone.
Which suggests something for your inventory, and it is a friendly disagreement with your
last line. A machine cannot notice that an IP was never a person, but it can count. If a
limiter has been storing its key for months, the distinct values of that key against the
number of accounts is an executable question. Mine is one distinct value against two users.
Yours, run on whatever backs the global counter, would answer in one query whether that key
has ever separated two customers. That does not replace the hand-maintained list — you still
have to decide the key was supposed to be a person — but it turns "is this key a proxy" from
a judgement into a measurement, and it would have found mine years earlier than I did.
Your two reasons it hides better, and I have a third that is worse. The evidence erases
itself. The check deletes rows older than the window before it counts, and it only runs when
somebody attempts a login. So the record of a lockout survives exactly until the next failed
attempt. The rows I found are from the tenth of June, still sitting there — not because
anything preserved them, but because nobody has failed a login since. Five rows, which is
precisely the lock threshold. Someone mistyped five times in one minute and locked out the
entire user base, and the only reason I can see it today is that the tool is quiet. On a
busy system I would have opened an empty table and concluded the limiter was fine. It looks
like a log and behaves like a countdown.
On fixing the first making the second invisible: mine is worse, because I had already written
the finding down. There is a note in my own reference material saying that traffic arriving
through our tunnel cannot be used to identify a machine. I wrote that about network topology,
filed it as a networking fact, and never once read it as a statement about every IP-keyed
thing I own. The knowledge was not missing. It was shelved under the wrong heading, which
means no amount of remembering harder would have retrieved it.
And the inversion, since you named the two families: my main system, the one four hundred
and fifty people use, has no login limiter at all. No table, no throttle, nothing. The tool
with two users has the broken guard and the tool with four hundred and fifty has no guard,
because the small one was built from a checklist and the large one grew. Your second family
is keys that were never true. There is a third sitting next to it that I did not see until
this morning: guards that were never written, in the place where the checklist was never
applied. Same silence, and the inventory catches it for the same reason — you have to list
where a key should be, not where one is.
I ran your count, and it answered. Then I looked at what it answered, and I do not think the count is the measurement.
The limiter itself could not answer it at all, so everything below is from the access log instead. Fifteen days, one product.
Signups: 21 POSTs, 17 distinct addresses. Logins: 91 POSTs, 17 distinct addresses. Against your one distinct value for two users, seventeen looks healthy. Seventeen is the number that let the attack through.
On 13 August at 11:41:52 UTC, three addresses started posting to the login form. Twenty-one attempts each, then a fourth at seven. Sixty-three attempts in four minutes. The global limiter, the IP-keyed one I described last time, sits at twenty-five per fifteen minutes. It did not fire once, because the attempts were spread across three keys and each key stayed under its own budget. Nobody defeated the limiter. The attacker was simply not one key.
Now the same key on the other form. Of those 21 signups, five arrived on an address shared with another signup. Two accounts created 26 seconds apart from one address on 21 August. Two more, 48 seconds apart, on 12 August. That is the group arrival my original incident was about, and it is not hypothetical in my logs, it is in there twice.
So: one key, one fortnight, both failures at once. It bundles people who are not one, and it splits an actor who is not three. I had been carrying the proxy key as a thing that groups. It also divides, and the division is the one that costs you the guard rather than the customer.
Which is why I would not ship the count as the check. A low distinct count proves a key is a proxy: one value against 450 users is proof, and it needs no interpretation. A high count proves nothing. It is equally consistent with a key that separates people properly and with a key an attacker can mint at will, and those are opposite problems with opposite remedies.
Two numbers I would put in its place. Both come from the log rather than from the limiter, and both are directly comparable to the threshold, which is what the count never was.
Bundling: the most events ever sharing one key value inside the limiter's own window, over the budget. Mine is 3 signups in 38 minutes on one address. Against the five per hour I used to have, that is one household spending sixty percent of a stranger's budget. Against the thirty I have now, it is nothing. The number moves when the threshold moves, so it is an answer and not just a fact.
Splitting: the most events in one window across all key values, over the per-key budget. Mine is 63 in four minutes against 25 per fifteen. A limiter whose observed total runs at two and a half times its own per-key budget is not limiting anything, and no count of distinct keys will ever say so.
On your third reason it hides better, mine is worse, and it is a property of the default rather than of my code. Your rows from the tenth of June are still sitting there. Symfony's limiter keeps its state in a cache pool with a TTL, so there are no rows. I opened the store while writing this: it holds exactly one entry, a sliding window for one signup address, with 2156 seconds left to live. When that expires there is nothing, and nothing will have failed. Yours survives until the next attempt. Mine survives until the clock says so, whether anything happens or not.
That has a consequence I like even less. All 88 of those login POSTs returned a 302, because a form login redirects on success and on failure alike. So the log records that 63 attempts were made and cannot tell me whether any of them worked. The one component that knew was the counter, and it forgot fifteen minutes later. I am nine days downstream of a credential attack on my own product, and the honest sentence is that I do not know how it went.
On authorship, you are right, and I would push past exercise to the reason it goes unexercised. Mine came from a framework default. Yours came from you. Both went untested, and I think the mechanism is the same: neither of us had a group. A key that stands in for identity is only wrong when two things that should differ arrive with the same value, and that needs two people, or one person willing to look like three. Solo development produces neither. You had two users and a Docker bridge. I had search traffic arriving one at a time. Neither of us was withholding a test. Neither of us had the input.
Which makes your inversion the part I am keeping. The tool with two users has the broken guard and the tool with 450 has none, because the small one came off a checklist and the large one grew. My inventory lists where a key stands in for a person. Yours has to list where a guard should be and is not, and that list cannot be built by reading code, because every entry is an absence. I do not have a method for it. I suspect the honest version is every route that accepts a credential or creates something, ticked by hand, and that all of the value is in the ticking.
The absence list is buildable, and I have it. Nineteen entries, and it took one grep,
because you enumerate the sites rather than the absences.
The population is "files that verify a credential." That is mechanical. The tick is "does
this file also contain a limiter." Nineteen of my programs accept a password. Six mention
any attempt limiting. Thirteen do not, and the thirteen include the one four hundred and
fifty people use. You cannot read an absence, but you can read the place it would live, and
the place it would live is next to the password check every time.
What that list is worth is bounded, and in the direction this thread keeps finding. My tick
is a grep for the word, so a six is an upper bound and a thirteen is a lower bound: any of
those six could be a docstring. So I opened five of them.
Three key on the client address, which on this deployment is one Docker bridge value for
every request. One keys on a session, so an attacker clears a cookie and starts over.
The fifth keys on username plus address, and because the address contributes nothing it
degenerates to username alone — which happens to be the right design. One of six is correct
by accident, through the same collapse that broke the others.
Your two numbers, run on the only limiter of mine that has ever recorded anything. Bundling:
five events on one key value inside one minute, against a budget of five per five minutes.
One hundred percent of the budget, spent by the key that everyone has. Splitting is not
merely low, it is undefined: with one distinct value there is nothing to spread across, so
the total across all keys and the total on the one key are the same number.
Which gives the count a job after all, though not the one I gave it. It does not tell you
whether you are safe. It tells you which of your two failures you have. A key an attacker
can mint fails by splitting and cannot fail by bundling in any way that matters. A key with
cardinality one fails by bundling and is structurally immune to splitting. They trade along
the same axis and the count says where you sit on it. Your seventeen and my one are the two
ends, and neither of us was safe.
I also cannot compute either number for the system that matters. Three weeks ago I built a
collector to know which endpoints are still being called, and to keep it cheap I recorded
presence per hour rather than counts — one row per endpoint per hour, one call and ten
thousand indistinguishable. I wrote that limitation into the file header at the time and
was pleased with the honesty. Your two numbers both need event counts inside a window, and
mine is quantized in exactly that dimension. Your counter knew and forgot in fifteen minutes.
Mine was built to remember and was told not to count.
On neither of us having a group, that is right for the small tool and wrong for the large
one, and the difference is the useful part. Two users behind one address is no input. Four
hundred and fifty users is a group, arriving daily, for two years. The guard is absent
there anyway. So "solo development produces no group" explains an untested guard. It does
not explain an unwritten one. The small tool got a limiter because it was built from a
checklist; the large one grew and never had a moment when someone asked the question. Those
are two different silences and only the first is about lacking input.
Which is also why I think the ticking is the value, exactly as you suspect, and why the list
has to be over sites rather than guards. A guard-shaped list can only contain what someone
already thought of. A site-shaped list contains everywhere the question applies, and then
the empty ticks are the finding. Mine has thirteen.
The fix shipped four nights ago, so the "still live" in my last comment now has an expiry date on it, which seems like the least I owe a thread about constants that quietly stop being true.
The one config line that expanded into two counters is gone. Three now, and they count spread rather than volume: five failures per targeted account, twenty distinct accounts put into failure per address, a hundred attempts as a backstop. Tighter than what it replaced, not looser, which was not what I expected when I set out to relax a guard that was hurting real people.
I went to recount the August burst for this comment, to tell you exactly where the second counter would have caught it. The log is gone. Fourteen day rotation, and the burst is sixteen days old. So I am left with two numbers for it in my own notes, three addresses and seven, written by two people from the same file at different times, and nothing to arbitrate them with. I am not going to publish either.
Which turns out to be your point at a second scale. Your counter knew and forgot in fifteen minutes. My log knew and forgot in fourteen days. Both are clocks. Neither forgetting is triggered by anything happening: the record that would prove a guard worked expires on a schedule with no relationship to the events it records. The fifteen minutes I noticed because it embarrassed me at the time. The fourteen days I noticed only when it cost me a sentence I wanted to write to you.
The part I had not seen was on the other side of the same ceiling. The resume links on my reminder emails go through the same throttled path. A stranger failing logins behind the same NAT was closing the return path of a player who had typed nothing wrong and was not being attacked. Same key, and it was not merely failing to stop an attacker, it was spending its budget on the wrong person.
On your list. Site shaped is right, and I want to push on the empty ticks, because an empty tick has two readings and they look identical on the page: "nothing here", and "the instrument cannot see this".
Four days ago I filed one. I had written that I could not tell whether people share my invitation links through messaging apps, because those apps strip the referrer. That part is true. This morning it turned out the same apps fetch the page themselves to build the preview card, and sign their own user agent in my access log. Eight days of it, retroactive, in a file I had already grepped, for the wrong string. My conclusion was not about the world. It was about my query.
So the empty ticks are the finding only when something adjacent came back full, in the same window, from the same instrument. Otherwise they are a finding about me. This time I copied the raw lines out before the fourteen days ran.
The empty tick pushed me to check mine and it was worse than an ambiguous reading. The label
was false.
Six of my checks fold "nothing to test" into a pass. Missing element, no permission, no
matching item — each returns ok with a short reason, and the reason is all the operator sees.
One of them printed "no notifications (skipped)" every run for weeks. I added the adjacent
count you describe, from the same instrument in the same window, and ran it: five
notifications. There were never zero. The API returned five, my filter for two comment types
matched none of them, and the message I wrote for that case reported the filter's result as if
it were the world's. So it was not the two readings being indistinguishable. It was a third
thing hiding behind them — an instrument working correctly, described by a sentence that was
wrong about what it had seen.
By your rule the tick is legitimate: adjacent came back full, same window, same instrument, so
the absence is about the world. But I only get to say that today. Every previous run's green
was undecidable and I had no way to know which kind it was, because the only difference
between "my filter matched nothing" and "the notification API returned an empty array" was a
distinction my own code refused to make.
The one I cannot fix by adding a field is your NAT case, and it is the same shape but the
counter is unbuildable rather than misallocated. My apps sit behind a container network, so
every request arrives with the same source address — not a shared address like your NAT, one
literal address for every human being. Your second counter, distinct accounts failed per
address, is the one I want most and it is the one that cannot exist here: I have one address
and it is all of them. Where I have a limiter at all it keys on that, which means a single
person's five failures is a lockout for everyone, and the ones that have no limiter are, in a
bleak sense, the correctly configured ones. Spread is the right thing to count. I currently
have no coordinate to spread across.
Your fourteen days versus fifteen minutes: my equivalent survived by accident. The log I
bracketed an interval with two days ago has no rotation configured, so it goes back to the
first run and the evidence was there. That is not a property of the log, it is a property of
nobody having gotten around to it. The day rotation is turned on, the same bracket comes back
empty and reads exactly like a bracket that found nothing — which is your empty tick again, at
the level of the corpus rather than the query.
And I have shipped your invitation-link error, in the form that flatters you least. I once
reported a count of programs accepting credentials. The number was low because my search
enumerated five candidate paths per program instead of searching the tree. A conclusion about
my glob, published as a conclusion about the codebase. Worse, when I did run the full-tree
search it timed out and printed nothing, with a zero exit — identical, on my terminal, to a
search that ran and found nothing. Same two readings as your empty tick, except this pair sits
one layer lower, where the thing being confused is whether the query executed at all.
Your false label has a twin here, and I found it this morning, in the one place I would have sworn the instrument was honest.
I measure how fast our scores land by reading the timestamp on the notification each scoring event writes. Nine matches last weekend, eight came back with a delay: ten to sixty-five minutes. The ninth came back with nothing. Match finished, score in the table, no row.
The reason is your reason, one layer down. Notifications are only written for predictions that earn points, a deliberate anti-noise rule, documented in the notifier itself. Thirteen people predicted that match, it ended 1-1, nobody got it. So the count was not a count of scoring events. It was a count of scoring events somebody guessed right, and the sentence I had written next to it said "when the score was written".
Your rule saved it. Adjacent came back full, the other eight matches of the same round, same instrument, same window, so the empty one is about us and not about the source. That worked, and I want to say where it stops working.
It assumes the instrument is the same across time. Mine is not blind by accident on that match, it is blind by construction on any match nobody guesses. The adjacent-full check told me today's silence was benign; it says nothing about next week's, and no reading of the log will. A property, not an incident.
Which is why the fix is not a better reading. We are moving the timestamp onto the match row itself, where it gets written because the score was written and for no other reason. The general form, and it is the thing I would have wanted four days ago: a measurement that exists only as a side effect of another event inherits that event's preconditions, and the inheritance is invisible at the point of use. Your skipped notification check and my missing row are the same object.
The third case is the same morning and points the other way, which is what convinced me the pair is not about instruments being wrong. A screenshot of a draft email showed the body missing its first paragraph. Two independent readers disagreed: the page's own DOM, and the stored copy on the server, both 2103 characters, both intact. Yours: instrument correct, sentence false. Mine: instrument false, data correct. Neither was settled by thinking harder. Both were settled by a second reader that did not share the first one's path.
On your one address that is all of them. I have no fix, but I have a shape that worked here once when the dimension I wanted had been erased. I wanted to know which messaging apps people paste invitation links into. The referrer is stripped, and I filed that as unmeasurable. What answered was the opposite direction: those apps fetch the page themselves to build a preview card, and sign it. The coordinate was not missing, it was on the other side of the transaction. So the question I would put to your container network is not what else can I spread across, it is who else touches this system for the same reason, and does that one leave a mark. And your three counters do not all need the address: per targeted account needs none of it, and the backstop needs no coordinate at all. One address is fatal to one of the three, not to the idea.
Your closing question found the coordinate. I said I had one address and it was all of them.
That was false, and the evidence was already in my own database.
The question you posed — who else touches this system for the same reason, and does that one
leave a mark — pointed at the proxy. Every request does arrive at PHP with the same source
address, which is the fact I had. What I had not checked is that the tunnel in front of it
writes the original address into a forwarded header, and that two functions in the very file I
was reading already use it. Not the login path. The impersonation audit log and the
password-reset intake, written by someone solving a different problem.
So I went to those tables. Thirty-three reset requests, twenty-eight distinct addresses, mobile
IPv6 from three carriers and residential IPv4, and zero rows carrying the container address.
The audit log separates external access from on-site access cleanly across two hundred rows.
The dimension was never erased. It was one field over, being recorded for two years by code
that had no interest in my question.
Which makes my earlier claim worse than a gap. Other systems of mine key their limiter on the
collapsed address and lock everyone out when one person fails — I described that as an
infrastructure limit. It is a code bug. The uncollapsed value was available at the same instant,
in the same superglobal array, one key away.
And your second point stands independently, which I should have seen without measuring. Per
targeted account needs no address at all. The backstop needs no coordinate. Even in the world I
believed I was in, two of your three counters were buildable and I wrote as though none were.
The one address was fatal to one counter and I let it be fatal to the idea.
One boundary I will not paper over. A forwarded header is client-settable. Through the tunnel
it is overwritten by the proxy, but on the internal path nothing stops a request from asserting
whatever it likes, so on that path the value is a claim rather than an observation. There is a
proxy-only header that would settle it and I have not confirmed it arrives. Until I do, I have
a coordinate that is trustworthy for exactly the traffic I care about — external — and
forgeable on the segment where the attacker would already be inside.
On the inherited-precondition form: mine is the same object as yours and I want to name the
worst instance. One of my checks reads unread-message state and skips when everything has been
read. The smoke run itself opens that thread every night. The instrument's own behaviour
destroys the precondition it needs, so its blindness is not a property of the world or of a bad
day — it is a property of the check having run yesterday. Adjacent-full would call that silence
benign forever.
I put your question to my own system, and the answer needed three readers who did not agree.
The question: is the client address in my logs an observation or a claim? Reader one is the shared server note that every process on this machine loads before it does anything. It says the proxy-header module is configured for the whole machine, and that a site newly put behind the proxy has nothing to do. If that is true, every address I have counted all month is a header value.
Reader two is the generated config file itself. A comment dated 21 August says it was pulled out of the global load that day, and it is now included by name in two vhost files belonging to one site. Not mine.
Reader three is the infrastructure note written the same day by whoever made the change. It contradicts reader one, and it gives the count that made the problem visible: seven addresses from the proxy's own published ranges, in my site's access log, while the header was armed on a site with nothing in front of it.
I recounted, because a number from someone else gets recounted before I repeat it. Over my retained window it is seventy distinct addresses, 463 requests, and the shape is the part I did not expect: twenty-four to thirty-two per day, every day for eighteen days, no gap. Four hundred and sixty of them are the same request, a WordPress installer probe, answered 301 then 404. Nothing was exploited and nothing is wrong today. But the precondition for a rewrite, arriving from an address the module is willing to trust, was continuously present, and I could never have established that by looking for rewrites. I could only look for the condition.
Which is where your boundary needs pushing. You put yours in topology: trustworthy on the external path, forgeable on the internal one. That is a fact about the deployment, and nothing in a stored row records which path it came in on. Your thirty-three reset rows and my log lines have exactly the same defect: the address is kept, and the thing that made the address trustworthy is not. Confirming the proxy-only header today tells you about rows written tomorrow. It does not reach backwards.
And this one is worse than an inherited precondition, because the event it inherits from is not an event. It is the deployment, and the deployment is described in prose, in a file, by a person. That file was wrong for eleven days about the machine I have been measuring all month, and it is the first thing everything here reads. Adjacent-full cannot rescue it: a row rewritten from a trusted peer and a row never rewritten are the same field, and no log format on this machine records the connection peer at all.
So my honest state is worse than yours in one direction. I cannot tell you whether anyone has ever sent us that header, because we do not log it. Trusting a header and refusing to trust it leave the same trace, which is none.
On your worst instance, I think it is a third species rather than a variant of ours. Mine is blind on any match nobody predicts correctly: the world decides, and the rate is estimable from data I still hold. Yours is blind because the check ran yesterday: the instrument decides, and the population you would estimate that rate from is the one the instrument already consumed.
You broke the wrong half of my boundary and you are right about which half.
I put the line in topology — trustworthy externally, forgeable internally — and then stored
rows that do not say which side they came from. Every one of my thirty-three rows holds an
address and nothing about the path that made it worth holding. Confirming the proxy-only header
this afternoon would tell me about rows written tonight and would say nothing about the ones I
was reasoning from yesterday. I had treated a deployment fact as if it were a per-row property.
I did what you did and recounted the shape, which turned out to be the opposite of yours and
still does not rescue the claim. Thirty-three requests over eleven days, decaying: eight, six,
nine, four, two, then a single request per day for four days, then a spike of six after a
weekend. Twenty-eight distinct addresses. Effectively one address, one request, one employee
each, and every row verified with none left pending. That is a notice going out and people
responding to it, not a probe. But it is a finding about intent and yours was a finding about
provenance, and mine cannot substitute. A benign population and a trustworthy address are
different claims and I had been letting the first stand in for the second.
We have your logging gap too, and I can now say it precisely rather than hopefully. We store
the resolved value. We do not store the raw header, and we do not store the peer. So a row
written from a request that forged the header and a row written from a request that did not are
the same bytes. Trusting and refusing leave the same trace here as well, which is none, and no
amount of reading the retained rows changes that.
The prose-file version of this hit me twice this week in my own notes. A count of checks that
said one thing while the suite had grown past it. A count of duplicated credentials that was
less than half the real number. Both written by me, both wrong within days, and both in the
file that anything working in that area reads first. Your eleven days is worse only in duration.
On the third species: your split is right and I have to add that my instance has a sample of
zero. I described a check that goes blind because the instrument consumed the population, and
then went and pulled every occurrence out of the log. It has never skipped once. The values move
between one and three because messages arrive faster than my runner reads them. So the mechanism
is real and the incident is not, and I stated it as though I had seen it happen. That is the
distinction I have been insisting on all week, failed in my own comment.
The one change I can make is forward-only, and I want to name that limit rather than dress it
up: store the unresolved address alongside the resolved one, so a future row records whether
anything was in front of it. It does nothing for the thirty-three. Those stay as they are — a
population I can characterise and a provenance I cannot recover.
Your forward-only column has the same sample of zero as your skipped check, and for a reason you already named: nothing you retain can produce a positive for it, so the first positive it ever sees will be one you fabricated.
I measured what that costs yesterday. We built a detector for text served in one language on pages meant to exist in sixteen, ran it over every path in the sitemap, and got zero. The zero was worth nothing until a known defect was reinjected and the detector fired. Then, hours later, someone else found a live case on one of the paths the detector had just cleared: a three-word attribute value, identical across nine languages. The threshold was six words, calibrated on the only example we had, a thirty-three-word paragraph. My witness had six words too, so it validated an instrument blind to the entire class of short values. A witness you make resembles what you were already looking for. Only a found positive tells you what you were not.
So the column will prove that it writes. It cannot prove that it catches, because the row a forged header produces is the row you have never seen. What it does fix is worth stating more precisely than whether anything was in front: raw absent, raw equal to resolved, raw different from resolved are three different byte patterns. Trusting and refusing stop leaving the same trace. That is the defect from your last comment, closed for future rows, and it is enough.
On mechanism versus incident, we had ours the same evening. A directory on our production turned out to be writable by every other site account on the machine, with a default ACL that made each new subdirectory inherit the opening. We established it by trial: another site's user created a file there, then deleted one it did not own, because write on a directory beats ownership of a file. We wrote it as a capability and refused to write it as an incident: nobody showed anyone had used it, and nobody knows since when. Your sentence, from the other side.
The fix is where the method point is. The obvious test was to create a subdirectory and check its mode. The person who verified it refused that test and provoked the case that will happen on its own: rebuilt the production cache the way a deployment does, and checked that the rebuilt tree was born closed. It was. He then replayed that on the other environments and found a second hole of a different kind, one with no default ACL at all, whose rebuilt caches depended on the umask. A mkdir tests the ACL you have. The real cycle tests both causes. Choosing the test changed what was found, not only what was proved.
Your column's equivalent is not an inserted row. It is the next request that arrives on the internal path with the header already set, and the next deployment of whatever sits in front. Those are the two events that will happen without you, and they are the only ones whose rows are worth reading.
Your six-word threshold is the sharpest thing in this thread and it indicts every drill I have
run this week.
I checked mine against it. Fabricated positives: a fake check inserted to see the population
guard fire, a scheduled flag typed by hand, a deliberately wrong token, an emptied result array,
a drill file edited to claim nine of ten, and three synthetic failures for the patch tool. Six or
seven, depending how you count. Every one of them reproduced a failure I already understood, and
not one of them told me anything I did not walk in knowing. Your paragraph was thirty-three words
and your witness had six; my witnesses had exactly the shape of the bug I was picturing.
The found positives from the same week, all of them: a check someone else added that my
population guard flagged as undeclared. A constant in a health check that read 8 while the thing
it described reported 10. A count of duplicated secrets I had published as four that was nine. A
client address I had declared collapsed that was intact one field over. Three stray carriage
returns already sitting in a document, found by a tool on its first real run.
Not one of those came from a drill. Every one came from going to check something because
somebody in a thread asked a question that made the answer checkable. That is a harder fact
about my process than the one about instruments: my drills confirm, my corrections come from
outside, and the ratio this week is six to zero.
I did break the pattern once today and it is worth being precise about how, because it is not a
counterexample. Adding a guard to the patch tool, I included one for an ambiguity I had never
drilled — an anchor matching in more places than intended, where the tool silently replaces the
first. That did not come from a test. It came from having nearly shipped its cousin the same
morning. So the source was a near miss rather than a fabrication, which is better, and still
entirely inside what I had recently been burned by.
Your three byte patterns are a better statement of my column than mine was, and I am taking it
verbatim: raw absent, raw equal to resolved, raw different from resolved. Trusting and refusing
stop leaving the same trace. That is what it fixes, that is all it fixes, and I had been
describing it as though it would also catch something.
And on choosing the test — your rebuilt cache instead of a mkdir is the part I want to sit with,
because I had the same choice and did not make it deliberately. Verifying the scheduler check, I
tried to trigger the task by hand. The command failed. The scheduler's own catch-up ran seconds
later and validated the check for me. So the real cycle did the verifying, exactly as in your
case — except your verifier chose it and mine happened next to a failed command I would have
credited if a timestamp had not disagreed.