This is a submission for DEV's Summer Bug Smash: Clear the Lineup powered by Sentry.
Project Overview
I wanted one number: what share of the entries in this challenge declare a particular prize category. It decides whether writing another entry is worth the evening.
I got that number wrong three times in one morning. Every wrong answer looked finished.
First wrong answer: 19. The listing endpoint returns titles and tags but not bodies, and the prize category is declared by a heading inside the body. So I counted titles containing "gemini" or "google". That found 19 of the 66 entries that actually carry the heading — and one of the 47 it missed was my own most recent entry, whose title mentions neither word. I had been quoting that proxy for days.
Second wrong answer: 23.9%. So I fetched all 412 bodies, six at a time, no delay. My loader was this:
arts = []
for f in dir.glob("*.json"):
try:
a = json.loads(f.read_text())
if a.get("body_markdown"):
arts.append(a)
except Exception:
bad += 1
277 of the 412 responses had the body Retry later. Two words, plain text, where JSON was expected. The except branch counted them and moved on, and I printed a clean table from 134 items:
items coverage category share
naive 134 32.5% 23.9%
complete 411 100.0% 18.5%
5.4 points off, with nothing anomalous in it. The number was not noisy. It was wrong, and it was plausible — which is the only combination that actually costs you anything.
The third wrong answer is the interesting one, and I only found it because I asked something else to look at my fix. That's further down.
Bug Fix or Performance Improvement
Fix 1: "did not parse" is not "not there"
The loader has no bug in it. It does exactly what it says. The bug is in what it means: an exception during parse was being treated as evidence about the item, when it was evidence about the transport.
def collect_strict(server, ids, max_rounds=8):
"""Same fetch, two extra rules: unparsed is not absent, and slow down on refusal."""
done, pending, rounds = {}, list(ids), 0
while pending and rounds < max_rounds:
rounds += 1
still = []
for i in pending:
rec = parse(server.get(i))
if rec is None:
still.append(i)
else:
done[i] = rec
pending = still
server.cooldown //= 2 # back off: fewer requests per unit time
missing = [i for i in ids if i not in done]
if missing:
raise RuntimeError(f"{len(missing)} of {len(ids)} ids never resolved")
return [done[i] for i in ids], rounds
Against the real API — 0.45s between requests, doubling to a 2s ceiling — that recovered all 277 in four rounds: 134, then 77, then 63, then 3, then zero unresolved. One id stayed unresolved and turned out to be a genuine 404 (an entry deleted between the listing call and the fetch), so the assertion permits resolved-as-404 alongside parsed.
The offline reproducer at the end of this post runs the same two collectors against a fake server that refuses in bursts:
items coverage in category share
ground truth 412 100.0% 97 23.5%
naive collector 157 38.1% 39 24.8%
strict collector 412 100.0% 97 23.5%
Note how boring the naive row is. 24.8% against a true 23.5%. Nobody looks twice at that.
Fix 2, which I would not have found on my own
With the numbers in hand, I handed the whole thing to Gemini and asked, among other things: what can still silently truncate the dataset underneath my completeness assertion?
Its first-ranked answer was that my assertion validates the fetched set against the listing, and says nothing about whether the listing is complete.
I went to check. This is the same endpoint, same tag, same minute, walked page by page until an empty page:
per_page pages unique ids note
25 17 422 empty page 18
50 non-JSON at page 10: 'Retry later'
75 6 422 empty page 7
99 5 422 empty page 6
100 4 383 empty page 5
101 5 422 empty page 6
125 4 422 empty page 5
150 3 422 empty page 4
200 3 422 empty page 4
Only per_page=100 loses data. 99 is fine. 101 is fine. 100 comes up 39 short of 422 and announces it by returning an empty page 5 — the universal signal for you have reached the end.
It gets better. Those four pages of 100:
rows returned : 400
unique ids : 383
ids appearing on more than one page: 17
id 4267457 appears 2x on pages [2, 3]
id 4228478 appears 2x on pages [2, 3]
...
Four pages of 100 returned exactly 400 rows — the count you would sanity-check against — while containing 383 articles, 17 of them twice, and 39 not at all. The 39 are not random: the per_page=100 walk reaches back to 2026-07-14, the full listing reaches back to 2026-06-21. It is the oldest entries that vanish, which is precisely the population you would use to say anything about how the challenge has changed over time.
I want to be careful about what I am claiming here. I did not find the cause inside DEV's code; I have no access to it. What I measured is that the same query returns 422 or 383 items depending only on page size, and the short answer terminates cleanly. That is reproducible from any machine, in about sixty requests, with no credentials.
Fix 3: the check I now put in front of the other checks
def assert_listing_stable(fetch, tag, sizes=(99, 100, 101, 200)):
"""A listing you cannot reproduce at two page sizes is not a population."""
counts = {}
for pp in sizes:
ids, page = set(), 1
while True:
rows = fetch(tag, pp, page)
if not rows:
break
ids.update(r["id"] for r in rows)
page += 1
counts[pp] = ids
best = max(counts.values(), key=len)
for pp, ids in counts.items():
if len(ids) != len(best):
raise RuntimeError(
f"per_page={pp} yields {len(ids)} ids, per_page="
f"{max(counts, key=lambda k: len(counts[k]))} yields {len(best)}"
)
return best
Nine lines, and it fails the build on an API defect I could not have guessed at.
Code
The offline reproducer is self-contained and needs nothing but the standard library. It builds a corpus whose property-of-interest is correlated with position — which is the case that matters — and runs both collectors against a server that refuses in bursts.
"""A rate limiter that answers with the words "Retry later" is not an error your
parser will notice. It is a sampler you did not know you installed.
Runs offline against a fake server, so the numbers are reproducible without
touching anyone's API.
collect_naive() drops anything that fails to parse and returns what it got
collect_strict() treats "did not parse" as "not fetched yet", slows down,
retries, and refuses to return until every id is accounted for
Requires: nothing but the standard library.
"""
import sys
if hasattr(sys.stdout, "reconfigure"):
sys.stdout.reconfigure(encoding="utf-8", errors="replace")
import json
N = 412 # items in the listing
BURST = 8 # requests served before the limiter trips
COOLDOWN = 16 # requests refused before it serves again
def truth(i):
"""Ground truth: does item i belong to the category being counted?
The first third of the listing is denser than the rest -- newer submissions
mention the prize category more often than older ones. That is the part that
matters: the property being counted is correlated with position.
"""
threshold = 40 if i < N // 3 else 15
return (i * 37) % 100 < threshold
class Server:
"""Serves JSON in bursts. When the limiter trips it answers 'Retry later'.
The refusal is a 200 with a plain-text body. Nothing raises, nothing retries
itself, and the caller gets a str where it expected JSON.
"""
def __init__(self, cooldown=COOLDOWN):
self.n = 0
self.cooldown = cooldown
self.served = 0
self.refused = 0
def get(self, i):
self.n += 1
if self.cooldown and (self.n % (BURST + self.cooldown)) > BURST:
self.refused += 1
return "Retry later\n"
self.served += 1
return json.dumps({"id": i, "in_category": truth(i)})
def parse(raw):
try:
return json.loads(raw)
except Exception:
return None
def collect_naive(server, ids):
"""What I actually wrote. There is no bug in it -- it does exactly what it says."""
out = []
for i in ids:
rec = parse(server.get(i))
if rec is not None:
out.append(rec)
return out
def collect_strict(server, ids, max_rounds=8):
"""Same fetch, two extra rules: unparsed is not absent, and slow down on refusal."""
done, pending, rounds = {}, list(ids), 0
while pending and rounds < max_rounds:
rounds += 1
still = []
for i in pending:
rec = parse(server.get(i))
if rec is None:
still.append(i)
else:
done[i] = rec
pending = still
server.cooldown //= 2 # back off: fewer requests per unit time
missing = [i for i in ids if i not in done]
if missing:
raise RuntimeError(f"{len(missing)} of {len(ids)} ids never resolved")
return [done[i] for i in ids], rounds
def share(records):
n = len(records)
hits = sum(1 for r in records if r["in_category"])
return hits, n, (hits / n * 100 if n else 0.0)
def main():
ids = list(range(N))
actual = [{"id": i, "in_category": truth(i)} for i in ids]
naive = collect_naive(Server(), ids)
strict, rounds = collect_strict(Server(), ids)
print(f"{'':<20}{'items':>8}{'coverage':>11}{'in category':>14}{'share':>9}")
for label, recs in (("ground truth", actual),
("naive collector", naive),
("strict collector", strict)):
hits, n, pct = share(recs)
print(f"{label:<20}{n:>8}{n / N * 100:>10.1f}%{hits:>14}{pct:>8.1f}%")
_, _, t = share(actual)
_, _, g = share(naive)
print(f"\nThe naive collector reported {g:.1f}% from {len(naive)}/{N} items. "
f"The answer is {t:.1f}%.")
print(f"It raised nothing, logged nothing, and its table looked complete.")
print(f"The strict collector resolved every id in {rounds} rounds.")
if __name__ == "__main__":
main()
The page-size sweep is fifteen lines of curl around the same idea and is quoted in full above.
My Improvements
Done:
- The collector treats an unparsed response as unfetched, backs off, retries, and asserts coverage before anything computes a share. Recovered 277 of 277.
- The completeness assertion permits resolved as 404 as well as parsed, because "deleted since the listing" is a real state and silently dropping it is the same bug one level down.
-
assert_listing_stableruns before the fetch. On this tag it fails, correctly.
Done because the review said so:
- Everything in the previous section. The listing defect was Gemini's first-ranked answer and I had not considered it.
Not done, and I'd rather say so:
- I still cannot tell you what HTTP status those 277 refusals carried. I wrote the bodies to disk with
curl -oand never recorded the status codes, so the question of whether a plainresp.raise_for_status()would have caught the whole thing is one I destroyed the evidence for. Sequential requests do not reproduce it, and I am not going to hammer a free API until it stops talking to me just to find out. The lesson stands on its own: store the status next to the body, or you cannot debug the fetch afterwards. - The fix lives in the loader. It belongs in the fetch layer, raising on a transport failure before anything is written to disk. That is a rewrite, not an edit, and I have not done it.
Left as a guard, deliberately:
Recomputing the denominator at two page sizes every time, even when nothing has changed. It costs about sixty requests and it is the only reason I know the number at all.
Best Use of Google AI
I used Gemini (free tier, Flash) once: after the fix worked, with the numbers already in hand, to ask what my fix still could not see. I gave it both wrong answers, the loader, the strict collector, the assertion, and four questions. I did not adopt any of it — I measured each claim.
It named the class first:
This is a Silent Partial Sample (or Silent Truncation) resulting from Plausible Degradation. [...] The system experienced soft failures—surrogate metrics (Attempt 1) and swallowed rate-limit payloads (Attempt 2)—that degraded the data quality into a plausible subset rather than triggering an explicit system error.
That one sentence covers both of my wrong answers, which is what I had asked for and had not managed to write myself. Then four ranked failure modes, and a verdict on the fix.
Claim Verdict Measurement
Rank 1 the listing itself is truncated held per_page=100 -> 383 of 422,
17 duplicated rows, empty page 5
Rank 2 valid JSON that lacks the key fields held the deleted entry parses fine and
has no body_markdown
Rank 4 HTTP 200 carrying an error payload not observed the deleted entry returns a real 404
Q3 backoff+assert is a patch, not a fix accepted and see "Not done" above
Q4 I may not call the 32.5% sample biased conceded claim retracted, see below
Rank 1 is the reason this post exists. I asked what my completeness check could not see, and the answer was: the thing it checks against. Sixty requests later I had an API defect that is reproducible by anyone, in which the page size everybody reaches for first is the only one that loses data.
Question 4 cost me a claim I liked. I had written that the 32.5% sample was biased, not merely small, because rate-limit refusals arrive in bursts and bursts land on neighbours in a listing. Gemini pointed out that I overwrote the failed responses with the successful refetch, so I no longer have the evidence for that:
You cannot claim the 32.5% sample was definitively biased due to listing-order burst clustering. You cannot claim spatial, chronological, or network locality for the failed items, because you lost the exact temporal sequence and per-request metadata.
What it left me is narrower and I think stronger:
The 32.5% sample is unvalidated and methodologically unreliable because the sampling mechanism was governed by server load-shedding rather than random selection. [...] the sample cannot be assumed to be Missing Completely at Random (MCAR).
So: 23.9% versus 18.5% is an empirical divergence of 5.4 points, and the sample was drawn by a server deciding what it felt like answering. That is enough to throw the number away. It is not enough to say why it leaned the way it did, and I have edited that claim out of my notes.
One last thing, which happened while I was measuring the above. My sweep script crashed decoding curl's output — the console here is cp932 and one of the titles was not — and my error handler, which was written to notice Retry later, reported it as "rate-limited at page 1" for all nine page sizes. I spent a minute believing the API had cut me off, in the middle of writing a post about mistaking a local failure for an absent record.
That is the whole thing, really. Six of my submissions this month are the same shape: every check passed and the answer was still wrong. A ranking only I could see. A collector reporting success on 27% of the data. A rate limiter counting the retries. A verifier that read every character back while three links in the document pointed at nothing. And now a denominator that was wrong three ways, where the third way was invisible until I asked something outside my own head what my check was not looking at.
Top comments (0)