Eighteen places across twelve Telegram bots marked a scheduled message as delivered before the send resolved. I know the number is eighteen because a checker counted them, and I can still reproduce that count by pointing the same checker at the commit before the fix. Violation messages below are trimmed for width, and the list is cut after the first two failing bots.
$ node scripts/sendorder-gate.mjs --repo /tmp/pre-fix-tree
PASS sendorder-gate anon
PASS sendorder-gate anonsay
FAIL sendorder-gate birthday
- dmOne: markDm() is not confirmed inside the try/catch that guards its send
FAIL sendorder-gate countdown
- tick: markDone() is not confirmed inside the try/catch that guards its send
- tick: markProgress() is not confirmed inside the try/catch that guards its send
...
sendorder-gate: 21 bot(s) checked, 18 violation(s)
Each of those lines is a message that one transient failure would have deleted. No retry, no error anywhere a person would look, and no counter that would have gone down.
The shape of it
Here is the trivia bot's weekly leaderboard, as it stood before the fix. The message body is elided, everything else is verbatim.
const text = /* rendered leaderboard */;
try { await bot.api.sendMessage(g.chat_id, text); sent += 1; }
catch (e) { console.log("weekly report failed", g.chat_id, String(e).slice(0, 100)); }
await store(env).markWeeklyReportSent(g.chat_id, week); // mark even on failure to avoid retry storms
await store(env).touchLastSent(g.chat_id);
Read it once and it looks defensive. There's a try. There's a catch. There's even a comment explaining the design. Read it twice and the mark sits after the catch, so it runs on the failure path too.
The due-query is the only retry that exists. On each tick weeklyReportDue asks whether this group already has a row for this week's index, and only a group with no row gets a send. The instant markWeeklyReportSent inserts that row, the group is out for the week. A 429 during that one send costs the group its leaderboard, and the log line is the only trace.
No, the client is not retrying for you
That was the first thing I checked, because if grammY were retrying under me the whole class of bug would be cosmetic.
$ grep -rn 'autoRetry\|apiThrottler\|api.config.use' --include='*.ts' */src/ kit/
$ grep -rl 'sendMessage' --include='*.ts' */src/ | wc -l
40
The second command is the control. Forty files call sendMessage. None of them install the auto-retry plugin or the throttler. So a 429 propagates as a thrown GrammyError, the catch logs 100 characters of it, and the mark below fires anyway.
Some sites were worse than the trivia one. The reminder bot's deliverDue did this, and that cron is the fleet's flagship retention mechanism. The failure mode there is a person's reminder vanishing at the exact moment it was supposed to arrive. The RSVP bot's one-hour reminder had a candidate window of (now, now + 3600], so once reminded = 1 no future tick could match that row again. The birthday greeting version, which is where I first met this bug, marked the year handled. Miss it and the next chance is in 365 days.
Three fixes, three recurrences
I found this in the birthday bot and fixed it. Sprints later I found the same shape in the focus timer and the standup bot's weekly report and fixed those. The standup case is the one that stings. The daily report two lines above was marked correctly, because that's the line the earlier fix was written for, and the Friday summary directly below it was not.
Then I found it in eleven more bots.
Each fix was right. Each was applied only to the bots under review that day. The rule itself lived in a decisions file as a sentence, and a sentence in a decisions file does not read your new cron for you. After the third recurrence I stopped writing the rule down and started compiling it.
The gate
kit/sendorder.ts is a set of pure functions over source text. No filesystem, no grammY import, no Workers runtime. It extracts the scheduled() method, walks every function reachable from it to a bounded depth, and asserts one invariant. Every delivery-marking call must be provably downstream of a send that resolved.
Provably downstream has exactly three legal shapes, and they came from reading the code that was already correct rather than from inventing a style.
A. The mark sits inside the same try as a qualifying send, after it.
B. The mark sits inside that try's own catch, the "mark on success, and on a permanent 403 too" shape, so a user who blocked the bot doesn't become an infinite retry.
C. The mark is gated by an if whose condition names a flag that is only ever assigned inside a try containing a send. The birthday bot already did this and it reads well.
let sent = 0;
try {
await bot.api.sendMessage(g.chat_id, text, { reply_markup: new InlineKeyboard().text(t("en", "btn_upcoming"), "bday:upcoming") });
sent = 1;
await store(env).incr("greetings_sent");
} catch (e) { console.log("greet failed", g.chat_id, String(e).slice(0, 100)); }
if (shouldMarkGreeted(sent === 1)) await store(env).markGreeted(g.chat_id, b.user_id, localDate(now(), g.tz_min).year);
The check itself is fifteen lines.
export function markIsGuarded(body: string, at: number, pairs: TryCatch[], sends: number[]): boolean {
for (const p of pairs) {
const sendBefore = sends.some((s) => s >= p.tryStart && s < p.tryEnd && s < at);
if (sendBefore && at >= p.tryStart && at < p.tryEnd) return true; // (A)
const hasSend = sendBefore || sends.some((s) => s >= p.tryStart && s < p.tryEnd);
if (hasSend && p.catchStart >= 0 && at >= p.catchStart && at < p.catchEnd) return true; // (B)
}
const cond = enclosingIfCondition(body, at);
if (!cond) return false;
const idents = cond.match(/[A-Za-z_$][\w$]*/g) ?? [];
for (const id of idents.slice(0, MAX_CALLS)) {
if (flagSetInSendTry(body, id, pairs, sends)) return true; // (C)
}
return false;
}
One thing that turned out to matter more than the invariant. The checker follows delegation up to three calls deep, so a caller whose try only awaits postQuestion(...) still sees the send that postQuestion performs. Without that, half the fleet would have failed for a reason a human reader would immediately call wrong, and a checker that cries wolf gets switched off in a week.
The parts that were harder than the rule
Scanning source with string functions is fine right up until it isn't, and three specific things bit.
Comments and quotes. The bracket matcher has to skip // and /* */ whole. Miss that and an apostrophe inside a doc comment, in a phrase as ordinary as "the recipient's own", opens a string that never closes, and every bracket for the rest of the file stops counting. The scan doesn't crash. It just quietly stops finding anything, which is the worst way for a gate to fail.
TypeScript return types. async function greet(env: Env): Promise<{ greeted: number; dms: number }> { has two opening braces and the first one is not the body. A naive indexOf("{") after the parameter list truncates the extracted function mid-signature. The fix walks from the end of the parameter list, treating <, ( and [ as nesting, and takes the first { at depth zero. It also bails at a depth-zero ;, because an expression-bodied arrow has no block at all, and otherwise the scan fuses two unrelated functions into one body.
Global regex state. Every scanning function builds its regex literal on entry instead of sharing a module-level one. A global-flagged RegExp carries lastIndex across calls, so one shared instance leaks scan position between unrelated strings. That is a bug you find at 2 a.m. and never once suspect.
There's also one name on an exclusion list. markSeen matches the mark[A-Z] pattern but it's the content-rotation dedupe, not a retry gate, so flagging it would be noise. I excluded it by name after checking every due* and claim* query in the fleet's storage layer to confirm none of them filter on it. Exclusions are where a gate goes soft, so the reason belongs in the code next to the exclusion, not in a commit message.
A checker needs its own known-bad input
The unit tests don't just assert the checker passes on the fixed tree. That would prove nothing, because a function that returns [] unconditionally also passes on the fixed tree.
The fixtures reproduce the historical bugs. One reconstructs the birthday markGreeted shape and asserts exactly one violation naming markGreeted. One reconstructs the focus timer's two and asserts both. One is the standup file where the daily report is correct and only the Friday summary is wrong, and it asserts a violation count of one with the right name. The good fixtures matter just as much, including the delegation case and a conditional send with an unconditional mark in the same try, which is legal and must not fire. Fourteen tests, and the ones I'd keep if I could only keep half are the known-bad ones.
Where it runs
The gate is wired into the smoke script and into the preflight of both deploy scripts, next to a handle checker that does a similar job for a different mistake. A violation is a non-zero exit and the deploy refuses to start.
$ node scripts/sendorder-gate.mjs
...
sendorder-gate: 21 bot(s) checked, 0 violation(s)
Each bot also carries its own copy of the checker and its own test file, and the fleet-wide runner imports each bot's own copy rather than the canonical one. That's deliberate. If a per-bot copy drifts out of sync it fails visibly instead of passing its own stale rules.
What this actually cost, and what it saved
Here's the part that makes the whole story easier to tell honestly. Across all fifteen bots with a scheduled send, no scheduled message has ever been delivered to a real user. I confirmed that with wrangler tail held across a cron boundary on every worker. The cron fires, the due-query runs, it returns zero rows, and the loop body never executes. Nobody had yet created a habit, a reminder or an event for the cron to act on.
So the eighteen bugs cost exactly zero real messages. The fleet is twenty-one bots, twelve people, and no payments. I'm not going to dress that up.
What it saved is the fourth recurrence. This defect has a habit of arriving whenever someone writes a new cron under time pressure, and it's invisible in every place you'd normally look, because the failure produces a log line rather than a missing counter. The value of turning it into a gate isn't the eighteen it found. It's that the shape can't be written again without the build stopping.
If you run scheduled sends against any API that can rate-limit you, go look at where your mark sits relative to your await. My guess is that at least one of them is on the wrong side, and that it's the one you'd miss most.
The reminder bot from the examples is NudgeRemindBot. The checker is about 400 lines of TypeScript with no dependencies, and the three legal shapes plus the delegation-following are the whole of it.
Top comments (0)