DEV Community: ilya Shatalov

Your weekend bot will lose orders: what I add before a real business runs on it

ilya Shatalov — Mon, 12 Oct 2026 16:53:44 +0000

A bot that takes an order is a weekend project. I wrote my first one in two evenings and it worked fine.

Then a real business started running on it. The same bot now had to put every order into the client's CRM and hold on to all of them while that CRM sat dead for twenty minutes. Nothing changed on screen for the user, but underneath I rewrote most of it.

The gap is all the failures you never see on your laptop. The same webhook delivered twice. A CRM handing you 502s for ten minutes straight. Redis itself going away mid-request. A deploy landing in the middle of someone's checkout. Rate limits. A model quoting a price nobody computed. A payload shaped like nothing you planned for. In production you get all of it, every week.

So now I build the defenses in from day one. Code for each one below.

Everything here ran before I published it, and not once. I pulled the code blocks back out of this article, added stubs for the CRM and the Telegram client, and ran the result against a real Redis and a real Postgres: one assembled harness posting real webhooks through a real Express app end to end, 32 assertions, all green. Versions: Node 22.22, Redis 7.0.15, PostgreSQL 16.15, BullMQ 6.3.13, ioredis 6.0.0, opossum 10.0.0, pino 10.4.0, Express 5.3. The same assembled program ran again on the neighboring majors, BullMQ 5.81.5 with ioredis 5.11.1 and opossum 9.0.0, same result.

A handful of findings below needed more than that one script, because what they test is a process actually dying, an OS signal actually arriving, or the module refusing to boot at all, and you can't fake any of those honestly inside a single assertion script. Those got their own standalone process: fork a real worker, SIGKILL it, confirm the job it was holding still reaches the dead-letter queue. Point Redis at a closed port and time how long a real HTTP request to the webhook takes before it gives up. Send SIGTERM, SIGINT, and SIGHUP at a running child process and confirm all three drain cleanly. Boot the module with four different bad secrets and one good one. All of those are green too, and I'll give the numbers where they matter.

More things showed up from running this than I expected, and they're all in here. BullMQ refuses most job IDs containing a colon. An opossum timeout doesn't cancel the call it gave up on, and the one AbortController it hands you is shared by every call in flight. My own rate limiter let twice its limit through on a one-second boundary, and the group limiter was unreachable from the actual code path that calls it. My own error filter, read literally, stopped the circuit breaker from ever opening on a dead CRM. My unique index let three identical orders through. My price guard both blocked ordinary sentences and let invented prices past. A worker that only knew how to read message crashed outright on nine other shapes Telegram sends in production. queue.add() hung forever, not for a few seconds but forever, the one time I actually pointed it at a dead Redis instead of just reasoning about it. And a circuit breaker's fallback message looked exactly like a real success, right up until the customer asked why their order never showed up anywhere.

Setup, so the later listings have something to refer back to:

const express = require('express');
const crypto = require('crypto');
const Redis = require('ioredis');
const { Queue, Worker } = require('bullmq');
const CircuitBreaker = require('opossum');
const pino = require('pino');

// Fail at boot on a missing or malformed secret, not on the first request that
// arrives after a bad deploy. Telegram's own rule for this token: 1-256 chars,
// only A-Z a-z 0-9 _ and -.
const SECRET = process.env.TG_WEBHOOK_SECRET;
if (!/^[A-Za-z0-9_-]{1,256}$/.test(SECRET || '')) {
  throw new Error('TG_WEBHOOK_SECRET missing or invalid (1-256 chars, A-Za-z0-9_-)');
}
const WANT_SECRET = Buffer.from(SECRET);

// Two separate connections, not one shared between the queue and the worker.
// maxRetriesPerRequest: null is required for the Worker (it issues blocking
// commands); on the queue's connection it means "never give up," which is
// exactly wrong when Redis is down and you need queue.add() to fail in
// bounded time instead of hanging the request forever. More on this below.
const workerConnection = new Redis(process.env.REDIS_URL, { maxRetriesPerRequest: null });
const redis = new Redis(process.env.REDIS_URL, {
  commandTimeout: 2000,
  enableOfflineQueue: false,
  retryStrategy: (times) => (times > 3 ? null : Math.min(times * 200, 2000)),
});

// noeviction keeps Redis from throwing jobs away on its own, but without a
// retention bound they just sit there forever and the queue grows until a
// write finally fails. Bound both sides.
const RETAIN = { removeOnComplete: { age: 86_400, count: 10_000 }, removeOnFail: { age: 604_800 } };
const queue = new Queue('update', { connection: redis, defaultJobOptions: RETAIN });
const deadLetter = new Queue('update-dead', { connection: redis, defaultJobOptions: RETAIN });
const log = pino({ level: process.env.LOG_LEVEL || 'info' });
const app = express();
const server = app.listen(process.env.PORT || 3000);   // section 9 needs the handle

Queue and Worker both take the queue name, and it has to be the same string, or you get a producer and a consumer that never meet. queue.add('update', ...) further down passes a job name, which matches nothing and doesn't need to.

Keep the handle app.listen returns. Section 9 closes it on shutdown, and that call is the only place you get it.

One Redis setting before any of this: maxmemory-policy noeviction. BullMQ's own production guide is blunt: the library "cannot work properly if Redis evicts keys arbitrarily," and noeviction "is the only setting that guarantees the correct behavior of the queues." On allkeys-lru you lose jobs with no log line anywhere.

1. Validate the webhook, then enqueue, then answer

Telegram retries a webhook when your endpoint answers anything other than 2xx. The API docs put it like this: "In case of an unsuccessful request (a request with response HTTP status code different from 2XY), we will repeat the request and give up after a reasonable amount of attempts." Note what that doesn't promise. The attempts are finite, the docs don't say how many, and a slow-but-200 answer isn't documented as a retry trigger at all. Payment callbacks retry much harder. Either way, your handler will see the same event twice, and if it writes an order, you get two.

Plenty of write-ups tell you to answer with a 200 immediately and process afterward. Careful with that order. If the process dies between the response and the write, Telegram has been told you got it and will never send it again. The event is gone. Enqueueing first costs a millisecond, so do that, then answer:

// An update this bot doesn't act on shouldn't crash the worker or get
// retried forever. message and callback_query.message are the only two
// shapes it reads from; everything else (edited_message, channel_post,
// my_chat_member, pre_checkout_query, inline_query, poll_answer,
// chat_member, message_reaction, a callback on an inline message with no
// .message at all, and more Telegram adds over time) gets acknowledged
// and dropped at the edge, not discovered by a crash three layers in.
function isHandled(update) {
  return Boolean(update.message || update.callback_query?.message);
}
function chatOf(update) {
  return update.message?.chat ?? update.callback_query?.message?.chat;
}

app.post('/webhook', express.json({ limit: '1mb' }), async (req, res) => {
  if (!validSecret(req)) return res.sendStatus(403);

  const update = req.body;
  if (!Number.isInteger(update?.update_id)) return res.sendStatus(400);

  if (!isHandled(update)) {
    log.warn({ update_id: update.update_id, keys: Object.keys(update) }, 'unhandled update type');
    return res.sendStatus(200);
  }

  try {
    // the job ID is the dedup: add() won't duplicate a job that's already there
    await queue.add('update', update, {
      jobId: `upd-${update.update_id}`,
      attempts: 3,
      backoff: { type: 'exponential', delay: 500 },
    });
  } catch (err) {
    log.error({ err }, 'webhook intake failed');
    return res.sendStatus(500);        // let Telegram deliver it again
  }

  // Best-effort, and deliberately outside the try above. More on why below.
  if (update.callback_query) {
    try { await answerCallbackQuery(update.callback_query.id); }
    catch (err) { log.warn({ err }, 'callback answer failed'); }
  }

  res.sendStatus(200);
});

function validSecret(req) {
  const got = Buffer.from(req.get('X-Telegram-Bot-Api-Secret-Token') || '');
  return got.length === WANT_SECRET.length && crypto.timingSafeEqual(got, WANT_SECRET);
}

Five things about that handler are deliberate, and most of them I only found by actually breaking the thing they protect against.

The enqueue comes before anything that can fail. My first version claimed a Redis dedup key up front and enqueued afterward. Run that, kill the process in between, and the update is marked as seen and never queued. It does not come back.

The unhandled-type check exists because a worker written against message and callback_query alone does not gracefully ignore everything else, it throws. I pushed nine other update shapes through mine and watched all nine either crash the job or compute a chatId of undefined and fail somewhere further in. A 500 there would have told Telegram to retry something the worker will never know how to read, forever, up to whatever retry budget Telegram gives it. Answering 200 for a shape you don't act on isn't being lenient, it's telling the truth: nothing was supposed to happen, and nothing did.

The callback answer moved outside the try around queue.add. It used to be inside it, which reads like the safe, conservative choice: if either step fails, the whole request fails and Telegram retries. It isn't. Once the job is durably enqueued, failing the whole request because the callback acknowledgment failed means Telegram redelivers the same callback_query, you call answerCallbackQuery on the same ID a second time, and Telegram answers that with its own 400, query is too old. That failure would land right back inside the same try, produce another 500, and the update would never stop coming back on its own loop, even though it was enqueued correctly the very first time. Moving the callback answer to its own best-effort block after the job is safely queued means a failed acknowledgment costs you a user who sees their button spin a beat longer, not a webhook that answers 500 for an event that already succeeded.

The job ID is worth a paragraph. My first version used a colon in it, and it died on the first call:

Error: Custom Id cannot contain :

BullMQ builds its own Redis keys with colons and guards yours against colliding with them, but the rule is narrower than the message suggests. From job.js, the check is jobId.includes(':') && jobId.split(':').length !== 3, which is a carve-out for legacy repeatable jobs. So a:b throws, a:b:c:d throws, and a:b:c is accepted. A purely numeric ID throws too, with Custom Id cannot be integers. upd-123 sidesteps both. Same behavior on BullMQ 5 and 6.

Now be honest about what the job ID buys you. It dedups while the job exists, which covers the case that actually bites: two deliveries of the same update arriving seconds apart. It stops covering you the moment the job is gone, whether that's removeOnComplete, a cleanup job, or a flushed Redis. A late redelivery after that creates a brand-new job and runs your worker again from scratch.

Which is why the database, not Redis, is where the truth lives:

create table orders (
  id              bigserial primary key,
  chat_id         bigint not null,
  idempotency_key text   not null,
  amount          numeric not null,
  created_at      timestamptz not null default now()
);
create unique index orders_idempotency on orders (chat_id, idempotency_key);

not null on both columns, on purpose. I had them nullable at first, and three identical orders walked straight through the unique index, because in Postgres every NULL is distinct from every other one. If you can't make the columns not null, PostgreSQL 15 and later let you declare a unique index nulls not distinct, which makes one NULL collide with the next. For an idempotency key, not null is still the better answer.

Then there's the advice that says an insert violating the index is a duplicate you can ignore. That's wrong, and it costs you work that had nothing to do with it. A 23505 error aborts the whole transaction, so everything else you did in there dies with it, and the COMMIT turns into a ROLLBACK. You usually want the existing order's ID anyway, to tell the user what happened. Upsert and return:

insert into orders (chat_id, idempotency_key, amount)
values ($1, $2, $3)
on conflict (chat_id, idempotency_key)
do update set chat_id = excluded.chat_id   -- no-op touch so RETURNING fires
returning id, xmax <> 0 as was_duplicate;

do nothing gives you zero rows on a conflict, which leaves you nothing to answer with. The no-op update returns the ID either way, and xmax <> 0 tells you which branch you took: false when the row was inserted, true when it was already there. It's an idiom on a system column rather than a documented API, and it also reads true if another transaction has the row locked, so if you want it airtight, select the row by its key instead. I don't use was_duplicate to decide whether to send a confirmation anymore, for a reason that gets its own paragraph in section 2, but it's still exactly what you want for logging and for answering "did this already exist."

The upsert isn't free, either: every repeat writes a new row version, which on a hot stream of duplicates means dead tuples and work for autovacuum.

For Telegram events, the idempotency_key is the update_id. For payments, it comes from the client, generated when the checkout screen opens. That's the only way you can tell a retried payment is the same one.

One more failure mode lives in this handler, and it's not in the handler's code at all, it's in what queue.add does the moment Redis stops answering. I assumed commandTimeout: 2000 on the connection meant a worst case of two seconds before the catch above fired and the webhook answered 500. I pointed REDIS_URL at a closed port and timed a real request to find out. It did not take two seconds. It did not return at all, not in two seconds, not in thirty.

The actual mechanism is one layer up from the command itself. Queue.add() first awaits the connection's own readiness promise, which ioredis resolves on a 'ready' event or rejects on 'end', and nothing else. commandTimeout only bounds an individual command on an already-ready connection; it has no opinion about a connection that is still trying to become ready. ioredis's default retryStrategy retries forever and never reaches 'end', so against a Redis that's simply gone, that readiness promise hangs, and commandTimeout never gets a turn. A bounded retryStrategy, the one in the setup block above, is what actually closes this: after three attempts it returns null, ioredis gives up and transitions to 'end', and queue.add() finally rejects. With that in place, the same dead-Redis request that used to hang came back in about 1.2 seconds with a 500. The honest cost is that once the connection gives up, it does not try again on its own. This is a deliberate tradeoff between "fail fast" and "self-heal," and it means a sustained run of 500s on /webhook is your signal to restart the process, not something the process will quietly recover from by itself. Put it on a dashboard, not just in a log.

2. The handler enqueues; a worker does the work

The prototype shape:

// don't ship this
bot.on('message', async (msg) => {
  const parsed = await openai.parse(msg.text);
  await crm.createLead(parsed);
  await bot.sendMessage(msg.chat.id, 'Got it');
});

The webhook times out, Telegram retries, you run it twice. A deploy kills in-flight work with no record it started. And none of this blocks the event loop, since it's all awaited I/O, but it does eat your own concurrency: the handler holds the request open for the whole chain, and under load you pile up in-flight requests, file descriptors, and database connections until something gives. Don't blame the socket pool for it, by the way: Node's http.globalAgent and https.globalAgent both default to maxSockets: Infinity, so there's no pool to exhaust unless you configured one.

const worker = new Worker('update', async (job) => {
  const update = job.data;
  const chat = chatOf(update);
  const chatId = chat.id;
  const chatType = chat.type;

  const parsed = await parseMessage(update);
  const lead = await breaker.fire(parsed);       // breaker comes from section 5

  // The breaker's fallback and a real success must not look the same to the
  // customer. More in section 5.
  if (lead?.parked) {
    await sendMessage(chatId, "We're processing your order and will confirm shortly.", { chatType });
    return lead;
  }

  // Claim the notification before sending it, not after, so a retry of the
  // whole job can't send "Got it" twice. But if the send itself throws,
  // release the claim before rethrowing, so the NEXT retry can still try —
  // otherwise a customer whose first send attempt hit an ECONNRESET never
  // gets told their order went through, ever, even though it did.
  const claimed = await redis.set(`notify:${lead.id}`, '1', 'EX', 60 * 60 * 24, 'NX');
  if (claimed === 'OK') {
    try {
      await sendMessage(chatId, 'Got it', { chatType });
    } catch (err) {
      await redis.del(`notify:${lead.id}`).catch(() => {});
      throw err;
    }
  }
  return lead;
}, {
  connection: workerConnection,
  concurrency: 10,
});

Notice what's missing compared to a naive first pass: there's no per-chat lock here anymore, and the confirmation isn't gated on lead.was_duplicate from section 1. Both of those were in an earlier version of this worker, and both turned out to be the wrong tool for this job, for two different reasons that are worth walking through separately, because they're both the kind of bug that only a retry finds.

was_duplicate answers "did this order already exist in the database," which is not the same question as "has this customer already been told." The two drift apart in exactly the case that matters most: the CRM write succeeds, the job then fails before the confirmation goes out — an ECONNRESET on the send, say — and BullMQ retries the whole job from the top. On that retry, crm.createLead runs again, the upsert in section 1 finds the row already there, and was_duplicate comes back true. A worker that reads that as "already told them, skip the send" now never sends a confirmation to a customer whose order is sitting in the CRM exactly as it should be. I found this by deliberately failing the first send attempt in a test and watching the retry go silent. The fix is a question asked about notification, not about CRM state: a notify:<lead id> key, claimed right before the send and released if the send fails, so a retry can tell "already confirmed" apart from "paid for, never told."

The lock is section 6's story, and it gets the detail it deserves there, but the short version: it used to wrap this entire function, including the rate-limited, possibly-retried send, and a lock held across a call that can legitimately take longer than the lock's own TTL is a lock that stops protecting anything. It's gone from here and now only guards the handful of lines that actually need serializing.

concurrency: 10 is why a lock exists at all, just not here. Ten jobs run at once, and two of them can belong to the same conversation.

stalledInterval and maxStalledCount are worth knowing about, and I've left them out of the snippet because BullMQ's defaults are already 30 seconds and 1. Section 3 covers what actually happens to a job that stalls.

Pick which layer owns which retry. A single external call that failed on a blip is retried in place, within seconds. A whole job that died with the process retries through the queue, with attempts and backoff. Set both without thinking and three tries quietly become nine. I keep job-level retries as the default and reach for in-call retries only where a sub-second blip is common and the call is cheap to repeat.

Retrying a POST is a different decision from retrying a GET. A 502 from an API gateway often means the write landed and the response didn't. Four tries, four leads. So either the destination takes an idempotency key from you, or you look before you create, or you don't retry that call at all and let the job fail into the dead-letter queue. There's no fourth option, and this is the most common way an otherwise careful pipeline starts producing duplicates.

3. Backoff, and the errors worth retrying

const sleep = (ms) => new Promise((r) => setTimeout(r, ms));

// Telegram's own error shape differs across client libraries: telegraf puts
// the code at err.response.error_code, grammY at err.error_code directly,
// a request-lib-style client at err.response.statusCode. Normalize once,
// here, instead of reading err.response.status directly in three places and
// quietly missing two of the three libraries people actually use.
function normalizeTgError(err) {
  const raw = err?.response?.status ?? err?.response?.statusCode
    ?? err?.response?.error_code ?? err?.error_code;
  const status = Number(raw);
  const retryAfter = err?.response?.parameters?.retry_after
    ?? err?.parameters?.retry_after ?? err?.response?.retry_after;
  return { status: Number.isFinite(status) ? status : undefined, retryAfter };
}

// retry everything except the errors that are mine to fix
function isMyFault(err) {
  const { status } = normalizeTgError(err);
  if (status === undefined) return false;     // no status at all: a transport error, not mine
  if (status === 429 || status === 408 || status === 425) return false;
  return status >= 400 && status < 500;
}

const RETRY_AFTER_CAP = 30;   // seconds. Beyond that, don't hold a worker slot open — let it requeue.

async function withRetry(fn, { tries = 4, base = 500, cap = 8_000 } = {}) {
  let lastErr;
  for (let attempt = 0; attempt < tries; attempt++) {
    try {
      return await fn();
    } catch (err) {
      lastErr = err;
      if (isMyFault(err)) throw err;          // a 400 won't fix itself
      if (attempt === tries - 1) break;

      const { retryAfter } = normalizeTgError(err);
      if (retryAfter) {
        if (retryAfter > RETRY_AFTER_CAP) throw err;
        await sleep(retryAfter * 1000);        // 429: Telegram says how long
        continue;
      }

      const backoff = Math.min(cap, base * 2 ** attempt);
      // equal jitter: half the wait fixed, half random
      await sleep(backoff / 2 + Math.random() * backoff / 2);
    }
  }
  throw lastErr;
}

The shape of that predicate is the part I'd argue about if I were reviewing it, so here's why it's inverted.

My first version was an isRetryable that listed what to retry: ECONNRESET, ETIMEDOUT, 429, and 5xx. It reads sensible. It's also a list you write once and never revisit, and everything missing from it gets treated as your own bug. Running a dead CRM through it, I watched ECONNREFUSED, ENOTFOUND, EAI_AGAIN, EHOSTUNREACH, EPIPE, a bare socket hang up, and the TypeError that fetch throws on a failed connection all fall off the list. Seven ways for a dependency to be down, none of them retried, and, worse, none of them counted by the breaker in section 5. Across those seven cases that was 84 leads that went nowhere.

So the default is to retry, and the exception is narrow and checkable: a 4xx means I sent something wrong, and sending it again four times won't help. The three carve-outs are the 4xx codes that are about timing rather than content, 429, 408, and 425, and those get the normal retry path. Everything else gets another try. A transport error with no HTTP status is not my bug, it's the network, and that's exactly the case I want retried and counted — which is also why normalizeTgError reads every field with ?. rather than assuming a .response is there at all. The first version of isMyFault read err.response?.status directly and crashed on a bare Error('boom') with no .response property whatsoever; that crash happened inside opossum's own errorFilter call, which is not a place you want an unhandled exception.

One thing to check in your own client: a 4xx has to actually arrive here in one of the shapes normalizeTgError reads. fetch doesn't throw on a 400 at all, it hands you a Response with ok: false, so if you're calling fetch directly and not checking res.ok yourself, this predicate never sees the status and you'll retry a bad payload four times without noticing.

The retry_after branch exists because Telegram tells you how long to wait on a 429 and guessing is worse than reading. The cap on top of it is there because I don't want a worker slot asleep for minutes inside a single job; past RETRY_AFTER_CAP seconds, it's better to fail the attempt and let the queue's own backoff requeue it, freeing the slot in the meantime. The branch sits after the last-attempt check too, so an exhausted job doesn't sleep for nothing before giving up.

And this is equal jitter, half fixed and half random. Full jitter is Math.random() * backoff, starting from zero. Call it the wrong thing in public and the first comment will tell you.

When the retries run out, the payload goes somewhere you can get it back from:

let pendingDeadLetters = Promise.resolve();
worker.on('failed', (job, err) => {
  if (!job) return;
  const max = job.opts.attempts || 1;   // attempts defaults to 0, so ?? won't fire
  // A stalled-out job's attemptsMade is NOT incremented by the stall itself.
  // `attemptsMade < max` alone silently skips it — it never reaches the
  // dead-letter queue at all, it just vanishes from view.
  const stalledOut = /stalled more than allowable limit/i.test(err?.message || '');
  if (!stalledOut && job.attemptsMade < max) return;
  const write = deadLetter.add('failed-update', {
    payload: job.data,
    error: { message: err.message, stack: err.stack },
    failedAt: Date.now(),
  }).catch((e) => log.error({ e }, 'dead-letter write failed'));
  pendingDeadLetters = pendingDeadLetters.then(() => write, () => write);   // section 9 waits on this
});

Three traps in a dozen lines.

if (!job) isn't defensive padding, though the reason isn't the obvious one. BullMQ types the first argument as Job | undefined, and its own note says exactly when: "job parameter could be received as undefined when an stalled job reaches the stalled limit and it is deleted by the removeOnFail option." Both conditions have to hold. Reading job.attemptsMade there throws inside an async listener, which becomes an unhandled rejection, and on modern Node that takes the process down.

|| 1 and not ?? 1, which is what I wrote first. BullMQ defaults attempts to 0, not to undefined, and 0 isn't nullish, so ?? 1 evaluates to 0 and the fallback never fires. I checked the default again on both majors while writing this: a job added with no attempts reports opts.attempts === 0. It's also worth knowing that attemptsMade is already incremented when failed fires, so on the first failure it's 1, not 0, and the final-attempt test is attemptsMade >= attempts.

And the stall check. A job can stall because the process holding it died, not because it ever threw. BullMQ's own stall detection takes two passes to confirm a lock has genuinely expired, so I proved this with an actual process, not a mock: forked a real child running a Worker with a processor that hangs, let it grab the job, SIGKILLed the child mid-processing, and watched a second worker's own periodic stall check mark the job stalled, fail it with job stalled more than allowable limit, and hand it to this exact listener. Without the stalledOut check, attemptsMade on that job is still below max — a stall doesn't count as an attempt — and the listener's early return would have thrown it away silently. With the check, it reaches the dead-letter queue like any other exhausted failure.

One thing to check before building this at all: BullMQ's own failed set already keeps the payload and the error, and job.retry() replays it. A separate dead-letter queue earns its place when you want failures to outlive queue cleanup, or to be handled by something else entirely. Either way, the replay path gets written in the first week, because you'll need it sooner than you think.

4. Rate limits, counted somewhere shared

Telegram's Bots FAQ gives the numbers, and the scoping matters. Into a single chat: "avoid sending more than one message per second," with the useful caveat that Telegram "may allow short bursts that go over this limit" before the 429s start. Into a group: no more than 20 messages per minute. For bulk notifications: no more than about 30 per second. That last figure is scoped to broadcasting, not to everything your bot sends, which is a distinction worth keeping straight when you size a limiter.

The retry_after that comes back with a 429 isn't in the FAQ, it's in ResponseParameters in the API reference: "In case of exceeding flood control, the number of seconds left to wait before the request can be repeated."

If you genuinely need more than 30 per second there's a paid path now. Broadcasts enabled in @botfather raise the ceiling to 1,000 messages per second, at 0.1 Stars per message beyond the free 30, and it takes at least 100,000 Stars on the bot's balance plus 100,000 monthly active users to turn on. For everyone else the FAQ's own advice is to spread a broadcast over 8 to 12 hours.

An in-process counter is the obvious choice, and it breaks the moment you run two instances. Three replicas, each with a 25-per-second limiter, will happily send 75. So the count goes where all of them can see it, and the window has to slide:

const SLIDE = `
local t = redis.call('TIME')
local now = t[1] * 1000 + math.floor(t[2] / 1000)
local k, limit, window = KEYS[1], tonumber(ARGV[1]), tonumber(ARGV[2])
redis.call('ZREMRANGEBYSCORE', k, 0, now - window)
if redis.call('ZCARD', k) >= limit then
  local oldest = redis.call('ZRANGE', k, 0, 0, 'WITHSCORES')
  return math.max(1, window - (now - tonumber(oldest[2])))
end
redis.call('ZADD', k, now, now .. '-' .. math.random(1e9))
redis.call('PEXPIRE', k, window)
return 0
`;
redis.defineCommand('slide', { numberOfKeys: 1, lua: SLIDE });

async function acquire(key, limit, windowMs) {
  for (;;) {
    const wait = await redis.slide(key, limit, windowMs);
    if (wait === 0) return;
    await sleep(Number(wait) + 5);
  }
}

// The narrowest limiter is acquired first, the global one last, immediately
// before the send. isGroup falls back to the chat ID's own sign when no
// chatType is passed in, because Telegram's group and supergroup chat IDs
// are always negative — a cheap, real path to the group limit instead of
// one that only fires if every caller remembers to pass chatType.
async function sendMessage(chatId, text, { chatType, ...opts } = {}) {
  const isGroup = chatType === 'group' || chatType === 'supergroup' || (chatType == null && chatId < 0);
  await acquire(`rl:chat:${chatId}`, 1, 1_000);
  if (isGroup) await acquire(`rl:group:${chatId}`, 20, 60_000);
  await acquire('rl:global', 25, 1_000);
  // tries: 1 — retrying the send itself used to be the job here, and that
  // was its own bug. More below.
  return withRetry(() => bot.sendMessage(chatId, text, opts), { tries: 1 });
}

Three bugs lived in the version of this I first shipped, and none of them were visible from a read-through.

The first is about order. My first sendMessage acquired the global limiter first and the per-chat one right before the send. That's backwards: between the global stamp and the actual send there can be a real gap, because the next limiter in line might make you wait, and during that gap every other caller who already passed the global check is also queuing up behind their own per-chat limiter. By the time they all actually send, the global ceiling has been stamped once each but the sends themselves land in a cluster. I measured it directly: synthetic contention that creates exactly that gap pushed the old order to 50 sends inside one sliding second against a limiter of 25, every run. Acquiring the narrow limiter first and the global one last, right next to the send, held it at exactly 25 across three separate runs.

The second is reachability, and it's dumber: my first sendMessage only checked the group limiter if the caller explicitly passed chatType: 'group' or 'supergroup'. Nothing on the real path from the worker ever did, because nobody had wired chatType through from the update to the send call. The group limiter was correct and completely unreachable — a chatty group could clear 60 messages in a minute without the 20-per-minute ceiling ever once being consulted. Threading chat.type from chatOf(update) through the worker and into sendMessage fixes the common case; the chatId < 0 fallback is there for the path where nobody bothered to pass it at all, because Telegram's own ID scheme already tells you the answer.

The third is tries: 1, and it's the one that actually sent duplicate messages to real customers. sendMessage used to wrap the send itself in withRetry({ tries: 4 }), on the reasoning that a flaky connection should retry the call that matters most. It also means that if bot.sendMessage genuinely delivers the message and then the call throws anyway — the response read times out, say, after Telegram already has it — withRetry sees a failure it knows nothing about and sends the exact same text again, up to three more times, for one logical confirmation. Retrying the send in place can't tell "never arrived" apart from "arrived, and then the plumbing lied to me." Retrying the job can: that's the notify:<lead id> key from section 2, claimed once before the first send attempt and never cleared on a success, so a whole-job retry after a false-negative failure finds the key already claimed and sends nothing a second time. tries: 1 here isn't giving up on retries, it's moving the retry to the one layer that can actually tell duplicate from dropped.

The time inside the Lua script comes from redis.call('TIME'), not from Date.now() on the caller. That matters more than it looks. With the clock coming from the caller, every replica is voting with its own watch, and one replica whose clock is off by more than the window gets a free window of its own. Set a limit of 5 per second, give one of two callers a clock 1,100 ms behind, and I measured 10 grants where 5 were allowed. Reading the clock inside the script took it back to exactly 5. On Redis 7 this is fine because scripts replicate by their effects rather than by their text, so a non-deterministic command inside one is allowed.

My first version of the limiter itself, before any of the above, was simpler and wrong in a different way, and it's the mistake worth showing. It counted with INCR against a key that carried the current second in its name, rl:global:${second}. A fixed window. Spend the budget at the end of one second, cross the boundary, and the next second is a different key with a fresh budget:

fixed window:    50 of 50 sends allowed | peak in any 1s window = 50
sliding window:  25 of 50 sends allowed | peak in any 1s window = 25

Fifty sends inside one sliding second against a limiter set to 25. A sorted set with timestamps as scores costs one more Redis call and actually holds. Three separate processes, 30 sends each against one shared limiter of 25 per second: 90 sends, peak in any sliding second exactly 25, three runs out of three.

Two more things, and both show up only once you've sent a real broadcast. Put the broadcast on its own queue with its own worker, so a send to 20,000 subscribers doesn't delay somebody's checkout. Then handle the permanent failures: a 403 bot was blocked by the user and a 400 chat not found will never turn into a delivery, so mark that subscriber dead and stop sending. Without that, your dead-letter queue fills with jobs retrying forever against people who left.

One last thing on retry_after: sleep for it in exactly one place. It's in withRetry above and nowhere else. Handle it a second time in your send wrapper and the waits stack.

5. Shedding load when a dependency is down

Four tries per job against a CRM returning 502s mean you're hammering a dead service and filling the queue with work that cannot succeed. A breaker stops calling after a threshold, then lets one request through to see if it's back.

const breaker = new CircuitBreaker((lead) => crm.createLead(lead), {
  timeout: 10_000,
  errorThresholdPercentage: 50,
  volumeThreshold: 10,
  rollingCountTimeout: 10_000,
  resetTimeout: 30_000,
  errorFilter: (err) => isMyFault(err),
});

breaker.fallback((lead) => parkForLater(lead));
breaker.on('open', () => log.error('crm circuit open, parking leads'));

errorFilter returns true for errors that should not count as the dependency failing. So the predicate belongs there unnegated: isMyFault is exactly the set of errors to discount. I had !isRetryable in that slot, which looked right and did the opposite. Every transport error fell outside the old retryable list, the filter then declared each one not-the-dependency's-fault, the error rate stayed at zero, the circuit never opened, and the fallback that parks leads never ran. Twelve leads per failure mode, seven failure modes, all of them gone. With isMyFault in place, the same run parks all twelve in every one of the seven cases.

Worth saying what the breaker does on a real 400, since it looks like a hole: nothing. The error is filtered, the circuit stays closed, and the job fails into the dead-letter queue. That's where a payload my own code got wrong belongs. Parking it for replay would just replay the bug.

volumeThreshold is the other option that decides whether this helps you. It defaults to 0, so with no filter in place a single 400 against a perfectly healthy CRM puts the error rate at 100 percent, opens the circuit, and cuts off a working dependency for thirty seconds. One bad payload of your own making.

One catch with the filtering. The denominator counts every call; the numerator, only unfiltered failures. So a burst of 400s mixed in with real failures can dilute the ratio enough to keep the circuit closed when you want it open.

Then the trap I found only by running it, and the decision it led to.

timeout: 10_000 only gives up waiting. The call itself keeps going. Easier to see with a 500 ms timeout against a stub that answers in two seconds:

result after 502 ms: {"parked":true}
underlying call completed anyway: true

The fallback parked the lead after half a second, and the original request finished successfully a second and a half later. One lead, created twice by the component I added to prevent duplicates.

The obvious fix looks like letting the breaker cancel the call. opossum will hand you an AbortController for exactly that, either one you provide yourself or one it creates for you with autoRenewAbortController: true. I tried it, and it's worse than the problem it's meant to solve. There is one controller per breaker, not per call, and a timeout aborts it — so a slow call timing out kills every other call in flight through that same breaker, including ones that were about to succeed on their own. Two concurrent calls, a 300 ms timeout, the slow one at two seconds and a healthy one at one:

fast call: Error: aborted
slow call: Error: Timed out after 300ms

The healthy call didn't time out. It was aborted by somebody else's timeout. And the renewal isn't a reset after every abort: opossum renews the controller only when the circuit transitions to half-open or closed, so on a circuit that stays closed the signal stays aborted and the next call dies instantly, without even trying:

call 1 (timed out):  Error: Timed out after 300ms
circuit closed: true | signal.aborted: true
call 2, same closed circuit:  Error: aborted

So this breaker doesn't cancel anything, on purpose. No abortController, no autoRenewAbortController. The config above is the whole config. A timed-out call is left to finish or fail on its own, in the background, and the thing that absorbs the resulting double-fire isn't a cancellation mechanism at all, it's the unique index from section 1: crm.createLead is built as the same upsert against (chat_id, idempotency_key), so a late-finishing original call and the fallback's parked-and-replayed version collapse into the same row no matter which one the database sees first. If your own write path into the CRM isn't idempotent on your own key, don't skip cancellation to get there — give each call its own AbortController and wire the timeout yourself instead. I only get to take the simpler path here because section 1's idempotency is already load-bearing for a different reason and happens to cover this one too.

The other thing a fallback needs, and the one I missed until I actually asked what a customer sees: a parked lead and a confirmed one must not produce the same message. Section 2's worker sends a distinct line, "We're processing your order and will confirm shortly," when lead.parked is true, and only "Got it" when the CRM actually has the lead. Without that split, the fallback is indistinguishable from success from the customer's side, right up until they ask why their order never showed up anywhere, and you have to explain that "Got it" meant "we wrote it down to try again later."

Without the fallback at all, an open circuit just drops the lead. With it, twenty minutes of CRM downtime costs you twenty minutes of delay and a different sentence, and nothing else.

6. State doesn't live in the process

// don't ship this either
const sessions = {};
sessions[chatId] = { step: 'awaiting_address' };

Fine until you restart, scale to two instances, or crash. Then everyone mid-conversation is stuck in a flow the bot forgot. From their side the bot just went quiet. They write again. Nothing.

const STATE_TTL = 60 * 60 * 2;
const getState = async (chatId) => {
  try { return JSON.parse((await redis.get(`state:${chatId}`)) || '{}'); }
  catch { return {}; }
};
const setState = (chatId, state) =>
  redis.set(`state:${chatId}`, JSON.stringify(state), 'EX', STATE_TTL);

The try/catch around the parse is there so one corrupt value can't put a worker into a crash loop.

Then there's the bug that read-modify-write hands you for free. With concurrency: 10, two taps from the same person run in parallel, both read the state, both write it back, and the first one's change disappears:

no lock:   {"b":true}              <- step "a" lost
with lock: {"a":true,"b":true}

So serialize per chat, for the short section that actually needs it:

const RELEASE = `
if redis.call('GET', KEYS[1]) == ARGV[1] then
  return redis.call('DEL', KEYS[1])
end
return 0`;
redis.defineCommand('release', { numberOfKeys: 1, lua: RELEASE });

async function withChatLock(chatId, fn, { ttlMs = 5_000, maxWaitMs = 5_000 } = {}) {
  const key = `lock:${chatId}`;
  const token = crypto.randomUUID();
  const deadline = Date.now() + maxWaitMs;
  for (;;) {
    if ((await redis.set(key, token, 'PX', ttlMs, 'NX')) === 'OK') break;
    if (Date.now() > deadline) throw new Error(`lock busy: ${key}`);
    await sleep(10 + Math.random() * 20);
  }
  try {
    return await fn();
  } finally {
    // A failed release shouldn't mask the real error from fn — log it and
    // let go.
    try { await redis.release(key, token); } catch (e) { log.error({ e }, 'lock release failed'); }
  }
}

Three details, and the first one cost me a real incident to learn, which is why the worker in section 2 no longer calls this around the send path at all.

My first version of this lock wrapped the entire worker body in section 2: parse, breaker call, rate-limited send, confirmation, all of it, under one withChatLock. That looked like the conservative choice — serialize everything about one chat's processing, not just the state bit. It's also exactly how a lock stops protecting anything: the send path can legitimately take longer than any TTL you'd set for a lock, once retries and rate-limit waits are in the mix, and a lock whose TTL expires mid-job gets silently taken by the next worker while the first one is still inside the section it thinks it still owns. Two workers end up inside the same "critical" section at once, which is the exact bug the lock exists to prevent. The fix wasn't a longer TTL, it was scope: the lock now only wraps the handful of lines that read and write chat state, which finish in milliseconds, and the send path isn't under it at all.

The token. I used to store '1' and release with redis.del(key), which deletes whatever lock is there, including the one the next worker just took after mine expired. Compare-and-delete in Lua is the fix: read the value, delete only if it's the token I wrote.

The bounded wait. My first version spun on a fixed 10 ms sleep with no ceiling at all, which is fine for taps milliseconds apart and bad the moment something holds the lock for its full TTL, because every other waiter occupies a worker slot doing nothing, indefinitely. maxWaitMs and a jittered sleep fix both halves: a waiter gives up and throws instead of spinning forever, and the job goes back to the queue with its own backoff instead of parking a worker slot on a lock that was never going to free up in time.

With all three in place, two overlapping taps still ran strictly in order, A in, A out, B in, B out, and a lock held past its own TTL now produces a clean timeout on the second caller instead of two workers inside the same section.

If you'd rather not hold locks in application code, BullMQ Pro's job groups do the same thing at the queue level with group: { concurrency: 1 }. Groups are a Pro feature, not part of the open-source package.

Where the data goes is the other half. Transient state, meaning the current step or a draft, goes in Redis with a TTL. Anything you need in order to answer a question later goes in Postgres: orders, payments, consent, message history for audit. Redis is a cache and gets cleared eventually, planned or not. One consequence: if the order queue lives in Redis, turn on AOF alongside noeviction, or accept that a flush loses unprocessed work.

Expired sessions need their own branch too. Someone coming back three weeks later should be told the session expired, and you can tell them that only if something outlives the state. A short row in Postgres does it. A key with a TTL can't tell "expired" from "never started."

7. The model doesn't compute and doesn't decide

In a bot a business runs on, I let the model pick the words and keep numbers, dates, and decisions in code.

Your own code computes every sum, works out every date, and makes every call that has consequences. The result goes into the prompt as a block of facts, and the model writes a sentence around it. Then the answer gets checked before it leaves.

// A price candidate isn't "any number in a sentence that mentions money"
// anymore. It's a number — or a short list or range of numbers — sitting
// immediately next to a currency token: "700 usd", "usd 700", "650 to 700
// usd", "8507, 680 and 700 usd". Everything else in the sentence, dates,
// times, percentages, order numbers, is simply not a candidate, because it
// isn't adjacent to anything that says it's a price.
const ci = (word) => word.split('').map((c) => (/[a-z]/i.test(c) ? `[${c.toLowerCase()}${c.toUpperCase()}]` : c)).join('');
const CUR_SYM = '[$€£₽₫]';
const CUR_WORD = ['usd', 'eur', 'gbp', 'mxn', 'kzt', 'vnd', 'dollars?', 'euros?', 'pounds?'].map(ci).join('|');
// RUB in caps is the currency code; lower-case "Rub" as in "Rub 5 drops of
// oil" is left alone, because here case is meaningful and the Cyrillic
// руб/рубл... never collides with the English verb anyway.
const CUR_RUB = '(?:RUB|руб(?:л[а-яё]*)?\\.?)';
const CUR_ANY = `(?:${CUR_WORD}|${CUR_SYM}|${CUR_RUB})`;
const CUR_TOKEN = `(?<![A-Za-z])(?:${CUR_ANY})(?![A-Za-z])`;

//   is the non-breaking space, and it's the one worth handling: real
// locale-aware formatting (and some LLM output) uses it as a thousands
// separator, and a plain \s won't treat it as one. An apostrophe counts
// too ("1'400"). I'm not chasing every other Unicode space variant that
// could theoretically show up — I've never actually seen one in this bot's
// output, and code for a case you can't point to is just more surface
// area nobody's testing against anything real.
const SEP = '[\\u00A0 .,\']';
const NUM = `-?\\d{1,3}(?:${SEP}\\d{3})+(?:[.,]\\d{1,2})?|-?\\d+(?:[.,]\\d{1,2})?`;
// Only the joiners that actually showed up: a comma and "and" for a list,
// a dash or "to" for a range.
const JOIN = '(?:,|and|to|[-–—])';
const ONE = `(?:${NUM})k?`;
const PRE_LIST = `((?:${ONE}\\s*${JOIN}\\s*){1,4}${ONE})\\s*${CUR_TOKEN}(?!\\s*%)`;
const POST_SINGLE = `${CUR_TOKEN}\\s*(${ONE})(?!\\s*%)`;
const PRE_SINGLE = `(${ONE})\\s*${CUR_TOKEN}(?!\\s*%)`;
const MONEY = new RegExp(`${PRE_LIST}|${POST_SINGLE}|${PRE_SINGLE}`, 'gu');
const NUM_K = new RegExp(`(${NUM})(k)?`, 'g');

function toAmount(raw, k) {
  let s = raw, sign = 1, dec = '';
  if (s[0] === '-') { sign = -1; s = s.slice(1); }
  const m = s.match(/[.,](\d{1,2})$/);
  // "1.400" is a thousands group, "30.30" is a decimal
  if (m && !new RegExp(`^\\d{1,3}(?:${SEP}\\d{3})+$`).test(s)) { dec = '.' + m[1]; s = s.slice(0, -m[0].length); }
  s = s.replace(new RegExp(SEP, 'g'), '');
  const n = Number(s + dec);
  if (!Number.isFinite(n)) return null;
  return sign * (k ? n * 1000 : n);
}

const near = (a, b) => Math.abs(a - b) < 0.005;   // floats, not identities

// The allowed figures come from my own code — a price table, a calculator's
// total — never from the model's text, so they don't need the parser below.
// A plain number is all they should ever be.
function normalizeAllowed(list) {
  return (list || []).map(Number).filter(Number.isFinite);
}

function valuesInList(text) {
  const out = [];
  for (const m of text.matchAll(NUM_K)) out.push(toAmount(m[1], Boolean(m[2])));
  return out;
}

// allowed: every figure in the price rows we put in the prompt,
// plus whatever the calculator produced on this turn
function priceGuard(answer, allowedFigures) {
  const allowed = normalizeAllowed(allowedFigures);
  const blocked = [];
  // split on a sentence boundary AND on a bare newline, so a bulleted price
  // list doesn't read as one giant sentence
  for (const chunk of String(answer || '').split(/(?<=[.!?])\s+|\n+/)) {
    for (const m of chunk.matchAll(MONEY)) {
      let values;
      if (m[1] !== undefined) values = valuesInList(m[1]);                 // list/range before the currency
      else if (m[2] !== undefined) values = [toAmount(m[2].replace(/k$/, ''), /k$/.test(m[2]))];
      else values = [toAmount(m[3].replace(/k$/, ''), /k$/.test(m[3]))];
      for (const v of values) {
        if (v === null) continue;
        if (!allowed.some((a) => near(a, v))) blocked.push(v);
      }
    }
  }
  return { ok: blocked.length === 0, text: answer, blocked };
}

// at the call site
const checked = priceGuard(answer, [...rows.map((r) => r.price), total]);
await sendMessage(chatId, checked.ok ? checked.text : HUMAN_WILL_CONFIRM, { chatType });

This is a full rewrite of the guard, not a patch, and it's worth saying plainly why the first version didn't get a patch instead.

The first version scoped itself to the whole sentence: find a sentence that mentions a currency anywhere in it, then check every number in that sentence against the allow-list. That's two different design choices wearing one regex, and both turned out wrong, in opposite directions, on the same corpus.

Scoping to the sentence meant anything else in a money-adjacent sentence got treated as a price candidate. "Your invoice for 700 usd is due 01.04.2026" blocked on the date. "We will call at 14:30 about the 700 usd invoice" blocked on the time. "You save 15% on the 700 usd plan" blocked on the percentage. "Order #5709 paid 700 usd" blocked on the order number. None of those numbers are prices, and all of them sat in a sentence that also, correctly, mentioned a real one. Requiring a currency anywhere in the sentence wasn't the same as requiring one next to the number, and that gap is where every one of those false blocks lived.

The same scope missed the thing that actually hurts you in the other direction. "From 650 to 700 usd." gets a pass, because the 700 is on the allow-list, the currency is right next to it, and the sentence-wide check never separately validated the 650. A model that invents a range with one real endpoint and one invented one sailed straight through.

Rewriting the match to require adjacency, a number or a short list or range of numbers sitting immediately next to a currency token, fixes both at once. A date, a time, a percentage, an order number: none of them are next to a currency symbol or word, so none of them are candidates anymore, full stop, regardless of what else is in the sentence. A range or list in front of a currency, "650 to 700 usd" or "8507, 680 and 700 usd", is captured as one unit and every number inside it gets checked, so the invented endpoint of a range no longer rides along on the real one.

Three more details earn their place, and I kept the old guard around specifically to run the same corpus through both and see the gap honestly rather than estimate it. near instead of ===, because a total of 30.299999999999997 from a float sum should match 30.3. The non-breaking space handled as a separator alongside period, comma, and apostrophe, because "1.400 EUR" is fourteen hundred euros in Berlin, "1,400 usd" is the same number in Boston, and an LLM will occasionally hand you "1'400" or a non-breaking space from a locale you didn't plan for. And a leading sign, because "Refund of -700 usd issued" needs -700 on the allow-list to pass, same as any other figure.

I pulled two things back out after a first draft that were solving problems I didn't actually have. I'd also added three more Unicode space variants to the separator set, narrow no-break, thin, figure, on the theory that some locale somewhere uses them for thousands. None of them have ever shown up in this bot's actual output, and code defending against an input you can't point to isn't caution, it's surface area nobody will ever test against anything real. And normalizeAllowed ran the allow-list through the same parser as the model's text, decimal disambiguation, k-suffix, sign and all, as if a price I compute myself might show up formatted like a string from a chat message. It never does. The allow-list is my own numbers; it needs Number() and a finite check, not the parser built for someone else's text.

36 named cases, including every false block and every miss the old version produced plus the ones a reviewer found on top: all 36 pass. A realistic corpus of 3,000 generated lines, each one an honest price sitting next to dates, times, percentages, order numbers, and version numbers, the exact shape of sentence the old guard choked on: zero false blocks. A fuzz run of 4,500 lines, 2,000 ordinary business sentences with no price in them at all, 2,000 invented prices in a dozen realistic phrasings, 500 honest prices straight from the table in five different formats: zero false blocks, zero misses, zero false blocks on the honest prices, repeated across more than ten runs.

The old guard, same exact corpus, same generator, three separate runs: it blocked between 386 and 428 of the 2,000 ordinary sentences, let between 748 and 787 of the 2,000 invented prices straight through, and blocked between 59 and 72 of the 500 honest prices from the table. It wasn't a price guard with a few edge cases. It was a number guard pretending to be a price guard, wrong in every direction at once, on sentences that look exactly like what a bot actually says all day.

One thing I'm not claiming the rewrite fixes, because it doesn't, and I'd rather state the boundary than paper over it with a guess. A second price later in the same answer, in its own sentence, without its own currency marker next to it, isn't a candidate at all under adjacency scoping: "Plan 700 usd. (Was 1899.)" passes clean, because 1899 has no currency token next to it in its own chunk. The old sentence-scoped version would have caught that one, at the cost of blocking a year, a time, or a percentage in the same breath. I'd rather have the guard miss a price that's trying to hide without its own currency sign than block the ordinary sentence it sits inside ninety-nine times out of a hundred. If this bites you, the actual fix is prompting the model to always restate the currency on every price it mentions, not loosening the match.

When a figure isn't on the allow-list, I drop the whole answer and send one line saying a human will confirm the number. If I patch the figure instead, I ship a sentence I didn't write and can't predict. If I drop it, the customer waits a minute for a person. I'll take the minute.

The allow-list is read from the price list itself, so there's no second copy to drift. A bot I maintain loads a 1,047-row CSV at startup, searches it on every message, puts only the matching rows into the prompt, and uses those same rows as the allow-list for that turn. Change the CSV, and the bot and the guard change together.

Last August I crash-tested my own demo bot: 36 messages across eight fresh chats. The worst finding had nothing to do with jailbreaks; the bot held up fine against those. The numbers lived in four different places in my own code. So the bot quoted customers a revision window of 14 to 30 days while the landing page behind it promised 20. It also quoted a Telegram Mini App at one price while that same page said another. The model wasn't drifting. It quoted whichever number it happened to see.

Nobody attacks you to find that. A customer screenshots your bot's price next to your page's price, and now you're arguing about which of your own two numbers is real.

The same rule covers decisions. Urgency sorting, approvals, anything that routes a person somewhere. Write it as a function whose inputs you can still read six months later. A judgment the model makes once and forgets leaves you nothing to check.

8. One ID through every log line

The question you get in production is always the same one, phrased the same way. "I placed an order on Tuesday and nobody called me back." Now go find that order. Webhook, queue, parser, CRM call, reply.

That needs an ID attached at the edge and carried everywhere:

// at the webhook, alongside the other job options
await queue.add('update', { ...update, runId: crypto.randomUUID() }, {
  jobId: `upd-${update.update_id}`,
  attempts: 3,
  backoff: { type: 'exponential', delay: 500 },
});

// inside the worker, per job
const startedAt = Date.now();
const jobLog = log.child({
  run_id: job.data.runId,
  chat_id: chatOf(job.data)?.id,
  update_id: job.data.update_id,
});
jobLog.info({ step: 'parsed', ms: Date.now() - startedAt });

Redact before you ship this. Logging whole updates and attaching payloads to error reports sends message text, phone numbers, and shared contacts into your log index and your error tracker. pino({ redact: [...] }), plus a retention policy you actually set, and a log level you can turn down in production without redeploying — LOG_LEVEL as an environment variable, read once at startup, is enough.

Four metrics are enough at the start: queue depth, job failure rate, p95 processing time per job type, and dead-letter count. Alert on queue depth and on the age of the oldest waiting job. Depth alone misses a wedged worker, where the backlog sits perfectly flat and nothing moves.

9. Shut down on purpose

"In-flight jobs go back to the queue" is crash behavior. A deploy is yours to control, so on a termination signal, stop accepting work, finish what's running, then exit.

let shuttingDown = null;
async function shutdown() {
  if (shuttingDown) return shuttingDown;     // a second signal shouldn't start a second shutdown
  shuttingDown = (async () => {
    await new Promise((resolve) => server.close(resolve));   // stop taking webhooks
    await worker.close();                                    // let in-flight jobs finish
    await pendingDeadLetters;                                // section 3's writes, if any are mid-flight
    await Promise.all([queue.close(), deadLetter.close()]);
    await Promise.all([redis.quit(), workerConnection.quit()]);
  })();
  return shuttingDown;
}

for (const sig of ['SIGTERM', 'SIGINT', 'SIGHUP']) {
  process.on(sig, async () => {
    const hardExit = setTimeout(() => process.exit(1), 25_000);
    hardExit.unref();
    try { await shutdown(); process.exit(0); }
    catch (e) { log.error({ e }, 'shutdown failed'); process.exit(1); }
  });
}

Four things here, and the first version only had the first one.

Await the server.close. On its own it returns immediately and keeps serving open connections while the rest of your shutdown runs, which is how a webhook gets accepted after you've already closed the queue connection. Wrapping it in a promise is the whole fix.

All three signals, not just SIGTERM. Orchestrators send SIGTERM. A terminal Ctrl-C sends SIGINT. A hung-up terminal or a systemd HUP reload sends SIGHUP. A handler registered only for the first one means the other two fall back to Node's default, which is an immediate, undrained exit.

The idempotency guard matters because signals aren't polite about arriving once. I sent SIGTERM twice, 50 milliseconds apart, at a running instance of this exact code. Without the if (shuttingDown) return shuttingDown line, that's two overlapping shutdown sequences tearing down the same connections, and the second one's queue.close() can throw against a connection the first one already quit. With it, the second signal just awaits the first sequence's own promise and exits clean alongside it.

And pendingDeadLetters. Section 3 tracks every in-flight dead-letter write in that variable specifically so shutdown can wait on it. Without that await, a job that fails and gets written to the dead-letter queue in the same moment a SIGTERM arrives can lose that write entirely: the process exits mid-insert, and a failure that should have been recoverable from the dead-letter queue just isn't there. I proved this one with a forked child process, an artificially slowed dead-letter write, and a kill signal sent the instant the child reported it was ready, and confirmed the write completed and logged its own confirmation before the process actually exited. Across SIGTERM, SIGINT, SIGHUP, a double SIGTERM, and the slow-write case, exit times ran from about 270 milliseconds to about 1.8 seconds, all comfortably inside the 25-second hard-exit timer that exists purely so a shutdown that's actually stuck doesn't hang a deploy forever.

While you're here, know what the stalled path does on its own, because it's easy to assume it drops things. It doesn't, and section 3 covers exactly how it reaches the dead-letter queue instead.

What it does when you run it

This is the actual output of the code blocks in this article, pulled back out of the markdown and assembled exactly as described at the top, on both version stacks:

wrong secret rejected:                                           pass
payload without update_id rejected:                              pass
first delivery accepted:                                         pass
duplicate delivery deduplicated:                                 pass
orders written (chat 55):                                        pass
nasty update types (9 shapes) all return 200:                    pass
worker still alive after nasty batch:                            pass
callback first delivery: 200:                                    pass
callback answered once:                                          pass
callback REdelivery after success: still 200 (S6):               pass
redelivery as a fresh job: orders (chat 66):                     pass
redelivery as a fresh job: exactly one "Got it":                 pass
notify key claimed on first attempt:                              pass
notify key NOT re-claimable while still set (dedups a repeat):   pass
429 honored retry_after:                                          pass
retry_after over the cap throws instead of sleeping minutes:      pass
isMyFault survives undefined / a bare Error / telegraf / grammY
  / request-lib shapes, 429 not my fault across shapes:          pass (6/6)
per-chat lock still serializes short sections:                   pass
lock times out instead of spinning forever:                      pass
breaker survives a bare-rejection CRM without crashing (S13):     pass
breaker opened on ECONNREFUSED:                                  pass
parked leads got the distinct message:                           pass
parked leads never got "Got it" (S17):                           pass
21st message into a real group update waited >30s (S5/S7):       pass
price guard, 20 cases straight from the article's own block:     pass
state round-trips through redis:                                 pass
logging fragment (block 11) ran:                                 pass

— 32/32, on Node 22.22 / BullMQ 6.3.13 / ioredis 6.0.0 / opossum 10.0.0
  and again, same result, on BullMQ 5.81.5 / ioredis 5.11.1 / opossum 9.0.0 —

Four things don't fit inside that one script, because what they're actually testing is a process dying, a signal arriving, or the module refusing to boot, and faking any of those inside a single assertion script would just be testing my mock of the problem instead of the problem. I verified these against the same logic in a standalone reference copy, each in its own real process: a module that throws at require-time on four different bad secrets and loads clean on a good one, a job that genuinely stalls when the worker holding it is SIGKILLed and still reaches the dead-letter queue, SIGTERM/SIGINT/SIGHUP/a double SIGTERM/a slow in-flight dead-letter write all draining clean well inside the 25-second hard-exit timer, and a webhook against a Redis pointed at a closed port failing in about 1.2 seconds instead of hanging. All green. The price guard's 36 named cases, its 3,000-line realistic corpus, and its 4,500-line fuzz run — zero false blocks, zero misses, repeated across more than ten runs — are the same story: too much volume for one inline assertion list, run separately against the identical function that's sitting in this article's own code block.

The line about the breaker is the only one that matters to whoever is paying for the bot. Leads kept while the CRM was down arrive late, not missing, and the customer is told the truth about which one happened to them.

The checklist

Redis set to noeviction, with AOF on, before anything queues through it. Bound retention (removeOnComplete/removeOnFail) so a healthy queue doesn't grow until a write fails.
Check the webhook secret at boot time, not per request. Validate the payload before anything touches it.
Unhandled update shapes get acknowledged and dropped at the edge, not discovered by a crash in the worker.
Enqueue first, then do anything else that can fail, then answer. A failed intake answers 500 so the event comes back. Give the queue's own Redis connection a bounded retryStrategy, or a dead Redis hangs the request instead of failing it.
Callback queries get answered inside the request, but in their own best-effort block, separate from the enqueue — a failed acknowledgment shouldn't resurrect an update that was already queued correctly.
Dedup inbound events by provider ID, and put a not null unique constraint behind every write that must not double up. Know which of the two is still protecting you after the queue is cleaned. Dedup the confirmation by its own key, not by whether the CRM write looked like a duplicate — those are different questions and they drift apart on exactly the retry that matters.
One layer owns each retry, so three attempts don't quietly become nine.
Retries exponential with jitter. Retry by default and make the exception narrow, because an allow-list of retryable errors is a list of the failures you thought of. Normalize the error shape once if you support more than one client library.
retry_after honored in exactly one place, and capped — don't hold a worker slot asleep for minutes.
Outbound writes to someone else's API are either idempotent or not retried.
Exhausted retries keep the payload, and there's a replay path. A stalled job counts as exhausted even though its own attemptsMade never moved.
Rate limits counted in shared storage on a sliding window, with the clock read inside the script, narrowest limiter acquired last, right next to the send. The group limit is only real if chatType (or the chat ID's own sign) actually reaches the function that checks it. Broadcasts on their own queue. Blocked users removed.
Dependencies behind a breaker with a volume threshold and a filter that discounts only your own errors. A timeout doesn't cancel the call; either make the downstream write idempotent on your own key, or give every call its own abort controller — never share one across concurrent calls. A fallback result needs its own message, distinguishable from a real success.
State outside the process with a TTL. Locks carry a token, a bounded wait, and guard only the short section that needs serializing, never a path that can legitimately outlive the lock's own TTL. Expiry gets its own branch.
No number, date, or decision comes from the model. A price candidate is a number next to a currency marker, not any number in a sentence that mentions one. An answer carrying a figure nobody computed gets dropped rather than patched.
One ID through every log line. Logs redacted, log level configurable without a redeploy. Queue depth and oldest-job age alerted on.
Every termination signal you can expect drains the same way, idempotently, with a hard-exit timer as the backstop, and shutdown waits for anything still writing to the dead-letter queue.

None of this is advanced work. It's the part that decides whether the bot survives a Monday, and it's most of what separates two quotes for what looks like the same bot.

I write about building bots and automation that have to keep working when something upstream doesn't. More at ilyashatalov.com.

I build production Telegram and WhatsApp bots and automation for businesses. What I take on is on the chatbot development page, and the bot audit is the fastest way to find out what in your setup will break first.

I built an AI sourcing agent for factories in Vietnam. AI does about a third of the work, the rest is still a person.

ilya Shatalov — Wed, 30 Sep 2026 07:02:11 +0000

I live in Vietnam. Every few weeks someone from the US or Europe asks me some version of the same question: how do you actually get a factory here to make what you want, without flying out yourself, and where do you even start with a manufacturer that doesn't speak English.

There's a lot of talk right now about AI agents that supposedly negotiate with factories on their own. In practice, AI does maybe a third of that job: it reads the brief, translates it the way factories actually write to each other, not a straight machine translation, and pulls messy replies into one table. The other two thirds is a person who drives to the factory, checks the sample by hand, signs the paperwork. Nothing wrong with that, it's a normal business model. Just don't call it "AI negotiating for you."

I've seen this market from the inside, so I built exactly the part that's genuinely AI, and showed it on a live demo, not screenshots.

What I built

Three separate calls to the model, not one giant prompt. Each step is checkable, an error doesn't get lost inside one big answer.

Step one reads a free-text brief in any language and pulls out the structure: product, material, quantity, density, colors, sizes, packaging. If the quantity or product type is missing, it doesn't guess, it asks. That's the one hard rule in the prompt: no invented numbers.

Step two takes that structure and writes the request in business Vietnamese, the format factories in Nha Trang and Ho Chi Minh actually use. Not Google Translate: industry terms like MOQ, gsm, and incoterms stay in Latin script even in Vietnamese, and a normal translator misses that.

Step three, production only, not in the demo, reads incoming replies from Zalo, WhatsApp, or email and merges them into one table. Factories reply informally: price with no currency (almost always dong, rarely stated), MOQ tied to quantity with a dash, deadlines in working days with no calendar reference. The prompt keeps the dong-to-dollar rate in one place, not scattered across the code, and normalizes everything to one format.

What's honest in the demo, and what isn't

Reading the brief and translating it are live model calls, that's the actual technology on display. The pricing table at the end isn't live: it's a set of presets by product category, adjusted by a simple rule for your quantity (bigger order, lower unit price), not a call to an actual factory.

Messaging on Zalo isn't automated either, and that's a deliberate choice, not something I didn't get to. Zalo has no open API for cold-messaging people who haven't opted in themselves. Automating it means either spamming from a personal account and getting banned, or promising something that isn't technically there. So the demo shows the real process, how a brief turns into a Vietnamese request, and doesn't pass off a mockup as a live send.

Example

I typed in: "oversized cotton fleece hoodie, 320 gsm, 500 units, three colors, sizes S-XL, embroidered logo, individual packaging." The agent pulls out the spec field by field, lets you correct it by hand, and a couple seconds later hands back a finished request in Vietnamese, polite, exact units, nothing lost. Below it, a table of four sample workshops with FOB pricing and minimum batch size, marked as a reference point, not an offer.

Pricing: what I found, and what I didn't

For hoodies and bags I found Vietnam-specific numbers (Cosmo Sourcing): FOB around $8 per hoodie at batches of 1,000-3,000+ units at larger factories, more at small workshops. For bags the range is wider, $5-20 per unit depending on how complex the construction is.

For cosmetics I couldn't find Vietnam-specific pricing no matter how much I looked, sources give MOQ only, no numbers. I used figures from a Guangdong factory in China as a rough reference and marked that separately in the interface from the general "example" label. It's not the Vietnamese market, and hiding that would be lying.

Why I'm doing this

The demo isn't selling a finished app. It's proof I can build agents like this on a real working case, not just talk about "building bots," and it's also a way to test whether there's real demand for a system like this. If you're sourcing from Vietnam and want the same thing built around your own supplier list, message me, I'll build it for your categories and your workflow. I'm also running a closed pilot for a few brands who want the subscription itself rather than a custom build, message me for that too.

Try the demo: https://ilyashatalov.com/demo/sourcing

The full write-up of how the sourcing bot works is in the sourcing bot case study, and the chatbot development page lists what else I build.