DEV Community: Boris Kl

Your Telegram bot isn't crashing randomly. It's on a timer.

Boris Kl — Mon, 20 Jul 2026 22:23:35 +0000

The ticket always reads the same way: "the bot keeps crashing randomly, please fix."

So you pull the logs. And it's not ra
ndom at all. The bot dies on a schedule — every few hours, almost to the minute. Like clockwork.

That rhythm is a gift. Random failures are the hard ones. They smell like memory leaks, race conditions, bad hardware. A failure that lands on a schedule means something expires. A socket. A token. A connection some box in the middle decided was idle. You're not hunting a ghost. You're looking for a timer.

Here are the three timers I find over and over in Telegram bots that "crash randomly." Between them they cover most of the rescues I get called in for.

Timer #1: the connection that quietly went stale

A long-polling bot holds one HTTP connection open to Telegram and waits. The code looks alive. The process is running. But between your VPS and Telegram sits a NAT table or a proxy. It has an idle timeout. After a fixed window, it drops the connection and tells no one.

No error on your side. No error on Telegram's side. The socket is just... gone. Your bot keeps waiting on a connection that no longer exists. Depending on the library and its settings, it throws two hours later. Or it hangs forever.

The fix isn't a bigger server and it isn't switching libraries. It's assuming the connection will die and reconnecting like you mean it:

import time, logging, requests

def poll_forever(token):
    offset = 0
    backoff = 1
    while True:
        try:
            r = requests.get(
                f"https://api.telegram.org/bot{token}/getUpdates",
                params={"offset": offset, "timeout": 50},
                timeout=(5, 60),   # connect, read — never infinite
            )
            r.raise_for_status()
            backoff = 1
            for upd in r.json()["result"]:
                offset = upd["update_id"] + 1
                handle(upd)
        except requests.RequestException as e:
            logging.warning("poll failed: %s — reconnecting in %ss", e, backoff)
            time.sleep(backoff)
            backoff = min(backoff * 2, 60)

Three things matter here. A real read timeout, set a bit above the long-poll window. Backoff that grows, so a Telegram hiccup doesn't turn into a hammering loop. And a log line every single time it happens. That last one is the sleeper. If reconnects are silent, you'll never know if they fire once a week or once a minute.

Frameworks like aiogram and python-telegram-bot handle much of this for you. But I keep meeting bots where someone turned retries off, or wrapped the library in their own loop that swallows the one exception that mattered.

Timer #2: two transports fighting over one bot

Telegram gives you two ways to receive updates: getUpdates polling or a webhook. One bot token gets exactly one. If a webhook is set and something calls getUpdates, Telegram answers with 409 Conflict, and plenty of homegrown loops treat that as a fatal crash.

How do you end up here without noticing? Easier than you'd think. Someone tested a webhook months ago and never deleted it. A deploy script starts a second copy before the first one dies. Or systemd restarts the service, the old process lingers for a minute still holding the connection, and the new one comes up screaming.

The symptoms look mystical: duplicate messages, updates arriving twice, the bot answering some users and ghosting others, crashes that only happen after a deploy. The cause is boring: two consumers, one queue.

The check takes ten seconds:

curl "https://api.telegram.org/bot$TOKEN/getWebhookInfo"

If you're polling and that response shows a URL — there's your bug. Clear it with deleteWebhook and pick one transport for good. Then make sure only one copy of the bot can run. On systemd that's the default, unless your unit file fights it. The ghost-process case usually comes down to KillMode and TimeoutStopSec that don't match how the bot really shuts down. I've watched a service restart-loop for hours because a dead process refused to let go of port and token.

Timer #3: the request with no timeout

This one doesn't kill the process. It's worse — the bot stays green in every dashboard while doing absolutely nothing.

Somewhere in a handler there's a call to an outside API. Weather, payments, a CRM, your own backend. It's written as requests.get(url), no timeout, because it worked fine in testing. Then one day that API stops answering but keeps the TCP connection open. The default timeout in requests is none. So the handler blocks forever. And in a single-threaded polling loop, that means the whole bot is now a very quiet piece of furniture.

Users say "the bot stopped replying." Monitoring says everything's fine. The process has been "up" for nine days.

Every outbound call gets a timeout. No exceptions, including the calls to Telegram itself:

requests.get(url, timeout=(5, 30))  # 5s to connect, 30s to read

And if the bot does real work, handlers shouldn't share one thread with the poller at all. Even a small thread pool changes the math. One dead upstream API costs you one worker, not the whole bot.

Make the next failure introduce itself

The pattern behind all three: the bot had no way to tell you what was going on. So the fix that outlasts any single bug is a heartbeat. One log line per minute: poll count, last-update time. Plus an alert when the process restarts more than a couple of times an hour. It's cheap to add. And it turns the next "it crashes randomly" into a two-minute read, because now the logs show the rhythm.

Uptime for a bot isn't a feature you bolt on at the end. It's a mindset. Assume every connection dies. Assume every API hangs. Assume every deploy leaves a ghost behind. Then write the ten extra lines that expect all of it.

I build and rescue Telegram bots for clients, and this trio is where I look first, before reading a line of business logic.

What's the strangest clockwork failure you've traced back to a timeout? And do you run bots on webhooks or polling — and why? Genuinely curious what breaks for other people.

I've shipped production code since 1999. AI didn't make me faster where it counts.

Boris Kl — Fri, 17 Jul 2026 10:39:05 +0000

I wrote my first paid line of code in 1999. Servers you could hear. Deploys over FTP. If something broke at 2am, you SSH'd in and read logs until your eyes hurt.

So when people tell me AI made them 10x faster, I want to believe it. I use it every day — I build bots on Claude and GPT, I've got AI running in production on my own projects. It's the best tool I've picked up in years.

But faster? Where it actually counts, no. And I think a lot of us are measuring the wrong thing.

Typing was never the bottleneck

Here's the uncomfortable part. The slow bit of this job was never the typing.

I can bang out a CRUD endpoint in ten minutes with or without AI. That was never what ate my week. What ate my week was understanding why the payment webhook fired twice, or why a site behind a CDN was fast in the test and dead for real users, or which of forty plugins was quietly holding the main thread hostage.

AI is fantastic at producing code. It's much weaker at understanding a system it can't see — your traffic, your data shape, the three weird decisions someone made in 2021 that everything now depends on. That understanding is the job. The code is just what falls out at the end.

So when AI writes the endpoint in 30 seconds, I didn't save a week. I saved ten minutes. The week is still there, waiting, in the part AI can't do for me.

The trap I catch myself in

There's a subtler cost, and it's the one I actually worry about.

When a tool hands you working code instantly, it's really tempting to skip the understanding step. It runs, tests are green, ship it. I've done it. I've shipped something an AI wrote, felt great about the speed, and then spent two days a week later untangling a problem I'd have seen in five minutes if I'd read the thing properly the first time.

The code "worked." That's the trap. Working and understood are not the same state, and production only cares about the second one.

I saw this on a real job — I'll keep it vague. A bot was sending conversion events to an ad platform and the numbers were off. The generated code was clean. It sent the events, got a 200 back, looked perfect in every test. The actual problem was two layers up in how an ID got passed between systems, somewhere no amount of "make the code correct" would ever fix, because the code was correct. You only find that by understanding the whole path. No model was going to hand me that. I had to sit and think.

What actually got faster

I don't want to sound like AI is a toy. It's not. Some things genuinely changed:

The boring first draft. Boilerplate, config, a test harness — AI gets me to something I can react to instead of a blank file. Reacting is faster than starting.
Unfamiliar territory. Dropped into a language or an API I don't use often, AI is a better first guide than skimming docs for an hour.
The rubber duck that talks back. Half the value is just explaining the problem out loud to something that pushes back. I catch my own bad assumptions faster.

Notice what those have in common. They speed up the edges — the start, the unfamiliar, the stuck. None of them touch the core: knowing the system well enough to make the right call. That part still runs at human speed, and I've made peace with it.

Why 1999 me would recognize this

The tools have changed beyond recognition. The habit that separated the people who could fix things from the people who could only write things — that hasn't changed at all.

Back then it was the guy who actually read the manual versus the guy who copy-pasted from a forum until it compiled. Both shipped. One of them could fix it at 2am. AI just made the copy-paste path frictionless and fast, which means the gap between "it runs" and "I understand it" is easier than ever to skip over. The discipline of not skipping it is worth more now, not less.

That's the thing I'd tell someone starting today. Let AI write the code. Then read every line like you'll be the one paged when it breaks — because you will be. Speed at typing is cheap. Understanding is the whole job, and it's the one thing nobody can generate for you.

I run production systems and build AI-driven tools for a living — the kind of work where "it ran in the demo" isn't good enough. If your stack has one of those problems where the code looks right but the behavior is wrong, that's the part I actually enjoy.

Genuine question for the room: where has AI actually made you faster — the core work, or just the edges around it? I'd like to know if I'm the outlier here or if we're all quietly measuring the wrong number.

Cloudflare's Flexible SSL looks secure. It isn't.

Boris Kl — Fri, 03 Jul 2026 10:45:41 +0000

A client called: "the site has the padlock, we're done with SSL, right?" Pulled up the Cloudflare panel — Flexible mode. The padlock was real. The encryption wasn't.

This catches more sites than it should, and the fix is straightforward once you see what's happening.

What Cloudflare's SSL modes actually mean

There are four, and only one is what most people think they have:

Off — no HTTPS. Plain HTTP both ways. Nobody picks this on purpose anymore.
Flexible — HTTPS between the visitor and Cloudflare. Plain HTTP between Cloudflare and your origin. The padlock is the browser's, but the second half of the trip is naked.
Full — HTTPS both ways. Cloudflare doesn't check the origin's certificate, so a self-signed cert is fine. Encrypted, but not authenticated.
Full (strict) — HTTPS both ways, and Cloudflare verifies the origin cert against a real CA. This is the one you want.

Flexible is dangerous because the browser shows the same lock as Full Strict. The site feels secure to the user, to the developer, to whatever scanner pings it from outside. But the traffic between Cloudflare and the origin server is plaintext.

The threat model people miss

"Plain HTTP between Cloudflare and origin — so what, that's a private link?"

It isn't. The path from Cloudflare to your origin crosses public internet. Any network in between — a transit provider, a hosting peer, an exit on a managed VPN, a misconfigured router — can read it. Session cookies, login posts, form data, anything not separately encrypted. You spent the budget on Cloudflare to protect the wire, and the back half of the wire is still wide open.

This is the part the padlock can't tell you about. The browser only sees its own leg of the connection. Cloudflare's leg to your box is invisible to the user, and that's exactly where the leak is.

The fix

Move to Full (strict). This is two steps.

1. Put a real cert on your origin.

If you control the server and run nginx/Apache, the easy answer is Let's Encrypt with certbot — free, 90-day renewal, automated. It's a 10-minute install on most stacks.

If you can't reach the origin from the public internet (it's locked to Cloudflare's IPs, or behind a firewall that won't let Let's Encrypt's HTTP-01 challenge through), use Cloudflare's Origin CA. It's a separate cert authority Cloudflare runs specifically to issue long-lived (up to 15-year) certs that only Cloudflare itself trusts. Generate one in the Cloudflare dashboard, install on your origin, done.

2. Switch the SSL/TLS mode in Cloudflare to Full (strict).

Cloudflare dashboard → SSL/TLS → Overview. Flip from Flexible to Full Strict.

There's an "Always Use HTTPS" toggle nearby — turn it on too. Without it, an attacker can downgrade the first request, the user clicks a plain http:// link, and the connection rides plaintext until Cloudflare upgrades it. Force HTTPS at the edge.

Watch out for the redirect loop

A mistake I see: site is on Flexible, the origin redirects HTTP→HTTPS at the app level (WordPress with the siteurl set to https://, for example). Cloudflare hits the origin on HTTP, the origin sends back a 301 to https://, Cloudflare follows it back to itself, and you get an infinite loop.

Symptom: page never loads, browser eventually shows "too many redirects."

Fix: don't redirect at the origin when you're on Flexible. Or, better, fix the actual problem — move to Full Strict and let the origin always speak HTTPS.

How to check what you have right now

In the Cloudflare dashboard:

SSL/TLS → Overview shows the current mode in plain words.
SSL/TLS → Edge Certificates shows the cert Cloudflare presents to visitors.
Edge → Origin Server → look at the cert source — that's the one to check is real.

From outside, you can also hit your origin's IP directly with curl --resolve to confirm it serves HTTPS:

curl -I --resolve yourdomain.com:443:<origin_ip> https://yourdomain.com/

If that errors with "SSL certificate problem" or "connection refused on 443," your origin isn't speaking HTTPS at all — Flexible was hiding that.

The padlock is a useful signal but it doesn't tell the whole story. Flexible SSL exists for a reason — early days, lots of origins didn't have certs — but it's a 2014 compromise and shouldn't be running anywhere in 2026. If you're holding client sites behind Cloudflare, audit the SSL mode on each one. It's a five-second check that catches a real problem.

I do this kind of edge-hardening for agencies running multiple client sites on Cloudflare — if your SSL setup hasn't been looked at in a while, happy to take a look.

Your Meta CAPI events from a Telegram bot are losing attribution. Here's the fix.

Boris Kl — Mon, 29 Jun 2026 19:37:36 +0000

A funnel I see all the time: a Meta ad sends people to a landing page, the page says "message us on Telegram," and a bot takes it from there — qualifies the lead, takes the order, whatever. The conversion happens inside the bot, server-side, so you fire it to Meta with the Conversions API instead of the Pixel.

Makes sense. Except attribution quietly falls apart, and it took me a while to see why.

The click ID is the whole game

Meta matches a server event back to the ad click using fbc. The format is fixed:

fb.1.<timestamp>.<fbclid>

That <fbclid> comes off the landing URL as ?fbclid=... when someone arrives from the ad. No fbc, and your CAPI event still lands — but match quality drops and Meta can't tie the conversion to the click. Which is the one thing you actually wanted.

So the job is: capture fbclid on the landing page, carry it into the bot, rebuild fbc there, send it with the event.

"Carry it into the bot" is where it breaks.

Telegram's deep link won't hold it

You move someone from web into a bot with a deep link:

https://t.me/your_bot?start=<payload>

The bot gets <payload> as the argument to /start. Perfect place to stash the click ID, right?

No. Telegram caps that start payload at 64 characters, and only allows A-Z a-z 0-9 _ -. A real fbclid runs way past that — the ones I've measured sit around 170+ characters. It does not fit. Truncate it and it's no longer a valid click ID. URL-encode it and it gets longer. There's no squeezing the real value through that gate.

This is the part nobody warns you about. The naive build looks fine in testing with a short fake fbclid, then drops attribution in production with real ones.

The fix: hand over a token, not the value

Don't pass the click ID. Pass a short token that points to it.

On the landing page, when the visitor taps through to Telegram:

Generate a short random token (a dozen URL-safe chars is plenty).
Store { fbclid, click timestamp, utm params } against that token server-side — Redis, a KV store, a tiny table.
Build the deep link with the token: https://t.me/your_bot?start=ab12cd34ef.

In the bot, on /start:

bot.start(async (ctx) => {
  const token = ctx.startPayload;            // "ab12cd34ef"
  const click = await store.get(token);      // { fbclid, ts, utm }

  if (click) {
    const fbc = `fb.1.${click.ts}.${click.fbclid}`;
    await sendCapiEvent({
      event_name: "Lead",
      action_source: "chat",   // it happened in a messaging app, not a website
      user_data: { fbc /* + whatever else you have */ },
      // event_id if you also fire a Pixel, so Meta dedups
    });
  }
  // no token? organic /start. Fine — just lower match quality, no fbc.
});

The token is short, so it sails through the 64-char limit. The real fbclid never has to travel through Telegram at all.

Two details that bite

Use the click timestamp, not the bot-open time. Store ts when they hit the landing page. People tap an ad now and open the bot tomorrow — if you stamp fbc with the moment /start fired, the window's off. Carry the original click time with the token.

action_source should tell the truth. A bot conversion isn't a website event — it happened in a messaging app, so the value is chat. (business_messaging is Meta's own channels like WhatsApp, not Telegram — don't borrow it.) Meta accepts non-website sources; misreporting it just muddies your own data later.

And give tokens a TTL. A click that never opens the bot leaves a dangling entry — expire them after a day or two.

None of this is exotic. It's one indirection — a token standing in for a value that's too big to move. But the failure is silent: events keep arriving, dashboards look populated, and attribution is just... soft, with no error to tell you why. If you're running paid traffic into a chat funnel, check whether your click IDs are actually surviving the handoff.

I build these bot funnels and the tracking behind them — if yours is leaking attribution somewhere, happy to take a look.

Approval-driven server ops: how I let contractors restart nginx without ever giving them SSH

Boris Kl — Fri, 26 Jun 2026 23:04:18 +0000

I run a small WordPress + Cloudflare agency. Two recurring pains finally pushed me to build something instead of complaining:

Pain 1: contractor access drift. I'd hand a contractor root SSH for "just this one task". Three jobs later, I'd realize four ex-contractors still had access I never rotated. The "I'll rotate keys later" lie compounds quickly.

Pain 2: 3am Telegram from a client. "Site is down" while I'm on the subway with no laptop. Find Wi-Fi, SSH in, systemctl restart nginx. Five minutes of actual work, ninety minutes of friction.

The constraint I wanted: a contractor should be able to restart nginx on a specific server, and nothing else. Not "a contractor with a limited shell". Not "a contractor with sudo restricted via sudoers". A contractor with no shell at all, and a chat command that does exactly one thing.

This article walks the architecture I ended up with. Real components, real grammar — names from the actual source tree. If it's useful, fork the idea — or if you'd rather not build it, link at the end.

The shape of the solution

Three components:

Operator (Telegram)
  APPROVE: action + nonce + ts (+ TOTP)
        ↓
Python control plane
  - aiogram bot
  - audit hash-chain (SHA-256)
  - policy engine + runtime state (SQLite WAL)
        ↓ mTLS              ↓ HTTPS
   Go agent           Cloudflare API
   (customer's        (under-attack mode,
    server)            challenges, blocks)
   diagnostics +
   allowlisted ops

The Go agent sits on the managed server. It speaks one protocol — mTLS HTTP — to the Python control plane. It never accepts shell. It accepts a fixed list of operations declared by a capability manifest loaded at startup.

The control plane talks to operators via a Telegram bot (aiogram) and to Cloudflare via their REST API. It runs the policy engine, persists state in SQLite (WAL mode), and writes a hash-chained audit log.

An operator's only interface is a Telegram chat. They never touch the server.

The APPROVE grammar

Every mutating action goes through this grammar. There is no other path.

APPROVE: <action> <client_id> nonce=<16-hex> ts=<unix_epoch>

With TOTP enabled (env var SECMON_TELEGRAM_TOTP_SECRET):

APPROVE: <action> <client_id> otp=<6digits> nonce=<16-hex> ts=<unix_epoch>

Rules the control plane enforces against the message:

Action allowlist. <action> must be in the canonical ActionID enum. Today that's restart-web, restart-php, restart-db, cf-under-attack, cf-managed-challenge, cf-targeted-block, cf-pattern-block, cf-rate-limit, enable-cf-auto. New actions require code + manifest + tests, not configuration. That's deliberate friction.
TTL. |now - ts| <= 300 seconds. Stale approvals get rejected, not silently applied.
One-time nonce. The 16-hex nonce is claimed atomically against a SQLite-backed nonce store. Replay = explicit rejection, not silent dedup.
TOTP second factor (when configured). Six-digit OTP from a pyotp-compatible secret. The control plane refuses the approval if the TOTP doesn't match.
Telegram sender allowlist. SECMON_TELEGRAM_ALLOWED_USER_IDS is checked before parsing. Unknown sender → message logged, action not even parsed.

If any of those fails, the agent never gets a request — the rejection happens in the control plane, gets written to the audit log with reason, and the operator gets a curt Telegram reply explaining what was wrong.

The capability manifest

The agent advertises its capabilities via a JSON document, signed at deploy time with HMAC-SHA256.

canonical = JSON of manifest body (keys sorted, no "signature" field)
signature = HMAC-SHA256(canonical, SECMON_MANIFEST_SECRET).hexdigest()

When the agent comes up, it loads capabilities.manifest.json. When the control plane dispatches an action, it pulls the agent's manifest, verifies the HMAC, and refuses anything not in the signed list. The agent cannot lie about what it supports — and the control plane cannot accidentally route a restart-db to a host whose manifest only declared restart-web.

This is the cheap version of the right idea. The expensive version is signed-everything-everywhere. HMAC-SHA256 with a shared secret is enough for "operator wants X, agent is allowed to do X". For multi-tenant trust I'd reach for asymmetric, but for a self-hosted agent paired with a single control plane, shared HMAC is good enough.

The audit log — hash-chained, verifiable offline

Every approval, every dispatch, every Cloudflare auto-mitigation gets a JSON record in an append-only JSONL file. Each record carries prev_hash and record_hash:

canonical   = JSON of record with prev_hash + record_hash stripped, keys sorted
record_hash = sha256(prev_hash || "|" || canonical).hexdigest()
prev_hash   = record_hash of the preceding line

The first record has prev_hash = "". Every subsequent record locks in everything before it. Tamper with any line, every line after fails verification.

Verification has a CLI: secmon-audit-verify. Walks the chain, returns line numbers of any break.

The threat model is "compromised SaaS cannot lie about what your servers did, after the fact". I think that's the audit threat model that actually matters for an agency that has to show a client what happened during an incident.

(The audit log is separate from the licensing JWT, which uses Ed25519. Different concern. Don't conflate them.)

What it deliberately is NOT

This is just as important as the feature list:

Not a monitoring stack. Use Prometheus, UptimeRobot, Grafana. This tool reacts to incidents — it doesn't detect them. The diagnostics module on the agent is read-only triage (/diag), not continuous monitoring.
Not a remote desktop. No raw shell exec. If you want SSH, you want SSH; this isn't a worse SSH.
Not DDoS protection. Cloudflare is your shield. The control plane can auto-dispatch a Cloudflare under-attack mode in response to certain patterns — but auto-dispatch has a hard kill-switch (SECMON_AUTO_CF_DISABLED=1) checked on every dispatch path. Manual APPROVE: cf-under-attack still works as an explicit operator override, with auto_dispatch_overridden_by=approve written to audit.
Not a ticketing system. No "open ticket for this incident". The audit log is the record, your team's normal tools do the workflow.

Every feature I didn't add is a feature I don't have to maintain. At my scale, that math dominates.

Threat model: assume the bot token leaks

The Telegram bot token gets passed around. Plan for the leak.

If the bot token leaks alone:

Attacker can spam the bot's Telegram channel.
Attacker cannot invoke an action, because:
- Their Telegram user ID isn't in SECMON_TELEGRAM_ALLOWED_USER_IDS. Message gets logged, parsing never starts.
- Even if they were on the allowlist, they don't have the TOTP secret.
- Even if they had the TOTP, they can't reuse a stolen nonce (one-time, atomic claim).
- Even with a fresh nonce, |now - ts| > 300s kills any captured approval.
Attacker cannot elevate themselves into the allowlist or the TOTP secret — both are env-var on the control plane host. Compromising the host changes the threat model entirely.

To actually run an operation against a server, the attacker would need all of:

Bot token (Telegram side)
TOTP secret (env var on control plane)
Telegram user ID on the allowlist (env var on control plane)
A fresh, unused nonce (or write access to the nonce store)
A timestamp within ±300s of dispatch

That's the design point. Each layer is cheap to add and forces an attacker to compromise distinct surfaces.

What I'd do differently

A few honest reflections:

TOTP belongs in v1, not v3. I shipped the bot + APPROVE grammar first, added nonce/TTL in a "security hardening" sprint, added TOTP in another security hardening sprint. Should have been one combined hardening pass before the first real customer touched it. I got away with it because the first customer was me, but I wouldn't ship a v2 without it.
The hash-chain is great until it isn't. A corrupt last line used to be swallowed silently — new events would be written with prev_hash="" and produce a "verifiable but forked" log. Fixed by explicitly propagating AuditChainError on read failure or JSON parse failure, with an opt-in manual recovery path. Lesson: tamper-evidence is a property of the whole chain, including the read path, not just the write path.
Per-tenant Cloudflare credentials need their own design. I had a single CF token at the start. That works for one client. For multi-client it became per-(client_id, zone_id) rows encrypted at rest with Fernet (SECMON_DATA_KEY). Earlier this had a one-key-per-deployment shape and I had to migrate — should have started multi-tenant from the first commit.
Diagnostics is bigger than ops. The agent's read-only diagnostics module (HTTP, web, PHP, DB, disk, SSL, flood patterns) is about 1.5K lines. The mutating-ops surface is smaller. That ratio is honest: you spend much more time looking than fixing, and read-only checks have to be deeply specific or they're noise.

Closing

If you run client infrastructure and the "shared root password" thing makes you wince, the architecture above is buildable. The Go agent is a single binary with no runtime deps. The control plane is Python with SQLite. Nothing exotic in the stack.

If you'd rather not build it: I packaged this up as secmon.io. Pricing is $49/mo Starter (1 server, 4h response), $199 Pro (5 servers, 1h response), $599 Agency (20 servers, monthly incident review). 7-day pilot on all tiers. The agent source is currently closed; whether to open-source it under AGPL is on the roadmap if there's demand — happy to discuss the trade-offs in comments.

If you'd like to be one of the first ten pilots, the contact form is on the homepage. Honest feedback is more useful to me right now than your money.

Critique very welcome — what's broken in the threat model? What's the obvious bigger competitor I missed?

Your page loads fast but still feels slow? It's INP, not load time

Boris Kl — Tue, 16 Jun 2026 10:27:25 +0000

Your Lighthouse report is mostly green. LCP is fine, CLS is fine, the page loads fast. Then the real-world score drops and you can't see why. Nine times out of ten the culprit is INP — and it's the one metric a quick Lighthouse run barely shows you.

What INP actually measures

INP, short for Interaction to Next Paint, replaced FID as a Core Web Vital in March 2024. FID only looked at the delay before your first interaction. INP looks at all of them, the whole time someone uses the page, and reports close to the worst one.

So it's not a loading metric. It's a responsiveness metric. It answers a different question: when I tap, click, or type, how long until the screen actually changes? Google's buckets are simple — 200ms or under is good, over 500ms is poor.

That's why a site can load in a second and still fail. Loading fast and responding fast are two different jobs, done by two different things.

Why it's invisible in a normal audit

LCP and CLS happen during load, so a lab tool catches them every run. INP only happens when a human interacts. Lighthouse doesn't tap your buttons, so its number is an estimate at best. You can have a green lab report and a red field score at the same time, and that gap is exactly where people get stuck.

To see the real number, measure interactions as they happen:

import { onINP } from 'web-vitals';
onINP(function (metric) {
  console.log('INP', metric.value, metric.entries);
});

That logs the actual slow interaction and the element behind it. Now you're fixing a real thing instead of guessing.

What's really slow

INP is almost always one thing: the main thread was busy when the user acted. The browser can't paint the response until the current JavaScript task finishes, so a long task blocks the interaction.

The usual sources:

A heavy event handler doing real work on every click or keystroke.
Third-party scripts like chat widgets, analytics, and tag managers, running long tasks at the wrong moment.
Layout thrash: reading and writing the DOM in a loop so the browser recalculates over and over.
Framework hydration waking the whole page up at once.

The fixes that move it

Break up long tasks. If a handler does a lot, let the browser breathe partway through instead of holding the thread:

async function onClick() {
  doUrgentPart();          // update the UI first
  await yieldToMain();     // give the browser a turn to paint
  doExpensivePart();       // the rest can wait a tick
}

yieldToMain is a one-line helper around scheduler.yield() where it's supported, or a setTimeout(0) fallback. The trick is to paint the response before the slow work, not after.

Beyond that: defer scripts the page doesn't need to react, audit third-party widgets for the ones that run long tasks, debounce expensive handlers, and batch your DOM reads and writes so the browser isn't recalculating layout on every line.

The honest part

I won't promise you a magic number — INP depends on your scripts, your theme, and what your users actually click. But it's measurable, and the field data shows the difference plainly once the long tasks are gone.

I keep my own WordPress sites in the green on Core Web Vitals, and INP is the one I watch most now, because it's the one that quietly fails while everything else looks fine. If your lab report is green but the real score isn't, stop staring at LCP. Go measure an interaction.

Your Telegram bot replies twice? It's timing, not a logic bug

Boris Kl — Mon, 15 Jun 2026 13:23:49 +0000

A Telegram bot replies to the same message twice. An n8n flow processes an order, then processes it again ten seconds later. The owner reads the handler code, finds nothing wrong, and assumes the logic is broken.

It usually isn't. These bugs are almost always about timing, not logic — and once you know the three places timing bites, they stop being mysterious.

1. The webhook you never answered

Telegram (and most webhook senders) wait for an HTTP 200. If your endpoint does the work first and answers afterward, a slow database call or a third-party API can push you past the timeout. The sender assumes delivery failed and sends the same update again. Now your "double reply" isn't a logic bug — it's the same event arriving twice because you were too slow to say "got it."

The fix is to acknowledge first, process second:

@app.post("/webhook")async def webhook(request):
    update = await request.json()
    queue.put_nowait(update)   # hand off
    return Response(status=200) # answer immediately

Return 200 the moment you've safely stored the update. Do the real work in a background task or a worker. The sender stops retrying, and the duplicates dry up.

2. No dedup, so retries become real work

Answering fast helps, but retries still happen — network blips, restarts, a sender that's feeling anxious. The honest assumption is: every event can arrive more than once. So make handling it twice harmless.

Every Telegram update has an update_id. Every message has a message_id. Most webhook payloads have some stable id. Key on it:

if await seen.exists(update_id):
    return            # already handled, do nothing
await seen.add(update_id, ttl=86400)
await handle(update)

seen can be Redis, a unique column in your database, anything that's shared across workers. The point is that "process this order" runs once even if the event shows up three times. People call this idempotency; it just means doing it again changes nothing.

3. Two messages, one piece of state, no lock

This is the one that looks the most like a logic bug and isn't. A user double-taps a button. Two updates arrive almost together. Both handlers read "balance: 100", both subtract 30, both write "70". You charged once for two actions, or booked the same slot twice.

Nothing in the logic is wrong. The two runs just overlapped. The fix is to stop them from overlapping on the same state:

async with lock(f"user:{user_id}"):
    balance = await get_balance(user_id)
    await set_balance(user_id, balance - 30)

A per-user lock (Redis SET NX, a database row lock, whatever you have) means update B waits for update A to finish before it touches the same row. In n8n the same idea shows up as a queue or a "wait for previous execution" step instead of letting every webhook fire its own parallel run.

The part that saves you next time

Most of these never get diagnosed because they're invisible. The handler "works" when you test it by hand — you can't tap fast enough to cause the race, and your local webhook answers instantly. It only breaks under real traffic, at 3am, where you're not looking.

So log the timing, not just the errors. Log the update_id on the way in and the way out. Log when a lock is contended. The first time you see the same update_id logged twice, the whole thing stops being a mystery and becomes a one-line fix.

I run Telegram bots and n8n in production every day, and I've hit all three of these. None of them were in the logic. They were in the gaps between events — and that's almost always where to look first.

Sending Telegram Bot Conversions to Meta? Don't Reach for business_messaging

Boris Kl — Sun, 14 Jun 2026 13:45:21 +0000

A bot was firing Subscribe and Purchase events from Telegram straight to Meta's Conversions API, and every call came back with a 400:

"error_user_title": "Missing Messaging Channel Parameter",
"error_user_msg": "A messaging channel parameter is required when provided
                   action source is business_messaging. Valid value could be
                   messenger, whatsapp and instagram."

The payload looked fine — event_name, event_time, a hashed external_id, and action_source: 'business_messaging'. So why the 400?

The gotcha

business_messaging is not a generic "it happened in a chat" source. Meta ties it to its own messaging products, and it demands a companion messaging_channel whose only valid values are messenger, whatsapp, instagram. Telegram isn't on that list — there's no channel you can hand it — so the request can never validate.

The instinct is to try app next. Don't. app drags in a required app_data block: the extinfo array, advertiser tracking flags, the whole mobile-SDK surface. You don't have that from a bot, and you don't want to fake it.

The fix

For a self-hosted Telegram bot, the right source is plain other:

"action_source": "other"

other has no extra mandatory fields. You need event_name, event_time, action_source, and a user_data with at least one identifier. A SHA-256 hashed Telegram user id as external_id is enough to clear the 400. One-line change.

Making it actually attribute

Not crashing is the low bar. To tie a Subscribe or Purchase back to the ad that caused it, you need the click id:

Capture fbc. Your ad sends people to a deep link — t.me/yourbot?start=.... Meta appends fbclid to that destination. Pack the fbclid into the start payload, read it on /start, and build fbc = fb.1.[unix_time].[fbclid]. Send it in user_data next to external_id. This is the single biggest lever for matching.
Purchase needs money. Add custom_data with value and currency, or there's no ROAS to compute later.
Verify before you trust it. Events Manager has a Test Events tab — send with a test_event_code and watch the events land and match before you point real traffic at it.
Dedupe if a web pixel fires the same events: same event_id on both sides and Meta collapses them.

The 400 is a five-second fix. The attribution is the part that actually pays for itself.

Lighthouse Gave My Site 100/100. The Site Was Down.

Boris Kl — Thu, 11 Jun 2026 18:56:58 +0000

Yesterday I ran PageSpeed Insights on a site I manage. Performance: 100/100. Green circle, confetti, the works.

One problem: the screenshot in the report showed a Cloudflare block page — "Sorry, you have been blocked."

Lighthouse didn't measure my site. It measured the error page my WAF served to Google's crawler. And error pages are, of course, blazing fast.

How this happens

If you put Cloudflare in front of a site and turn the security dial up (Bot Fight Mode, aggressive WAF rules, country blocks), you'll eventually block more than bots:

PageSpeed Insights / Lighthouse — measures a block page, reports nonsense
Uptime monitors — see HTTP 403 with a 200-ish body, or vice versa, and lie to you either way
Google's crawler itself — and that one quietly costs you rankings

The nasty part is the silence. Nothing looks broken from your own browser, because you're whitelisted by your own cookies, IP reputation, or login session. The tools just start telling you fairy tales.

The five-minute audit

Open Cloudflare → Security → Events. Filter the last 7 days. Look at what's actually being challenged or blocked — you'll usually find a legit service in there within a minute.
Check the user agents: Chrome-Lighthouse, GoogleOther, Googlebot, your uptime checker. If they show up here, that traffic never reached your site.
Verify bots properly: Cloudflare has a "Verified Bots" category — allow it instead of hand-maintaining user-agent allowlists (user agents are trivially faked; verified-bot checks aren't).
Re-run your measurement and look at the rendered screenshot, not just the score. The screenshot is the only part of a Lighthouse report that can't lie to you.

Rules I now follow

Never trust a perfect score. 100/100 on a real WordPress/commerce site is a smell, not an achievement. Real sites have real images and real JavaScript.
Check the screenshot first, score second.
After every WAF change, re-test from outside: different network, curl with a Googlebot UA, or just PageSpeed Insights — and read the Events log after.
Monitoring that runs behind your own allowlist isn't monitoring. It's a mirror.

Cloudflare is still the best free thing that ever happened to small sites — I run my own production behind it and it has eaten real attack waves for breakfast. But a security layer you configured and never audited is just a random traffic filter with good branding.

Five minutes in the Events log. That's the whole tip.

I set up Claude Code for a real production project. Here's what actually earned its keep

Boris Kl — Sat, 06 Jun 2026 13:53:19 +0000

Everyone's got a "10 AI coding tricks" post. This isn't that. This is what's left after three weeks of running Claude Code on a real project — a bilingual booking bot for a beauty salon (Telegram + WhatsApp, Postgres, Google Calendar) — once the novelty wore off and only the useful parts survived.

Out of the box, Claude Code is a very smart intern with amnesia. Every session it shows up brilliant and clueless. The whole game is fixing the clueless part. Four things did that for me: a CLAUDE.md file, two custom agents, one skill, and two hooks. Everything else I tried, I deleted.

CLAUDE.md: the file that pays rent every single day

CLAUDE.md sits in your repo root and gets read at the start of every session. Mine started as three lines. It grew every time the assistant did something I had to undo.

That's the trick, honestly. Don't write CLAUDE.md upfront — grow it from failures. Mine now includes things like:

## Architecture rules
- Business logic lives in src/core/ and must not know about
  Telegram or WhatsApp. Channel code lives in src/adapters/.
- All times stored in UTC; convert only for display.
- Booking creation must stay double-booking-safe — never remove
  locks or constraints around it.

## Working agreements
- Before "done": run typecheck && lint && test and show the result.
- Schema changes go through a migration file. Always.
- Prefer the smallest diff that does the job.

Each of those lines exists because the assistant once did the opposite. It put Telegram-specific code in core logic — new rule. It "fixed" a timezone bug by converting at storage time — new rule. It reported "done" with failing types — new rule.

Three weeks in, I almost never repeat an instruction. That file is the difference between an assistant and a goldfish.

Custom agents: the reviewer I argue with

Custom agents live in .claude/agents/ as markdown files with a system prompt. You invoke them for a specific job, they do it with their own instructions and tool limits, and they don't pollute your main session's context.

The one that earns its keep daily is a code reviewer:

---
name: code-reviewer
description: "Reviews changes for bugs and security issues"
  before they are committed.
tools: Read, Grep, Glob, Bash
---

You are a strict but practical code reviewer.
Check, in this order: correctness (timezone boundaries,
double-booking windows), security (unvalidated webhook input,
SQL built by concatenation, missing signature checks),
project rules from CLAUDE.md, and whether behavior changed
without a test changing.
Report findings ordered by severity, with file:line and a
concrete fix. If something is fine, don't pad the review.

The point isn't that it catches everything. The point is that it's a different context with one job. The main session wrote the code and is biased toward liking it. The reviewer agent reads it cold. It regularly catches things the main session waved through — a webhook handler that trusted message_id without checking the signature, a slot calculation that broke across midnight.

It found the midnight bug before my client's customers did. That one agent paid for the whole setup.

A skill that stops me from skipping steps

Skills are reusable workflows — a SKILL.md file describing a procedure the assistant follows when you invoke it. I have exactly one that matters, /add-feature:

Restate what we're building, confirm.
List files that will change and why. Smallest possible diff.
Implement, following CLAUDE.md.
Write tests for the changed units.
Run the code-reviewer agent on the diff. Fix what it finds.
Summarize: what changed, how to try it, what I must do manually.

Nothing clever. It's a checklist. But here's the thing about checklists — they work precisely because on the fifth feature of the day, I would skip the review step. The skill doesn't get tired at 11pm. Pilots figured this out decades ago; we're just catching up.

Hooks: the two-line insurance policy

Hooks run shell commands on events. I only need two.

The first blocks any edit to secrets files. The assistant has no business touching .env, ever, and now it physically can't — a PreToolUse hook checks the file path and exits with an error if it looks like secrets. Cost me five minutes to write. Worth it the first time a refactor tried to "helpfully" update an env var.

The second runs the typecheck after every file edit and pipes problems straight back into the session. The assistant sees its own type errors immediately instead of discovering them at the end, which means it fixes them while the context is hot. This one change cut my "it said done but nothing compiles" rate to roughly zero.

{
  "hooks": {
    "PreToolUse": [{
      "matcher": "Edit|Write",
      "hooks": [{ "type": "command",
        "command": "bash .claude/hooks/protect-secrets.sh" }]
    }],
    "PostToolUse": [{
      "matcher": "Edit|Write",
      "hooks": [{ "type": "command",
        "command": "npm run --silent typecheck" }]
    }]
  }
}

What I tried and deleted

For honesty's sake: I also built an agent for writing commit messages (the main session does this fine), a skill for deployments (too risky to automate, I want my hands on that), and a hook that auto-ran the full test suite on every edit (made everything crawl — the typecheck is the right granularity; full tests run at review time).

If a piece of setup doesn't save you something every day, it's not configuration, it's clutter.

The honest summary

Claude Code without setup is a talented freelancer on their first day, every day. With a grown-from-failures CLAUDE.md, one cold-eyed reviewer agent, one checklist skill and two hooks, it's closer to a colleague who's been on the project for a month.

The setup took me about two hours total, spread over days, mostly as reactions to things that annoyed me. The payback is that I now ship features for a production bot — payments, reminders, a wait-list — in evenings, alone, without the quality dropping.

Start with CLAUDE.md. Add a reviewer agent the first time you catch a bug you should've caught. Grow the rest from your own failures — they're better teachers than my list anyway.

One year of self-hosted n8n on a $6 Hetzner VPS

Boris Kl — Wed, 27 May 2026 11:49:40 +0000

One year of self-hosted n8n on a $6 Hetzner VPS

Twelve months ago I moved my workflow automation off Zapier and onto a single Hetzner CX22 — €4.51/mo, 2 vCPU, 4 GB RAM, 40 GB disk. One Docker host, one n8n container, one Postgres, one Caddy reverse proxy. It's run four production workflows continuously since then, with one outage I'll get to below.

This post is not a "n8n vs Zapier" pitch. It's a year of operating notes — what stayed cheap, what broke, what I'd do differently.

The actual setup

Hetzner Cloud CX22 (Falkenstein)
├── Docker
│   ├── n8n (latest stable)
│   ├── postgres:15
│   └── caddy (with automatic TLS)
├── UFW (22, 80, 443 only)
└── borgbackup → Hetzner Storage Box (€3.81/mo)

The Caddy bit matters more than people think. n8n's built-in HTTP is fine for localhost, but webhook receivers need real TLS, and Caddy gives you ACME, HTTP→HTTPS redirect, and per-domain certificates with zero config. Caddyfile is six lines. You don't have to think about it again.

What's running

Four workflows. None of them invented; all real:

Telegram bot dispatcher. Inbound webhook → routing logic → either a Postgres write or a downstream service call. About 40 events/day average, occasional 200-event spikes.
RSS aggregator → Telegram channel. Polls 12 feeds every 15 min, dedupes by URL hash in Postgres, posts new items to a private channel. ~30 posts/day.
Form submission → CRM-lite. A few WordPress sites hit a webhook on form submit; n8n writes to Postgres, sends an email confirmation, and logs to a Discord channel for me.
Daily reporting cron. Pulls metrics from three internal APIs at 06:00, builds a markdown digest, emails it, also posts it to Slack.

None of these need millisecond latency. All of them benefit from being one config-pull away from changing.

The cost breakdown (12 months)

Item	Monthly	Annual
Hetzner CX22	€4.51	€54.12
Storage Box (backup)	€3.81	€45.72
Domain (.dev)	—	€12
Total	~€9.20	~€112

Equivalent Zapier seat for the same task volume would have been ~$30-50/month depending on the plan, so we're looking at roughly €350-500 saved over the year. Not life-changing. The real win is something else, which I'll get to.

What broke (the one outage)

Month four. n8n upgraded from v1.x to a major release. I'd been running docker compose pull weekly without pinning, because "it's been fine." The upgrade introduced a breaking change to how credentials were stored. Container started; UI loaded; every workflow showed "credentials missing" and refused to execute.

Root cause: I had no version-pin and no upgrade test. The backup was fine (borg snapshots intact), but the restore-and-investigate took me a Saturday afternoon.

What I changed:

Pinned n8n image to a specific minor version (n8nio/n8n:1.45.x).
Added a "staging" branch on a second Hetzner VPS (€3/mo CX21) that gets the upgrade first.
Subscribed to the n8n releases RSS feed so I see breaking changes before I pull.

In hindsight: a SaaS would have done the upgrade for me and either Bigger Things would have broken (multi-tenant blast radius) or none of this would have ever happened. Pick your trade.

The actual win (it's not the money)

The €350/year doesn't matter. What matters is that workflows live in a git-tracked YAML I own, on infrastructure I own.

When a workflow changes, I commit the n8n export. When something breaks, I can diff yesterday's export against today's and see what shifted. When the credentials database gets weird, I open psql and look at the rows. When the webhook target changes, I write the new URL in a Caddyfile and reload — no support ticket, no rate limit on changes, no "this requires an upgrade to the Team plan."

On Zapier, the same change graph is a black box. Some changes are free, some require the next plan tier, and you don't always know which until the click. With n8n on a box you control, the question "can I do this?" reduces to "is it physically possible?" — and the answer is almost always yes.

Things I'd do differently if starting today

Pin the image from day one. Whatever the cost in "missing the new shiny feature for a week" is dwarfed by the cost of an unscheduled Saturday.
Use external Postgres, not the docker-compose one. Hetzner offers managed Postgres now. €11/mo, automatic backups, no "my container restarted and ate the WAL" risk. I'd take the €11 hit gladly.
Don't put auth on the webhook receivers via n8n itself. Put it at Caddy or a separate gateway. n8n's auth model exists, but you can't reuse it for non-n8n endpoints, and you'll regret the coupling.
Write the runbook first, not after the first outage. "How do I restore from borg," "how do I roll the credentials key," "where are the env files" — five minutes to write, an hour to rediscover when stressed.
Don't put more than 10 workflows on one box. Memory usage scales with concurrent execution, and a runaway loop in one workflow will starve the others. If you go past 10, split into two n8n instances, not one.

When NOT to self-host

This setup works because the four workflows are mine, the data is mine, and downtime measured in hours (not minutes) is acceptable. If any of those three change, the calculus changes.

If a client depends on the webhook receiver having 99.95% uptime, this single-box setup is wrong. Use n8n Cloud or a multi-node deployment.
If the workflows touch regulated data (HIPAA, PCI, GDPR's stricter applications), don't reach for the cheapest box. Use a vendor who'll sign a DPA and an audit-ready hosting tier.
If you're a team of more than three and people need fine-grained access, n8n self-host's RBAC is workable but not great. The Cloud tier handles teams better.
If your time is worth more than €30/month, and the workflows are simple enough that Zapier or Make.com handles them without ceremony, the savings aren't worth the operating load. Pay for the SaaS.

The five-line take

Self-hosted n8n on a cheap VPS is one of those rare cases where the "boring" answer is also the cheap one and also the powerful one. Run it for a year before you decide it's not for you. Pin your versions. Write the runbook. Don't put it on the same box as anything else important.

— Boris (@lamastoma)

Publishing checklist

☐ Set published: true
☐ Add cover image (1000×420 — Hetzner ANGE + n8n logo composite? or just terminal screenshot)
☐ Tags: n8n, selfhosted, automation, devops — Dev.to limits to 4
☐ Canonical URL: leave blank (Dev.to is canonical)
☐ Once published, share Fiverr profile URL in bio (not in body of article)
☐ Comment-engagement plan: monitor for first 24h, reply to every comment, no defensive corrections

A Production Python Telegram Bot Was Crashing Every 2 Hours. The Fix Was 18 Lines.

Boris Kl — Wed, 20 May 2026 13:28:23 +0000

"If you see cascading errors, find the first thing that fails and stop reading the log there. Everything after the first failure is the system reacting to the first failure."

A production Python Telegram bot I was looking after started crashing every 2-3 hours. The traceback was a horror show — TelegramRetryAfter, then asyncio.TimeoutError, then sqlite3.OperationalError: database is locked, then 47 leaked sessions, then the process got OOM-killed, then systemd restarted it. Then it happened again, 140 minutes later, like clockwork.

The temptation when you see this kind of cascade is to throw the whole architecture out. "SQLite can't handle our scale, let's move to Postgres." "Bare asyncio is too low-level, let's add a queue." "Let's rewrite it in Go."

I didn't do any of those things. The fix was 18 lines of code in one middleware file. The bot has been up for weeks since.

Here's the diagnosis, the fix, and the takeaway. The code is real (anonymized of any client specifics) and the numbers are real.

The symptoms

Stack: Python 3.12, aiogram 3.x, SQLite for user state, asyncio everywhere. Volume: about 4,000 daily incoming messages. Not high-throughput.

The log every 140 minutes looked like this:

[14:22:01] ERROR  aiogram.TelegramRetryAfter: flood control, retry in 28s
[14:22:03] ERROR  asyncio.TimeoutError in update handler
[14:22:05] WARNING bot.session not closed (47 active)
[14:22:08] ERROR  sqlite3.OperationalError: database is locked
[14:22:14] ERROR  ...same pattern, multiplying...
[14:22:20] ERROR  process killed by OOM
[14:22:21] INFO   systemd: restarted

Process up ~140 minutes. Then the cascade. Then restart. Repeat.

What looked plausible (and was wrong)

When I started looking, the first hypothesis was "SQLite is the bottleneck — it can't handle the concurrency." That's the most obvious thing to say when you see database is locked in a log.

It was wrong. Here's why I dropped it after 30 minutes:

4,000 messages a day is nothing for SQLite. SQLite handles tens of thousands of writes per second on modest hardware. If we were hitting a SQLite ceiling, we'd be hitting it under steady load, not in sudden bursts. The 140-minute interval was the giveaway — something was accumulating, not saturating.

The second hypothesis was "We're hitting Telegram API rate limits." That's what TelegramRetryAfter literally says. But again, 4,000 messages a day = roughly 1 message every 20 seconds on average. Telegram's per-bot rate limit is 30 messages per second. We weren't even in the same order of magnitude.

So whatever was happening was bursty, not steady-state. And the bot was somehow turning a steady stream of inbound updates into a burst of outbound API calls.

The actual root cause

Here's what was happening, step by step:

A user sends a message. aiogram receives it as an update.
The handler runs, does some work, and sends a reply to Telegram.
Normally: that reply goes out, the handler returns, the asyncio task ends, the bot.session HTTP connection is released.
What actually happened: no throttle middleware existed. If 5-10 users happened to message in the same second (which happens during peak hours), the bot fired 5-10 outbound sendMessage API calls concurrently.
Five or ten outbound requests inside one second pushed us past Telegram's per-second rate limit. Telegram answered with 429 Too Many Requests and a retry_after header.
aiogram raised TelegramRetryAfter. But the handler that raised it was waiting on the API response — it couldn't release its HTTP session until the retry window closed (28 seconds in the log above).
While that handler was waiting, the next inbound update hit the same handler code. Another async task spawned. Another bot.session connection opened. Another wait.
Now we have two stuck tasks, each holding a connection, each blocked on retry_after. Both tasks also need to update the user's row in SQLite. SQLite locks the row for the first writer. The second writer waits. Deadlock potential.
Multiply this by 10 minutes of bursty traffic. Now you have 47 leaked sessions, an SQLite deadlock, and a Python process eating memory because tasks aren't completing.
OOM killer hits. Systemd restarts. Cycle resets.

The cascade had one cause: no rate limit on the bot's inbound side. Everything downstream was just the system reacting to the upstream pressure.

The fix — 18 lines

A throttle middleware. Drop incoming updates from a user if they already had a message in the last second. That's it.

# middleware.py
from aiogram import BaseMiddleware
from aiogram.types import Update
from cachetools import TTLCache


class ThrottleMiddleware(BaseMiddleware):
    """Drop second-message-within-N-seconds per user.

    Without this, bursty inbound traffic translates 1:1 into bursty
    outbound API calls and trips Telegram's flood control.
    """

    def __init__(self, rate_limit: float = 1.0):
        self.cache = TTLCache(maxsize=10_000, ttl=rate_limit)

    async def __call__(self, handler, event: Update, data):
        user_id = event.message.from_user.id if event.message else None
        if user_id and user_id in self.cache:
            return  # silently drop — user is over their rate limit
        if user_id:
            self.cache[user_id] = True
        return await handler(event, data)

And wire it up plus a clean shutdown:

# main.py
from aiogram import Bot, Dispatcher

bot = Bot(token=BOT_TOKEN)
dp = Dispatcher()

dp.update.middleware(ThrottleMiddleware(rate_limit=1.0))


async def on_shutdown():
    """Close the bot session explicitly. Otherwise sessions leak
    on graceful shutdown and the next start hits a connection pool
    in a weird state.
    """
    await bot.session.close()


dp.shutdown.register(on_shutdown)

That's 18 lines of production code plus one test:

# test_middleware.py
import pytest
from middleware import ThrottleMiddleware


@pytest.mark.asyncio
async def test_throttle_drops_rapid_second_message(mocker):
    middleware = ThrottleMiddleware(rate_limit=1.0)
    handler = mocker.AsyncMock(return_value="processed")

    event = make_event(user_id=123)  # helper to build a fake aiogram Update

    # First message — goes through
    result1 = await middleware(handler, event, {})
    assert result1 == "processed"

    # Second message same user, same second — dropped
    result2 = await middleware(handler, event, {})
    assert result2 is None

    handler.assert_called_once()

That's the whole patch.

Why this works

The fix doesn't make SQLite faster. It doesn't add a queue. It doesn't change anything about how the handlers process messages. It just stops the upstream pressure before it cascades downstream.

Once incoming updates are rate-limited per-user at 1 per second, the bot never has 10 concurrent outbound API calls. It has at most 1-2. Telegram never gets angry. TelegramRetryAfter never fires. Handlers never get stuck waiting. Sessions never leak. SQLite never sees concurrent writes for the same row.

The cascade isn't a chain. It's a tree, and the throttle cuts the tree at the root.

The result

Numbers (real, from production):

First 4 hours after deploy: zero TelegramRetryAfter. Zero TimeoutError. Session count stable at 1-2 (vs. climbing past 40 every two hours before).
First 24 hours: zero errors of any kind in the log.
First 7 days: zero crashes. Zero systemd restarts.

Bot has been up continuously since deploy. Same SQLite. Same asyncio. Same handlers. The only thing that changed is the throttle middleware.

What I'd tell a junior on the team

A few generic takeaways that apply far beyond this specific bug:

1. Find the first failure in the log and stop reading. When you see cascading errors, everything after the first failure is the system reacting to the first failure. Don't try to "fix" the downstream errors. Find the upstream cause.

2. Upstream backpressure is the cause about 80% of the time when you see async-Python cascades. When the downstream component (SQLite, HTTP client, worker pool) looks stuck, it's almost always waiting for something the upstream is doing too fast. Rate-limit the upstream first.

3. The temptation to rewrite is almost always wrong early in diagnosis. "Rewrite in Go" / "switch to Postgres" / "add a queue" are valid responses to real scale problems. They're not valid responses to "I haven't figured out the bug yet." Spend an hour with the actual logs first.

4. Volume matters less than burstiness. A system handling 4k messages/day average can absolutely fall over from 10 messages in one second. The metric you care about is peak concurrency, not total throughput.

5. Test the throttle as a unit, not as an integration. The fix above has one test (12 lines). It doesn't try to spin up a real bot. It just verifies the middleware behavior in isolation. That's enough — the actual production behavior is downstream of this contract holding.

Code

The middleware and the test are public:

→ github.com/lamas51/claude-code-templates (case studies folder)

Same project also has Claude Code agent/skill/hook templates I deploy across Go, Python, and WordPress projects — feel free to fork.

About me

I'm Boris — IT-pro since 1999. I run production code across Go, Python, and React, mostly for small and mid-size businesses. Last 18 months I've been heavy on Claude Code workflow.

If you have a production Python service throwing similar cascades and want help diagnosing it, I take this kind of work through Fiverr (clean scope, escrow, no off-platform contact):

→ fiverr.com/lamastoma — Python / n8n / Telegram bot bug fixing in 24 hours

Open to questions in the comments — happy to dig into specifics if you're seeing something similar.

Anonymized — no client data, the diagnosis flow and final patch are the actual ones I shipped.

DEV Community: Boris Kl

Your Telegram bot isn't crashing randomly. It's on a timer.

Timer #1: the connection that quietly went stale

Timer #2: two transports fighting over one bot

Timer #3: the request with no timeout

Make the next failure introduce itself

I've shipped production code since 1999. AI didn't make me faster where it counts.

Typing was never the bottleneck

The trap I catch myself in

What actually got faster

Why 1999 me would recognize this

Cloudflare's Flexible SSL looks secure. It isn't.

What Cloudflare's SSL modes actually mean

The threat model people miss

The fix

Watch out for the redirect loop

How to check what you have right now

Your Meta CAPI events from a Telegram bot are losing attribution. Here's the fix.

The click ID is the whole game

Telegram's deep link won't hold it

The fix: hand over a token, not the value

Two details that bite

Approval-driven server ops: how I let contractors restart nginx without ever giving them SSH

The shape of the solution

The APPROVE grammar

The capability manifest

The audit log — hash-chained, verifiable offline

What it deliberately is NOT

Threat model: assume the bot token leaks

What I'd do differently

Closing

Your page loads fast but still feels slow? It's INP, not load time

What INP actually measures

Why it's invisible in a normal audit

What's really slow

The fixes that move it

The honest part

Your Telegram bot replies twice? It's timing, not a logic bug

1. The webhook you never answered

2. No dedup, so retries become real work

3. Two messages, one piece of state, no lock

The part that saves you next time

Sending Telegram Bot Conversions to Meta? Don't Reach for business_messaging

The gotcha

The fix

Making it actually attribute

Lighthouse Gave My Site 100/100. The Site Was Down.

How this happens

The five-minute audit

Rules I now follow

I set up Claude Code for a real production project. Here's what actually earned its keep

CLAUDE.md: the file that pays rent every single day

Custom agents: the reviewer I argue with

A skill that stops me from skipping steps

Hooks: the two-line insurance policy

What I tried and deleted

The honest summary

One year of self-hosted n8n on a $6 Hetzner VPS

One year of self-hosted n8n on a $6 Hetzner VPS

The actual setup

What's running

The cost breakdown (12 months)

What broke (the one outage)

The actual win (it's not the money)

Things I'd do differently if starting today

When NOT to self-host

The five-line take

Publishing checklist

See also

A Production Python Telegram Bot Was Crashing Every 2 Hours. The Fix Was 18 Lines.

The symptoms

What looked plausible (and was wrong)

The actual root cause

The fix — 18 lines

Why this works

The result

What I'd tell a junior on the team

Code

About me