DEV Community: Boris Kl

Your n8n workflow stopped three days ago. Nobody noticed.

Boris Kl — Tue, 11 Aug 2026 17:05:54 +0000

The message always has the same shape: "Hey, are the leads still coming through? I don't think I've seen one since Tuesday."

You open n8n. The workflow is right there. Green. "Active." Looks perfectly healthy. Then you check the executions list and your stomach drops — the last successful run was three days ago. Every trigger since has failed, silently, and the automation just... stopped mattering. No alert. No email. Nothing. It failed the way a light bulb fails: quietly, and you only notice in the dark.

That gap — "active" on the canvas but dead in reality — is the single most expensive thing about self-hosted automation. Not the crash. The silence around the crash. Here's why it happens and how to make the next failure announce itself instead of hiding.

Why "active" lies to you

The word "active" on a workflow only means one thing: the trigger is armed. It says nothing about whether runs are succeeding. A workflow can be active and failing every single time, forever, and n8n will keep showing that calm green dot.

Three failures cause almost all of the silent ones I get called in for.

An upstream API changed and nobody told you. A field got renamed. An auth token expired. The endpoint now returns a 429 under load. Your node throws, the execution stops, and unless you wired up something to catch it, that error dies inside the execution log where no human ever looks.

A trigger that quietly stopped firing. Polling triggers depend on a schedule. Webhook triggers depend on the sending service still pointing at your URL. If the n8n process restarted and the queue setup is off, the schedule may not come back. The workflow looks active. The trigger is asleep.

The process itself fell over. Self-hosted n8n on a small VPS hits the OOM killer more than people expect. A big run spikes memory. Linux kills the process. Systemd may or may not bring it back cleanly. Any workflow mid-flight is just gone. No summary. No "3 workflows were interrupted" notice.

Every one of these looks identical from the dashboard: green, active, fine. That's the whole trap.

Make failure loud: the error workflow

The single highest-value thing you can build in n8n is a workflow whose only job is to tell you when other workflows fail.

n8n has this built in and almost nobody turns it on. You create one workflow that starts with the Error Trigger node. Then in the settings of every other workflow, you set that error workflow as the handler. Now any unhandled failure, anywhere, fires it — and you route that to wherever you actually look. Telegram. Slack. Email. Whatever you'll see within the hour.

The message that error workflow sends should carry enough to act on without opening n8n:

❌ Workflow failed: "Lead intake → CRM"
Node: HTTP Request (Create contact)
Error: 401 Unauthorized
Time: 2m ago
Execution: <link straight to the run>

That's the difference between "I found out three days later from a client" and "I knew two minutes after it broke." One error workflow covers your whole instance. You build it once. It's the closest thing to free insurance that self-hosted automation has.

The failures an error workflow can't catch

Here's the part people miss. An Error Trigger only fires when a workflow runs and throws. It cannot fire when the workflow never runs at all — a dead poller, a webhook pointed at a URL that no longer exists, a process that's been down for a day. Silence can't trigger an alert about silence.

So you also need a heartbeat working the other way. A dead man's switch.

Pick your most important workflow. Have it write a timestamp somewhere cheap on every successful run — a row in a database, a value in Redis, a ping to a free uptime monitor. Then set that monitor to scream if the timestamp goes stale. If "last success" is older than, say, twice the normal interval, something upstream is dead even though nothing threw an error.

The logic is boring and that's the point:

On every successful run:
  update last_success = now()

Separate check, every 15 min:
  if (now() - last_success) > 2 × expected_interval:
    alert("Workflow X hasn't succeeded in too long — trigger may be dead")

Error workflow catches the loud failures. Heartbeat catches the quiet ones. You need both, because they fail in opposite directions.

Stop feeding the OOM killer

If your process keeps dying on a small box, throwing more RAM at it is the lazy fix. Two settings do more.

Cap how much execution history n8n keeps in the database. Logs with no limit are a classic slow bleed of memory and disk. And for workflows that move real volume, turn on queue mode. Executions then run in separate worker processes instead of piling into one. One heavy run costs you a worker, not the whole instance. A crash stops being an all-or-nothing event.

The mindset that outlasts any single fix

Automation you can't see is a risk wearing the mask of an asset. The goal was never "set it and forget it." It's "set it, and get told the moment it breaks."

So the checklist is short. One error workflow wired into every workflow you care about. A heartbeat on the critical ones, watching for silence. Bounded execution history. Queue mode once volume is real. None of it is fancy. All of it turns "a client noticed before I did" into "I fixed it before the client noticed."

I build and rescue self-hosted n8n setups for people running real business processes through them, and the first thing I check is never the broken node — it's whether anything would have told them it broke.

So, honest question: if your most important automation died right now, how would you find out — and how long would that take? If the answer is "a customer would tell me," that's the actual bug. What's your alerting setup for the stuff running quietly in the background?

Your MCP server works in the demo. Here's why the model still ignores your tool.

Boris Kl — Tue, 04 Aug 2026 22:02:30 +0000

The server was done. tools/list returned all six tools. The inspector lit up green on every one. I wired it into a real chat, asked the kind of question it was built for, and the model reached for the wrong tool. Then it made something up instead of calling the right one at all.

Nothing was broken. The protocol was fine. The tools ran fine when I called them by hand. The problem was quieter than a bug: a tool the model can technically see, and a tool the model actually uses at the right moment, are two different things.

Here are the four ways that gap shows up, and how to close each one.

Trap 1: a vague description is a dead tool

The model doesn't pick tools by their name. It picks by their description. That's the whole game, and it's the thing people skip.

A tool called get_data with the description "fetches data" is invisible in practice. The model has no idea when to reach for it versus the five other tools that also, in some sense, fetch data. So it guesses, or it skips.

Compare these two:

// the model shrugs at this one
{
  "name": "get_data",
  "description": "Fetches data"
}

// the model knows exactly when to use this one
{
  "name": "search_orders",
  "description": "Find orders by customer email or order ID. Returns status, items, and ship date. Use this before answering any question about a specific order — do not guess from memory."
}

Write the description for the model, not for a docs page. Say when to call it. Say what comes back. Say how it differs from the neighbor tool that looks similar. That last part matters more than people expect — most "the model called the wrong tool" problems are really "two descriptions that sound the same."

Trap 2: a fat response poisons the context

A tool returned four thousand tokens of raw JSON to answer "how many open orders are there?" The number was in there. So was everything else. The model waded through the dump, half-answered, and then the next few tool calls came back sloppy because the context was now full of junk it didn't need.

Your tool's return value isn't a database export. It's an input to the model's next thought. Every token you hand back is a token it has to read on every following turn.

So return what's needed to make a decision, not everything you have. If the question is a count, return a count. If it's a list, paginate it and say there's more. Add a summary field the model can lean on instead of parsing the raw rows. A tool that answers tightly keeps the whole conversation sharp. A tool that dumps drags everything after it down with it.

Trap 3: the schema lies about what it needs

Your input schema is a contract. When it's wrong, the model holds up its end and the server still breaks.

The usual version: a field is really optional, but the schema doesn't mark it, so the model dutifully invents a value to fill it. Or a field can be null in practice, but nothing says so, so the model sends a string and your handler throws. Then the tool 500s with a stack trace the model can't do anything with, and it either retries the same broken call or gives up.

Two fixes, and you need both. Make the schema tell the truth — mark what's actually required, mark what's nullable, add a one-line description on each field the way you did for the tool itself. And validate the input on the way in, so that when something's still off, you return a plain-English error the model can read and correct, not a language-runtime traceback.

Trap 4: one tool fails and the whole agent stalls

Five tools, and number three throws an unhandled exception. If that bubbles up as a hard error, the agent loop can just stop. One flaky dependency and the model is stuck, staring at a failure it has no way to route around.

Catch the error inside the tool and hand it back as something the model can read:

{ "error": "Order service timed out. Try again, or ask the user to retry in a minute." }

Now the model has options. It can retry, try a different tool, or tell the user plainly. A readable error keeps the agent moving. A thrown exception freezes it. Same failure underneath — completely different experience on top.

How to actually test this

Here's the trap in the trap: the inspector being green tells you the tools run. It tells you nothing about whether the model chooses them right. Those are separate questions, and only the second one matters to a user.

So test the second one:

Run a real conversation, not a manual tool call. Watch which tool the model reaches for at each step. Wrong pick? That's a description problem, almost every time.
Log every tool call the model makes — name and arguments. The arguments show you what the model thought your schema wanted, which is often not what you thought it said.
Throw an ambiguous request at it, the kind that could match two tools, and see if it lands on the right one. If it coin-flips, your two descriptions aren't distinct enough yet.

That loop — real chat, watch the choices, fix the descriptions, run it again — is the work. The protocol is the easy part. Getting the model to use your tools the way you meant is the part that takes iterations.

I build MCP servers and wire up Claude Code setups for teams that want the model to actually use their tools, not just expose them — and this is where most of the time goes, long after tools/list is green.

So, honest question: what's the one MCP tool the model keeps refusing to call for you — and did rewriting its description finally fix it, or is it still sitting there ignored? Tell me what it does. I've got a guess about the description.

Nobody checks the VPS between deploys — until it's mining crypto for someone else.

Boris Kl — Tue, 28 Jul 2026 05:31:09 +0000

A friend runs an agency. Six people, forty-something client sites, each on its own small VPS. One Tuesday a client emails asking why their server is "running hot." Load average through the roof. Fan noise from a box that's supposed to just serve a WordPress site. My friend logs in, runs top, and there it is: a process with a random name, eating every core, phoning home to an IP nobody knows. A miner. It had been there, quietly, for three weeks.

Nobody had touched that server in three weeks. That's exactly the problem.

The gap between "deployed" and "watched"

Most small ops setups have decent coverage for one moment: the deploy. CI runs, the app boots, maybe a health check pings once. Then everyone moves on to the next client, the next ticket. What happens on day 12, or day 40, when nobody's looking? Nothing. Nothing gets checked, because nothing triggered a check. The server just sits there, open to whatever the internet throws at it. Until a client notices the fan. Or the CPU bill. Or worse.

Scale that across an agency running dozens of servers and the truth gets ugly fast: most of that fleet is watched by nobody, most of the time. Not because anyone's careless. Because there was never a cheap way to watch it all the time.

Full security monitoring is the wrong first step

The instinct is to reach for a proper monitoring agent. Install a daemon, ship logs somewhere, wire up a SIEM. That's real work, real cost, real upkeep. For a five-person agency running forty small VPS, it's way too much for the problem at hand. Nobody's rolling out a full security stack just to catch "someone added a cron job."

But you don't need the whole stack to catch the miner. You need one small thing: notice when something changes that shouldn't have.

Drift, not intrusion detection

Think about what actually shifts on a server between deploys, when nothing legitimate is going on. New listening ports. New processes that weren't there last time. New or edited cron entries. New user accounts. A real deploy changes some of this too — but on purpose, in a known, expected way. An intrusion changes it in a way nobody asked for.

So the check isn't "is this server secure." That's a much bigger question. The check is smaller: is this server the same shape it was yesterday? If not, was that change supposed to happen? That's drift detection, and it's cheaper than it sounds. No agent daemon required.

A basic version is just three snapshots and a diff:

# collect_state.sh — run via cron, once per hour
{
  echo "### ports"
  ss -tulpn
  echo "### processes"
  ps -eo pid,user,cmd --sort=pid
  echo "### cron"
  for u in $(cut -f1 -d: /etc/passwd); do crontab -u "$u" -l 2>/dev/null; done
} > /var/log/state/snapshot-$(date +%s).txt

# compare against the last snapshot, alert on anything new
diff "$PREV_SNAPSHOT" "$LATEST_SNAPSHOT" | grep '^>' > /tmp/new_lines.txt
if [ -s /tmp/new_lines.txt ]; then
  curl -s -X POST "$WEBHOOK_URL" -d "New on $(hostname): $(cat /tmp/new_lines.txt)"
fi

That's it. No agent on the box, nothing running in the background, just a cron job and a diff. It won't catch a careful attacker who cleans up after themselves, and it's not a substitute for patching or a firewall. But it catches exactly the case above: something new showed up and stayed, because nobody was ever going to notice a strange process name on a server they hadn't opened in three weeks.

Where this earns its keep

The value isn't in any single alert. It's the fact that a server nobody has logged into for a month still gets checked every hour, automatically, for free. For an agency carrying dozens of client boxes, that's the gap between "we'll find out when the client complains" and "we already know." The miner in my friend's story would have shown up in the first diff, the same hour it started. Not three weeks later, in a support ticket about fan noise.

I build this kind of lightweight, agentless drift-checking for people who manage a stack of client VPS and don't have a security team. The whole point is that it costs almost nothing to run and doesn't need anyone babysitting it.

So: if someone SSH'd into one of your servers right now and left a new process running quietly in the background, would anything tell you? Or would you find out the way my friend did?

Your Telegram bot isn't crashing randomly. It's on a timer.

Boris Kl — Mon, 20 Jul 2026 22:23:35 +0000

The ticket always reads the same way: "the bot keeps crashing randomly, please fix."

So you pull the logs. And it's not ra
ndom at all. The bot dies on a schedule — every few hours, almost to the minute. Like clockwork.

That rhythm is a gift. Random failures are the hard ones. They smell like memory leaks, race conditions, bad hardware. A failure that lands on a schedule means something expires. A socket. A token. A connection some box in the middle decided was idle. You're not hunting a ghost. You're looking for a timer.

Here are the three timers I find over and over in Telegram bots that "crash randomly." Between them they cover most of the rescues I get called in for.

Timer #1: the connection that quietly went stale

A long-polling bot holds one HTTP connection open to Telegram and waits. The code looks alive. The process is running. But between your VPS and Telegram sits a NAT table or a proxy. It has an idle timeout. After a fixed window, it drops the connection and tells no one.

No error on your side. No error on Telegram's side. The socket is just... gone. Your bot keeps waiting on a connection that no longer exists. Depending on the library and its settings, it throws two hours later. Or it hangs forever.

The fix isn't a bigger server and it isn't switching libraries. It's assuming the connection will die and reconnecting like you mean it:

import time, logging, requests

def poll_forever(token):
    offset = 0
    backoff = 1
    while True:
        try:
            r = requests.get(
                f"https://api.telegram.org/bot{token}/getUpdates",
                params={"offset": offset, "timeout": 50},
                timeout=(5, 60),   # connect, read — never infinite
            )
            r.raise_for_status()
            backoff = 1
            for upd in r.json()["result"]:
                offset = upd["update_id"] + 1
                handle(upd)
        except requests.RequestException as e:
            logging.warning("poll failed: %s — reconnecting in %ss", e, backoff)
            time.sleep(backoff)
            backoff = min(backoff * 2, 60)

Three things matter here. A real read timeout, set a bit above the long-poll window. Backoff that grows, so a Telegram hiccup doesn't turn into a hammering loop. And a log line every single time it happens. That last one is the sleeper. If reconnects are silent, you'll never know if they fire once a week or once a minute.

Frameworks like aiogram and python-telegram-bot handle much of this for you. But I keep meeting bots where someone turned retries off, or wrapped the library in their own loop that swallows the one exception that mattered.

Timer #2: two transports fighting over one bot

Telegram gives you two ways to receive updates: getUpdates polling or a webhook. One bot token gets exactly one. If a webhook is set and something calls getUpdates, Telegram answers with 409 Conflict, and plenty of homegrown loops treat that as a fatal crash.

How do you end up here without noticing? Easier than you'd think. Someone tested a webhook months ago and never deleted it. A deploy script starts a second copy before the first one dies. Or systemd restarts the service, the old process lingers for a minute still holding the connection, and the new one comes up screaming.

The symptoms look mystical: duplicate messages, updates arriving twice, the bot answering some users and ghosting others, crashes that only happen after a deploy. The cause is boring: two consumers, one queue.

The check takes ten seconds:

curl "https://api.telegram.org/bot$TOKEN/getWebhookInfo"

If you're polling and that response shows a URL — there's your bug. Clear it with deleteWebhook and pick one transport for good. Then make sure only one copy of the bot can run. On systemd that's the default, unless your unit file fights it. The ghost-process case usually comes down to KillMode and TimeoutStopSec that don't match how the bot really shuts down. I've watched a service restart-loop for hours because a dead process refused to let go of port and token.

Timer #3: the request with no timeout

This one doesn't kill the process. It's worse — the bot stays green in every dashboard while doing absolutely nothing.

Somewhere in a handler there's a call to an outside API. Weather, payments, a CRM, your own backend. It's written as requests.get(url), no timeout, because it worked fine in testing. Then one day that API stops answering but keeps the TCP connection open. The default timeout in requests is none. So the handler blocks forever. And in a single-threaded polling loop, that means the whole bot is now a very quiet piece of furniture.

Users say "the bot stopped replying." Monitoring says everything's fine. The process has been "up" for nine days.

Every outbound call gets a timeout. No exceptions, including the calls to Telegram itself:

requests.get(url, timeout=(5, 30))  # 5s to connect, 30s to read

And if the bot does real work, handlers shouldn't share one thread with the poller at all. Even a small thread pool changes the math. One dead upstream API costs you one worker, not the whole bot.

Make the next failure introduce itself

The pattern behind all three: the bot had no way to tell you what was going on. So the fix that outlasts any single bug is a heartbeat. One log line per minute: poll count, last-update time. Plus an alert when the process restarts more than a couple of times an hour. It's cheap to add. And it turns the next "it crashes randomly" into a two-minute read, because now the logs show the rhythm.

Uptime for a bot isn't a feature you bolt on at the end. It's a mindset. Assume every connection dies. Assume every API hangs. Assume every deploy leaves a ghost behind. Then write the ten extra lines that expect all of it.

I build and rescue Telegram bots for clients, and this trio is where I look first, before reading a line of business logic.

What's the strangest clockwork failure you've traced back to a timeout? And do you run bots on webhooks or polling — and why? Genuinely curious what breaks for other people.

I've shipped production code since 1999. AI didn't make me faster where it counts.

Boris Kl — Fri, 17 Jul 2026 10:39:05 +0000

I wrote my first paid line of code in 1999. Servers you could hear. Deploys over FTP. If something broke at 2am, you SSH'd in and read logs until your eyes hurt.

So when people tell me AI made them 10x faster, I want to believe it. I use it every day — I build bots on Claude and GPT, I've got AI running in production on my own projects. It's the best tool I've picked up in years.

But faster? Where it actually counts, no. And I think a lot of us are measuring the wrong thing.

Typing was never the bottleneck

Here's the uncomfortable part. The slow bit of this job was never the typing.

I can bang out a CRUD endpoint in ten minutes with or without AI. That was never what ate my week. What ate my week was understanding why the payment webhook fired twice, or why a site behind a CDN was fast in the test and dead for real users, or which of forty plugins was quietly holding the main thread hostage.

AI is fantastic at producing code. It's much weaker at understanding a system it can't see — your traffic, your data shape, the three weird decisions someone made in 2021 that everything now depends on. That understanding is the job. The code is just what falls out at the end.

So when AI writes the endpoint in 30 seconds, I didn't save a week. I saved ten minutes. The week is still there, waiting, in the part AI can't do for me.

The trap I catch myself in

There's a subtler cost, and it's the one I actually worry about.

When a tool hands you working code instantly, it's really tempting to skip the understanding step. It runs, tests are green, ship it. I've done it. I've shipped something an AI wrote, felt great about the speed, and then spent two days a week later untangling a problem I'd have seen in five minutes if I'd read the thing properly the first time.

The code "worked." That's the trap. Working and understood are not the same state, and production only cares about the second one.

I saw this on a real job — I'll keep it vague. A bot was sending conversion events to an ad platform and the numbers were off. The generated code was clean. It sent the events, got a 200 back, looked perfect in every test. The actual problem was two layers up in how an ID got passed between systems, somewhere no amount of "make the code correct" would ever fix, because the code was correct. You only find that by understanding the whole path. No model was going to hand me that. I had to sit and think.

What actually got faster

I don't want to sound like AI is a toy. It's not. Some things genuinely changed:

The boring first draft. Boilerplate, config, a test harness — AI gets me to something I can react to instead of a blank file. Reacting is faster than starting.
Unfamiliar territory. Dropped into a language or an API I don't use often, AI is a better first guide than skimming docs for an hour.
The rubber duck that talks back. Half the value is just explaining the problem out loud to something that pushes back. I catch my own bad assumptions faster.

Notice what those have in common. They speed up the edges — the start, the unfamiliar, the stuck. None of them touch the core: knowing the system well enough to make the right call. That part still runs at human speed, and I've made peace with it.

Why 1999 me would recognize this

The tools have changed beyond recognition. The habit that separated the people who could fix things from the people who could only write things — that hasn't changed at all.

Back then it was the guy who actually read the manual versus the guy who copy-pasted from a forum until it compiled. Both shipped. One of them could fix it at 2am. AI just made the copy-paste path frictionless and fast, which means the gap between "it runs" and "I understand it" is easier than ever to skip over. The discipline of not skipping it is worth more now, not less.

That's the thing I'd tell someone starting today. Let AI write the code. Then read every line like you'll be the one paged when it breaks — because you will be. Speed at typing is cheap. Understanding is the whole job, and it's the one thing nobody can generate for you.

I run production systems and build AI-driven tools for a living — the kind of work where "it ran in the demo" isn't good enough. If your stack has one of those problems where the code looks right but the behavior is wrong, that's the part I actually enjoy.

Genuine question for the room: where has AI actually made you faster — the core work, or just the edges around it? I'd like to know if I'm the outlier here or if we're all quietly measuring the wrong number.

Cloudflare's Flexible SSL looks secure. It isn't.

Boris Kl — Fri, 03 Jul 2026 10:45:41 +0000

A client called: "the site has the padlock, we're done with SSL, right?" Pulled up the Cloudflare panel — Flexible mode. The padlock was real. The encryption wasn't.

This catches more sites than it should, and the fix is straightforward once you see what's happening.

What Cloudflare's SSL modes actually mean

There are four, and only one is what most people think they have:

Off — no HTTPS. Plain HTTP both ways. Nobody picks this on purpose anymore.
Flexible — HTTPS between the visitor and Cloudflare. Plain HTTP between Cloudflare and your origin. The padlock is the browser's, but the second half of the trip is naked.
Full — HTTPS both ways. Cloudflare doesn't check the origin's certificate, so a self-signed cert is fine. Encrypted, but not authenticated.
Full (strict) — HTTPS both ways, and Cloudflare verifies the origin cert against a real CA. This is the one you want.

Flexible is dangerous because the browser shows the same lock as Full Strict. The site feels secure to the user, to the developer, to whatever scanner pings it from outside. But the traffic between Cloudflare and the origin server is plaintext.

The threat model people miss

"Plain HTTP between Cloudflare and origin — so what, that's a private link?"

It isn't. The path from Cloudflare to your origin crosses public internet. Any network in between — a transit provider, a hosting peer, an exit on a managed VPN, a misconfigured router — can read it. Session cookies, login posts, form data, anything not separately encrypted. You spent the budget on Cloudflare to protect the wire, and the back half of the wire is still wide open.

This is the part the padlock can't tell you about. The browser only sees its own leg of the connection. Cloudflare's leg to your box is invisible to the user, and that's exactly where the leak is.

The fix

Move to Full (strict). This is two steps.

1. Put a real cert on your origin.

If you control the server and run nginx/Apache, the easy answer is Let's Encrypt with certbot — free, 90-day renewal, automated. It's a 10-minute install on most stacks.

If you can't reach the origin from the public internet (it's locked to Cloudflare's IPs, or behind a firewall that won't let Let's Encrypt's HTTP-01 challenge through), use Cloudflare's Origin CA. It's a separate cert authority Cloudflare runs specifically to issue long-lived (up to 15-year) certs that only Cloudflare itself trusts. Generate one in the Cloudflare dashboard, install on your origin, done.

2. Switch the SSL/TLS mode in Cloudflare to Full (strict).

Cloudflare dashboard → SSL/TLS → Overview. Flip from Flexible to Full Strict.

There's an "Always Use HTTPS" toggle nearby — turn it on too. Without it, an attacker can downgrade the first request, the user clicks a plain http:// link, and the connection rides plaintext until Cloudflare upgrades it. Force HTTPS at the edge.

Watch out for the redirect loop

A mistake I see: site is on Flexible, the origin redirects HTTP→HTTPS at the app level (WordPress with the siteurl set to https://, for example). Cloudflare hits the origin on HTTP, the origin sends back a 301 to https://, Cloudflare follows it back to itself, and you get an infinite loop.

Symptom: page never loads, browser eventually shows "too many redirects."

Fix: don't redirect at the origin when you're on Flexible. Or, better, fix the actual problem — move to Full Strict and let the origin always speak HTTPS.

How to check what you have right now

In the Cloudflare dashboard:

SSL/TLS → Overview shows the current mode in plain words.
SSL/TLS → Edge Certificates shows the cert Cloudflare presents to visitors.
Edge → Origin Server → look at the cert source — that's the one to check is real.

From outside, you can also hit your origin's IP directly with curl --resolve to confirm it serves HTTPS:

curl -I --resolve yourdomain.com:443:<origin_ip> https://yourdomain.com/

If that errors with "SSL certificate problem" or "connection refused on 443," your origin isn't speaking HTTPS at all — Flexible was hiding that.

The padlock is a useful signal but it doesn't tell the whole story. Flexible SSL exists for a reason — early days, lots of origins didn't have certs — but it's a 2014 compromise and shouldn't be running anywhere in 2026. If you're holding client sites behind Cloudflare, audit the SSL mode on each one. It's a five-second check that catches a real problem.

I do this kind of edge-hardening for agencies running multiple client sites on Cloudflare — if your SSL setup hasn't been looked at in a while, happy to take a look.

Your Meta CAPI events from a Telegram bot are losing attribution. Here's the fix.

Boris Kl — Mon, 29 Jun 2026 19:37:36 +0000

A funnel I see all the time: a Meta ad sends people to a landing page, the page says "message us on Telegram," and a bot takes it from there — qualifies the lead, takes the order, whatever. The conversion happens inside the bot, server-side, so you fire it to Meta with the Conversions API instead of the Pixel.

Makes sense. Except attribution quietly falls apart, and it took me a while to see why.

The click ID is the whole game

Meta matches a server event back to the ad click using fbc. The format is fixed:

fb.1.<timestamp>.<fbclid>

That <fbclid> comes off the landing URL as ?fbclid=... when someone arrives from the ad. No fbc, and your CAPI event still lands — but match quality drops and Meta can't tie the conversion to the click. Which is the one thing you actually wanted.

So the job is: capture fbclid on the landing page, carry it into the bot, rebuild fbc there, send it with the event.

"Carry it into the bot" is where it breaks.

Telegram's deep link won't hold it

You move someone from web into a bot with a deep link:

https://t.me/your_bot?start=<payload>

The bot gets <payload> as the argument to /start. Perfect place to stash the click ID, right?

No. Telegram caps that start payload at 64 characters, and only allows A-Z a-z 0-9 _ -. A real fbclid runs way past that — the ones I've measured sit around 170+ characters. It does not fit. Truncate it and it's no longer a valid click ID. URL-encode it and it gets longer. There's no squeezing the real value through that gate.

This is the part nobody warns you about. The naive build looks fine in testing with a short fake fbclid, then drops attribution in production with real ones.

The fix: hand over a token, not the value

Don't pass the click ID. Pass a short token that points to it.

On the landing page, when the visitor taps through to Telegram:

Generate a short random token (a dozen URL-safe chars is plenty).
Store { fbclid, click timestamp, utm params } against that token server-side — Redis, a KV store, a tiny table.
Build the deep link with the token: https://t.me/your_bot?start=ab12cd34ef.

In the bot, on /start:

bot.start(async (ctx) => {
  const token = ctx.startPayload;            // "ab12cd34ef"
  const click = await store.get(token);      // { fbclid, ts, utm }

  if (click) {
    const fbc = `fb.1.${click.ts}.${click.fbclid}`;
    await sendCapiEvent({
      event_name: "Lead",
      action_source: "chat",   // it happened in a messaging app, not a website
      user_data: { fbc /* + whatever else you have */ },
      // event_id if you also fire a Pixel, so Meta dedups
    });
  }
  // no token? organic /start. Fine — just lower match quality, no fbc.
});

The token is short, so it sails through the 64-char limit. The real fbclid never has to travel through Telegram at all.

Two details that bite

Use the click timestamp, not the bot-open time. Store ts when they hit the landing page. People tap an ad now and open the bot tomorrow — if you stamp fbc with the moment /start fired, the window's off. Carry the original click time with the token.

action_source should tell the truth. A bot conversion isn't a website event — it happened in a messaging app, so the value is chat. (business_messaging is Meta's own channels like WhatsApp, not Telegram — don't borrow it.) Meta accepts non-website sources; misreporting it just muddies your own data later.

And give tokens a TTL. A click that never opens the bot leaves a dangling entry — expire them after a day or two.

None of this is exotic. It's one indirection — a token standing in for a value that's too big to move. But the failure is silent: events keep arriving, dashboards look populated, and attribution is just... soft, with no error to tell you why. If you're running paid traffic into a chat funnel, check whether your click IDs are actually surviving the handoff.

I build these bot funnels and the tracking behind them — if yours is leaking attribution somewhere, happy to take a look.

Approval-driven server ops: how I let contractors restart nginx without ever giving them SSH

Boris Kl — Fri, 26 Jun 2026 23:04:18 +0000

I run a small WordPress + Cloudflare agency. Two recurring pains finally pushed me to build something instead of complaining:

Pain 1: contractor access drift. I'd hand a contractor root SSH for "just this one task". Three jobs later, I'd realize four ex-contractors still had access I never rotated. The "I'll rotate keys later" lie compounds quickly.

Pain 2: 3am Telegram from a client. "Site is down" while I'm on the subway with no laptop. Find Wi-Fi, SSH in, systemctl restart nginx. Five minutes of actual work, ninety minutes of friction.

The constraint I wanted: a contractor should be able to restart nginx on a specific server, and nothing else. Not "a contractor with a limited shell". Not "a contractor with sudo restricted via sudoers". A contractor with no shell at all, and a chat command that does exactly one thing.

This article walks the architecture I ended up with. Real components, real grammar — names from the actual source tree. If it's useful, fork the idea — or if you'd rather not build it, link at the end.

The shape of the solution

Three components:

Operator (Telegram)
  APPROVE: action + nonce + ts (+ TOTP)
        ↓
Python control plane
  - aiogram bot
  - audit hash-chain (SHA-256)
  - policy engine + runtime state (SQLite WAL)
        ↓ mTLS              ↓ HTTPS
   Go agent           Cloudflare API
   (customer's        (under-attack mode,
    server)            challenges, blocks)
   diagnostics +
   allowlisted ops

The Go agent sits on the managed server. It speaks one protocol — mTLS HTTP — to the Python control plane. It never accepts shell. It accepts a fixed list of operations declared by a capability manifest loaded at startup.

The control plane talks to operators via a Telegram bot (aiogram) and to Cloudflare via their REST API. It runs the policy engine, persists state in SQLite (WAL mode), and writes a hash-chained audit log.

An operator's only interface is a Telegram chat. They never touch the server.

The APPROVE grammar

Every mutating action goes through this grammar. There is no other path.

APPROVE: <action> <client_id> nonce=<16-hex> ts=<unix_epoch>

With TOTP enabled (env var SECMON_TELEGRAM_TOTP_SECRET):

APPROVE: <action> <client_id> otp=<6digits> nonce=<16-hex> ts=<unix_epoch>

Rules the control plane enforces against the message:

Action allowlist. <action> must be in the canonical ActionID enum. Today that's restart-web, restart-php, restart-db, cf-under-attack, cf-managed-challenge, cf-targeted-block, cf-pattern-block, cf-rate-limit, enable-cf-auto. New actions require code + manifest + tests, not configuration. That's deliberate friction.
TTL. |now - ts| <= 300 seconds. Stale approvals get rejected, not silently applied.
One-time nonce. The 16-hex nonce is claimed atomically against a SQLite-backed nonce store. Replay = explicit rejection, not silent dedup.
TOTP second factor (when configured). Six-digit OTP from a pyotp-compatible secret. The control plane refuses the approval if the TOTP doesn't match.
Telegram sender allowlist. SECMON_TELEGRAM_ALLOWED_USER_IDS is checked before parsing. Unknown sender → message logged, action not even parsed.

If any of those fails, the agent never gets a request — the rejection happens in the control plane, gets written to the audit log with reason, and the operator gets a curt Telegram reply explaining what was wrong.

The capability manifest

The agent advertises its capabilities via a JSON document, signed at deploy time with HMAC-SHA256.

canonical = JSON of manifest body (keys sorted, no "signature" field)
signature = HMAC-SHA256(canonical, SECMON_MANIFEST_SECRET).hexdigest()

When the agent comes up, it loads capabilities.manifest.json. When the control plane dispatches an action, it pulls the agent's manifest, verifies the HMAC, and refuses anything not in the signed list. The agent cannot lie about what it supports — and the control plane cannot accidentally route a restart-db to a host whose manifest only declared restart-web.

This is the cheap version of the right idea. The expensive version is signed-everything-everywhere. HMAC-SHA256 with a shared secret is enough for "operator wants X, agent is allowed to do X". For multi-tenant trust I'd reach for asymmetric, but for a self-hosted agent paired with a single control plane, shared HMAC is good enough.

The audit log — hash-chained, verifiable offline

Every approval, every dispatch, every Cloudflare auto-mitigation gets a JSON record in an append-only JSONL file. Each record carries prev_hash and record_hash:

canonical   = JSON of record with prev_hash + record_hash stripped, keys sorted
record_hash = sha256(prev_hash || "|" || canonical).hexdigest()
prev_hash   = record_hash of the preceding line

The first record has prev_hash = "". Every subsequent record locks in everything before it. Tamper with any line, every line after fails verification.

Verification has a CLI: secmon-audit-verify. Walks the chain, returns line numbers of any break.

The threat model is "compromised SaaS cannot lie about what your servers did, after the fact". I think that's the audit threat model that actually matters for an agency that has to show a client what happened during an incident.

(The audit log is separate from the licensing JWT, which uses Ed25519. Different concern. Don't conflate them.)

What it deliberately is NOT

This is just as important as the feature list:

Not a monitoring stack. Use Prometheus, UptimeRobot, Grafana. This tool reacts to incidents — it doesn't detect them. The diagnostics module on the agent is read-only triage (/diag), not continuous monitoring.
Not a remote desktop. No raw shell exec. If you want SSH, you want SSH; this isn't a worse SSH.
Not DDoS protection. Cloudflare is your shield. The control plane can auto-dispatch a Cloudflare under-attack mode in response to certain patterns — but auto-dispatch has a hard kill-switch (SECMON_AUTO_CF_DISABLED=1) checked on every dispatch path. Manual APPROVE: cf-under-attack still works as an explicit operator override, with auto_dispatch_overridden_by=approve written to audit.
Not a ticketing system. No "open ticket for this incident". The audit log is the record, your team's normal tools do the workflow.

Every feature I didn't add is a feature I don't have to maintain. At my scale, that math dominates.

Threat model: assume the bot token leaks

The Telegram bot token gets passed around. Plan for the leak.

If the bot token leaks alone:

Attacker can spam the bot's Telegram channel.
Attacker cannot invoke an action, because:
- Their Telegram user ID isn't in SECMON_TELEGRAM_ALLOWED_USER_IDS. Message gets logged, parsing never starts.
- Even if they were on the allowlist, they don't have the TOTP secret.
- Even if they had the TOTP, they can't reuse a stolen nonce (one-time, atomic claim).
- Even with a fresh nonce, |now - ts| > 300s kills any captured approval.
Attacker cannot elevate themselves into the allowlist or the TOTP secret — both are env-var on the control plane host. Compromising the host changes the threat model entirely.

To actually run an operation against a server, the attacker would need all of:

Bot token (Telegram side)
TOTP secret (env var on control plane)
Telegram user ID on the allowlist (env var on control plane)
A fresh, unused nonce (or write access to the nonce store)
A timestamp within ±300s of dispatch

That's the design point. Each layer is cheap to add and forces an attacker to compromise distinct surfaces.

What I'd do differently

A few honest reflections:

TOTP belongs in v1, not v3. I shipped the bot + APPROVE grammar first, added nonce/TTL in a "security hardening" sprint, added TOTP in another security hardening sprint. Should have been one combined hardening pass before the first real customer touched it. I got away with it because the first customer was me, but I wouldn't ship a v2 without it.
The hash-chain is great until it isn't. A corrupt last line used to be swallowed silently — new events would be written with prev_hash="" and produce a "verifiable but forked" log. Fixed by explicitly propagating AuditChainError on read failure or JSON parse failure, with an opt-in manual recovery path. Lesson: tamper-evidence is a property of the whole chain, including the read path, not just the write path.
Per-tenant Cloudflare credentials need their own design. I had a single CF token at the start. That works for one client. For multi-client it became per-(client_id, zone_id) rows encrypted at rest with Fernet (SECMON_DATA_KEY). Earlier this had a one-key-per-deployment shape and I had to migrate — should have started multi-tenant from the first commit.
Diagnostics is bigger than ops. The agent's read-only diagnostics module (HTTP, web, PHP, DB, disk, SSL, flood patterns) is about 1.5K lines. The mutating-ops surface is smaller. That ratio is honest: you spend much more time looking than fixing, and read-only checks have to be deeply specific or they're noise.

Closing

If you run client infrastructure and the "shared root password" thing makes you wince, the architecture above is buildable. The Go agent is a single binary with no runtime deps. The control plane is Python with SQLite. Nothing exotic in the stack.

If you'd rather not build it: I packaged this up as secmon.io. Pricing is $49/mo Starter (1 server, 4h response), $199 Pro (5 servers, 1h response), $599 Agency (20 servers, monthly incident review). 7-day pilot on all tiers. The agent source is currently closed; whether to open-source it under AGPL is on the roadmap if there's demand — happy to discuss the trade-offs in comments.

If you'd like to be one of the first ten pilots, the contact form is on the homepage. Honest feedback is more useful to me right now than your money.

Critique very welcome — what's broken in the threat model? What's the obvious bigger competitor I missed?

Your page loads fast but still feels slow? It's INP, not load time

Boris Kl — Tue, 16 Jun 2026 10:27:25 +0000

Your Lighthouse report is mostly green. LCP is fine, CLS is fine, the page loads fast. Then the real-world score drops and you can't see why. Nine times out of ten the culprit is INP — and it's the one metric a quick Lighthouse run barely shows you.

What INP actually measures

INP, short for Interaction to Next Paint, replaced FID as a Core Web Vital in March 2024. FID only looked at the delay before your first interaction. INP looks at all of them, the whole time someone uses the page, and reports close to the worst one.

So it's not a loading metric. It's a responsiveness metric. It answers a different question: when I tap, click, or type, how long until the screen actually changes? Google's buckets are simple — 200ms or under is good, over 500ms is poor.

That's why a site can load in a second and still fail. Loading fast and responding fast are two different jobs, done by two different things.

Why it's invisible in a normal audit

LCP and CLS happen during load, so a lab tool catches them every run. INP only happens when a human interacts. Lighthouse doesn't tap your buttons, so its number is an estimate at best. You can have a green lab report and a red field score at the same time, and that gap is exactly where people get stuck.

To see the real number, measure interactions as they happen:

import { onINP } from 'web-vitals';
onINP(function (metric) {
  console.log('INP', metric.value, metric.entries);
});

That logs the actual slow interaction and the element behind it. Now you're fixing a real thing instead of guessing.

What's really slow

INP is almost always one thing: the main thread was busy when the user acted. The browser can't paint the response until the current JavaScript task finishes, so a long task blocks the interaction.

The usual sources:

A heavy event handler doing real work on every click or keystroke.
Third-party scripts like chat widgets, analytics, and tag managers, running long tasks at the wrong moment.
Layout thrash: reading and writing the DOM in a loop so the browser recalculates over and over.
Framework hydration waking the whole page up at once.

The fixes that move it

Break up long tasks. If a handler does a lot, let the browser breathe partway through instead of holding the thread:

async function onClick() {
  doUrgentPart();          // update the UI first
  await yieldToMain();     // give the browser a turn to paint
  doExpensivePart();       // the rest can wait a tick
}

yieldToMain is a one-line helper around scheduler.yield() where it's supported, or a setTimeout(0) fallback. The trick is to paint the response before the slow work, not after.

Beyond that: defer scripts the page doesn't need to react, audit third-party widgets for the ones that run long tasks, debounce expensive handlers, and batch your DOM reads and writes so the browser isn't recalculating layout on every line.

The honest part

I won't promise you a magic number — INP depends on your scripts, your theme, and what your users actually click. But it's measurable, and the field data shows the difference plainly once the long tasks are gone.

I keep my own WordPress sites in the green on Core Web Vitals, and INP is the one I watch most now, because it's the one that quietly fails while everything else looks fine. If your lab report is green but the real score isn't, stop staring at LCP. Go measure an interaction.

Your Telegram bot replies twice? It's timing, not a logic bug

Boris Kl — Mon, 15 Jun 2026 13:23:49 +0000

A Telegram bot replies to the same message twice. An n8n flow processes an order, then processes it again ten seconds later. The owner reads the handler code, finds nothing wrong, and assumes the logic is broken.

It usually isn't. These bugs are almost always about timing, not logic — and once you know the three places timing bites, they stop being mysterious.

1. The webhook you never answered

Telegram (and most webhook senders) wait for an HTTP 200. If your endpoint does the work first and answers afterward, a slow database call or a third-party API can push you past the timeout. The sender assumes delivery failed and sends the same update again. Now your "double reply" isn't a logic bug — it's the same event arriving twice because you were too slow to say "got it."

The fix is to acknowledge first, process second:

@app.post("/webhook")async def webhook(request):
    update = await request.json()
    queue.put_nowait(update)   # hand off
    return Response(status=200) # answer immediately

Return 200 the moment you've safely stored the update. Do the real work in a background task or a worker. The sender stops retrying, and the duplicates dry up.

2. No dedup, so retries become real work

Answering fast helps, but retries still happen — network blips, restarts, a sender that's feeling anxious. The honest assumption is: every event can arrive more than once. So make handling it twice harmless.

Every Telegram update has an update_id. Every message has a message_id. Most webhook payloads have some stable id. Key on it:

if await seen.exists(update_id):
    return            # already handled, do nothing
await seen.add(update_id, ttl=86400)
await handle(update)

seen can be Redis, a unique column in your database, anything that's shared across workers. The point is that "process this order" runs once even if the event shows up three times. People call this idempotency; it just means doing it again changes nothing.

3. Two messages, one piece of state, no lock

This is the one that looks the most like a logic bug and isn't. A user double-taps a button. Two updates arrive almost together. Both handlers read "balance: 100", both subtract 30, both write "70". You charged once for two actions, or booked the same slot twice.

Nothing in the logic is wrong. The two runs just overlapped. The fix is to stop them from overlapping on the same state:

async with lock(f"user:{user_id}"):
    balance = await get_balance(user_id)
    await set_balance(user_id, balance - 30)

A per-user lock (Redis SET NX, a database row lock, whatever you have) means update B waits for update A to finish before it touches the same row. In n8n the same idea shows up as a queue or a "wait for previous execution" step instead of letting every webhook fire its own parallel run.

The part that saves you next time

Most of these never get diagnosed because they're invisible. The handler "works" when you test it by hand — you can't tap fast enough to cause the race, and your local webhook answers instantly. It only breaks under real traffic, at 3am, where you're not looking.

So log the timing, not just the errors. Log the update_id on the way in and the way out. Log when a lock is contended. The first time you see the same update_id logged twice, the whole thing stops being a mystery and becomes a one-line fix.

I run Telegram bots and n8n in production every day, and I've hit all three of these. None of them were in the logic. They were in the gaps between events — and that's almost always where to look first.

Sending Telegram Bot Conversions to Meta? Don't Reach for business_messaging

Boris Kl — Sun, 14 Jun 2026 13:45:21 +0000

A bot was firing Subscribe and Purchase events from Telegram straight to Meta's Conversions API, and every call came back with a 400:

"error_user_title": "Missing Messaging Channel Parameter",
"error_user_msg": "A messaging channel parameter is required when provided
                   action source is business_messaging. Valid value could be
                   messenger, whatsapp and instagram."

The payload looked fine — event_name, event_time, a hashed external_id, and action_source: 'business_messaging'. So why the 400?

The gotcha

business_messaging is not a generic "it happened in a chat" source. Meta ties it to its own messaging products, and it demands a companion messaging_channel whose only valid values are messenger, whatsapp, instagram. Telegram isn't on that list — there's no channel you can hand it — so the request can never validate.

The instinct is to try app next. Don't. app drags in a required app_data block: the extinfo array, advertiser tracking flags, the whole mobile-SDK surface. You don't have that from a bot, and you don't want to fake it.

The fix

For a self-hosted Telegram bot, the right source is plain other:

"action_source": "other"

other has no extra mandatory fields. You need event_name, event_time, action_source, and a user_data with at least one identifier. A SHA-256 hashed Telegram user id as external_id is enough to clear the 400. One-line change.

Making it actually attribute

Not crashing is the low bar. To tie a Subscribe or Purchase back to the ad that caused it, you need the click id:

Capture fbc. Your ad sends people to a deep link — t.me/yourbot?start=.... Meta appends fbclid to that destination. Pack the fbclid into the start payload, read it on /start, and build fbc = fb.1.[unix_time].[fbclid]. Send it in user_data next to external_id. This is the single biggest lever for matching.
Purchase needs money. Add custom_data with value and currency, or there's no ROAS to compute later.
Verify before you trust it. Events Manager has a Test Events tab — send with a test_event_code and watch the events land and match before you point real traffic at it.
Dedupe if a web pixel fires the same events: same event_id on both sides and Meta collapses them.

The 400 is a five-second fix. The attribution is the part that actually pays for itself.

Lighthouse Gave My Site 100/100. The Site Was Down.

Boris Kl — Thu, 11 Jun 2026 18:56:58 +0000

Yesterday I ran PageSpeed Insights on a site I manage. Performance: 100/100. Green circle, confetti, the works.

One problem: the screenshot in the report showed a Cloudflare block page — "Sorry, you have been blocked."

Lighthouse didn't measure my site. It measured the error page my WAF served to Google's crawler. And error pages are, of course, blazing fast.

How this happens

If you put Cloudflare in front of a site and turn the security dial up (Bot Fight Mode, aggressive WAF rules, country blocks), you'll eventually block more than bots:

PageSpeed Insights / Lighthouse — measures a block page, reports nonsense
Uptime monitors — see HTTP 403 with a 200-ish body, or vice versa, and lie to you either way
Google's crawler itself — and that one quietly costs you rankings

The nasty part is the silence. Nothing looks broken from your own browser, because you're whitelisted by your own cookies, IP reputation, or login session. The tools just start telling you fairy tales.

The five-minute audit

Open Cloudflare → Security → Events. Filter the last 7 days. Look at what's actually being challenged or blocked — you'll usually find a legit service in there within a minute.
Check the user agents: Chrome-Lighthouse, GoogleOther, Googlebot, your uptime checker. If they show up here, that traffic never reached your site.
Verify bots properly: Cloudflare has a "Verified Bots" category — allow it instead of hand-maintaining user-agent allowlists (user agents are trivially faked; verified-bot checks aren't).
Re-run your measurement and look at the rendered screenshot, not just the score. The screenshot is the only part of a Lighthouse report that can't lie to you.

Rules I now follow

Never trust a perfect score. 100/100 on a real WordPress/commerce site is a smell, not an achievement. Real sites have real images and real JavaScript.
Check the screenshot first, score second.
After every WAF change, re-test from outside: different network, curl with a Googlebot UA, or just PageSpeed Insights — and read the Events log after.
Monitoring that runs behind your own allowlist isn't monitoring. It's a mirror.

Cloudflare is still the best free thing that ever happened to small sites — I run my own production behind it and it has eaten real attack waves for breakfast. But a security layer you configured and never audited is just a random traffic filter with good branding.

Five minutes in the Events log. That's the whole tip.