Killing duplicate Telegram posts with idempotent sends and rate limits

Our channel bot used to publish the same announcement twice, sometimes three times. Not because the code was clever — because it was naive: a retry after a timeout, a restart halfway through a broadcast, an admin tapping the "publish" button twice. Each of those produced a fresh sendMessage call, and the Bot API happily accepted it.

Here is the setup that fixed it for us. It is roughly 45 minutes of work at an intermediate level, and it is built from three cheap layers: a content fingerprint in SQLite, a per-minute token bucket, and a permalink check after the fact.

The whole flow looks like this:

publisher -> [1] idempotency key -> [2] rate limiter -> [3] sendMessage
                  |                      |                     |
             SQLite outbox          token bucket          message_id
             (UNIQUE key)         (per-minute cap)             |
                  |                                           |
                  +---------- skip on duplicate <--------------+

What you need before you start

Python 3.9+ with requests, a bot token from @botfather, and a private test channel where that bot is an administrator with the "Post messages" permission. Note both the numeric channel id (-100...) and the public username, since both end up in the code. Keep the Bot API docs open in a tab — the limits below are what we worked with, but Telegram adjusts them, so verify the current numbers before you hardcode production values.

mkdir -p ~/tg-dedup && cd ~/tg-dedup
python3 -m venv .venv && source .venv/bin/activate
pip install requests

export TG_BOT_TOKEN="<token-from-BotFather>"
export TG_CHANNEL="-1001234567890"
export TG_CHANNEL_USER="my_test_channel"

The token lives in an environment variable and never in the repository. Quick sanity check: echo "${TG_BOT_TOKEN:0:10}" should print the first ten characters.

Layer 1: fingerprint the content, not the click

An idempotency key is a fingerprint of an operation. Run the same operation twice, get the same fingerprint, and the second run is thrown away instead of executed. For us the fingerprint is a SHA-256 hash of the channel id plus the normalized message text, truncated to 32 hex characters.

Normalization matters more than the hash. Collapsing whitespace and lowercasing means "Hello World" and "hello world" produce one key — which is exactly the duplicate we kept getting from a re-rendered template.

# dedup.py
import hashlib
import sqlite3

SCHEMA = """
CREATE TABLE IF NOT EXISTS outbox (
    idem_key   TEXT PRIMARY KEY,
    channel_id TEXT NOT NULL,
    text       TEXT NOT NULL,
    status     TEXT NOT NULL DEFAULT 'pending',   -- pending | sent | failed
    message_id INTEGER,
    attempts   INTEGER NOT NULL DEFAULT 0,
    created_at TEXT NOT NULL DEFAULT (datetime('now')),
    sent_at    TEXT
);
"""


def idempotency_key(channel_id: str, text: str) -> str:
    normalized = " ".join(text.split()).strip().lower()
    raw = f"{channel_id}:{normalized}".encode()
    return hashlib.sha256(raw).hexdigest()[:32]


def init_db(path: str = "outbox.db") -> None:
    con = sqlite3.connect(path)
    con.executescript(SCHEMA)
    con.commit()
    con.close()


if __name__ == "__main__":
    init_db()

Check that normalization actually works before moving on:

python3 dedup.py
python3 -c "
from dedup import idempotency_key
print(idempotency_key('-1001234567890', 'Hello  World'))
print(idempotency_key('-1001234567890', 'hello world'))
"

Both lines must be identical 32-character hex strings. If they differ, your normalization is broken and every downstream layer is decoration.

Layer 2: let SQLite decide who wins

Because idem_key is the TEXT PRIMARY KEY, the database itself becomes the arbiter. The second insert of the same key conflicts, and ON CONFLICT DO NOTHING turns that conflict into a silent no-op. Two processes cannot both walk away thinking they own the message.

def claim(key: str, text: str) -> bool:
    """Insert first; False means someone already owns this key."""
    con = sqlite3.connect(DB)
    cur = con.execute(
        "INSERT INTO outbox (idem_key, channel_id, text) VALUES (?, ?, ?) "
        "ON CONFLICT(idem_key) DO NOTHING",
        (key, CHANNEL, text),
    )
    con.commit()
    claimed = cur.rowcount == 1
    con.close()
    return claimed

The ordering is the point: the row is written before the network call, not after. If the process dies between the insert and the send, you are left with an explicit pending row you can inspect and decide about — instead of an invisible duplicate you find out about from a subscriber.

Layer 3: a token bucket plus retry_after

One thing worth internalizing: sendMessage has no idempotency parameter. Deduplication is entirely your job (confirm the current parameter list in the docs). The second part of the job is not getting throttled during a broadcast loop.

The limiter below is a token bucket — it keeps timestamps of recent sends and sleeps when the per-minute cap is reached. We sized it at 20 per minute for a single channel, based on the documented figure of roughly 20 messages per minute per group or channel and a global cap near 30 messages per second.

# publisher.py
import os
import sqlite3
import sys
import time

import requests

from dedup import idempotency_key, init_db

BOT_TOKEN = os.environ["TG_BOT_TOKEN"]
CHANNEL = os.environ["TG_CHANNEL"]
USERNAME = os.environ["TG_CHANNEL_USER"]
API = f"https://api.telegram.org/bot{BOT_TOKEN}"
DB = "outbox.db"


class RateLimiter:
    def __init__(self, per_minute: int = 20):
        self.per_minute = per_minute
        self.stamps = []

    def wait(self) -> None:
        now = time.monotonic()
        self.stamps = [t for t in self.stamps if now - t < 60]
        if len(self.stamps) >= self.per_minute:
            time.sleep(60 - (now - self.stamps[0]) + 0.5)
        self.stamps.append(time.monotonic())


def publish(text: str, limiter: RateLimiter) -> None:
    key = idempotency_key(CHANNEL, text)
    if not claim(key, text):
        print(f"skip: duplicate idem_key={key}")
        return

    r = None
    for _ in range(3):
        limiter.wait()
        r = requests.post(
            f"{API}/sendMessage",
            json={"chat_id": CHANNEL, "text": text, "disable_web_page_preview": True},
            timeout=15,
        )
        if r.status_code != 429:
            break
        retry_after = r.json().get("parameters", {}).get("retry_after", 30)
        print(f"429: sleeping {retry_after}s")
        time.sleep(retry_after)

    body = r.json()
    if not body.get("ok"):
        print(f"failed: {body.get('description')}")
        return

    message_id = body["result"]["message_id"]
    con = sqlite3.connect(DB)
    con.execute(
        "UPDATE outbox SET status='sent', message_id=?, sent_at=datetime('now') "
        "WHERE idem_key=?",
        (message_id, key),
    )
    con.commit()
    con.close()
    print(f"sent: message_id={message_id} https://t.me/{USERNAME}/{message_id}")


if __name__ == "__main__":
    init_db()
    publish(" ".join(sys.argv[1:]), RateLimiter(per_minute=20))

The retry loop gives up after three attempts, and it reads retry_after from the response instead of using a fixed sleep. That detail is what stops a second 429 from landing one second after the first.

Step 4: confirm the post is really there

Run it once:

python3 publisher.py "Hello from the outbox"

Expected output:

sent: message_id=42 https://t.me/my_test_channel/42

Then open the permalink. This third layer is the one people skip: an HTTP 200 only proves Telegram accepted something. A permalink check is what catches a wrong channel id, a wrong username, or a post that was deleted right after publishing.

sqlite3 outbox.db "SELECT status, message_id FROM outbox;"

Run the script again with the same text and you should see skip: duplicate idem_key=... and no new row.

Things I keep an eye on

• Limits drift. The 30/sec global and ~20/min per-channel figures were accurate when I built this; re-read the docs before setting production values.
• One worker only. The token bucket lives in process memory, so it does nothing for you across multiple workers. Run a single publisher and queue the work.
• Failed sends stay visible. A pending row after a crash is a feature, not a bug — but somebody has to look at it and decide whether to retry.
• Environment variables, always. The token in a shell export is boring and that is precisely why it works.

What I took away

Duplicate posts are rarely a Telegram problem. They are an ordering problem: we called the network before we recorded the intent, so the system had no memory of what it had already done. Writing the intent first, letting a unique key arbitrate, throttling to a documented rate, and verifying with a permalink cost us a table and about 90 lines of Python — and the duplicates stopped.

If you want more write-ups like this one, this article was originally published on AI Release — https://ai-release.org. There is also a channel with new guides at https://t.me/ai_release1.