<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: parsa shahmohammadi</title>
    <description>The latest articles on DEV Community by parsa shahmohammadi (@parsa_shahmohammadi).</description>
    <link>https://dev.to/parsa_shahmohammadi</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4251601%2F58c116bd-16d5-4341-a833-e5686fd741c6.png</url>
      <title>DEV Community: parsa shahmohammadi</title>
      <link>https://dev.to/parsa_shahmohammadi</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/parsa_shahmohammadi"/>
    <language>en</language>
    <item>
      <title>MikroTik Firewall Hardening on RouterOS 7: A Safe Baseline (Without Locking Yourself Out)</title>
      <dc:creator>parsa shahmohammadi</dc:creator>
      <pubDate>Wed, 18 Nov 2026 12:14:15 +0000</pubDate>
      <link>https://dev.to/parsa_shahmohammadi/mikrotik-firewall-hardening-on-routeros-7-a-safe-baseline-without-locking-yourself-out-1c85</link>
      <guid>https://dev.to/parsa_shahmohammadi/mikrotik-firewall-hardening-on-routeros-7-a-safe-baseline-without-locking-yourself-out-1c85</guid>
      <description>&lt;p&gt;A MikroTik router with a public IP gets scanned within minutes, just like a server. The usual problems are the same every time: Winbox or SSH open to the internet, the DNS resolver answering the whole world (and being used for DNS amplification attacks), the default &lt;code&gt;admin&lt;/code&gt; user, and firmware that hasn't been updated in years.&lt;/p&gt;

&lt;p&gt;This guide gives you a &lt;strong&gt;RouterOS 7 baseline&lt;/strong&gt; you can apply to a small office or home-lab router, step by step, in an order that won't cut you off halfway.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Assumptions: &lt;code&gt;ether1&lt;/code&gt; is the WAN (internet) port and &lt;code&gt;bridge&lt;/code&gt; is your LAN, as in the default configuration. Change the names to match your router before pasting anything.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  Step 0: Turn on Safe Mode first
&lt;/h2&gt;

&lt;p&gt;This one habit prevents most lockouts. In a terminal session press &lt;strong&gt;Ctrl+X&lt;/strong&gt; (in Winbox, click &lt;strong&gt;Safe Mode&lt;/strong&gt; at the top left). While Safe Mode is on, if your session drops because of a change you just made, RouterOS &lt;strong&gt;rolls back every change made in that session&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Press Ctrl+X again to leave Safe Mode and keep the changes, once you've confirmed you can still connect.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 1: Interface lists
&lt;/h2&gt;

&lt;p&gt;The firewall rules below refer to lists, not ports, so they keep working if you add a second WAN or LAN interface later. If you started from the default configuration, the lists already exist and you can skip this.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;/interface list
add name=WAN
add name=LAN
/interface list member
add list=WAN interface=ether1
add list=LAN interface=bridge
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Step 2: Protect the router itself (input chain)
&lt;/h2&gt;

&lt;p&gt;The &lt;code&gt;input&lt;/code&gt; chain is traffic &lt;strong&gt;to the router&lt;/strong&gt;: Winbox, SSH, DNS, the web UI. The rule is simple: accept what belongs to existing connections, accept the LAN, drop everything else.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;/ip firewall filter
add chain=input action=accept connection-state=established,related,untracked comment="input: accept established,related,untracked"
add chain=input action=drop connection-state=invalid comment="input: drop invalid"
add chain=input action=accept protocol=icmp comment="input: accept ICMP"
add chain=input action=accept dst-address=127.0.0.1 comment="input: accept to local loopback (CAPsMAN)"
add chain=input action=drop in-interface-list=!LAN comment="input: drop all not coming from LAN"
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The last rule closes &lt;strong&gt;Winbox (8291), SSH (22), the web UI, the API and DNS (53)&lt;/strong&gt; to the internet in one line. If you need remote management, use a VPN (step 5) instead of opening these ports.&lt;/p&gt;

&lt;p&gt;Order matters: RouterOS reads the rules top to bottom and stops at the first match. Use &lt;code&gt;/ip firewall filter print&lt;/code&gt; and make sure the &lt;code&gt;drop&lt;/code&gt; rule is last in the input chain.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 3: Protect the LAN (forward chain)
&lt;/h2&gt;

&lt;p&gt;The &lt;code&gt;forward&lt;/code&gt; chain is traffic &lt;strong&gt;through the router&lt;/strong&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;/ip firewall filter
add chain=forward action=fasttrack-connection connection-state=established,related hw-offload=yes comment="forward: fasttrack"
add chain=forward action=accept connection-state=established,related,untracked comment="forward: accept established,related,untracked"
add chain=forward action=drop connection-state=invalid comment="forward: drop invalid"
add chain=forward action=drop connection-state=new connection-nat-state=!dstnat in-interface-list=WAN comment="forward: drop all from WAN not DSTNATed"
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The last rule means nothing from the internet can open a connection into your LAN unless you created a port forward (dst-nat) for it on purpose.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;FastTrack speeds things up but bypasses queues and some firewall features (mangle, simple queues) for the fast-tracked traffic. If you shape bandwidth with queues, remove the first rule.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  Step 4: Close the doors you don't use
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;# services: keep only what you use, and only from the LAN
/ip service
set telnet disabled=yes
set ftp disabled=yes
set www disabled=yes
set api disabled=yes
set api-ssl disabled=yes
set ssh address=192.168.88.0/24
set winbox address=192.168.88.0/24

# layer-2 discovery and MAC access only on the LAN
/ip neighbor discovery-settings set discover-interface-list=LAN
/tool mac-server set allowed-interface-list=LAN
/tool mac-server mac-winbox set allowed-interface-list=LAN

# bandwidth test server is a DoS helper if left on
/tool bandwidth-server set enabled=no
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Replace &lt;code&gt;192.168.88.0/24&lt;/code&gt; with your LAN subnet.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;DNS:&lt;/strong&gt; if LAN clients use the router as their DNS server, &lt;code&gt;allow-remote-requests=yes&lt;/code&gt; is fine, &lt;strong&gt;as long as&lt;/strong&gt; the input &lt;code&gt;drop&lt;/code&gt; rule from step 2 is in place. Without it, your router becomes an open resolver that attackers use to amplify DDoS traffic. This is one of the most common MikroTik misconfigurations on the internet.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 5: Remote access through WireGuard, not open ports
&lt;/h2&gt;

&lt;p&gt;RouterOS 7 includes WireGuard. A minimal server side:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;/interface wireguard add name=wg-admin listen-port=13231
/ip address add address=10.10.10.1/24 interface=wg-admin
/interface wireguard peers add interface=wg-admin public-key="CLIENT_PUBLIC_KEY" allowed-address=10.10.10.2/32
/interface list member add list=LAN interface=wg-admin

/ip firewall filter
add chain=input action=accept protocol=udp dst-port=13231 comment="input: allow WireGuard" place-before=[find comment="input: drop all not coming from LAN"]
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Get the router's public key with &lt;code&gt;/interface wireguard print&lt;/code&gt; and put it in your client config. Because &lt;code&gt;wg-admin&lt;/code&gt; is in the LAN list, connected admins can reach Winbox and SSH. Nobody else can.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 6: Users and updates
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;/user add name=netadmin group=full password="a-long-unique-password"
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Log in as &lt;code&gt;netadmin&lt;/code&gt;, check that it works, then remove the default account:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;/user remove admin
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then update RouterOS and the RouterBOARD firmware:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;/system package update check-for-updates
/system package update install
# after the reboot:
/system routerboard upgrade
/system reboot
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Most mass compromises of MikroTik routers (including large botnets) used vulnerabilities that had been patched long before. Updating is the single most effective step on this list.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 7: Check from the outside
&lt;/h2&gt;

&lt;p&gt;Your own LAN can't tell you what the internet sees. Check the router's public IP from outside:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;From a machine outside your network: &lt;code&gt;nmap -Pn -p 22,53,80,443,8291,8728 YOUR_PUBLIC_IP&lt;/code&gt;. Everything should show as filtered or closed.&lt;/li&gt;
&lt;li&gt;Check DNS specifically: &lt;code&gt;dig @YOUR_PUBLIC_IP example.com&lt;/code&gt; from outside should &lt;strong&gt;time out&lt;/strong&gt;, not answer.&lt;/li&gt;
&lt;li&gt;Or use a browser-based &lt;a href="https://configserver.cloud/%D8%AA%D8%B3%D8%AA-%D9%BE%D9%88%D8%B1%D8%AA-%D8%A8%D8%A7%D8%B2/" rel="noopener noreferrer"&gt;open port checker&lt;/a&gt; for single ports (Persian interface: enter the IP and port).&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Common mistakes
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Mistake&lt;/th&gt;
&lt;th&gt;Result&lt;/th&gt;
&lt;th&gt;Fix&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Adding the input &lt;code&gt;drop&lt;/code&gt; rule from a WAN connection&lt;/td&gt;
&lt;td&gt;You lock yourself out&lt;/td&gt;
&lt;td&gt;Safe Mode (step 0), or work from the LAN&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;drop&lt;/code&gt; rule placed above the &lt;code&gt;accept&lt;/code&gt; rules&lt;/td&gt;
&lt;td&gt;LAN can't reach the router&lt;/td&gt;
&lt;td&gt;Check order with &lt;code&gt;/ip firewall filter print&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;allow-remote-requests=yes&lt;/code&gt; without an input firewall&lt;/td&gt;
&lt;td&gt;Open DNS resolver, used in DDoS attacks&lt;/td&gt;
&lt;td&gt;Step 2 drop rule&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Winbox open to the internet "with a strong password"&lt;/td&gt;
&lt;td&gt;Brute-force and exploit attempts all day&lt;/td&gt;
&lt;td&gt;VPN only (step 5)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;RouterOS never updated&lt;/td&gt;
&lt;td&gt;Known, public exploits work&lt;/td&gt;
&lt;td&gt;Step 6&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  A shortcut
&lt;/h2&gt;

&lt;p&gt;If you'd rather not assemble the rules by hand, the &lt;a href="https://configserver.cloud/%D8%A7%D8%B3%DA%A9%D8%B1%DB%8C%D9%BE%D8%AA-%D9%81%D8%A7%DB%8C%D8%B1%D9%88%D8%A7%D9%84-%D9%85%DB%8C%DA%A9%D8%B1%D9%88%D8%AA%DB%8C%DA%A9/" rel="noopener noreferrer"&gt;MikroTik firewall script generator for RouterOS 7&lt;/a&gt; builds a script like the one above from your interface names and options (interface in Persian, output is a standard RouterOS script). Read whatever it produces before you paste it, the same as any script from the internet. For office networks that need more than a baseline (VLANs, multiple WANs, site-to-site VPN), there's also &lt;a href="https://configserver.cloud/%DA%A9%D8%A7%D9%86%D9%81%DB%8C%DA%AF-%D9%85%DB%8C%DA%A9%D8%B1%D9%88%D8%AA%DB%8C%DA%A9/" rel="noopener noreferrer"&gt;MikroTik configuration as a service&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;References:&lt;/strong&gt; &lt;a href="https://help.mikrotik.com/docs/spaces/ROS/pages/48660574/Filter" rel="noopener noreferrer"&gt;MikroTik: Firewall filter&lt;/a&gt;, &lt;a href="https://help.mikrotik.com/docs/spaces/ROS/pages/328353/Securing+your+router" rel="noopener noreferrer"&gt;MikroTik: Securing your router&lt;/a&gt;, &lt;a href="https://help.mikrotik.com/docs/spaces/ROS/pages/69664792/WireGuard" rel="noopener noreferrer"&gt;MikroTik: WireGuard&lt;/a&gt;&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Parsa Shahmohammadi works with the team at &lt;a href="https://configserver.cloud" rel="noopener noreferrer"&gt;ConfigServer.cloud&lt;/a&gt;, which configures and secures Linux servers and MikroTik networks for businesses in Iran.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>networking</category>
      <category>security</category>
      <category>mikrotik</category>
      <category>tutorial</category>
    </item>
    <item>
      <title>Deploy a Python Telegram Bot with Webhooks on a VPS (systemd, Nginx, Let's Encrypt)</title>
      <dc:creator>parsa shahmohammadi</dc:creator>
      <pubDate>Wed, 18 Nov 2026 10:31:39 +0000</pubDate>
      <link>https://dev.to/parsa_shahmohammadi/deploy-a-python-telegram-bot-with-webhooks-on-a-vps-systemd-nginx-lets-encrypt-31b6</link>
      <guid>https://dev.to/parsa_shahmohammadi/deploy-a-python-telegram-bot-with-webhooks-on-a-vps-systemd-nginx-lets-encrypt-31b6</guid>
      <description>&lt;p&gt;Most Telegram bot tutorials stop at &lt;code&gt;run_polling()&lt;/code&gt; on a laptop. That works until you close the laptop. This guide takes a Python bot to a VPS and runs it the way production bots usually run: &lt;strong&gt;webhooks behind Nginx with HTTPS&lt;/strong&gt;, started and restarted by &lt;strong&gt;systemd&lt;/strong&gt;, with a &lt;strong&gt;secret token&lt;/strong&gt; so only Telegram can call it.&lt;/p&gt;

&lt;p&gt;You need:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;A Linux VPS (the commands use Ubuntu 24.04)&lt;/li&gt;
&lt;li&gt;A domain or subdomain pointing at the server (an &lt;code&gt;A&lt;/code&gt; record such as &lt;code&gt;bot.example.com&lt;/code&gt;)&lt;/li&gt;
&lt;li&gt;A bot token from &lt;a href="https://t.me/BotFather" rel="noopener noreferrer"&gt;@BotFather&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Polling or webhook?
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;Polling&lt;/th&gt;
&lt;th&gt;Webhook&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;How it works&lt;/td&gt;
&lt;td&gt;Your bot keeps asking Telegram for updates&lt;/td&gt;
&lt;td&gt;Telegram sends each update to your HTTPS URL&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Needs a public HTTPS endpoint&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Idle cost&lt;/td&gt;
&lt;td&gt;Constant requests&lt;/td&gt;
&lt;td&gt;None until a message arrives&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Good for&lt;/td&gt;
&lt;td&gt;Development, small bots&lt;/td&gt;
&lt;td&gt;Production, many users&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Telegram only delivers webhooks to ports &lt;strong&gt;443, 80, 88 or 8443&lt;/strong&gt;, and only over HTTPS. Nginx on 443 in front of the bot takes care of both.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Hosting from Iran?&lt;/strong&gt; Telegram is filtered in Iran, so a bot on an Iranian server has no direct route to &lt;code&gt;api.telegram.org&lt;/code&gt;. Most bots for Iranian users run on a server abroad (Germany is a common choice) and serve the same users without problems.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  Step 1: A user and a virtual environment for the bot
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;sudo apt update &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; sudo apt install -y python3-venv nginx
sudo adduser --system --group --home /opt/mybot mybot
sudo -u mybot python3 -m venv /opt/mybot/venv
sudo -u mybot /opt/mybot/venv/bin/pip install &lt;span class="s2"&gt;"python-telegram-bot[webhooks]&amp;gt;=21,&amp;lt;22"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The &lt;code&gt;[webhooks]&lt;/code&gt; extra installs the small web server that &lt;code&gt;python-telegram-bot&lt;/code&gt; uses to receive updates. The bot runs as its own system user, so a bug in the bot cannot touch the rest of the server.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 2: The bot
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;/opt/mybot/bot.py&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;logging&lt;/span&gt;
&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;os&lt;/span&gt;

&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;telegram&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;Update&lt;/span&gt;
&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;telegram&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;ext&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;Application&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;CommandHandler&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;ContextTypes&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;MessageHandler&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;filters&lt;/span&gt;

&lt;span class="n"&gt;logging&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;basicConfig&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;level&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;logging&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="no"&gt;INFO&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;


&lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;start&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;update&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nc"&gt;Update&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;context&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nc"&gt;ContextTypes&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="no"&gt;DEFAULT_TYPE&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;update&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;message&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;reply_text&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"Hi! I'm running on a webhook."&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;


&lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;echo&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;update&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nc"&gt;Update&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;context&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nc"&gt;ContextTypes&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="no"&gt;DEFAULT_TYPE&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;update&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;message&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;reply_text&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;update&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;message&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;


&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;main&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;app&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;Application&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;builder&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;token&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;environ&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"BOT_TOKEN"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;build&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;app&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;add_handler&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;CommandHandler&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"start"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;start&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;app&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;add_handler&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;MessageHandler&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;filters&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="no"&gt;TEXT&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&lt;/span&gt; &lt;span class="o"&gt;~&lt;/span&gt;&lt;span class="n"&gt;filters&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="no"&gt;COMMAND&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;echo&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="n"&gt;app&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;run_webhook&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="n"&gt;listen&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"127.0.0.1"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;          &lt;span class="c1"&gt;# only Nginx can reach it&lt;/span&gt;
        &lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;8000&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;url_path&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"telegram"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;webhook_url&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"https://{os.environ['BOT_DOMAIN']}/telegram"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;secret_token&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;environ&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"WEBHOOK_SECRET"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;drop_pending_updates&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;


&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;__name__&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s2"&gt;"__main__"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="nf"&gt;main&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;run_webhook&lt;/code&gt; registers the webhook with Telegram on start-up (&lt;code&gt;setWebhook&lt;/code&gt;) and serves it. With &lt;code&gt;secret_token&lt;/code&gt; set, Telegram sends the value in the &lt;code&gt;X-Telegram-Bot-Api-Secret-Token&lt;/code&gt; header, and the library rejects any request where it is missing or wrong.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 3: Keep the secrets out of the code
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;sudo tee /etc/mybot.env &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; /dev/null &lt;span class="o"&gt;&amp;lt;&amp;lt;&lt;/span&gt;&lt;span class="s1"&gt;'EOF'&lt;/span&gt;
&lt;span class="nv"&gt;BOT_TOKEN&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;123456:ABC-your-token
&lt;span class="nv"&gt;BOT_DOMAIN&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;bot.example.com
&lt;span class="nv"&gt;WEBHOOK_SECRET&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;change-me-to-a-long-random-string
EOF
sudo chown root:mybot /etc/mybot.env
sudo chmod 640 /etc/mybot.env
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Generate the secret with &lt;code&gt;openssl rand -hex 32&lt;/code&gt;. Telegram allows only the characters &lt;code&gt;A-Z&lt;/code&gt;, &lt;code&gt;a-z&lt;/code&gt;, &lt;code&gt;0-9&lt;/code&gt;, &lt;code&gt;_&lt;/code&gt; and &lt;code&gt;-&lt;/code&gt;, up to 256 of them, and hex output fits.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 4: A systemd service
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;/etc/systemd/system/mybot.service&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ini"&gt;&lt;code&gt;&lt;span class="k"&gt;[Unit]&lt;/span&gt;
&lt;span class="na"&gt;Description&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nc"&gt;Telegram&lt;/span&gt; &lt;span class="n"&gt;bot&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;webhook&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="na"&gt;After&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;network&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;online&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;target&lt;/span&gt;
&lt;span class="na"&gt;Wants&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;network&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;online&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;target&lt;/span&gt;

&lt;span class="k"&gt;[Service]&lt;/span&gt;
&lt;span class="na"&gt;User&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;mybot&lt;/span&gt;
&lt;span class="na"&gt;Group&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;mybot&lt;/span&gt;
&lt;span class="na"&gt;WorkingDirectory&lt;/span&gt;&lt;span class="o"&gt;=/&lt;/span&gt;&lt;span class="n"&gt;opt&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;mybot&lt;/span&gt;
&lt;span class="na"&gt;EnvironmentFile&lt;/span&gt;&lt;span class="o"&gt;=/&lt;/span&gt;&lt;span class="n"&gt;etc&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;mybot&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;env&lt;/span&gt;
&lt;span class="na"&gt;ExecStart&lt;/span&gt;&lt;span class="o"&gt;=/&lt;/span&gt;&lt;span class="n"&gt;opt&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;mybot&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;venv&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;bin&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;python&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;opt&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;mybot&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;bot&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;py&lt;/span&gt;
&lt;span class="na"&gt;Restart&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;always&lt;/span&gt;
&lt;span class="na"&gt;RestartSec&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;

&lt;span class="c1"&gt;# basic sandboxing
&lt;/span&gt;&lt;span class="na"&gt;NoNewPrivileges&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;true&lt;/span&gt;
&lt;span class="na"&gt;ProtectSystem&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;strict&lt;/span&gt;
&lt;span class="na"&gt;ProtectHome&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;true&lt;/span&gt;
&lt;span class="na"&gt;PrivateTmp&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;true&lt;/span&gt;

&lt;span class="k"&gt;[Install]&lt;/span&gt;
&lt;span class="na"&gt;WantedBy&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;multi&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;user&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;target&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Don't start it yet. Telegram will refuse the webhook until HTTPS works.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 5: Nginx and a free certificate
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;/etc/nginx/sites-available/mybot&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight nginx"&gt;&lt;code&gt;&lt;span class="n"&gt;server&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;listen&lt;/span&gt; &lt;span class="mi"&gt;80&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="n"&gt;server_name&lt;/span&gt; &lt;span class="n"&gt;bot&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;example&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;com&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

    &lt;span class="n"&gt;location&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;telegram&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;proxy_pass&lt;/span&gt; &lt;span class="n"&gt;http&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="c1"&gt;//127.0.0.1:8000/telegram;&lt;/span&gt;
        &lt;span class="n"&gt;proxy_set_header&lt;/span&gt; &lt;span class="nc"&gt;Host&lt;/span&gt; $&lt;span class="n"&gt;host&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
        &lt;span class="n"&gt;proxy_set_header&lt;/span&gt; &lt;span class="no"&gt;X&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nc"&gt;Forwarded&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nc"&gt;For&lt;/span&gt; $&lt;span class="n"&gt;proxy_add_x_forwarded_for&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
        &lt;span class="n"&gt;proxy_set_header&lt;/span&gt; &lt;span class="no"&gt;X&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nc"&gt;Forwarded&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nc"&gt;Proto&lt;/span&gt; $&lt;span class="n"&gt;scheme&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="n"&gt;location&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="mi"&gt;404&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;sudo ln -s /etc/nginx/sites-available/mybot /etc/nginx/sites-enabled/
sudo nginx -t &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; sudo systemctl reload nginx

sudo apt install -y certbot python3-certbot-nginx
sudo certbot --nginx -d bot.example.com
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Certbot adds the &lt;code&gt;listen 443 ssl&lt;/code&gt; block, an HTTP-to-HTTPS redirect, and a timer that renews the certificate automatically.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 6: Start it and check what Telegram sees
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;sudo systemctl daemon-reload
sudo systemctl &lt;span class="nb"&gt;enable&lt;/span&gt; --now mybot
sudo journalctl -u mybot -f
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then ask Telegram what it thinks of your webhook:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;curl -s &lt;span class="s2"&gt;"https://api.telegram.org/bot&amp;lt;YOUR_TOKEN&amp;gt;/getWebhookInfo"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A healthy result shows your URL, &lt;code&gt;"pending_update_count": 0&lt;/code&gt;, and no &lt;code&gt;last_error_message&lt;/code&gt;. Send &lt;code&gt;/start&lt;/code&gt; to the bot and you should get a reply within a second.&lt;/p&gt;

&lt;h2&gt;
  
  
  When it doesn't work
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;What you see&lt;/th&gt;
&lt;th&gt;Likely cause&lt;/th&gt;
&lt;th&gt;Fix&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;last_error_message&lt;/code&gt;: "Wrong response from the webhook: 502 Bad Gateway"&lt;/td&gt;
&lt;td&gt;Bot not running, or a different port&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;systemctl status mybot&lt;/code&gt;, check &lt;code&gt;port=&lt;/code&gt; matches &lt;code&gt;proxy_pass&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;"Wrong response from the webhook: 404 Not Found"&lt;/td&gt;
&lt;td&gt;Path mismatch&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;url_path&lt;/code&gt;, &lt;code&gt;webhook_url&lt;/code&gt; and the Nginx &lt;code&gt;location&lt;/code&gt; must all say &lt;code&gt;/telegram&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;"Wrong response from the webhook: 403 Forbidden"&lt;/td&gt;
&lt;td&gt;Secret token mismatch&lt;/td&gt;
&lt;td&gt;Restart the bot so it registers the webhook again with the current secret&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;"SSL error" or certificate errors&lt;/td&gt;
&lt;td&gt;No valid certificate for the domain&lt;/td&gt;
&lt;td&gt;Re-run &lt;code&gt;certbot --nginx -d ...&lt;/code&gt;, check DNS points to this server&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;pending_update_count&lt;/code&gt; keeps growing&lt;/td&gt;
&lt;td&gt;Telegram can't reach you at all&lt;/td&gt;
&lt;td&gt;Firewall: &lt;code&gt;sudo ufw allow 'Nginx Full'&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Bot works, then stops after a reboot&lt;/td&gt;
&lt;td&gt;Service not enabled&lt;/td&gt;
&lt;td&gt;&lt;code&gt;systemctl enable mybot&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  Before you call it production
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Logs:&lt;/strong&gt; &lt;code&gt;journalctl -u mybot&lt;/code&gt; already keeps them. Add &lt;code&gt;SystemMaxUse=&lt;/code&gt; in &lt;code&gt;journald.conf&lt;/code&gt; if the bot is chatty.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;State:&lt;/strong&gt; anything you keep in Python variables is gone on restart. Use SQLite or Redis for user data.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Updates:&lt;/strong&gt; pin the major version (&lt;code&gt;&amp;gt;=21,&amp;lt;22&lt;/code&gt;) and read the changelog before you move to the next one.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Several bots on one server:&lt;/strong&gt; give each one its own user, port, &lt;code&gt;url_path&lt;/code&gt; and service file. One Nginx server block can route several paths.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If you'd rather not run the VPS yourself, managed &lt;a href="https://pikohost.com/telegram.html" rel="noopener noreferrer"&gt;Telegram bot hosting&lt;/a&gt; and &lt;a href="https://pikohost.com/python-host.html" rel="noopener noreferrer"&gt;Python hosting&lt;/a&gt; take care of the server side, and a &lt;a href="https://pikohost.com/germany-vps.html" rel="noopener noreferrer"&gt;VPS in Germany&lt;/a&gt; is a good fit when your users are in Iran (pages are in Persian).&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;References:&lt;/strong&gt; &lt;a href="https://github.com/python-telegram-bot/python-telegram-bot/wiki/Webhooks" rel="noopener noreferrer"&gt;python-telegram-bot: Webhooks&lt;/a&gt;, &lt;a href="https://core.telegram.org/bots/api#setwebhook" rel="noopener noreferrer"&gt;Telegram Bot API: setWebhook&lt;/a&gt;&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Parsa Shahmohammadi works on &lt;a href="https://pikohost.com" rel="noopener noreferrer"&gt;Pikohost&lt;/a&gt;, a hosting provider for Persian-speaking developers and businesses.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>python</category>
      <category>telegram</category>
      <category>devops</category>
      <category>tutorial</category>
    </item>
    <item>
      <title>Linux Server Monitoring with Prometheus, Grafana and Alertmanager (Docker Compose, Alerts That Matter)</title>
      <dc:creator>parsa shahmohammadi</dc:creator>
      <pubDate>Wed, 18 Nov 2026 10:31:18 +0000</pubDate>
      <link>https://dev.to/parsa_shahmohammadi/linux-server-monitoring-with-prometheus-grafana-and-alertmanager-docker-compose-alerts-that-1pgm</link>
      <guid>https://dev.to/parsa_shahmohammadi/linux-server-monitoring-with-prometheus-grafana-and-alertmanager-docker-compose-alerts-that-1pgm</guid>
      <description>&lt;p&gt;Most servers get monitored after the first outage. By then you're reading logs to find out what happened, when an alert could have told you an hour earlier that the disk was filling up.&lt;/p&gt;

&lt;p&gt;This guide sets up a small monitoring stack in &lt;strong&gt;one Docker Compose file&lt;/strong&gt;: node_exporter collects host metrics, Prometheus stores them and evaluates alert rules, Alertmanager sends the alerts (Telegram or email), and Grafana draws the dashboards. Then it adds the &lt;strong&gt;five alert rules&lt;/strong&gt; worth starting with, and leaves out the rest.&lt;/p&gt;

&lt;p&gt;You need a Linux server with Docker Engine and the Compose plugin (&lt;code&gt;docker compose version&lt;/code&gt; should work).&lt;/p&gt;

&lt;h2&gt;
  
  
  The layout
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;monitoring/
├── docker-compose.yml
├── prometheus/
│   ├── prometheus.yml
│   └── alerts.yml
└── alertmanager/
    └── alertmanager.yml
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Step 1: docker-compose.yml
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;services&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;prometheus&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;image&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;prom/prometheus:v2.53.0&lt;/span&gt;
    &lt;span class="na"&gt;volumes&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="s"&gt;./prometheus:/etc/prometheus:ro&lt;/span&gt;
      &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="s"&gt;prometheus-data:/prometheus&lt;/span&gt;
    &lt;span class="na"&gt;command&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="s"&gt;--config.file=/etc/prometheus/prometheus.yml&lt;/span&gt;
      &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="s"&gt;--storage.tsdb.retention.time=30d&lt;/span&gt;
    &lt;span class="na"&gt;ports&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="s2"&gt;"127.0.0.1:9090:9090"&lt;/span&gt;
    &lt;span class="na"&gt;restart&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;unless-stopped&lt;/span&gt;

  &lt;span class="na"&gt;alertmanager&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;image&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;prom/alertmanager:v0.27.0&lt;/span&gt;
    &lt;span class="na"&gt;volumes&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="s"&gt;./alertmanager:/etc/alertmanager:ro&lt;/span&gt;
    &lt;span class="na"&gt;ports&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="s2"&gt;"127.0.0.1:9093:9093"&lt;/span&gt;
    &lt;span class="na"&gt;restart&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;unless-stopped&lt;/span&gt;

  &lt;span class="na"&gt;node-exporter&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;image&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;prom/node-exporter:v1.8.1&lt;/span&gt;
    &lt;span class="na"&gt;command&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="s"&gt;--path.rootfs=/host&lt;/span&gt;
    &lt;span class="na"&gt;pid&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;host&lt;/span&gt;
    &lt;span class="na"&gt;volumes&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="s"&gt;/:/host:ro,rslave&lt;/span&gt;
    &lt;span class="na"&gt;restart&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;unless-stopped&lt;/span&gt;

  &lt;span class="na"&gt;grafana&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;image&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;grafana/grafana:11.1.0&lt;/span&gt;
    &lt;span class="na"&gt;environment&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="s"&gt;GF_SECURITY_ADMIN_PASSWORD__FILE=/run/secrets/grafana_admin&lt;/span&gt;
    &lt;span class="na"&gt;secrets&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="s"&gt;grafana_admin&lt;/span&gt;
    &lt;span class="na"&gt;volumes&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="s"&gt;grafana-data:/var/lib/grafana&lt;/span&gt;
    &lt;span class="na"&gt;ports&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="s2"&gt;"127.0.0.1:3000:3000"&lt;/span&gt;
    &lt;span class="na"&gt;restart&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;unless-stopped&lt;/span&gt;

&lt;span class="na"&gt;secrets&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;grafana_admin&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;file&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;./grafana_admin.txt&lt;/span&gt;

&lt;span class="na"&gt;volumes&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;prometheus-data&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;grafana-data&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two choices here matter more than they look:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Every port is bound to &lt;code&gt;127.0.0.1&lt;/code&gt;.&lt;/strong&gt; Prometheus, Alertmanager and node_exporter have no login of their own. Published on &lt;code&gt;0.0.0.0&lt;/code&gt;, they show your whole infrastructure to anyone who finds the IP. Docker also writes its own iptables rules, so a published port can be reachable even when UFW says it's closed. Reach the UIs through an SSH tunnel (&lt;code&gt;ssh -L 3000:127.0.0.1:3000 user@server&lt;/code&gt;) or a reverse proxy with authentication.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The image versions are pinned.&lt;/strong&gt; &lt;code&gt;latest&lt;/code&gt; changes under you on the next &lt;code&gt;docker compose pull&lt;/code&gt;. Upgrade on purpose, after reading the release notes.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Put the Grafana admin password in &lt;code&gt;grafana_admin.txt&lt;/code&gt; (&lt;code&gt;openssl rand -base64 24 &amp;gt; grafana_admin.txt &amp;amp;&amp;amp; chmod 600 grafana_admin.txt&lt;/code&gt;).&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 2: prometheus.yml
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;global&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;scrape_interval&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;30s&lt;/span&gt;
  &lt;span class="na"&gt;evaluation_interval&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;30s&lt;/span&gt;

&lt;span class="na"&gt;rule_files&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="s"&gt;/etc/prometheus/alerts.yml&lt;/span&gt;

&lt;span class="na"&gt;alerting&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;alertmanagers&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;static_configs&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;targets&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"alertmanager:9093"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;

&lt;span class="na"&gt;scrape_configs&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;job_name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;prometheus&lt;/span&gt;
    &lt;span class="na"&gt;static_configs&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;targets&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"localhost:9090"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;

  &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;job_name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;node&lt;/span&gt;
    &lt;span class="na"&gt;static_configs&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;targets&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"node-exporter:9100"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
        &lt;span class="na"&gt;labels&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;server&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"web-01"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;To watch more servers, run node_exporter on each one and add them as targets. Only let your Prometheus server reach their port 9100, at the firewall.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 3: The five alerts worth starting with
&lt;/h2&gt;

&lt;p&gt;Most teams start with too many alerts, get flooded and start ignoring all of them. These five catch the outages that actually happen on most Linux servers. &lt;code&gt;prometheus/alerts.yml&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;groups&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;node&lt;/span&gt;
    &lt;span class="na"&gt;rules&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;alert&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;InstanceDown&lt;/span&gt;
        &lt;span class="na"&gt;expr&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;up == 0&lt;/span&gt;
        &lt;span class="na"&gt;for&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;2m&lt;/span&gt;
        &lt;span class="na"&gt;labels&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="s"&gt;severity:&lt;/span&gt; &lt;span class="s"&gt;critical&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
        &lt;span class="na"&gt;annotations&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;summary&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"{{ $labels.instance }} is not responding"&lt;/span&gt;

      &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;alert&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;DiskWillFillIn24h&lt;/span&gt;
        &lt;span class="na"&gt;expr&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;predict_linear(node_filesystem_avail_bytes{fstype!~"tmpfs|overlay|squashfs"}[6h], 24 * 3600) &amp;lt; 0&lt;/span&gt;
        &lt;span class="na"&gt;for&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;30m&lt;/span&gt;
        &lt;span class="na"&gt;labels&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="s"&gt;severity:&lt;/span&gt; &lt;span class="s"&gt;warning&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
        &lt;span class="na"&gt;annotations&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;summary&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"{{ $labels.mountpoint }} on {{ $labels.instance }} will be full within 24h"&lt;/span&gt;

      &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;alert&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;DiskAlmostFull&lt;/span&gt;
        &lt;span class="na"&gt;expr&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;(1 - node_filesystem_avail_bytes{fstype!~"tmpfs|overlay|squashfs"} / node_filesystem_size_bytes{fstype!~"tmpfs|overlay|squashfs"}) * 100 &amp;gt; 90&lt;/span&gt;
        &lt;span class="na"&gt;for&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;10m&lt;/span&gt;
        &lt;span class="na"&gt;labels&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="s"&gt;severity:&lt;/span&gt; &lt;span class="s"&gt;critical&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
        &lt;span class="na"&gt;annotations&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;summary&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"{{ $labels.mountpoint }} on {{ $labels.instance }} is over 90% full"&lt;/span&gt;

      &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;alert&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;MemoryLow&lt;/span&gt;
        &lt;span class="na"&gt;expr&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;(1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100 &amp;gt; 90&lt;/span&gt;
        &lt;span class="na"&gt;for&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;10m&lt;/span&gt;
        &lt;span class="na"&gt;labels&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="s"&gt;severity:&lt;/span&gt; &lt;span class="s"&gt;warning&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
        &lt;span class="na"&gt;annotations&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;summary&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"Memory on {{ $labels.instance }} is over 90% used"&lt;/span&gt;

      &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;alert&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;CpuSaturated&lt;/span&gt;
        &lt;span class="na"&gt;expr&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) &amp;gt; 90&lt;/span&gt;
        &lt;span class="na"&gt;for&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;15m&lt;/span&gt;
        &lt;span class="na"&gt;labels&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="s"&gt;severity:&lt;/span&gt; &lt;span class="s"&gt;warning&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
        &lt;span class="na"&gt;annotations&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;summary&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"CPU on {{ $labels.instance }} above 90% for 15 minutes"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Why these:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;DiskWillFillIn24h&lt;/code&gt;&lt;/strong&gt; is the most useful alert most people don't have. &lt;code&gt;predict_linear&lt;/code&gt; looks at the last 6 hours and warns you while there's still time to act, not when the database has already crashed.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;for:&lt;/code&gt;&lt;/strong&gt; on every rule keeps a short spike (a backup, a deploy) from paging anyone.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;MemAvailable&lt;/code&gt;&lt;/strong&gt;, not &lt;code&gt;MemFree&lt;/code&gt;. Linux uses free memory as cache, so &lt;code&gt;MemFree&lt;/code&gt; is always low and means nothing.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Step 4: alertmanager.yml (Telegram or email)
&lt;/h2&gt;

&lt;p&gt;Alertmanager has a native Telegram receiver:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;route&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;receiver&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;telegram&lt;/span&gt;
  &lt;span class="na"&gt;group_by&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"alertname"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"instance"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
  &lt;span class="na"&gt;group_wait&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;30s&lt;/span&gt;
  &lt;span class="na"&gt;group_interval&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;5m&lt;/span&gt;
  &lt;span class="na"&gt;repeat_interval&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;4h&lt;/span&gt;

&lt;span class="na"&gt;receivers&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;telegram&lt;/span&gt;
    &lt;span class="na"&gt;telegram_configs&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;bot_token&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"123456:ABC-your-bot-token"&lt;/span&gt;
        &lt;span class="na"&gt;chat_id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;-1001234567890&lt;/span&gt;
        &lt;span class="na"&gt;parse_mode&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;HTML&lt;/span&gt;
        &lt;span class="na"&gt;send_resolved&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Create a bot with &lt;a class="mentioned-user" href="https://dev.to/botfather"&gt;@botfather&lt;/a&gt;, add it to a group, and get the group's &lt;code&gt;chat_id&lt;/code&gt; from &lt;code&gt;https://api.telegram.org/bot&amp;lt;TOKEN&amp;gt;/getUpdates&lt;/code&gt; after someone posts in the group. For email, use &lt;code&gt;email_configs&lt;/code&gt; with your SMTP server instead.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;repeat_interval: 4h&lt;/code&gt; matters: an unfixed problem reminds you every 4 hours instead of every 5 minutes.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 5: Start it and test an alert
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;docker compose up -d
docker compose ps
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;ul&gt;
&lt;li&gt;Prometheus targets: open &lt;code&gt;http://127.0.0.1:9090/targets&lt;/code&gt; through your tunnel. Both jobs should show &lt;strong&gt;UP&lt;/strong&gt;.&lt;/li&gt;
&lt;li&gt;Grafana: &lt;code&gt;http://127.0.0.1:3000&lt;/code&gt;, log in as &lt;code&gt;admin&lt;/code&gt;, add Prometheus as a data source (&lt;code&gt;http://prometheus:9090&lt;/code&gt;), then import dashboard &lt;strong&gt;1860&lt;/strong&gt; ("Node Exporter Full").&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Test the alert path before you need it:&lt;/strong&gt; run &lt;code&gt;docker compose stop node-exporter&lt;/code&gt;, wait about 3 minutes, and &lt;code&gt;InstanceDown&lt;/code&gt; should arrive in Telegram. Start it again and you should get the "resolved" message.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;An alert you have never seen fire is an alert you don't know works.&lt;/p&gt;

&lt;h2&gt;
  
  
  Common mistakes
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Mistake&lt;/th&gt;
&lt;th&gt;What happens&lt;/th&gt;
&lt;th&gt;Fix&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Ports published on &lt;code&gt;0.0.0.0&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;Metrics and admin UIs open to the internet, even with UFW on&lt;/td&gt;
&lt;td&gt;Bind to &lt;code&gt;127.0.0.1&lt;/code&gt;, use a tunnel or proxy with auth&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;No &lt;code&gt;for:&lt;/code&gt; on rules&lt;/td&gt;
&lt;td&gt;Alerts on every short spike, then everyone ignores them&lt;/td&gt;
&lt;td&gt;2 to 15 minutes depending on the rule&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Alerting on &lt;code&gt;MemFree&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;Constant false alarms&lt;/td&gt;
&lt;td&gt;Use &lt;code&gt;MemAvailable&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Never testing the alert path&lt;/td&gt;
&lt;td&gt;First real outage, no message&lt;/td&gt;
&lt;td&gt;Stop node-exporter once and watch the alert arrive&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Monitoring server is the server you monitor&lt;/td&gt;
&lt;td&gt;It goes down together with everything else&lt;/td&gt;
&lt;td&gt;Monitor from a second machine, or add an external uptime check&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  Where to go next
&lt;/h2&gt;

&lt;p&gt;Once host metrics are covered, the next layer is application metrics (Nginx, databases, queues) and logs in one place. If you'd rather have someone else run this, PikoSystem offers &lt;a href="https://pikosystem.com/en/devops/observability/" rel="noopener noreferrer"&gt;observability setup with Grafana and Prometheus&lt;/a&gt; and &lt;a href="https://pikosystem.com/en/managed-it/proactive-monitoring/" rel="noopener noreferrer"&gt;proactive server monitoring&lt;/a&gt;, and a &lt;a href="https://pikosystem.com/en/free-audit/server-monitoring/" rel="noopener noreferrer"&gt;free server monitoring review&lt;/a&gt; if you want a second opinion on an existing setup.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;References:&lt;/strong&gt; &lt;a href="https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/" rel="noopener noreferrer"&gt;Prometheus: alerting rules&lt;/a&gt;, &lt;a href="https://prometheus.io/docs/alerting/latest/configuration/" rel="noopener noreferrer"&gt;Alertmanager configuration&lt;/a&gt;, &lt;a href="https://github.com/prometheus/node_exporter" rel="noopener noreferrer"&gt;node_exporter&lt;/a&gt;&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Parsa Shahmohammadi works on &lt;a href="https://pikosystem.com/en/" rel="noopener noreferrer"&gt;PikoSystem&lt;/a&gt;, an IT services and server management team.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>devops</category>
      <category>monitoring</category>
      <category>linux</category>
      <category>docker</category>
    </item>
  </channel>
</rss>
