<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: sunah kim</title>
    <description>The latest articles on DEV Community by sunah kim (@pythona).</description>
    <link>https://dev.to/pythona</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4170698%2F17c9c746-a38d-4e91-a5ca-2a65d7821a71.png</url>
      <title>DEV Community: sunah kim</title>
      <link>https://dev.to/pythona</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/pythona"/>
    <language>en</language>
    <item>
      <title>30 Minutes After a CVSS 9.8 Alert — The Conclusion Was 'No Patch Yet' and a 50-Second Reboot</title>
      <dc:creator>sunah kim</dc:creator>
      <pubDate>Thu, 19 Nov 2026 11:54:13 +0000</pubDate>
      <link>https://dev.to/pythona/30-minutes-after-a-cvss-98-alert-the-conclusion-was-no-patch-yet-and-a-50-second-reboot-kmg</link>
      <guid>https://dev.to/pythona/30-minutes-after-a-cvss-98-alert-the-conclusion-was-no-patch-yet-and-a-50-second-reboot-kmg</guid>
      <description>&lt;h2&gt;
  
  
  Intro
&lt;/h2&gt;

&lt;p&gt;A post went around on social media: "Linux kernel TCP (IPv4) use-after-free, CVE-2026-98432, CVSS 9.8. If your cloud server has open TCP ports, update immediately." I run a small production server (Ubuntu 24.04, one VM, Caddy reverse proxy + Java app on Docker), so here's what I actually checked, in order, and where I got stuck.&lt;/p&gt;

&lt;p&gt;Spoiler: &lt;strong&gt;no fixed kernel existed yet for my kernel flavor.&lt;/strong&gt; Instead, I applied other security patches that had been sitting there waiting for a reboot (about 50 seconds of downtime), and left a one-liner that will tell me the moment a fix ships.&lt;/p&gt;

&lt;h2&gt;
  
  
  The setup
&lt;/h2&gt;



&lt;pre data-lang="mermaid"&gt;&lt;code&gt;flowchart LR
    U["Browser"] --&amp;gt;|"443"| C["Caddy container&amp;lt;br/&amp;gt;reverse proxy"]
    C --&amp;gt;|"8080"| A["app container&amp;lt;br/&amp;gt;Java 17"]
    A --&amp;gt; D["Managed DB&amp;lt;br/&amp;gt;(outside the VM)"]
    subgraph VM["1 VM · Ubuntu 24.04 · open ports 22/80/443"]
      C
      A
    end&lt;/code&gt;&lt;/pre&gt;



&lt;p&gt;The important part: &lt;strong&gt;no data lives on this server.&lt;/strong&gt; The DB and file storage are external, so the box is disposable — which made the later reboot decision easy. If the DB had been on the VM, I'd have started with a snapshot.&lt;/p&gt;

&lt;h2&gt;
  
  
  Pitfall 1: The server I meant to inspect wasn't there
&lt;/h2&gt;

&lt;p&gt;SSH to the IP in my notes: &lt;code&gt;Operation timed out&lt;/code&gt;. Not just 22 — 80 and 443 were closed too. Yet the service domain happily returned 200. The service had been &lt;strong&gt;migrated to a different VM&lt;/strong&gt; (new IP, different SSH key), and my notes and scripts still had the old one.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;dig +short app.example.com   &lt;span class="c"&gt;# where the domain points NOW&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; p &lt;span class="k"&gt;in&lt;/span&gt; 22 80 443&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="k"&gt;do&lt;/span&gt; nc -z -w 5 203.0.113.10 &lt;span class="nv"&gt;$p&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"$p open"&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"$p closed"&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="k"&gt;done&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Step zero of any security check is "which box is the service actually running on?" Don't trust an IP somebody (or an old doc) gave you — walk backwards from the domain. Then fix the notes the same day; the next person losing ten minutes to a timeout is you.&lt;/p&gt;

&lt;h2&gt;
  
  
  Pitfall 2: "CVSS 9.8" and "a patch exists for my server" are different sentences
&lt;/h2&gt;

&lt;p&gt;The post said "patches are rolling out for Ubuntu 22.04/24.04, run &lt;code&gt;apt upgrade&lt;/code&gt;." But the official Ubuntu CVE tracker showed, for my release:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Kernel package&lt;/th&gt;
&lt;th&gt;Status&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;linux&lt;/code&gt; (6.8 GA)&lt;/td&gt;
&lt;td&gt;Vulnerable, work in progress in 6.8.0-149&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;linux-aws&lt;/code&gt; (6.8)&lt;/td&gt;
&lt;td&gt;Vulnerable, work in progress&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;linux-aws-7.0&lt;/code&gt; (this server)&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;Vulnerable — no target version even listed&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;So &lt;strong&gt;running &lt;code&gt;apt upgrade&lt;/code&gt; today would not fix this CVE at all.&lt;/strong&gt; The post wasn't wrong, exactly — "rolling out" just doesn't mean "has reached every kernel flavor." Cloud kernels (&lt;code&gt;-aws&lt;/code&gt;/&lt;code&gt;-azure&lt;/code&gt;/&lt;code&gt;-gcp&lt;/code&gt;) and HWE series often lag behind the GA kernel.&lt;/p&gt;

&lt;p&gt;Finding out which flavor you're on:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;uname -r                                  &lt;span class="c"&gt;# e.g. 7.0.0-1012-aws → "aws" flavor, 7.0 HWE&lt;/span&gt;
dpkg -l &lt;span class="s1"&gt;'linux-image-*'&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; awk &lt;span class="s1"&gt;'/^ii/{print $2, $3}'&lt;/span&gt;
apt changelog linux-image-7.0.0-1014-aws 2&lt;span class="o"&gt;&amp;gt;&amp;amp;&lt;/span&gt;1 &lt;span class="o"&gt;|&lt;/span&gt; grep -i -m3 &lt;span class="s1"&gt;'CVE-2026-98432'&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"(not mentioned)"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;One detail: don't silence the changelog command with &lt;code&gt;2&amp;gt;/dev/null&lt;/code&gt;. If you do, you can't tell "empty because the fetch failed" from "empty because the CVE isn't mentioned."&lt;/p&gt;

&lt;h2&gt;
  
  
  Pitfall 3: "Everything is updated" — but the running kernel is old
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;apt list --upgradable&lt;/code&gt; showed 0 packages. Yet &lt;code&gt;uname -r&lt;/code&gt; said 1012, kernel 1014 was installed, and &lt;code&gt;/var/run/reboot-required&lt;/code&gt; was still waving its "System restart required" flag. Someone had &lt;strong&gt;installed the patches and then not rebooted for a week.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Replacing kernel files on disk changes nothing about the kernel running in memory. Libraries take effect on process restart; the kernel alone requires a reboot to be &lt;em&gt;applied&lt;/em&gt;.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;cat /var/run/reboot-required 2&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;/dev/null &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"no reboot needed"&lt;/span&gt;
cat /var/run/reboot-required.pkgs 2&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;/dev/null   &lt;span class="c"&gt;# which packages demand it&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  The 50-second reboot — reasoning and timeline
&lt;/h2&gt;

&lt;p&gt;Three reasons the reboot was safe: (1) both containers use &lt;code&gt;unless-stopped&lt;/code&gt;, so they come back with Docker; (2) no data on the box; (3) nothing downstream cascades if this server stops.&lt;br&gt;
&lt;/p&gt;

&lt;pre data-lang="mermaid"&gt;&lt;code&gt;timeline
    title Inside the 50 seconds (measured)
    10:43:17 : sudo reboot sent
    10:43:41 : SSH back · uname -r shows new kernel
             : both containers "Up 4 seconds"
             : site still 502
    10:44:05 : app "Started" log line
             : site 200 · API 401 = healthy&lt;/code&gt;&lt;/pre&gt;



&lt;p&gt;Right after recovery, &lt;code&gt;docker ps&lt;/code&gt; showed both containers &lt;code&gt;Up&lt;/code&gt; — and the site returned 502. The reverse proxy was awake, but the JVM behind it hadn't opened its port yet. Decide your "healthy means exactly these values" check in advance: for this app it's "site returns 200 AND an auth-required API returns 401" (meaning the request made it all the way to auth). &lt;strong&gt;Container Up ≠ service OK.&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;s&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;$(&lt;/span&gt;curl -s -o /dev/null -w &lt;span class="s1"&gt;'%{http_code}'&lt;/span&gt; https://app.example.com/&lt;span class="o"&gt;)&lt;/span&gt;
&lt;span class="nv"&gt;a&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;$(&lt;/span&gt;curl -s -o /dev/null -w &lt;span class="s1"&gt;'%{http_code}'&lt;/span&gt; https://app.example.com/api/auth/me&lt;span class="o"&gt;)&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt; &lt;span class="s2"&gt;"$s"&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; 200 &lt;span class="o"&gt;]&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="o"&gt;[&lt;/span&gt; &lt;span class="s2"&gt;"$a"&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; 401 &lt;span class="o"&gt;]&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"up: site=$s api=$a"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  What to leave behind when there is no patch: a daily one-liner
&lt;/h2&gt;

&lt;p&gt;Ports 80/443 can't be closed, so the only complete defense against this CVE is the patch — and I wrote that fact itself into the report. Then I left one line that catches the moment a fix ships:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;ssh server &lt;span class="s1"&gt;'sudo apt update -qq &amp;amp;&amp;amp; apt list --upgradable 2&amp;gt;/dev/null | grep linux-'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The other signal is my row on the tracker flipping to &lt;code&gt;Released&lt;/code&gt;. The day either one changes, I re-run Pitfall 3 and the reboot procedure. That's it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Takeaways
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Step zero is "which server?" — walk backwards from the domain&lt;/li&gt;
&lt;li&gt;A CVSS score tells you urgency; your kernel row on the tracker tells you what you can actually do&lt;/li&gt;
&lt;li&gt;Installed ≠ applied — for kernels, the reboot is the application, and &lt;code&gt;reboot-required&lt;/code&gt; is the signal&lt;/li&gt;
&lt;li&gt;Container Up ≠ service OK — define "healthy means exactly these values" ahead of time&lt;/li&gt;
&lt;li&gt;"No patch exists" is a valid conclusion; a daily one-liner catches the moment that changes&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The biggest thing I accomplished in 30 minutes after seeing that CVSS 9.8 post was confirming that &lt;em&gt;no patch existed yet&lt;/em&gt;. That confirmation was the report. The 50-second reboot was a bonus.&lt;/p&gt;

</description>
      <category>linux</category>
      <category>ubuntu</category>
      <category>security</category>
      <category>devops</category>
    </item>
    <item>
      <title>30 Minutes Before Deleting the Old Server — A Checklist for Logs, Configs, and Payment-Provider IPs</title>
      <dc:creator>sunah kim</dc:creator>
      <pubDate>Wed, 18 Nov 2026 11:53:04 +0000</pubDate>
      <link>https://dev.to/pythona/30-minutes-before-deleting-the-old-server-a-checklist-for-logs-configs-and-payment-provider-ips-2hkm</link>
      <guid>https://dev.to/pythona/30-minutes-before-deleting-the-old-server-a-checklist-for-logs-configs-and-payment-provider-ips-2hkm</guid>
      <description>&lt;h2&gt;
  
  
  Introduction
&lt;/h2&gt;

&lt;p&gt;The production server migration was done, and the old server had reached the "watch it for 2–3 days, then delete" stage. On Lightsail, a stopped instance still accrues charges, so "just leave it around for a while" isn't an option — deletion is the only path. And deletion is irreversible.&lt;/p&gt;

&lt;p&gt;This article is the checklist of what I actually did in the ~30 minutes before pressing the delete button: evacuating logs and configs, and cleaning up the origin-IP list registered with our payment provider. I also hit two shell gotchas along the way, so those are included too.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I evacuated
&lt;/h2&gt;

&lt;p&gt;What hadn't been moved to the new server wasn't the application itself — it was the "periphery":&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Apache access/error logs (25 files including rotated ones)&lt;/li&gt;
&lt;li&gt;143 files from EC-CUBE's &lt;code&gt;var/log&lt;/code&gt; (about 7 weeks' worth)&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;httpd.conf&lt;/code&gt;, &lt;code&gt;conf.d/&lt;/code&gt;, &lt;code&gt;php.d/&lt;/code&gt;, and Let's Encrypt renewal configs&lt;/li&gt;
&lt;li&gt;crontab, the rpm package list, and &lt;code&gt;yum history&lt;/code&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;91MB in total, placed on the new server at &lt;code&gt;/var/backup/old-prod-YYYYMMDD/&lt;/code&gt;, root-only (700).&lt;/p&gt;

&lt;h3&gt;
  
  
  Why keep the logs
&lt;/h3&gt;

&lt;p&gt;The access statistics for the month before the migration — the ratio of scan traffic to total requests, the number of source IPs behind login attacks — were all computed from these logs. They are the only baseline for comparing "did access patterns change?" after the migration.&lt;/p&gt;

&lt;p&gt;Same for the config snapshot. When you later wonder "this setting existed on the old server but not the new one," the snapshot is your only evidence.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# Example: capturing configs and metadata (on the old server)&lt;/span&gt;
rpm -qa &lt;span class="o"&gt;|&lt;/span&gt; sort &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; /tmp/rpm-list.txt
yum &lt;span class="nb"&gt;history&lt;/span&gt; list all &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; /tmp/yum-history.txt
crontab -l &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; /tmp/crontab-root.txt
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Gotcha 1: ssh inside a heredoc ate the rest of the script
&lt;/h2&gt;

&lt;p&gt;While writing the evacuation script as a heredoc, I ran into steps silently not executing.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;bash &lt;span class="o"&gt;&amp;lt;&amp;lt;&lt;/span&gt;&lt;span class="s1"&gt;'EOF'&lt;/span&gt;
ssh old-prod &lt;span class="s1"&gt;'rpm -qa'&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; rpm-list.txt   &lt;span class="c"&gt;# stdin gets consumed here&lt;/span&gt;
&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"next step"&lt;/span&gt;                         &lt;span class="c"&gt;# never runs&lt;/span&gt;
EOF
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The cause: &lt;code&gt;ssh&lt;/code&gt; reads standard input. The remainder of the heredoc (the following lines) gets passed to ssh, and the script ends early. The fix is &lt;code&gt;ssh -n&lt;/code&gt;, which detaches stdin.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;ssh -n old-prod &lt;span class="s1"&gt;'rpm -qa'&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; rpm-list.txt
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Gotcha 2: bundling multiple sources into one rsync hides failures
&lt;/h2&gt;

&lt;p&gt;Trying to finish the transfer in one shot, I passed several directories to a single rsync:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;rsync -av old-prod:/etc/httpd/ :/var/log/httpd/ :/etc/letsencrypt/ ./backup/
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This looks like it works, but when some paths fail while others succeed, the overall result is hard to judge — failures can get swallowed silently. For an evacuation, where the whole point is "everything made it," running individual rsyncs and checking each exit code was the reliable approach.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="k"&gt;for&lt;/span&gt; src &lt;span class="k"&gt;in&lt;/span&gt; /etc/httpd /var/log/httpd /etc/letsencrypt&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="k"&gt;do&lt;/span&gt;
  rsync -av &lt;span class="s2"&gt;"old-prod:${src}/"&lt;/span&gt; &lt;span class="s2"&gt;"./backup${src}/"&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"FAILED: ${src}"&lt;/span&gt;
&lt;span class="k"&gt;done&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Deletion order
&lt;/h2&gt;

&lt;p&gt;Once the evacuation was done, deletion proceeded in this order:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Take one Lightsail snapshot (billed only for actual used capacity — cheap insurance)&lt;/li&gt;
&lt;li&gt;Delete the instance&lt;/li&gt;
&lt;li&gt;Release the now-detached static IP (a static IP detached from an instance is billable)&lt;/li&gt;
&lt;li&gt;Remove the old IP from the origin-IP list registered with the payment provider (for both production and test environments)&lt;/li&gt;
&lt;li&gt;Update the asset inventory, SSH config, and ops notes with the new server's info&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Skip step 4 and an unused IP stays on the payment provider's allowlist. After release, that IP gets assigned to someone else — meaning "an IP that isn't ours is registered as our payment origin." This was cleanup that belonged on deletion day itself.&lt;/p&gt;

&lt;h2&gt;
  
  
  Summary
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Deletion is irreversible, so the criterion isn't "what do I need in order to delete" but "what will I miss after it's gone." Logs and configs are always the latter&lt;/li&gt;
&lt;li&gt;Use &lt;code&gt;-n&lt;/code&gt; with ssh inside heredocs — otherwise the rest of your script silently disappears&lt;/li&gt;
&lt;li&gt;Don't bundle evacuation rsyncs; run them one at a time where exit codes are visible&lt;/li&gt;
&lt;li&gt;"Deletion work" includes releasing the static IP and updating the payment provider's origin list&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The last command on migration day was &lt;code&gt;rsync&lt;/code&gt;, not &lt;code&gt;rm&lt;/code&gt; — that's this checklist in one sentence.&lt;/p&gt;

</description>
      <category>aws</category>
      <category>lightsail</category>
      <category>linux</category>
      <category>devops</category>
    </item>
    <item>
      <title>php_value Throws a 500 — Migrating to Amazon Linux 2023, Where mod_php Is Gone</title>
      <dc:creator>sunah kim</dc:creator>
      <pubDate>Tue, 17 Nov 2026 11:53:33 +0000</pubDate>
      <link>https://dev.to/pythona/phpvalue-throws-a-500-migrating-to-amazon-linux-2023-where-modphp-is-gone-4e6c</link>
      <guid>https://dev.to/pythona/phpvalue-throws-a-500-migrating-to-amazon-linux-2023-where-modphp-is-gone-4e6c</guid>
      <description>&lt;h2&gt;
  
  
  What happened
&lt;/h2&gt;

&lt;p&gt;This is from migrating an e-commerce site running on Amazon Linux 2 (PHP 8.2) to Amazon Linux 2023 (PHP 8.3). I moved the code and the database, started httpd, opened the top page — and got a &lt;strong&gt;500 Internal Server Error&lt;/strong&gt;. Nothing in the application log. Only Apache's error_log had this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;/var/www/app/html/.htaccess: Invalid command 'php_value',
perhaps misspelled or defined by a module not included in the server configuration
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The culprit was a block that had been sitting in the old server's &lt;code&gt;.htaccess&lt;/code&gt; for years:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight apache"&gt;&lt;code&gt;&lt;span class="n"&gt;php_value&lt;/span&gt; &lt;span class="n"&gt;upload_max_filesize&lt;/span&gt; &lt;span class="mi"&gt;500M&lt;/span&gt;
&lt;span class="n"&gt;php_value&lt;/span&gt; &lt;span class="n"&gt;post_max_size&lt;/span&gt; &lt;span class="mi"&gt;512M&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  The cause — AL2023's httpd has no mod_php
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;php_value&lt;/code&gt; and &lt;code&gt;php_flag&lt;/code&gt; are &lt;strong&gt;mod_php directives&lt;/strong&gt;. On AL2, httpd shipped with mod_php built in, so they worked. The standard setup on AL2023 is &lt;strong&gt;php-fpm + proxy_fcgi with the event MPM&lt;/strong&gt; — the mod_php module simply does not exist.&lt;/p&gt;

&lt;p&gt;When Apache finds a directive it cannot parse in &lt;code&gt;.htaccess&lt;/code&gt;, it returns 500 for every request to that directory. So this isn't "the PHP settings silently stop applying" — &lt;strong&gt;not a single page renders&lt;/strong&gt;. When a migration changes how the runtime executes, config files living inside the app directory are where the teeth come out.&lt;/p&gt;

&lt;h2&gt;
  
  
  The fix — relocate where the settings live
&lt;/h2&gt;

&lt;h3&gt;
  
  
  1. Neutralize &lt;code&gt;.htaccess&lt;/code&gt; with IfModule
&lt;/h3&gt;

&lt;p&gt;I wanted to keep compatibility with the old server, so instead of deleting the &lt;code&gt;php_value&lt;/code&gt; lines I wrapped them:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight apache"&gt;&lt;code&gt;&lt;span class="c1"&gt;# .htaccess after the fix
&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nc"&gt;IfModule&lt;/span&gt; &lt;span class="n"&gt;mod_php&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;
&lt;span class="n"&gt;php_value&lt;/span&gt; &lt;span class="n"&gt;upload_max_filesize&lt;/span&gt; &lt;span class="mi"&gt;500M&lt;/span&gt;
&lt;span class="n"&gt;php_value&lt;/span&gt; &lt;span class="n"&gt;post_max_size&lt;/span&gt; &lt;span class="mi"&gt;512M&lt;/span&gt;
&lt;span class="o"&gt;&amp;lt;/&lt;/span&gt;&lt;span class="nc"&gt;IfModule&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Where mod_php exists, the block still applies; where it doesn't, Apache skips it entirely. The same file can be deployed to both generations of servers during the transition.&lt;/p&gt;

&lt;h3&gt;
  
  
  2. Move the same values into php.ini
&lt;/h3&gt;

&lt;p&gt;Everything that used to live in &lt;code&gt;.htaccess&lt;/code&gt; went into &lt;code&gt;/etc/php.d/99-app.ini&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ini"&gt;&lt;code&gt;&lt;span class="c1"&gt;; /etc/php.d/99-app.ini
&lt;/span&gt;&lt;span class="na"&gt;memory_limit&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;512M&lt;/span&gt;
&lt;span class="na"&gt;post_max_size&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;512M&lt;/span&gt;
&lt;span class="na"&gt;upload_max_filesize&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;500M&lt;/span&gt;
&lt;span class="na"&gt;max_execution_time&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;
&lt;span class="na"&gt;opcache.memory_consumption&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;256&lt;/span&gt;
&lt;span class="na"&gt;opcache.max_accelerated_files&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;20000&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;With php-fpm, a &lt;code&gt;systemctl reload php-fpm&lt;/code&gt; picks these up.&lt;/p&gt;

&lt;h3&gt;
  
  
  3. Derive pm.max_children from available memory
&lt;/h3&gt;

&lt;p&gt;The default value will swap a small instance to death. Assuming &lt;code&gt;memory_limit 512M&lt;/code&gt;, I used 4 children for 512 MB of RAM and 8 for 1 GB.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ini"&gt;&lt;code&gt;&lt;span class="c1"&gt;; /etc/php-fpm.d/www.conf
&lt;/span&gt;&lt;span class="na"&gt;pm.max_children&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;8&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  4. Diff the extension list against the old server's php -m
&lt;/h3&gt;

&lt;p&gt;AL2023's default set is missing &lt;code&gt;zip&lt;/code&gt; and &lt;code&gt;exif&lt;/code&gt;. I captured &lt;code&gt;php -m&lt;/code&gt; on the old server, diffed it against the new one, and installed only the gap (&lt;code&gt;dnf install php8.3-zip&lt;/code&gt;, etc.). Skip this comparison and the features that break are the ones you rarely touch — like the admin panel's CSV upload — weeks later.&lt;/p&gt;

&lt;h3&gt;
  
  
  5. Regenerate caches as the web user
&lt;/h3&gt;

&lt;p&gt;Running cache warmup or &lt;code&gt;assets:install&lt;/code&gt; as root creates root-owned cache files, which become your next 500. Everything went through &lt;code&gt;sudo -u apache php bin/console ...&lt;/code&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Side quests along the way
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Lightsail's &lt;code&gt;Default SSH key&lt;/code&gt; is &lt;strong&gt;per-region&lt;/strong&gt;. The same pem happened to open the new instance only because it was in the same region.&lt;/li&gt;
&lt;li&gt;If rsync's source includes root-only files (sessions, etc.), use &lt;code&gt;--rsync-path="sudo rsync"&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;A default vhost in &lt;code&gt;ssl.conf&lt;/code&gt; without a ServerName logs &lt;code&gt;certificate does NOT include an ID&lt;/code&gt; on every start. Harmless, but it pollutes log monitoring, so I filled it in.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Results
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Response times: 0.17–0.43s → &lt;strong&gt;0.12–0.48s&lt;/strong&gt; (HTTP/2 + event MPM)&lt;/li&gt;
&lt;li&gt;OpenSSL 1.0.2k → &lt;strong&gt;3.5.8&lt;/strong&gt;
&lt;/li&gt;
&lt;li&gt;Support runway extended: OS until 2028-03, PHP 8.3 until 2027-12&lt;/li&gt;
&lt;li&gt;Vendor directory rebuilt via &lt;code&gt;composer install&lt;/code&gt; from a &lt;code&gt;composer.lock&lt;/code&gt; already validated in dev (13 installs / 102 upgrades, 46 seconds)&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Takeaway
&lt;/h2&gt;

&lt;p&gt;Ninety percent of a runtime migration is a table: &lt;strong&gt;where does the old server read each setting, and where does the new server expect it to be written?&lt;/strong&gt; php.ini values, &lt;code&gt;.htaccess&lt;/code&gt; values, the module list, vhosts, cron — build the table first, and the migration script is just a translation of that table.&lt;/p&gt;

&lt;p&gt;My migration script failed three times in dev while it got polished, then passed on the first try in production. That, I think, is exactly what dev environments are for.&lt;/p&gt;

</description>
      <category>php</category>
      <category>apache</category>
      <category>aws</category>
      <category>devops</category>
    </item>
    <item>
      <title>The Fix Only Existed in kernel6.12 — Switching Kernel Streams on Amazon Linux 2023</title>
      <dc:creator>sunah kim</dc:creator>
      <pubDate>Mon, 16 Nov 2026 11:53:44 +0000</pubDate>
      <link>https://dev.to/pythona/the-fix-only-existed-in-kernel612-switching-kernel-streams-on-amazon-linux-2023-27pa</link>
      <guid>https://dev.to/pythona/the-fix-only-existed-in-kernel612-switching-kernel-streams-on-amazon-linux-2023-27pa</guid>
      <description>&lt;h2&gt;
  
  
  What happened
&lt;/h2&gt;

&lt;p&gt;This was during the week of the CVSS 9.8 kernel TCP vulnerability (CVE-2026-98432). When I checked the Amazon Linux Security Center for the production Amazon Linux 2023 server I manage, I found an unusual situation:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The default &lt;code&gt;kernel&lt;/code&gt; package (6.1 series) — &lt;strong&gt;Pending Fix&lt;/strong&gt;
&lt;/li&gt;
&lt;li&gt;A separate package, &lt;code&gt;kernel6.12&lt;/code&gt; (6.12 series) — a fixed version with its ALAS (security advisory) &lt;strong&gt;already published&lt;/strong&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Within the same OS, whether a fix existed or not depended on the kernel stream. AL2023 ships a newer kernel stream as a separately named package, &lt;code&gt;kernel6.12&lt;/code&gt;, alongside the default 6.1 series. Instead of waiting for the 6.1 fix, &lt;strong&gt;switching streams meant I could patch today&lt;/strong&gt;. So that's what I did.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I did — four moves
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# 1. Install the 6.12 kernel (coexists with the 6.1 series)&lt;/span&gt;
sudo dnf install kernel6.12 kernel6.12-modules-extra

&lt;span class="c"&gt;# 2. Make 6.12 the kernel for the next boot&lt;/span&gt;
sudo grubby --set-default /boot/vmlinuz-6.12.*.amzn2023.x86_64

&lt;span class="c"&gt;# 3. Declare that future kernel updates should keep 6.12 as default&lt;/span&gt;
&lt;span class="c"&gt;#    /etc/sysconfig/kernel&lt;/span&gt;
&lt;span class="c"&gt;#    UPDATEDEFAULT=yes&lt;/span&gt;
&lt;span class="c"&gt;#    DEFAULTKERNEL=kernel6.12&lt;/span&gt;

&lt;span class="c"&gt;# 4. Reboot to apply&lt;/span&gt;
sudo reboot
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;dnf install&lt;/code&gt; doesn't replace the existing 6.1 kernel — it puts 6.12 &lt;strong&gt;next to it&lt;/strong&gt; in &lt;code&gt;/boot&lt;/code&gt;. Which one boots is decided by the grub default, which is why step 2 explicitly switches it with &lt;code&gt;grubby&lt;/code&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Pitfalls
&lt;/h2&gt;

&lt;h3&gt;
  
  
  1. &lt;code&gt;dnf update&lt;/code&gt; keeps you on 6.1 forever
&lt;/h3&gt;

&lt;p&gt;This is the biggest trap. &lt;code&gt;kernel&lt;/code&gt; and &lt;code&gt;kernel6.12&lt;/code&gt; are &lt;strong&gt;different package names&lt;/strong&gt;, so no amount of &lt;code&gt;dnf update&lt;/code&gt; will move you from 6.1 to 6.12 — it only takes you to the latest 6.1. That's why "I keep updating but the Security Center warning won't go away." A stream switch only happens through an explicit &lt;code&gt;dnf install kernel6.12&lt;/code&gt;.&lt;/p&gt;

&lt;h3&gt;
  
  
  2. Without &lt;code&gt;/etc/sysconfig/kernel&lt;/code&gt;, you'll eventually boot 6.1 again
&lt;/h3&gt;

&lt;p&gt;Stopping at &lt;code&gt;grubby --set-default&lt;/code&gt; leaves a trap armed. When the next kernel update lands, whether the new kernel becomes the default is decided by &lt;code&gt;DEFAULTKERNEL&lt;/code&gt; in &lt;code&gt;/etc/sysconfig/kernel&lt;/code&gt;. If that still says &lt;code&gt;kernel&lt;/code&gt; (the 6.1 series), &lt;strong&gt;the default can flip back to 6.1 on an update&lt;/strong&gt;.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ini"&gt;&lt;code&gt;&lt;span class="c1"&gt;# /etc/sysconfig/kernel
&lt;/span&gt;&lt;span class="na"&gt;UPDATEDEFAULT&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;yes&lt;/span&gt;
&lt;span class="na"&gt;DEFAULTKERNEL&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;kernel6&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="mi"&gt;12&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;"I switched today" and "I'll keep tracking the 6.12 series" are two different settings.&lt;/p&gt;

&lt;h3&gt;
  
  
  3. I kept 6.1 installed as a fallback — and paid for it
&lt;/h3&gt;

&lt;p&gt;I didn't uninstall the 6.1 series: it's the escape hatch if 6.12 fails to boot. That was the right call as insurance, but it has a cost. Security updates keep arriving for the unused 6.1 series, so &lt;code&gt;dnf needs-restarting -r&lt;/code&gt; now warns "kernel updated, reboot required" — about a stream that isn't even running. Before trusting that warning, I check &lt;code&gt;uname -r&lt;/code&gt; against &lt;code&gt;grubby --default-kernel&lt;/code&gt; to see whether the running stream was actually updated.&lt;/p&gt;

&lt;h3&gt;
  
  
  4. Production reboots follow a fixed template
&lt;/h3&gt;

&lt;p&gt;Production reboots always run in the same order: maintenance page ON → reboot → verify the real pages via curl with a bypass cookie → OFF. Since verification happens before reopening, users never see the 502s right after boot.&lt;/p&gt;

&lt;h2&gt;
  
  
  Verification
&lt;/h2&gt;

&lt;p&gt;Not straight to production: I ran the whole thing on a dev server with the same setup first (about 1 minute). On production, measured by the maintenance page, &lt;strong&gt;downtime was 21 seconds&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Three checks afterward:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# Is the running kernel the 6.12 series?&lt;/span&gt;
uname -r
&lt;span class="c"&gt;# → 6.12.x-x.amzn2023.x86_64&lt;/span&gt;

&lt;span class="c"&gt;# Is the relevant ALAS now shown as installed?&lt;/span&gt;
sudo dnf updateinfo list --installed &lt;span class="o"&gt;|&lt;/span&gt; grep -i &lt;span class="o"&gt;&amp;lt;&lt;/span&gt;ALAS ID&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;

&lt;span class="c"&gt;# Does the public page return 200?&lt;/span&gt;
curl -s -o /dev/null -w &lt;span class="s1"&gt;'%{http_code}\n'&lt;/span&gt; https://example.com/
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Takeaways
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;"No patch yet" is not a statement about the whole distro — read it &lt;strong&gt;per kernel stream&lt;/strong&gt;. On AL2023, &lt;code&gt;kernel&lt;/code&gt; (6.1) was Pending Fix while &lt;code&gt;kernel6.12&lt;/code&gt; already had the fix&lt;/li&gt;
&lt;li&gt;A stream switch never happens via &lt;code&gt;dnf update&lt;/code&gt;. It's four moves: &lt;code&gt;dnf install kernel6.12 kernel6.12-modules-extra&lt;/code&gt; → &lt;code&gt;grubby&lt;/code&gt; → &lt;code&gt;/etc/sysconfig/kernel&lt;/code&gt; → reboot&lt;/li&gt;
&lt;li&gt;If you keep the old stream as insurance, accept the false "reboot required" warnings that come with it&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The same week, an Ubuntu server I was also watching had no fix published for its kernel stream — nothing to do but wait. The AL2023 server, by switching streams, went from alert to applied fix within the day. Half of vulnerability response is locating &lt;strong&gt;which row in the security tracker's table is yours&lt;/strong&gt;. A different row means different options and a different deadline.&lt;/p&gt;

</description>
      <category>aws</category>
      <category>linux</category>
      <category>security</category>
      <category>devops</category>
    </item>
    <item>
      <title>672 Requests in a New Server's First 4.5 Hours — Zero Were Human</title>
      <dc:creator>sunah kim</dc:creator>
      <pubDate>Sun, 15 Nov 2026 11:53:22 +0000</pubDate>
      <link>https://dev.to/pythona/672-requests-in-a-new-servers-first-45-hours-zero-were-human-1a25</link>
      <guid>https://dev.to/pythona/672-requests-in-a-new-servers-first-45-hours-zero-were-human-1a25</guid>
      <description>&lt;h2&gt;
  
  
  What happened
&lt;/h2&gt;

&lt;p&gt;The day I spun up a new instance, attached a static IP, and switched DNS over, I got curious: who is actually using this dev server right now? So I aggregated the access log for the first four and a half hours, grouped by IP.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;$ awk &lt;span class="s1"&gt;'{print $1}'&lt;/span&gt; /var/log/httpd/access_log &lt;span class="o"&gt;|&lt;/span&gt; sort &lt;span class="o"&gt;|&lt;/span&gt; uniq -c &lt;span class="o"&gt;|&lt;/span&gt; sort -rn
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;672 requests total. The breakdown:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Source&lt;/th&gt;
&lt;th&gt;Count&lt;/th&gt;
&lt;th&gt;What it was&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Me (post-migration QA)&lt;/td&gt;
&lt;td&gt;119&lt;/td&gt;
&lt;td&gt;Manual checks&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;One cloud-provider IP&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;421&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;A vulnerability scan — in &lt;strong&gt;17 seconds&lt;/strong&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Certificate authority&lt;/td&gt;
&lt;td&gt;4&lt;/td&gt;
&lt;td&gt;Domain validation&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;~20 crawlers&lt;/td&gt;
&lt;td&gt;~120&lt;/td&gt;
&lt;td&gt;Hit &lt;code&gt;/&lt;/code&gt; and the login page, then leave&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The only human was me. Actual users: zero. The 421-request scan was the eye-opener: in seventeen seconds it swept through every "commonly forgotten file" on its list — &lt;code&gt;/wp-config.php.bak&lt;/code&gt;, &lt;code&gt;/.env&lt;/code&gt;, &lt;code&gt;/graphql&lt;/code&gt;, &lt;code&gt;/wrangler.toml&lt;/code&gt; — and got 404 or 403 for all of them. It was probing a non-WordPress server for WordPress config files. Scanners don't profile their target; they just run the whole list, top to bottom.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where I got tripped up
&lt;/h2&gt;

&lt;h3&gt;
  
  
  1. Scans arrive with your domain name within hours of the DNS switch
&lt;/h3&gt;

&lt;p&gt;My first reaction: "Scanning a raw IP, fine — but how do they already know the domain?" The answer is Certificate Transparency logs. When a TLS certificate is issued, the domain name is written to a public log. Scanners watch those logs, so "new certificate issued" equals "new server just appeared" in near real time. "Nobody knows this URL yet, so it's safe" simply isn't a thing.&lt;/p&gt;

&lt;h3&gt;
  
  
  2. You cannot filter scanners by User-Agent
&lt;/h3&gt;

&lt;p&gt;The scanner's UA was a perfectly ordinary &lt;code&gt;Mozilla/5.0 ...&lt;/code&gt; browser string. UA-based filtering is useless. Two signals actually separated bots from humans:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Requests per second&lt;/strong&gt; — no human opens 421 pages in 17 seconds&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Ratio of nonexistent paths&lt;/strong&gt; — legitimate users don't stumble into &lt;code&gt;/wp-config.php.bak&lt;/code&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# Status-code distribution per IP&lt;/span&gt;
$ awk &lt;span class="s1"&gt;'{print $1, $9}'&lt;/span&gt; access_log &lt;span class="o"&gt;|&lt;/span&gt; sort &lt;span class="o"&gt;|&lt;/span&gt; uniq -c &lt;span class="o"&gt;|&lt;/span&gt; sort -rn &lt;span class="o"&gt;|&lt;/span&gt; head -20
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  3. "Nothing in &lt;code&gt;last&lt;/code&gt;" does not mean "nobody logged in"
&lt;/h3&gt;

&lt;p&gt;I checked SSH history with &lt;code&gt;last&lt;/code&gt;, saw nothing suspicious, and almost called it a day. But key-based non-interactive execution (&lt;code&gt;ssh host 'command'&lt;/code&gt;) leaves no entry in &lt;code&gt;last&lt;/code&gt;. Absence of a login record is not absence of a login — worth keeping in mind before you write "no unauthorized access" in a report.&lt;/p&gt;

&lt;h3&gt;
  
  
  4. A zero-byte ssl_access_log scared me
&lt;/h3&gt;

&lt;p&gt;The HTTPS log was empty and for a moment I thought logging was broken. In reality, the virtual-host &lt;code&gt;CustomLog&lt;/code&gt; directive funneled both HTTP and HTTPS into the single &lt;code&gt;access_log&lt;/code&gt; file. When a log looks "missing," check where the config actually writes before panicking.&lt;/p&gt;

&lt;h2&gt;
  
  
  Verification
&lt;/h2&gt;

&lt;p&gt;For the 421 scan requests, the one thing I made sure to check was whether any of them got a 200.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;$ grep &lt;span class="s2"&gt;"&amp;lt;scanner-ip&amp;gt;"&lt;/span&gt; access_log &lt;span class="o"&gt;|&lt;/span&gt; awk &lt;span class="s1"&gt;'$9==200 {print $7}'&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; sort &lt;span class="o"&gt;|&lt;/span&gt; uniq -c
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The only 200s were redirects to the login page; not a single request reached a real file. Only after seeing that can you say "we were scanned, but nothing landed." Counting requests isn't log review — knowing &lt;strong&gt;what returned 200&lt;/strong&gt; is.&lt;/p&gt;

&lt;h2&gt;
  
  
  Takeaways
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;The default state of a public server isn't "unused" — it's "&lt;strong&gt;used only by bots&lt;/strong&gt;"&lt;/li&gt;
&lt;li&gt;Judge whether a server is idle by the number of &lt;em&gt;human&lt;/em&gt; requests, not total requests&lt;/li&gt;
&lt;li&gt;Your domain goes public the moment a certificate is issued. Lock down pre-launch servers like production&lt;/li&gt;
&lt;li&gt;Identify scanners by speed and nonexistent-path ratio, never by User-Agent&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The first visitor to any new server is not a customer. It's a scanner looking for your &lt;code&gt;.env&lt;/code&gt;. Lock the doors on day one accordingly.&lt;/p&gt;

</description>
      <category>security</category>
      <category>linux</category>
      <category>aws</category>
      <category>devops</category>
    </item>
    <item>
      <title>Where Should an AI Agent Go on an EC Site with 1–2 Orders a Week? The Numbers Said: Nowhere, Yet</title>
      <dc:creator>sunah kim</dc:creator>
      <pubDate>Sat, 14 Nov 2026 11:52:49 +0000</pubDate>
      <link>https://dev.to/pythona/where-should-an-ai-agent-go-on-an-ec-site-with-1-2-orders-a-week-the-numbers-said-nowhere-yet-500a</link>
      <guid>https://dev.to/pythona/where-should-an-ai-agent-go-on-an-ec-site-with-1-2-orders-a-week-the-numbers-said-nowhere-yet-500a</guid>
      <description>&lt;h2&gt;
  
  
  What happened
&lt;/h2&gt;

&lt;p&gt;I got a request along these lines: "Our dev server is sitting idle — let's put an AI agent on it and automate some work." The target was a small e-commerce site, and the first candidate everyone thought of was automating order fulfillment (shipping updates).&lt;/p&gt;

&lt;p&gt;Before building anything, I decided to look at the numbers. I aggregated the last 180 days of orders:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="nb"&gt;COUNT&lt;/span&gt;&lt;span class="cm"&gt;(*) AS orders,
       SUM(payment_method = 'credit') / COUNT(*)&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;credit_ratio&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;dtb_order&lt;/span&gt;
&lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="n"&gt;order_date&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="kc"&gt;CURRENT_DATE&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="kt"&gt;INTERVAL&lt;/span&gt; &lt;span class="mi"&gt;180&lt;/span&gt; &lt;span class="no"&gt;DAY&lt;/span&gt;
  &lt;span class="k"&gt;AND&lt;/span&gt; &lt;span class="n"&gt;order_status&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;IN&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'cancelled'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The result: &lt;strong&gt;39 orders in 180 days&lt;/strong&gt; — one or two a week. About 70% were card payments, and the payment itself is handled automatically by the payment processor. The only manual work is the shipping update, which adds up to roughly &lt;strong&gt;10 minutes per week&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;The job we wanted to automate turned out to be ten minutes a week. That's where I stopped and reconsidered.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why "automate fulfillment" was the wrong first project
&lt;/h2&gt;

&lt;p&gt;Several problems surfaced once I looked closely.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;1. Most of the automation doesn't need an LLM.&lt;/strong&gt; "When an order comes in, update the shipping status and send a notification" is a deterministic task with fixed inputs and outputs. That's cron + SQL + email — there's no reason to put an agent in the middle. Agents earn their keep where &lt;strong&gt;judgment, writing, or exceptions&lt;/strong&gt; are involved: reading logs and summarizing what's normal vs. concerning, spotting and explaining discrepancies in aggregates, drafting replies to customer inquiries, writing product descriptions and translations.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. Starting your first agent project with production DB writes maximizes the cost of failure.&lt;/strong&gt; Changing shipping status means mutating production data. Picking a task where a mistake directly affects customers as your &lt;em&gt;first&lt;/em&gt; agent project is backwards. Start with read-only daily reports, earn trust, then consider writes.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. A 512MB server can't run a local model anyway.&lt;/strong&gt; The whole premise was "the server is idle, let's use it" — but with 512MB of RAM, local LLMs are off the table. API-based agents use almost no memory on the host, so the server's spec is irrelevant. In other words, "the server is idle" was never a reason to introduce an agent in the first place.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;4. The dev server shares the production database.&lt;/strong&gt; If an agent is going to touch that DB at all, it needs its own read-only account:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="no"&gt;USER&lt;/span&gt; &lt;span class="s1"&gt;'agent_ro'&lt;/span&gt;@&lt;span class="s1"&gt;'localhost'&lt;/span&gt; &lt;span class="no"&gt;IDENTIFIED&lt;/span&gt; &lt;span class="k"&gt;BY&lt;/span&gt; &lt;span class="s1"&gt;'********'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;GRANT&lt;/span&gt; &lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="k"&gt;ON&lt;/span&gt; &lt;span class="n"&gt;shop_db&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="k"&gt;TO&lt;/span&gt; &lt;span class="s1"&gt;'agent_ro'&lt;/span&gt;@&lt;span class="s1"&gt;'localhost'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If you reuse the application's credentials, "it was only supposed to read" stops being a guarantee.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I actually did: a five-candidate table, ranked
&lt;/h2&gt;

&lt;p&gt;Instead of picking by gut feeling, I put the candidates in a table with two columns — "why an agent?" and "risk":&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Candidate&lt;/th&gt;
&lt;th&gt;Why an agent?&lt;/th&gt;
&lt;th&gt;Risk&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Daily report (orders + log summary)&lt;/td&gt;
&lt;td&gt;Reads logs, explains the situation in words&lt;/td&gt;
&lt;td&gt;Low (read-only)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Detecting &amp;amp; explaining aggregate mismatches&lt;/td&gt;
&lt;td&gt;Hypothesizes causes, writes them up&lt;/td&gt;
&lt;td&gt;Low (read-only)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Drafting inquiry replies&lt;/td&gt;
&lt;td&gt;Context-aware text generation&lt;/td&gt;
&lt;td&gt;Medium (human reviews before sending)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Product descriptions &amp;amp; translations&lt;/td&gt;
&lt;td&gt;Text generation itself&lt;/td&gt;
&lt;td&gt;Medium (human reviews before publishing)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Automating fulfillment&lt;/td&gt;
&lt;td&gt;— deterministic, no LLM needed&lt;/td&gt;
&lt;td&gt;High (production DB writes)&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Laid out like this, the original favorite — fulfillment — lands at the bottom. The agent-shaped work is the top four, and the natural order is: read-only first, then human-reviewed text generation, then (only once trusted) writes.&lt;/p&gt;

&lt;p&gt;The final decision was to &lt;strong&gt;put the whole thing on hold&lt;/strong&gt;. Saving ten minutes a week doesn't justify the cost of building, operating, and monitoring an agent. Making the table, ranking the candidates, and recording "not now" — that's a legitimate outcome too.&lt;/p&gt;

&lt;h2&gt;
  
  
  Takeaways
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;The first question for agent adoption isn't "what should we automate?" but &lt;strong&gt;"how many minutes a week is that job, and what breaks if it goes wrong?"&lt;/strong&gt;
&lt;/li&gt;
&lt;li&gt;Deterministic work is cron + SQL. Agents pay off where judgment, writing, or exceptions are involved&lt;/li&gt;
&lt;li&gt;First project: read-only. Production writes come last&lt;/li&gt;
&lt;li&gt;"The server is idle" is not a reason to adopt an agent (API-based agents barely use the host)&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The work you &lt;em&gt;want&lt;/em&gt; to automate and the work &lt;em&gt;worth&lt;/em&gt; automating are different things. Thirty-nine orders in 180 days made that clear.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>llm</category>
      <category>automation</category>
      <category>devops</category>
    </item>
    <item>
      <title>The 5 Questions an AI Agent Install Wizard Asks — How I Answered Them on a Work Mac</title>
      <dc:creator>sunah kim</dc:creator>
      <pubDate>Fri, 13 Nov 2026 11:53:38 +0000</pubDate>
      <link>https://dev.to/pythona/the-5-questions-an-ai-agent-install-wizard-asks-how-i-answered-them-on-a-work-mac-310n</link>
      <guid>https://dev.to/pythona/the-5-questions-an-ai-agent-install-wizard-asks-how-i-answered-them-on-a-work-mac-310n</guid>
      <description>&lt;h2&gt;
  
  
  Introduction
&lt;/h2&gt;

&lt;p&gt;I installed an open-source AI agent (Hermes Agent) on my work Mac. The installation itself took five minutes — but during those five minutes, the setup wizard asked me five questions. And while answering them, I realized that these five answers determine not how smart the agent is, but how &lt;strong&gt;safe&lt;/strong&gt; it is.&lt;/p&gt;

&lt;p&gt;The wizard asks:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Which model to use (choose from 5 free options)&lt;/li&gt;
&lt;li&gt;Where to execute commands (Local / Docker / SSH / cloud sandbox)&lt;/li&gt;
&lt;li&gt;Whether to enable an egress firewall on the Docker sandbox&lt;/li&gt;
&lt;li&gt;Whether to connect a messenger (Telegram, etc.)&lt;/li&gt;
&lt;li&gt;Whether to share usage statistics&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;This post walks through how I actually answered, what tripped me up, and the reasoning behind each choice.&lt;/p&gt;

&lt;h2&gt;
  
  
  Gotcha 1: Decide the execution environment before the model
&lt;/h2&gt;

&lt;p&gt;The wizard asks about the model first, but the priority is the other way around.&lt;/p&gt;

&lt;p&gt;This Mac holds SSH keys that reach production servers, and repositories containing DB credentials. If you pick &lt;code&gt;Local&lt;/code&gt; as the execution environment, the agent reads files and runs commands with your user privileges — which means &lt;strong&gt;everything the agent reads, and every command output, gets sent to the model provider&lt;/strong&gt;. If you're on a free model, that's a configuration you want to avoid even more.&lt;/p&gt;

&lt;p&gt;So I chose the Docker sandbox. The first run promptly failed:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Error: Cannot connect to the Docker daemon at unix:///var/run/docker.sock.
Is the docker daemon running?
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The cause was trivial — Docker Desktop simply wasn't running.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;open -a Docker   &lt;span class="c"&gt;# wait a few dozen seconds for the daemon&lt;/span&gt;
docker info      &lt;span class="c"&gt;# you're good once the Server section appears&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Re-ran the wizard and it went through. If you're a Mac user who "has Docker installed but rarely uses it," start by launching the daemon.&lt;/p&gt;

&lt;h2&gt;
  
  
  Gotcha 2: Assume free models log your input — and may disappear
&lt;/h2&gt;

&lt;p&gt;Free models often record your input and use it for training. This is especially true for models like &lt;code&gt;stealth/...&lt;/code&gt; where &lt;strong&gt;the provider is anonymous&lt;/strong&gt; — there's no way to verify where your data ends up. Free tiers also have tight rate limits and can be discontinued without notice, so they're a poor foundation for automation that runs daily.&lt;/p&gt;

&lt;p&gt;I picked a free large model from a clearly identified provider, on the premise that work files never leave the Docker sandbox anyway.&lt;/p&gt;

&lt;h2&gt;
  
  
  Gotcha 3: Don't answer an unfamiliar 5-way model choice with "I don't know"
&lt;/h2&gt;

&lt;p&gt;That said, out of the five model names listed, I had used exactly zero. If you settle this with "no idea, whatever," you won't be able to help the next person who faces the same screen.&lt;/p&gt;

&lt;p&gt;I did three things: state the limits of my evidence (all you can read from a model name is its approximate parameter scale), commit to one anyway, and — most importantly — &lt;strong&gt;confirm how to change it later&lt;/strong&gt;.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;hermes model        &lt;span class="c"&gt;# switch models interactively&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Once you know the change command exists, the initial choice becomes a provisional one. Don't let the model picker become a blocker.&lt;/p&gt;

&lt;h2&gt;
  
  
  The remaining two: egress firewall and messenger integration
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Egress firewall: Yes.&lt;/strong&gt; This puts outbound traffic from the container on an allowlist, closing off routes for the agent to send data somewhere unexpected. There is a side effect: package installation inside the container may get blocked. If &lt;code&gt;pip install&lt;/code&gt; or &lt;code&gt;npm install&lt;/code&gt; fails later, suspect this setting first.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Messenger integration: Skip.&lt;/strong&gt; It lets you command the agent via DM from Telegram and the like — but the moment you connect it, that messenger account becomes &lt;strong&gt;a remote shell into your Mac&lt;/strong&gt;. If the account gets compromised, the attacker can effectively run commands on your machine. Not on a work machine.&lt;/p&gt;

&lt;p&gt;I also turned off usage statistics sharing.&lt;/p&gt;

&lt;h2&gt;
  
  
  Verification: the final configuration
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Question&lt;/th&gt;
&lt;th&gt;Choice&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Model&lt;/td&gt;
&lt;td&gt;Free large model (clearly identified provider)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Execution environment&lt;/td&gt;
&lt;td&gt;Docker sandbox&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Egress firewall&lt;/td&gt;
&lt;td&gt;ON&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Messenger integration&lt;/td&gt;
&lt;td&gt;None&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Usage statistics&lt;/td&gt;
&lt;td&gt;Not shared&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The install took five minutes. Thinking took longer than typing.&lt;/p&gt;

&lt;h2&gt;
  
  
  Summary
&lt;/h2&gt;

&lt;p&gt;The security of an agent installation is decided not by "which model is smartest" but by &lt;strong&gt;what this process can read, and where what it reads goes&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;One more thing: the wizard's defaults are "the settings that let you try it fastest," not "the safest settings." Local execution, messenger integration recommended, firewall off — those were all the defaults. On a work machine, it's worth pausing to ask what an Enter-mashing install has just granted.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>security</category>
      <category>docker</category>
      <category>llm</category>
    </item>
    <item>
      <title>The Pre-Maintenance Notice I Send Before a 1-Minute Outage Is Just 4 Lines</title>
      <dc:creator>sunah kim</dc:creator>
      <pubDate>Thu, 12 Nov 2026 11:52:56 +0000</pubDate>
      <link>https://dev.to/pythona/the-pre-maintenance-notice-i-send-before-a-1-minute-outage-is-just-4-lines-3m69</link>
      <guid>https://dev.to/pythona/the-pre-maintenance-notice-i-send-before-a-1-minute-outage-is-just-4-lines-3m69</guid>
      <description>&lt;h2&gt;
  
  
  What happened
&lt;/h2&gt;

&lt;p&gt;A security patch batch included the kernel and glibc, which means a reboot. Thanks to the runbook I polished in &lt;a href="https://zenn.dev/pythona/articles/reboot-downtime-human-bottleneck" rel="noopener noreferrer"&gt;my previous post&lt;/a&gt;, the site is only down for just under a minute. Still, you can't silently restart a production e-commerce site during business hours, so I had to send an advance notice to the client (a Japanese company).&lt;/p&gt;

&lt;p&gt;My first draft was, I thought, thorough and honest: the vulnerability's severity, the CVE number, the risk of leaving it unpatched, why this week was preferable. Then I reread it and realized: &lt;strong&gt;the person receiving this will feel anxious before they feel informed.&lt;/strong&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  The trap — the more you explain "why", the more worried they get
&lt;/h2&gt;

&lt;p&gt;To an engineer, laying out all the reasoning is what honesty looks like. But the recipient is not an engineer. A CVE number or the phrase "severity: Important" is not decision-making input for them — it's an &lt;strong&gt;alarm&lt;/strong&gt;. Three paragraphs of justification read as "is this so dangerous that it needs three paragraphs?"&lt;/p&gt;

&lt;p&gt;The other trap was manufactured urgency. This patch was rated Important, not Critical. If I write "urgent action required" for an Important-grade patch, I have no words left for a real emergency. For a non-critical patch, the right ask is "let us do this during a quiet window this week" — not a midnight maintenance, not "immediately."&lt;/p&gt;

&lt;h2&gt;
  
  
  The fix — lock the notice to 4 lines
&lt;/h2&gt;

&lt;p&gt;The notice I actually sent boiled down to these four lines:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;1. When:      Thu, Nov 12, around 14:00. Total work time: ~5 minutes
2. Impact:    For about 1 minute, the site will be unreachable (no page / error page)
3. Verified:  The same update has been applied to our dev server and confirmed working
4. Follow-up: We will confirm normal operation and notify you once done
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;With greetings, the full email was under ten lines. The only technical content was one phrase — "for a server security update." No severity grade, no CVE number. What builds trust is not the detailed explanation; it's &lt;strong&gt;line 3: "already verified on the dev server."&lt;/strong&gt; That single line says "this is not a first attempt in production," and the reply came back as one line: "Understood, please go ahead."&lt;/p&gt;

&lt;p&gt;Technical details are for people who ask — and even then, one parenthetical is enough: "it updates glibc (a core Linux library)."&lt;/p&gt;

&lt;h2&gt;
  
  
  Before sending — confirm the patch is NOT already applied
&lt;/h2&gt;

&lt;p&gt;Right before sending the notice, I checked the server's state. If the nightly auto-patching had already applied the update, the notice itself would be pointless.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# When did dnf last run, and what did it do?&lt;/span&gt;
sudo dnf &lt;span class="nb"&gt;history&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; head -5

&lt;span class="c"&gt;# When was the last reboot?&lt;/span&gt;
uptime -s
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;I confirmed the update was not yet in &lt;code&gt;dnf history&lt;/code&gt; and that &lt;code&gt;uptime -s&lt;/code&gt; still showed last month's date — then sent the notice. Announcing an outage for work that turns out to be already done costs you credibility; the next notice gets taken less seriously.&lt;/p&gt;

&lt;p&gt;The post-work flow is simple: reboot → check the new kernel with &lt;code&gt;uname -r&lt;/code&gt; → confirm the main pages return 200 → reply with a single line: "Work completed; normal operation confirmed." A short completion notice also communicates "it went fine" better than a long one.&lt;/p&gt;

&lt;h2&gt;
  
  
  Takeaways
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Lock the pre-maintenance notice to four lines: &lt;strong&gt;when, impact, verified, follow-up&lt;/strong&gt;. A long explanation reads as anxiety fuel, not honesty&lt;/li&gt;
&lt;li&gt;For non-critical patches, ask for "a quiet window this week." Save the word "urgent" for actual emergencies&lt;/li&gt;
&lt;li&gt;Leave severity grades and CVE numbers out. The trust-building line is "already verified on the dev server"&lt;/li&gt;
&lt;li&gt;Before sending, confirm the patch is not already applied, with &lt;code&gt;dnf history&lt;/code&gt; and &lt;code&gt;uptime -s&lt;/code&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A short notice earns more trust than a long explanation. The moment your explanation gets long, the first thing the reader thinks is: "is something dangerous going on?"&lt;/p&gt;

</description>
      <category>sre</category>
      <category>linux</category>
      <category>devops</category>
      <category>communication</category>
    </item>
    <item>
      <title>The Kernel-Update Reboot Should Take 1 Minute. It Took 10 — the Slow Part Was Human</title>
      <dc:creator>sunah kim</dc:creator>
      <pubDate>Wed, 11 Nov 2026 11:54:31 +0000</pubDate>
      <link>https://dev.to/pythona/the-kernel-update-reboot-should-take-1-minute-it-took-10-the-slow-part-was-human-4k4o</link>
      <guid>https://dev.to/pythona/the-kernel-update-reboot-should-take-1-minute-it-took-10-the-slow-part-was-human-4k4o</guid>
      <description>&lt;h2&gt;
  
  
  What happened
&lt;/h2&gt;

&lt;p&gt;Our nightly auto-patching (a systemd timer running &lt;code&gt;dnf upgrade&lt;/code&gt;) deliberately does not reboot. On days when the batch includes packages like the kernel or glibc — things that only take effect after a restart — it just logs &lt;code&gt;REBOOT-NEEDED&lt;/code&gt; and leaves the reboot to a human.&lt;/p&gt;

&lt;p&gt;The 33 piled-up updates from &lt;a href="https://zenn.dev/pythona/articles/dnf-refresh-metadata-false-zero" rel="noopener noreferrer"&gt;my previous post&lt;/a&gt; included exactly that: kernel and glibc. Rebooting the production site follows a simple runbook: put up the maintenance page → reboot → verify → take the page down. The server itself is down for under a minute. Yet the maintenance page was actually visible for &lt;strong&gt;about 10 minutes&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;The slow part wasn't the server. It was me.&lt;/p&gt;

&lt;h2&gt;
  
  
  Procedure v1 failed — maintenance ON came too early
&lt;/h2&gt;

&lt;p&gt;My original procedure looked like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# v1: maintenance ON → update → reboot&lt;/span&gt;
sudo touch /var/www/html/.maintenance    &lt;span class="c"&gt;# maintenance page ON (flag file)&lt;/span&gt;
sudo dnf upgrade -y --releasever=latest  &lt;span class="c"&gt;# ← takes 1–2 minutes here&lt;/span&gt;
sudo reboot
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Downloading and applying packages takes a minute or two, during which the server can serve traffic just fine — but visitors see a maintenance page. Pointless downtime from the user's perspective. So I swapped the order:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# v2: update → maintenance ON → reboot&lt;/span&gt;
sudo dnf upgrade -y --releasever=latest
sudo touch /var/www/html/.maintenance
sudo reboot
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Now the only outage window is around the reboot itself — under a minute. In theory.&lt;/p&gt;

&lt;h2&gt;
  
  
  Measured: 10 minutes — the bottleneck was human
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Step&lt;/th&gt;
&lt;th&gt;Expected&lt;/th&gt;
&lt;th&gt;Measured&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;dnf download/apply (before maintenance)&lt;/td&gt;
&lt;td&gt;1–2 min&lt;/td&gt;
&lt;td&gt;~2 min (no impact)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Maintenance ON → reboot → OS back up&lt;/td&gt;
&lt;td&gt;~1 min&lt;/td&gt;
&lt;td&gt;~50 s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Human reopens browser SSH and logs back in&lt;/td&gt;
&lt;td&gt;(not budgeted)&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;~9 min&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Removal command (&lt;code&gt;rm&lt;/code&gt;)&lt;/td&gt;
&lt;td&gt;seconds&lt;/td&gt;
&lt;td&gt;seconds&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Checking &lt;code&gt;uptime -s&lt;/code&gt; afterwards showed the reboot itself finished in about 50 seconds. The remaining 9 minutes were me: the browser SSH session drops on reboot, I reopen the tab, retry a few connection errors wondering "is it up yet?", get distracted by something else, and finally log in and type the removal command.&lt;/p&gt;

&lt;p&gt;I had designed the procedure to minimize server downtime — but I was &lt;strong&gt;measuring the wrong thing&lt;/strong&gt;. What determines how long the maintenance page stays up isn't the reboot time; it's the time until a human types the next command.&lt;/p&gt;

&lt;h2&gt;
  
  
  The fix — take the human out of the removal step
&lt;/h2&gt;

&lt;p&gt;Two changes.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;(1) Prepare the one-liner to paste after reconnecting&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Right before rebooting, put the next command in a note:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;sudo rm -f /var/www/html/.maintenance &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; curl -sk -o /dev/null -w &lt;span class="s1"&gt;'%{http_code}\n'&lt;/span&gt; https://localhost/
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Reopen the session, paste, done. No "wait, what do I type after logging in?" time.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;(2) A systemd oneshot that clears the maintenance flag after reboot&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ini"&gt;&lt;code&gt;&lt;span class="c1"&gt;# /etc/systemd/system/maintenance-clear.service
&lt;/span&gt;&lt;span class="k"&gt;[Unit]&lt;/span&gt;
&lt;span class="na"&gt;Description&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nc"&gt;Clear&lt;/span&gt; &lt;span class="n"&gt;maintenance&lt;/span&gt; &lt;span class="n"&gt;flag&lt;/span&gt; &lt;span class="n"&gt;after&lt;/span&gt; &lt;span class="n"&gt;reboot&lt;/span&gt;
&lt;span class="na"&gt;After&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;httpd&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;service&lt;/span&gt;
&lt;span class="na"&gt;Requires&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;httpd&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;service&lt;/span&gt;

&lt;span class="k"&gt;[Service]&lt;/span&gt;
&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;oneshot&lt;/span&gt;
&lt;span class="c1"&gt;# Wait until localhost returns 200 before clearing the flag
&lt;/span&gt;&lt;span class="na"&gt;ExecStartPre&lt;/span&gt;&lt;span class="o"&gt;=/&lt;/span&gt;&lt;span class="n"&gt;usr&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;bin&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;bash&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;c&lt;/span&gt; &lt;span class="s1"&gt;'for i in $(seq 1 30); do curl -sfk -o /dev/null https://localhost/ &amp;amp;&amp;amp; exit 0; sleep 2; done; exit 1'&lt;/span&gt;
&lt;span class="na"&gt;ExecStart&lt;/span&gt;&lt;span class="o"&gt;=/&lt;/span&gt;&lt;span class="n"&gt;usr&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;bin&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;rm&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="kd"&gt;var&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;www&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;html&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;maintenance&lt;/span&gt;

&lt;span class="k"&gt;[Install]&lt;/span&gt;
&lt;span class="na"&gt;WantedBy&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;multi&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;user&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;target&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The key is the &lt;code&gt;ExecStartPre&lt;/code&gt; wait. If you remove the flag unconditionally at boot, the maintenance page disappears before the app is up and visitors see a 502. Verify "healthy means exactly this value" first, then remove. If the health check never passes, the flag stays and the maintenance page remains — the failure mode is safe.&lt;/p&gt;

&lt;p&gt;With this in place the human part of the runbook ends at "dnf → maintenance ON → reboot". I only come back later to check &lt;code&gt;uname -r&lt;/code&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Verification
&lt;/h2&gt;

&lt;p&gt;What I checked after the reboot:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;uptime -s                    &lt;span class="c"&gt;# reboot time (backs up the actual outage duration)&lt;/span&gt;
uname -r                     &lt;span class="c"&gt;# running on the new kernel?&lt;/span&gt;
sudo dnf &lt;span class="nb"&gt;history&lt;/span&gt; info last   &lt;span class="c"&gt;# what was applied&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;On top of that: during maintenance, I confirmed the four main pages returned 200 from inside using a bypass cookie; after removal, I recorded 200s and response times for the same four pages via their external URLs. The recovery time I log is not "when I removed the flag" but &lt;strong&gt;when the site looked healthy from outside&lt;/strong&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Takeaways
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Nightly auto-patching doesn't reboot by design. Kernel/glibc days need a human reboot&lt;/li&gt;
&lt;li&gt;Turn maintenance ON right before the reboot — showing it during the download is pointless exposure&lt;/li&gt;
&lt;li&gt;Measure downtime not as "time the server was down" but as "time until the human types the next command." Here: 50 seconds vs. 10 minutes&lt;/li&gt;
&lt;li&gt;Bundle the removal into the reconnect one-liner, or automate it with a health-checked oneshot, and the "human" row disappears from the timetable&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Add a "human" row to the time estimates in your runbooks. It's usually the longest one.&lt;/p&gt;

</description>
      <category>linux</category>
      <category>aws</category>
      <category>devops</category>
      <category>sre</category>
    </item>
    <item>
      <title>The 'Unpatched for 16 Days' Alert Was Already Patched — Why I Split Monitoring into Two Tiers</title>
      <dc:creator>sunah kim</dc:creator>
      <pubDate>Mon, 09 Nov 2026 11:53:54 +0000</pubDate>
      <link>https://dev.to/pythona/the-unpatched-for-16-days-alert-was-already-patched-why-i-split-monitoring-into-two-tiers-4h6i</link>
      <guid>https://dev.to/pythona/the-unpatched-for-16-days-alert-was-already-patched-why-i-split-monitoring-into-two-tiers-4h6i</guid>
      <description>&lt;h2&gt;
  
  
  What happened
&lt;/h2&gt;

&lt;p&gt;Every morning, an AI routine running in the cloud collects the latest security releases for PHP, Windows, the Linux kernel, Docker and so on from the web, and pushes a summary to my phone. One day the notification read:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;[URGENT] 16 days since the PHP 8.3.x security release.
Production server status unverified. Apply immediately.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;"Unpatched for 16 days." The same urgent alert came every single day. But when I actually SSH'd into the production server:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;$ &lt;/span&gt;php &lt;span class="nt"&gt;-v&lt;/span&gt;
PHP 8.3.&lt;span class="o"&gt;(&lt;/span&gt;patched version&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;cli&lt;span class="o"&gt;)&lt;/span&gt;
&lt;span class="nv"&gt;$ &lt;/span&gt;&lt;span class="nb"&gt;sudo &lt;/span&gt;dnf &lt;span class="nb"&gt;history &lt;/span&gt;list php&lt;span class="se"&gt;\*&lt;/span&gt; | &lt;span class="nb"&gt;head&lt;/span&gt; &lt;span class="nt"&gt;-3&lt;/span&gt;
ID | Command line               | Date
25 | update php&lt;span class="k"&gt;*&lt;/span&gt;                | 2 weeks ago
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;It had been patched long ago&lt;/strong&gt; — two days after the release, in fact. Yet the alert kept shouting "unpatched, urgent" for 16 days.&lt;/p&gt;

&lt;h2&gt;
  
  
  The cause: the alert routine can't reach the server
&lt;/h2&gt;

&lt;p&gt;The reason was simple: the cloud routine that crawls the web has &lt;strong&gt;no way to connect to the server&lt;/strong&gt;. It only knows "what has been released in the world," never "what is installed on this box." So it honestly keeps saying "production version unverified → possibly unpatched → urgent."&lt;/p&gt;

&lt;p&gt;It wasn't lying. But &lt;strong&gt;an urgent alert that arrives every day is no longer urgent.&lt;/strong&gt; Cry-wolf notifications get skimmed and ignored, and on the day a genuinely dangerous release lands, it arrives with exactly the same weight and drowns.&lt;/p&gt;

&lt;h2&gt;
  
  
  The fix: split "what's out" from "what's missing"
&lt;/h2&gt;

&lt;p&gt;I split the job into two tiers.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Tier&lt;/th&gt;
&lt;th&gt;Runs where&lt;/th&gt;
&lt;th&gt;Looks at&lt;/th&gt;
&lt;th&gt;Answers&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;① Collector&lt;/td&gt;
&lt;td&gt;Cloud (every morning)&lt;/td&gt;
&lt;td&gt;Vendor announcements, security trackers&lt;/td&gt;
&lt;td&gt;What has been released?&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;② Checker&lt;/td&gt;
&lt;td&gt;Local scheduled task&lt;/td&gt;
&lt;td&gt;Each server's package manager over SSH&lt;/td&gt;
&lt;td&gt;What is missing on &lt;em&gt;my&lt;/em&gt; servers?&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Tier ② reads tier ①'s summary as reference material, but &lt;strong&gt;the applied/not-applied verdict comes only from the package manager's output on the server&lt;/strong&gt;.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# Read-only check tier ② runs per server (example: Amazon Linux 2023)&lt;/span&gt;
&lt;span class="nb"&gt;sudo &lt;/span&gt;dnf updateinfo list &lt;span class="nt"&gt;--security&lt;/span&gt;   &lt;span class="c"&gt;# lists only unapplied security updates&lt;/span&gt;
rpm &lt;span class="nt"&gt;-q&lt;/span&gt; php-fpm kernel openssl         &lt;span class="c"&gt;# definitive installed versions&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If ①'s summary says "a PHP fix is out" but ②'s &lt;code&gt;dnf updateinfo&lt;/code&gt; shows nothing, the conclusion is "already applied." Conversely, anything &lt;code&gt;updateinfo&lt;/code&gt; reports is "missing" even if ① never mentioned it. With a single source of truth for the verdict, the ghost "16 days unpatched" alerts disappeared.&lt;/p&gt;

&lt;h2&gt;
  
  
  Gotchas
&lt;/h2&gt;

&lt;p&gt;Most of the friction turned out to be operational rules, not architecture.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;1. A summary covering every system is noise.&lt;/strong&gt; Mixing the PHP site, the internal Java system and the Windows servers into one digest meant that, in each per-site session, most of it was irrelevant. After the feedback "only tell me the parts that apply to me," tier ② now extracts only the lines matching each server's actual stack.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. Never execute commands found in the summary.&lt;/strong&gt; Tier ①'s summary is text derived from web pages. If tier ② naively runs commands written in it, &lt;strong&gt;web page content becomes commands executed on your server.&lt;/strong&gt; The summary is reference data only; tier ② runs its own fixed command list, nothing else.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. Decide the auto-apply scope up front.&lt;/strong&gt; OS security patches flow automatically through dev server → verification → production. Runtime major upgrades, library updates, and anything touching DB/DNS/payment settings are &lt;strong&gt;report-only&lt;/strong&gt; — a human decides.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;4. Production write commands were blocked by the agent's permission classifier.&lt;/strong&gt; I didn't work around it. Instead the report includes "the one line for a human to paste." The block itself is proof the boundary works.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;5. The first unattended run stalls on approval.&lt;/strong&gt; The very first scheduled execution sat in an approval dialog for 15 minutes. Now the runbook says: a human must open and approve the first run of any unattended job.&lt;/p&gt;

&lt;h2&gt;
  
  
  Verification
&lt;/h2&gt;

&lt;p&gt;Measured on the day of the split: tier ① claimed "2 urgent items." Tier ② showed only one applied to my site — and it was already patched. Verification took 30 seconds. Meanwhile, four OS patches that ①'s summary never mentioned were detected only by ②'s check. "What's out" and "what's missing" really do come from different sources, and now I had the numbers to show it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Takeaways
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Alerts only know "it's out"; servers only know "it's missing." The bridge between them doesn't have to be human eyeballs — one line of &lt;code&gt;rpm -q&lt;/code&gt; is enough.&lt;/li&gt;
&lt;li&gt;Base the applied/not-applied verdict solely on package manager output. Never let a web-derived summary judge — or execute.&lt;/li&gt;
&lt;li&gt;Document the auto-apply scope and the human-approval boundary before you automate.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;When urgent alerts arrive daily, they stop being urgent. The fastest way to reduce notifications isn't ignoring them — it's &lt;strong&gt;teaching the notifier what version you're actually running.&lt;/strong&gt;&lt;/p&gt;

</description>
      <category>security</category>
      <category>automation</category>
      <category>linux</category>
      <category>devops</category>
    </item>
    <item>
      <title>Don't Trust 'Zero Security Updates' on Amazon Linux 2023 — The Release-Pinning Trap</title>
      <dc:creator>sunah kim</dc:creator>
      <pubDate>Mon, 09 Nov 2026 02:22:27 +0000</pubDate>
      <link>https://dev.to/pythona/dont-trust-zero-security-updates-on-amazon-linux-2023-the-release-pinning-trap-3gcn</link>
      <guid>https://dev.to/pythona/dont-trust-zero-security-updates-on-amazon-linux-2023-the-release-pinning-trap-3gcn</guid>
      <description>&lt;h2&gt;
  
  
  What happened
&lt;/h2&gt;

&lt;p&gt;During a routine security check on a production Amazon Linux 2023 server, I ran the usual command:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;$ &lt;/span&gt;&lt;span class="nb"&gt;sudo &lt;/span&gt;dnf updateinfo list &lt;span class="nt"&gt;--security&lt;/span&gt;
Last metadata expiration check: 0:12:34 ago
&lt;span class="c"&gt;# (nothing)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Nothing. So: "zero pending security updates." I reported it as fully patched.&lt;/p&gt;

&lt;p&gt;The next day, while working on something else, I ran &lt;code&gt;dnf check-release-update&lt;/code&gt; — and the picture changed:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;$ &lt;/span&gt;&lt;span class="nb"&gt;sudo &lt;/span&gt;dnf check-release-update
WARNING:
  A newer release of &lt;span class="s2"&gt;"Amazon Linux"&lt;/span&gt; is available.

  Available Versions:
  Version 2023.9.20261104:
    Run the following &lt;span class="nb"&gt;command &lt;/span&gt;to upgrade to 2023.9.20261104:
      dnf upgrade &lt;span class="nt"&gt;--releasever&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;2023.9.20261104
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A newer release existed. Re-running the security check with &lt;code&gt;--releasever=latest&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;$ &lt;/span&gt;&lt;span class="nb"&gt;sudo &lt;/span&gt;dnf updateinfo list &lt;span class="nt"&gt;--security&lt;/span&gt; &lt;span class="nt"&gt;--releasever&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;latest
ALAS2023-2026-xxxx Important/Sec. openssh-&lt;span class="k"&gt;*&lt;/span&gt;
ALAS2023-2026-xxxx Important/Sec. libxml2-&lt;span class="k"&gt;*&lt;/span&gt;
ALAS2023-2026-xxxx Medium/Sec.    amazon-ssm-agent-&lt;span class="k"&gt;*&lt;/span&gt;
ALAS2023-2026-xxxx Important/Sec. kernel-&lt;span class="k"&gt;*&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Four security updates were waiting: openssh, libxml2, amazon-ssm-agent, and the kernel. My "zero" report from the day before was wrong.&lt;/p&gt;

&lt;h2&gt;
  
  
  The cause — AL2023 pins you to a release
&lt;/h2&gt;

&lt;p&gt;Amazon Linux 2023 uses a design called &lt;strong&gt;deterministic upgrades&lt;/strong&gt;. An instance's repositories are locked to the release it was created with (or last explicitly upgraded to), and a plain &lt;code&gt;dnf&lt;/code&gt; only sees updates &lt;em&gt;within that release&lt;/em&gt;.&lt;/p&gt;

&lt;p&gt;If you come from AL2 or CentOS, a quiet &lt;code&gt;yum check-update&lt;/code&gt; means you're current. On AL2023, it only means you're current &lt;em&gt;relative to the pinned release&lt;/em&gt;. Security fixes that landed in a newer release are invisible unless you pass &lt;code&gt;--releasever&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;It's a stability feature, not a bug — but combined with the assumption that "quiet output = safe," it's a nasty trap.&lt;/p&gt;

&lt;h2&gt;
  
  
  Gotchas
&lt;/h2&gt;

&lt;h3&gt;
  
  
  1. The day after patching, it still shows "zero"
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;sudo &lt;/span&gt;dnf upgrade &lt;span class="nt"&gt;--security&lt;/span&gt; &lt;span class="nt"&gt;--releasever&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;latest
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This pulls only the security packages from the new release and leaves &lt;code&gt;system-release&lt;/code&gt; untouched. Which means the next day, a check &lt;em&gt;without&lt;/em&gt; the option still reports zero — and you make the same mistake again. The fix is to bake &lt;code&gt;--releasever=latest&lt;/code&gt; permanently into your check scripts.&lt;/p&gt;

&lt;h3&gt;
  
  
  2. A false "reboot required"
&lt;/h3&gt;

&lt;p&gt;After applying, &lt;code&gt;dnf needs-restarting -r&lt;/code&gt; insisted a reboot was needed because "the kernel was updated." But the updated one was the &lt;strong&gt;unused spare kernel line (the 6.1-series &lt;code&gt;kernel&lt;/code&gt; package)&lt;/strong&gt; — the server actually runs the &lt;code&gt;kernel6.12&lt;/code&gt; package line.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;$ &lt;/span&gt;&lt;span class="nb"&gt;uname&lt;/span&gt; &lt;span class="nt"&gt;-r&lt;/span&gt;
6.12.x-x.amzn2023.x86_64
&lt;span class="nv"&gt;$ &lt;/span&gt;&lt;span class="nb"&gt;sudo &lt;/span&gt;grubby &lt;span class="nt"&gt;--default-kernel&lt;/span&gt;
/boot/vmlinuz-6.12.x-x.amzn2023.x86_64
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If the running kernel and the default boot kernel are unchanged, you don't need the reboot this warning asks for.&lt;/p&gt;

&lt;h3&gt;
  
  
  3. "Secure Boot not enabled" is not an error
&lt;/h3&gt;

&lt;p&gt;The &lt;code&gt;Secure Boot not enabled&lt;/code&gt; line printed during kernel installation is an informational message from the install scripts, not a failure. It looks scary the first time.&lt;/p&gt;

&lt;h2&gt;
  
  
  The actual procedure
&lt;/h2&gt;

&lt;p&gt;Since this was production, I dry-ran the transaction first with &lt;code&gt;--assumeno&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# 1. Preview (changes nothing)&lt;/span&gt;
&lt;span class="nb"&gt;sudo &lt;/span&gt;dnf upgrade &lt;span class="nt"&gt;--security&lt;/span&gt; &lt;span class="nt"&gt;--releasever&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;latest &lt;span class="nt"&gt;--assumeno&lt;/span&gt;
&lt;span class="c"&gt;#    → confirmed: 2 installs, 7 upgrades&lt;/span&gt;

&lt;span class="c"&gt;# 2. Apply&lt;/span&gt;
&lt;span class="nb"&gt;sudo &lt;/span&gt;dnf upgrade &lt;span class="nt"&gt;--security&lt;/span&gt; &lt;span class="nt"&gt;--releasever&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;latest &lt;span class="nt"&gt;-y&lt;/span&gt;

&lt;span class="c"&gt;# 3. Verify the default kernel didn't change&lt;/span&gt;
&lt;span class="nb"&gt;uname&lt;/span&gt; &lt;span class="nt"&gt;-r&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="nb"&gt;sudo &lt;/span&gt;grubby &lt;span class="nt"&gt;--default-kernel&lt;/span&gt;

&lt;span class="c"&gt;# 4. Swap the processes holding libxml2 (1–2 seconds)&lt;/span&gt;
&lt;span class="nb"&gt;sudo &lt;/span&gt;systemctl restart php-fpm httpd

&lt;span class="c"&gt;# 5. Check key pages return HTTP 200, then confirm the count is now truly zero&lt;/span&gt;
&lt;span class="nb"&gt;sudo &lt;/span&gt;dnf updateinfo list &lt;span class="nt"&gt;--security&lt;/span&gt; &lt;span class="nt"&gt;--releasever&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;latest
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;No reboot; the only service interruption was the 1–2 seconds of restarting &lt;code&gt;php-fpm&lt;/code&gt; / &lt;code&gt;httpd&lt;/code&gt;. The final check finally showed a &lt;em&gt;real&lt;/em&gt; zero.&lt;/p&gt;

&lt;h2&gt;
  
  
  Takeaways
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;AL2023 repositories are pinned to a release. "Zero" from a plain &lt;code&gt;dnf updateinfo&lt;/code&gt; means "zero &lt;strong&gt;within the pinned release&lt;/strong&gt;" — nothing more.&lt;/li&gt;
&lt;li&gt;Always pass &lt;code&gt;--releasever=latest&lt;/code&gt; for security checks and upgrades, and bake it into your scripts.&lt;/li&gt;
&lt;li&gt;When &lt;code&gt;needs-restarting&lt;/code&gt; demands a reboot for a kernel update, verify with &lt;code&gt;uname -r&lt;/code&gt; and &lt;code&gt;grubby --default-kernel&lt;/code&gt; whether the &lt;strong&gt;running kernel line&lt;/strong&gt; was actually the one updated.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A "zero" only becomes a fact once you know &lt;em&gt;zero relative to what&lt;/em&gt;. Errors stand out, so they get questioned; "nothing to report" never does. The quietest command is the most dangerous one — that was the lesson.&lt;/p&gt;

</description>
      <category>aws</category>
      <category>linux</category>
      <category>security</category>
      <category>devops</category>
    </item>
    <item>
      <title>Automating the AWS Console with an AI Agent: Everything Worked Except Four Save Buttons</title>
      <dc:creator>sunah kim</dc:creator>
      <pubDate>Thu, 05 Nov 2026 11:54:14 +0000</pubDate>
      <link>https://dev.to/pythona/automating-the-aws-console-with-an-ai-agent-everything-worked-except-four-save-buttons-2e2f</link>
      <guid>https://dev.to/pythona/automating-the-aws-console-with-an-ai-agent-everything-worked-except-four-save-buttons-2e2f</guid>
      <description>&lt;h2&gt;
  
  
  What I did
&lt;/h2&gt;

&lt;p&gt;As part of a server migration, I let an AI agent drive its built-in browser against the AWS Lightsail console and a payment provider's admin panel. The boundary between "safe to delegate" and "a human must click this" turned out to be much sharper than I expected, so here is a record of what worked, what got rejected, and why.&lt;/p&gt;

&lt;h2&gt;
  
  
  What worked
&lt;/h2&gt;

&lt;p&gt;Surprisingly many operations went through end to end:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Instance creation&lt;/strong&gt;: platform tab → blueprint → plan radio button → name field → create button, the whole flow&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Creating and attaching a static IP&lt;/strong&gt;&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Deletion confirmation dialogs&lt;/strong&gt;: even the two-step pattern of ticking a checkbox and typing "confirm" into a text box&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The payment provider's settings screen&lt;/strong&gt;: reading the current value of &lt;code&gt;&amp;lt;input name=url&amp;gt;&lt;/code&gt; and pre-filling the new one (a human pressed Save)&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The common thread: &lt;strong&gt;operating through the accessibility tree instead of coordinate clicks&lt;/strong&gt;. Rather than guessing "the button is around here" from a screenshot and clicking at coordinates, the agent runs &lt;code&gt;find&lt;/code&gt; against the accessibility tree, gets an element reference (ref), and issues the click against that ref. This sidesteps viewport scaling and zoom issues entirely.&lt;/p&gt;

&lt;h2&gt;
  
  
  Four places it got stuck
&lt;/h2&gt;

&lt;h3&gt;
  
  
  1. The firewall rule form — React controlled inputs
&lt;/h3&gt;

&lt;p&gt;Setting the port field's value programmatically made "443" appear on screen, but the form's internal state stayed empty, so saving failed validation. Switching to keyboard input instead appended to the existing value: "443443".&lt;/p&gt;

&lt;h3&gt;
  
  
  2. Cloudscape custom selects
&lt;/h3&gt;

&lt;p&gt;AWS's design system (Cloudscape) ships select boxes that look native but are custom components. Opening the dropdown and clicking an option did not update internal state, so the dependent "Attach" button stayed disabled.&lt;/p&gt;

&lt;h3&gt;
  
  
  3. DNS record editing — blocked by policy
&lt;/h3&gt;

&lt;p&gt;The agent's own automation policy classified DNS record changes as "high external impact" and refused the click outright. Not a technical limitation — a deliberate safety valve.&lt;/p&gt;

&lt;h3&gt;
  
  
  4. Session expiry
&lt;/h3&gt;

&lt;p&gt;The payment provider's admin panel logs you out after roughly ten minutes. The agent got logged out three times mid-task, and a human had to log back in each time.&lt;/p&gt;

&lt;h2&gt;
  
  
  The cause: what you see is not what the app knows
&lt;/h2&gt;

&lt;p&gt;Items 1 and 2 share a root cause: React controlled components. Writing to the DOM's &lt;code&gt;value&lt;/code&gt; property directly does not change React's state, so the screen and the application disagree.&lt;/p&gt;

&lt;p&gt;The known workaround is calling the native setter and then dispatching an &lt;code&gt;input&lt;/code&gt; event:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;input&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;document&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;querySelector&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;input[name="port"]&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;setter&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;Object&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;getOwnPropertyDescriptor&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
  &lt;span class="nb"&gt;window&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;HTMLInputElement&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;prototype&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;value&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;
&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="kd"&gt;set&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="nx"&gt;setter&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;call&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;input&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;443&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="nx"&gt;input&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;dispatchEvent&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;Event&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;input&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;bubbles&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt; &lt;span class="p"&gt;}));&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This gets some forms through, but components like Cloudscape's custom select have internal event plumbing complex enough that state still does not follow. There is a real gap between "automatable with effort" and "reliable enough to delegate".&lt;/p&gt;

&lt;h2&gt;
  
  
  The pattern that worked: reads are automated, saves are human
&lt;/h2&gt;

&lt;p&gt;After the trial and error, the smoothest division of labor was:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Reading, validating, and pre-filling values: automated. Save buttons and cutover buttons: human.&lt;/strong&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;The agent opens the settings screen, reads the current values, validates them, and types in the new ones. The human presses Save once. Mistakes go down, and the question "who approved this change?" has a clear answer. The more irreversible the operation — DNS cutover being the prime example — the more this split pays off.&lt;/p&gt;

&lt;h2&gt;
  
  
  Other gotchas
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Viewport emulation resets when the panel width changes, which silently breaks coordinate clicks. &lt;strong&gt;Use refs, not coordinates&lt;/strong&gt;
&lt;/li&gt;
&lt;li&gt;A cookie consent banner grabbed &lt;code&gt;role=dialog&lt;/code&gt; first, so the selector meant for the deletion confirmation dialog matched the banner instead&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Takeaways
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Most console automation works if you split it as "reads via JS, writes via ref clicks, final commit by a human"&lt;/li&gt;
&lt;li&gt;The two failure classes — &lt;strong&gt;React controlled inputs&lt;/strong&gt; (display diverges from state) and &lt;strong&gt;policy blocks&lt;/strong&gt; (safety valves by design) — should be assigned to the human from the start&lt;/li&gt;
&lt;li&gt;Coordinate clicks break with the environment; accessibility-tree refs don't&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;When someone asks "so it can do everything for you?", the honest answer was: "everything except four Save buttons."&lt;/p&gt;

</description>
      <category>aws</category>
      <category>automation</category>
      <category>ai</category>
      <category>react</category>
    </item>
  </channel>
</rss>
