<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: lastmileops</title>
    <description>The latest articles on DEV Community by lastmileops (@lastmileops).</description>
    <link>https://dev.to/lastmileops</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4202947%2F6848770a-17cb-4c82-b9af-22119b0a297b.png</url>
      <title>DEV Community: lastmileops</title>
      <link>https://dev.to/lastmileops</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/lastmileops"/>
    <language>en</language>
    <item>
      <title>Test your internal dashboard with two users before you cache it</title>
      <dc:creator>lastmileops</dc:creator>
      <pubDate>Sun, 08 Nov 2026 15:43:56 +0000</pubDate>
      <link>https://dev.to/lastmileops/test-your-internal-dashboard-with-two-users-before-you-cache-it-3cda</link>
      <guid>https://dev.to/lastmileops/test-your-internal-dashboard-with-two-users-before-you-cache-it-3cda</guid>
      <description>&lt;p&gt;A report is slow, so someone adds caching. The next person opens the dashboard and gets a result immediately. That feels like a successful optimization until they notice the first person's customer list.&lt;/p&gt;

&lt;p&gt;An older &lt;a href="https://discuss.streamlit.io/t/session-state-shared-between-different-users/44432" rel="noopener noreferrer"&gt;Streamlit forum thread&lt;/a&gt; contains reports of users seeing one another's data and a long discussion about session state and caching. It does not establish a single diagnosis for every app in that thread. It does suggest a useful test for any internal tool that started life with one user.&lt;/p&gt;

&lt;h2&gt;
  
  
  Name the audience of a cached result
&lt;/h2&gt;

&lt;p&gt;Two questions belong in the design review: who may read this result now, and which previous computation can supply it?&lt;/p&gt;

&lt;p&gt;The first is authorization. The second is cache identity. Neither can substitute for the other.&lt;/p&gt;

&lt;p&gt;Current &lt;a href="https://docs.streamlit.io/develop/concepts/architecture/caching" rel="noopener noreferrer"&gt;Streamlit caching documentation&lt;/a&gt; says cached values are shared across users by default, distinguishes copied data from shared resources, and offers session scope. Check the options supported by your installed version. Copying a cached object protects against some shared mutations; it does not decide whether its contents belong to the current reader.&lt;/p&gt;

&lt;p&gt;For a private report, consider the tenant, authenticated principal or effective visibility scope, requested filters, and data revision. If two users have different row-level permissions, a tenant ID alone is insufficient. A shared database connection is not proof that its query results have the same audience.&lt;/p&gt;

&lt;p&gt;Derive these values from trusted server-side context. A caller supplying a different user ID must not acquire that user's cached report.&lt;/p&gt;

&lt;h2&gt;
  
  
  Keep the access check on the path that runs
&lt;/h2&gt;

&lt;p&gt;A cache hit can skip the function body. An authorization check inside that body may therefore disappear precisely when a result is available quickly.&lt;/p&gt;

&lt;p&gt;This small standard-library example demonstrates the placement of the check. It is an executable teaching example with fictional users, not a Streamlit integration or a complete access-control system:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;functools&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;lru_cache&lt;/span&gt;

&lt;span class="n"&gt;grants&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;alice&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;team-a&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;bob&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;team-a&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)}&lt;/span&gt;
&lt;span class="n"&gt;rows&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;team-a&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;alice&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;A-001&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,),&lt;/span&gt;
    &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;team-a&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;bob&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;B-001&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,),&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="nd"&gt;@lru_cache&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;maxsize&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;128&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;cached_rows&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;tenant&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;principal&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;revision&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;[(&lt;/span&gt;&lt;span class="n"&gt;tenant&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;principal&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;revision&lt;/span&gt;&lt;span class="p"&gt;)]&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;report&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;principal&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;tenant&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;revision&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="c1"&gt;# Production: resolve identity and current grants on the server.
&lt;/span&gt;    &lt;span class="nf"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;principal&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;tenant&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;grants&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;raise&lt;/span&gt; &lt;span class="nc"&gt;PermissionError&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Access denied&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nf"&gt;cached_rows&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;tenant&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;principal&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;revision&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;expected&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;alice&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;A-001&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,),&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;bob&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;B-001&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,)}&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;order&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;alice&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;bob&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;bob&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;alice&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)):&lt;/span&gt;
    &lt;span class="n"&gt;cached_rows&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;cache_clear&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;principal&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;order&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;assert&lt;/span&gt; &lt;span class="nf"&gt;report&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;principal&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;team-a&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="n"&gt;expected&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;principal&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="n"&gt;grants&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;remove&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;alice&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;team-a&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
&lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="nf"&gt;report&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;alice&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;team-a&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;  &lt;span class="c1"&gt;# Alice's result is still cached.
&lt;/span&gt;&lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="nb"&gt;PermissionError&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;pass&lt;/span&gt;
&lt;span class="k"&gt;else&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;raise&lt;/span&gt; &lt;span class="nc"&gt;AssertionError&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;A warm cache bypassed revocation&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Both users belong to the same tenant and have different rows. Local checks on Python 3.14.4 passed this exact example in both warm-up orders. A deliberately broken cache that ignored the principal returned the first user's rows to the second and failed the assertions. The correct key passed again, including the warm-cache revocation check. These are local standard-library checks, not a live Streamlit test.&lt;/p&gt;

&lt;p&gt;Production policy can be more granular than this example. Partial row-level revocation is outside this demonstration. If a user retains access to the report while losing access to some rows, the cache identity needs the effective permission scope or a trustworthy policy revision as well. Check current authorization before returning the value, and arrange invalidation when the visibility rules change.&lt;/p&gt;

&lt;p&gt;Data freshness is a separate decision. A TTL limits age; it does not enforce access. A source revision should identify the data actually queried. Reading a revision and then querying a moving dataset needs a consistency rule too.&lt;/p&gt;

&lt;h2&gt;
  
  
  Use two independent sessions
&lt;/h2&gt;

&lt;p&gt;Start with two test accounts that have deliberately different visible records. Warm the cache as the first account, then request the equivalent report as the second. Reverse the order. Change a filter, change the dataset, and revoke access while the cached value remains warm.&lt;/p&gt;

&lt;p&gt;If your app supports logout and another login in the same browser, test that transition as well. Session-local storage still needs a lifecycle: remove or rebind private values when the authenticated identity changes. A fresh browser tab is a useful test setup, not a security boundary for a badly scoped server cache.&lt;/p&gt;

&lt;p&gt;Keep assertions about the returned records, not just the page heading or selected username. The dashboard can greet Bob perfectly while displaying Alice's rows.&lt;/p&gt;

&lt;p&gt;Streamlit documentation checked on 6 November 2026; version-specific caching options should be verified against the version you deploy.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;I use modern AI tools to help make my technical writing easier to read.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>python</category>
      <category>security</category>
      <category>webdev</category>
    </item>
    <item>
      <title>Give your helpdesk a diagnostic check, not SSH access</title>
      <dc:creator>lastmileops</dc:creator>
      <pubDate>Tue, 03 Nov 2026 14:45:29 +0000</pubDate>
      <link>https://dev.to/lastmileops/give-your-helpdesk-a-diagnostic-check-not-ssh-access-3n4n</link>
      <guid>https://dev.to/lastmileops/give-your-helpdesk-a-diagnostic-check-not-ssh-access-3n4n</guid>
      <description>&lt;p&gt;&lt;em&gt;A Linux example using Bash and jq; the code was checked locally.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;"Could you check whether the VPN hostname resolves?" It's a simple question with a ten-second answer. But if you have to&lt;br&gt;
track down the on-call network engineer first, those ten seconds can easily turn into a separate ticket and a wait.&lt;/p&gt;

&lt;p&gt;The helpdesk can answer questions like this on its own. What it needs is a check with clear boundaries.&lt;/p&gt;

&lt;p&gt;Shell access doesn't automatically make someone an administrator: that depends on the account's permissions. But even&lt;br&gt;
an unprivileged shell lets its user run commands, read accessible files and connect to accessible services. For a question&lt;br&gt;
like "does this host respond?", that's usually more access than the job needs.&lt;/p&gt;

&lt;p&gt;You can start with a small diagnostic script and three rules:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Only known targets.&lt;/strong&gt; The network team maintains the host list; neither the helpdesk nor the process running the check can change it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Only named checks.&lt;/strong&gt; The user chooses &lt;code&gt;dns&lt;/code&gt;, &lt;code&gt;port443&lt;/code&gt; or &lt;code&gt;ping&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Validate the input before running the check.&lt;/strong&gt; Return the result, including the reason for a refusal, in a clear format.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The allowed hostnames go in &lt;code&gt;/etc/diag/hosts.allow&lt;/code&gt;, one per line:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;vpn.internal
intranet.internal
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Here's the script:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;#!/bin/bash&lt;/span&gt;
&lt;span class="c"&gt;# diag.sh HOST CHECK — only allowed hosts and checks.&lt;/span&gt;
&lt;span class="nb"&gt;set&lt;/span&gt; &lt;span class="nt"&gt;-uo&lt;/span&gt; pipefail
&lt;span class="nb"&gt;export &lt;/span&gt;&lt;span class="nv"&gt;PATH&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;/usr/bin:/bin &lt;span class="nv"&gt;LC_ALL&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;C
&lt;span class="nb"&gt;readonly &lt;/span&gt;&lt;span class="nv"&gt;ALLOWLIST&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;/etc/diag/hosts.allow
&lt;span class="nv"&gt;host&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;1&lt;/span&gt;&lt;span class="k"&gt;:-}&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="nv"&gt;check&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;2&lt;/span&gt;&lt;span class="k"&gt;:-}&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;
emit&lt;span class="o"&gt;()&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
  jq &lt;span class="nt"&gt;-nc&lt;/span&gt; &lt;span class="nt"&gt;--arg&lt;/span&gt; host &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$host&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="nt"&gt;--arg&lt;/span&gt; check &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$check&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
    &lt;span class="nt"&gt;--arg&lt;/span&gt; result &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$1&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="nt"&gt;--arg&lt;/span&gt; reason &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;2&lt;/span&gt;&lt;span class="k"&gt;:-}&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
    &lt;span class="s1"&gt;'{host:$host, check:$check, result:$result}
     + (if $reason == "" then {} else {reason:$reason} end)'&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
fail&lt;span class="o"&gt;()&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt; emit refused &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$1&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="nb"&gt;exit &lt;/span&gt;2&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="o"&gt;}&lt;/span&gt;
&lt;span class="o"&gt;[[&lt;/span&gt; &lt;span class="nv"&gt;$# &lt;/span&gt;&lt;span class="nt"&gt;-eq&lt;/span&gt; 2 &lt;span class="o"&gt;]]&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; fail &lt;span class="s2"&gt;"expected HOST CHECK"&lt;/span&gt;
&lt;span class="o"&gt;[[&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$host&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt;~ ^[A-Za-z0-9][A-Za-z0-9.-]&lt;span class="o"&gt;{&lt;/span&gt;0,252&lt;span class="o"&gt;}&lt;/span&gt;&lt;span class="nv"&gt;$ &lt;/span&gt;&lt;span class="o"&gt;]]&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; fail &lt;span class="s2"&gt;"invalid host name"&lt;/span&gt;
&lt;span class="o"&gt;[[&lt;/span&gt; &lt;span class="nt"&gt;-r&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$ALLOWLIST&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="o"&gt;]]&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt; emit error &lt;span class="s2"&gt;"allowlist is unavailable"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="nb"&gt;exit &lt;/span&gt;1&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="o"&gt;}&lt;/span&gt;
&lt;span class="nb"&gt;grep&lt;/span&gt; &lt;span class="nt"&gt;-qxF&lt;/span&gt; &lt;span class="nt"&gt;--&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$host&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$ALLOWLIST&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; fail &lt;span class="s2"&gt;"host is not on the allowlist"&lt;/span&gt;
&lt;span class="k"&gt;case&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$check&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="k"&gt;in
  &lt;/span&gt;dns&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="nv"&gt;out&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;timeout &lt;/span&gt;5 getent hosts &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$host&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="si"&gt;)&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
      emit error &lt;span class="s2"&gt;"name lookup failed or timed out"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="nb"&gt;exit &lt;/span&gt;1&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;
    &lt;span class="nv"&gt;out&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;printf&lt;/span&gt; &lt;span class="s1"&gt;'%s\n'&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$out&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; | &lt;span class="nb"&gt;awk&lt;/span&gt; &lt;span class="s1"&gt;'{print $1}'&lt;/span&gt; | &lt;span class="nb"&gt;paste&lt;/span&gt; &lt;span class="nt"&gt;-sd&lt;/span&gt;, -&lt;span class="si"&gt;)&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
      emit error &lt;span class="s2"&gt;"name lookup output could not be formatted"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="nb"&gt;exit &lt;/span&gt;1&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;
    &lt;span class="p"&gt;;;&lt;/span&gt;
  port443&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="nv"&gt;out&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;timeout &lt;/span&gt;5 bash &lt;span class="nt"&gt;-c&lt;/span&gt; &lt;span class="s1"&gt;'exec 3&amp;lt;&amp;gt;"/dev/tcp/$1/443"'&lt;/span&gt; diag &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$host&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
      2&amp;gt;/dev/null &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="nb"&gt;echo &lt;/span&gt;open &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="nb"&gt;echo &lt;/span&gt;no_connection&lt;span class="si"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;;;&lt;/span&gt;
  ping&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="nv"&gt;out&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;timeout &lt;/span&gt;5 ping &lt;span class="nt"&gt;-c1&lt;/span&gt; &lt;span class="nt"&gt;-W2&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$host&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt;/dev/null 2&amp;gt;&amp;amp;1 &lt;span class="se"&gt;\&lt;/span&gt;
      &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="nb"&gt;echo &lt;/span&gt;reachable &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="nb"&gt;echo &lt;/span&gt;no_reply&lt;span class="si"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;;;&lt;/span&gt;
  &lt;span class="k"&gt;*&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; fail &lt;span class="s2"&gt;"unknown check (use dns, port443 or ping)"&lt;/span&gt; &lt;span class="p"&gt;;;&lt;/span&gt;
&lt;span class="k"&gt;esac&lt;/span&gt;
emit &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;out&lt;/span&gt;&lt;span class="k"&gt;:-&lt;/span&gt;&lt;span class="nv"&gt;none&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The allowlist path is fixed in the script: &lt;code&gt;DIAG_ALLOWLIST&lt;/code&gt; cannot be used to substitute another file. The script, the list&lt;br&gt;
and their parent directories must be protected against writes by both the helpdesk and the account that runs the check.&lt;br&gt;
A fixed path is no help if the caller can replace the file at that path.&lt;/p&gt;

&lt;p&gt;Here's what the helpdesk gets back:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;{"host":"vpn.internal","check":"dns","result":"10.0.4.12"}
{"host":"vpn.internal","check":"port443","result":"open"}
{"host":"vpn.internal;reboot","check":"dns","result":"refused","reason":"invalid host name"}
{"host":"db-prod","check":"dns","result":"refused","reason":"host is not on the allowlist"}
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;An attempt to append &lt;code&gt;;reboot&lt;/code&gt; stops at the character check. A name starting with &lt;code&gt;-&lt;/code&gt; is rejected too, and &lt;code&gt;--&lt;/code&gt; tells&lt;br&gt;
&lt;code&gt;grep&lt;/code&gt; where command options end and the name begins. The script refuses &lt;code&gt;db-prod&lt;/code&gt; because that host isn't on the list.&lt;/p&gt;

&lt;p&gt;Read the results literally. Here, &lt;code&gt;dns&lt;/code&gt; checks the system's name resolution, including &lt;code&gt;/etc/hosts&lt;/code&gt;.&lt;br&gt;
&lt;code&gt;open&lt;/code&gt; means a TCP connection to port 443 was established; it doesn't prove that TLS or the application itself is healthy.&lt;br&gt;
&lt;code&gt;no_connection&lt;/code&gt; and &lt;code&gt;no_reply&lt;/code&gt; mean no response was obtained under the conditions of the check. There can be several reasons for that.&lt;/p&gt;

&lt;p&gt;Now for the important part: how do you use this without SSH?&lt;/p&gt;

&lt;p&gt;An existing internal form or company bot needs just two fields: "Host" and "Check". The employee signs in through the&lt;br&gt;
company's usual mechanism, and the handler verifies that they are allowed to run diagnostics. Dropdowns make things&lt;br&gt;
convenient for the user; the server still needs to validate the arguments.&lt;/p&gt;

&lt;p&gt;The handler runs the script as a separate, unprivileged account on a machine that can reach the required hosts. That&lt;br&gt;
account cannot change the script, the target list or service settings. The employee gets no shell access to the machine.&lt;br&gt;
The command and its path are set on the server; the user supplies only the two values.&lt;/p&gt;

&lt;p&gt;For example, after checking access, the Python call looks like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;subprocess&lt;/span&gt;

&lt;span class="n"&gt;completed&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;subprocess&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;run&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;/opt/diag/diag.sh&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;host&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;check&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;
    &lt;span class="n"&gt;shell&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;False&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;stdin&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;subprocess&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;DEVNULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;capture_output&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;8&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;cwd&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;/&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;env&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;PATH&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;/usr/bin:/bin&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;LC_ALL&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;C&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The arguments are passed as a list, without building a shell command from the request text. The standard&lt;br&gt;
&lt;a href="https://docs.python.org/3/library/subprocess.html#security-considerations" rel="noopener noreferrer"&gt;&lt;code&gt;subprocess.run&lt;/code&gt;&lt;/a&gt; supports this directly.&lt;br&gt;
The server also supplies the environment: the user doesn't choose the program path, the host list or Bash settings.&lt;/p&gt;

&lt;p&gt;The handler returns the script's JSON to the employee and reports a launch error or timeout separately. The log records&lt;br&gt;
the user, target, check and result. The form or bot provides authorization and logging; the script itself doesn't implement them.&lt;/p&gt;

&lt;p&gt;That makes for a short path: choose &lt;code&gt;vpn.internal&lt;/code&gt; → click "Check DNS" → get an answer you can attach to a ticket.&lt;br&gt;
Restarting a service remains a separate action with separate permissions. It doesn't accidentally end up inside the diagnostics button.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Whenever I get the chance to make my geeky thoughts more pleasant for a wider audience, I don't shy away from what modern AI can do.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>bash</category>
      <category>sysadmin</category>
      <category>security</category>
      <category>devops</category>
    </item>
    <item>
      <title>Your certificate renewed. Does your service know?</title>
      <dc:creator>lastmileops</dc:creator>
      <pubDate>Tue, 27 Oct 2026 13:04:27 +0000</pubDate>
      <link>https://dev.to/lastmileops/your-certificate-renewed-does-your-service-know-emb</link>
      <guid>https://dev.to/lastmileops/your-certificate-renewed-does-your-service-know-emb</guid>
      <description>&lt;p&gt;&lt;em&gt;Tested with OpenSSL 3.6 and GNU date.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Automatic certificate renewal was invented to make expiry a non-event. Most of the time it is. But when something does break,&lt;br&gt;
it breaks quietly and with imagination:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;the DNS provider's API token expired, and domain validation no longer passes;&lt;/li&gt;
&lt;li&gt;a new firewall politely drops the HTTP challenge, and nobody knows;&lt;/li&gt;
&lt;li&gt;the deploy hook restarts a service that was renamed six months ago;&lt;/li&gt;
&lt;li&gt;renewal went perfectly, and the service keeps serving the old certificate from memory.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;And if you were counting on reminder emails from your CA: Let's Encrypt stopped sending them back in 2025. Nobody is going to&lt;br&gt;
remind you now except you.&lt;/p&gt;

&lt;p&gt;One simple idea covers that whole list: a check that &lt;strong&gt;doesn't trust&lt;/strong&gt; the renewal job at all. It connects to the service the way&lt;br&gt;
an ordinary client does and looks at the certificate that actually comes back. Not the file on disk, not the certbot log, but what&lt;br&gt;
your users' browsers see.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;#!/usr/bin/env bash&lt;/span&gt;
&lt;span class="c"&gt;# check-cert.sh HOST[:PORT] [MIN_DAYS] — check the certificate a service actually presents&lt;/span&gt;
&lt;span class="nb"&gt;set&lt;/span&gt; &lt;span class="nt"&gt;-uo&lt;/span&gt; pipefail
&lt;span class="nv"&gt;target&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$1&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="nv"&gt;min&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;2&lt;/span&gt;&lt;span class="k"&gt;:-&lt;/span&gt;&lt;span class="nv"&gt;14&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;
&lt;span class="nv"&gt;host&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;target&lt;/span&gt;&lt;span class="p"&gt;%%&lt;/span&gt;:&lt;span class="p"&gt;*&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="o"&gt;[[&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$target&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="k"&gt;*&lt;/span&gt;:&lt;span class="k"&gt;*&lt;/span&gt; &lt;span class="o"&gt;]]&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="nv"&gt;target&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$target&lt;/span&gt;&lt;span class="s2"&gt;:443"&lt;/span&gt;
&lt;span class="nv"&gt;pem&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt; | &lt;span class="nb"&gt;timeout &lt;/span&gt;10 openssl s_client &lt;span class="nt"&gt;-servername&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$host&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="nt"&gt;-connect&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$target&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; 2&amp;gt;/dev/null &lt;span class="se"&gt;\&lt;/span&gt;
      | openssl x509 2&amp;gt;/dev/null&lt;span class="si"&gt;)&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="nb"&gt;true
&lt;/span&gt;&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;[[&lt;/span&gt; &lt;span class="nt"&gt;-z&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$pem&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="o"&gt;]]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;then &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"UNREACHABLE: &lt;/span&gt;&lt;span class="nv"&gt;$target&lt;/span&gt;&lt;span class="s2"&gt; presented no certificate"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="nb"&gt;exit &lt;/span&gt;2&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;fi
&lt;/span&gt;&lt;span class="nv"&gt;end&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;openssl x509 &lt;span class="nt"&gt;-noout&lt;/span&gt; &lt;span class="nt"&gt;-enddate&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&amp;lt;&amp;lt;&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$pem&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; | &lt;span class="nb"&gt;cut&lt;/span&gt; &lt;span class="nt"&gt;-d&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nt"&gt;-f2&lt;/span&gt;&lt;span class="si"&gt;)&lt;/span&gt;
&lt;span class="nv"&gt;days&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;$((&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt; &lt;span class="si"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;date&lt;/span&gt; &lt;span class="nt"&gt;-u&lt;/span&gt; &lt;span class="nt"&gt;-d&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$end&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; +%s&lt;span class="si"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="si"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;date&lt;/span&gt; &lt;span class="nt"&gt;-u&lt;/span&gt; +%s&lt;span class="si"&gt;)&lt;/span&gt; &lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="m"&gt;86400&lt;/span&gt; &lt;span class="k"&gt;))&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;!&lt;/span&gt; openssl x509 &lt;span class="nt"&gt;-noout&lt;/span&gt; &lt;span class="nt"&gt;-checkhost&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$host&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&amp;lt;&amp;lt;&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$pem&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; | &lt;span class="nb"&gt;grep&lt;/span&gt; &lt;span class="nt"&gt;-q&lt;/span&gt; &lt;span class="s2"&gt;"does match"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;then
  &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"MISMATCH: &lt;/span&gt;&lt;span class="nv"&gt;$target&lt;/span&gt;&lt;span class="s2"&gt; presents a certificate not valid for &lt;/span&gt;&lt;span class="nv"&gt;$host&lt;/span&gt;&lt;span class="s2"&gt; (expires in &lt;/span&gt;&lt;span class="nv"&gt;$days&lt;/span&gt;&lt;span class="s2"&gt; days)"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="nb"&gt;exit &lt;/span&gt;2
&lt;span class="k"&gt;fi
if&lt;/span&gt; &lt;span class="o"&gt;((&lt;/span&gt; days &amp;lt; 0 &lt;span class="o"&gt;))&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;then &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"EXPIRED: &lt;/span&gt;&lt;span class="nv"&gt;$target&lt;/span&gt;&lt;span class="s2"&gt; expired on &lt;/span&gt;&lt;span class="nv"&gt;$end&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="nb"&gt;exit &lt;/span&gt;1&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;fi
if&lt;/span&gt; &lt;span class="o"&gt;((&lt;/span&gt; days &amp;lt; min &lt;span class="o"&gt;))&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;then &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"EXPIRING: &lt;/span&gt;&lt;span class="nv"&gt;$target&lt;/span&gt;&lt;span class="s2"&gt; expires in &lt;/span&gt;&lt;span class="nv"&gt;$days&lt;/span&gt;&lt;span class="s2"&gt; days (&lt;/span&gt;&lt;span class="nv"&gt;$end&lt;/span&gt;&lt;span class="s2"&gt;)"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="nb"&gt;exit &lt;/span&gt;1&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;fi
&lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"OK: &lt;/span&gt;&lt;span class="nv"&gt;$target&lt;/span&gt;&lt;span class="s2"&gt; valid for &lt;/span&gt;&lt;span class="nv"&gt;$days&lt;/span&gt;&lt;span class="s2"&gt; more days"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;What it says:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;OK: intranet.example.com:443 valid for 54 more days
EXPIRING: wiki.example.com:443 expires in 9 days (Nov  2 18:13:52 2026 GMT)
EXPIRED: old-vpn.example.com:443 expired on Feb  1 00:00:00 2025 GMT
MISMATCH: files.example.com:443 presents a certificate not valid for files.example.com (expires in 399 days)
UNREACHABLE: mail.example.com:465 presented no certificate
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Pick a threshold below the point where renewal should already have happened. From February 10, 2027, Let's Encrypt will issue&lt;br&gt;
64-day certificates by default, and it already recommends renewing at about two-thirds of the lifetime, which leaves roughly&lt;br&gt;
21 days. An alert at 14 days stays quiet on normal renewals and fires when one is a week late. A 21-day threshold would page&lt;br&gt;
someone on every routine renewal, and within a month everyone learns to ignore it. Which is exactly what we were trying to avoid.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;MISMATCH&lt;/code&gt; is the sneakiest one. A virtual host lost its own certificate and quietly serves the server's default one. Technically&lt;br&gt;
there is a certificate, and it's valid. Just not for this name. Your users' browsers will disagree with you.&lt;/p&gt;

&lt;p&gt;Run it for every name and port that matters, and &lt;strong&gt;from a different machine&lt;/strong&gt; than the one doing the renewals. Otherwise the&lt;br&gt;
check and the renewal break together and stay silent together.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;#!/usr/bin/env bash&lt;/span&gt;
&lt;span class="c"&gt;# check-all.sh — run check-cert.sh for every host:port in certs.txt, print only problems&lt;/span&gt;
&lt;span class="k"&gt;while &lt;/span&gt;&lt;span class="nb"&gt;read&lt;/span&gt; &lt;span class="nt"&gt;-r&lt;/span&gt; t _&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;do&lt;/span&gt;
  &lt;span class="o"&gt;[[&lt;/span&gt; &lt;span class="nt"&gt;-z&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$t&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$t&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="se"&gt;\#&lt;/span&gt;&lt;span class="k"&gt;*&lt;/span&gt; &lt;span class="o"&gt;]]&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="k"&gt;continue&lt;/span&gt;
  /opt/tools/check-cert.sh &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$t&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; 14 &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="nb"&gt;true
&lt;/span&gt;&lt;span class="k"&gt;done&lt;/span&gt; &amp;lt; /opt/tools/certs.txt | &lt;span class="nb"&gt;grep&lt;/span&gt; &lt;span class="nt"&gt;-v&lt;/span&gt; &lt;span class="s1"&gt;'^OK'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;0 7 &lt;span class="k"&gt;*&lt;/span&gt; &lt;span class="k"&gt;*&lt;/span&gt; &lt;span class="k"&gt;*&lt;/span&gt; &lt;span class="nv"&gt;out&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;/opt/tools/check-all.sh&lt;span class="si"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="o"&gt;[&lt;/span&gt; &lt;span class="nt"&gt;-z&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$out&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="o"&gt;]&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$out&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; | notify-oncall
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Send the result to a role or a shared queue, not to whoever set all this up once. The address that gets the alert should outlive&lt;br&gt;
the person who typed it in. Otherwise you end up like one thread on Spiceworks: renewal failed, and the notifications were going&lt;br&gt;
to the mailbox of a former contractor who was long gone.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What it doesn't do.&lt;/strong&gt; It doesn't validate the chain or revocation: if you need that, add &lt;code&gt;-verify_return_error&lt;/code&gt; and your own CA&lt;br&gt;
bundle to &lt;code&gt;s_client&lt;/code&gt;. Services that upgrade with STARTTLS (SMTP on 25 and 587, IMAP on 143) need &lt;code&gt;-starttls smtp&lt;/code&gt; or&lt;br&gt;
&lt;code&gt;-starttls imap&lt;/code&gt;. And it tells you &lt;em&gt;that&lt;/em&gt; something broke, not &lt;em&gt;why&lt;/em&gt;. For the why, keep a record of every renewal run: which&lt;br&gt;
version, when, and how it ended.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Whenever I get the chance to make my geeky thoughts more pleasant for a wider audience, I don't shy away from what modern AI can do.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>bash</category>
      <category>sysadmin</category>
      <category>security</category>
      <category>devops</category>
    </item>
    <item>
      <title>Run records and missed-run checks for scheduled scripts</title>
      <dc:creator>lastmileops</dc:creator>
      <pubDate>Fri, 23 Oct 2026 17:29:50 +0000</pubDate>
      <link>https://dev.to/lastmileops/run-records-and-missed-run-checks-for-scheduled-scripts-3cg</link>
      <guid>https://dev.to/lastmileops/run-records-and-missed-run-checks-for-scheduled-scripts-3cg</guid>
      <description>&lt;p&gt;&lt;em&gt;Tested on Linux with bash, jq and GNU date.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;A quiet Slack channel can mean two things: nothing went wrong, or the job never ran. Most small scheduled scripts can't&lt;br&gt;
tell those apart, because the only trace they leave is the alert they send when something looks bad.&lt;/p&gt;

&lt;p&gt;Two small additions fix that: write a record for every run, including the boring successful ones, and check separately&lt;br&gt;
that each job actually finished recently.&lt;/p&gt;
&lt;h3&gt;
  
  
  1. One line per run
&lt;/h3&gt;


&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;#!/usr/bin/env bash&lt;/span&gt;
&lt;span class="c"&gt;# run-logged.sh JOB_NAME COMMAND... — run a job and append one JSON line per run to $RUN_LOG&lt;/span&gt;
&lt;span class="nb"&gt;set&lt;/span&gt; &lt;span class="nt"&gt;-uo&lt;/span&gt; pipefail
&lt;span class="nv"&gt;job&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$1&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="nb"&gt;shift
&lt;/span&gt;&lt;span class="nv"&gt;log&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;RUN_LOG&lt;/span&gt;&lt;span class="k"&gt;:-&lt;/span&gt;&lt;span class="p"&gt;/var/log/run-records.jsonl&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;
&lt;span class="nv"&gt;rev&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;sha256sum&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;command&lt;/span&gt; &lt;span class="nt"&gt;-v&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$1&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="si"&gt;)&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; 2&amp;gt;/dev/null | &lt;span class="nb"&gt;cut&lt;/span&gt; &lt;span class="nt"&gt;-c1-12&lt;/span&gt;&lt;span class="si"&gt;)&lt;/span&gt;
&lt;span class="nv"&gt;start&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;date&lt;/span&gt; &lt;span class="nt"&gt;-u&lt;/span&gt; +%FT%TZ&lt;span class="si"&gt;)&lt;/span&gt;
&lt;span class="nv"&gt;out&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$@&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; 2&amp;gt;&amp;amp;1&lt;span class="si"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="nv"&gt;status&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;$?&lt;/span&gt;
&lt;span class="nv"&gt;end&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;date&lt;/span&gt; &lt;span class="nt"&gt;-u&lt;/span&gt; +%FT%TZ&lt;span class="si"&gt;)&lt;/span&gt;
jq &lt;span class="nt"&gt;-nc&lt;/span&gt; &lt;span class="nt"&gt;--arg&lt;/span&gt; job &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$job&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="nt"&gt;--arg&lt;/span&gt; rev &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;rev&lt;/span&gt;&lt;span class="k"&gt;:-&lt;/span&gt;&lt;span class="nv"&gt;unknown&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="nt"&gt;--arg&lt;/span&gt; start &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$start&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="nt"&gt;--arg&lt;/span&gt; end &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$end&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
      &lt;span class="nt"&gt;--argjson&lt;/span&gt; status &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$status&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="nt"&gt;--arg&lt;/span&gt; out &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;printf&lt;/span&gt; &lt;span class="s2"&gt;"%s"&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$out&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; | &lt;span class="nb"&gt;tail&lt;/span&gt; &lt;span class="nt"&gt;-c&lt;/span&gt; 2000&lt;span class="si"&gt;)&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
      &lt;span class="s1"&gt;'{job:$job, rev:$rev, start:$start, end:$end, status:$status, output:$out}'&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&amp;gt;&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$log&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;
&lt;span class="nb"&gt;exit&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$status&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;


&lt;p&gt;Use it in cron instead of calling the script directly:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;0 6 &lt;span class="k"&gt;*&lt;/span&gt; &lt;span class="k"&gt;*&lt;/span&gt; &lt;span class="k"&gt;*&lt;/span&gt; &lt;span class="nv"&gt;RUN_LOG&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;/var/log/run-records.jsonl /opt/tools/run-logged.sh nightly-report /opt/tools/nightly-report.sh
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Every run now leaves a line with the job name, a short hash of the program that ran, start and end time, exit status and&lt;br&gt;
the tail of its output — whether it found something or not. When someone asks "did it run on Tuesday, and which version?",&lt;br&gt;
the answer is one &lt;code&gt;jq&lt;/code&gt; away.&lt;/p&gt;
&lt;h3&gt;
  
  
  2. A separate check for missed runs
&lt;/h3&gt;


&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;#!/usr/bin/env bash&lt;/span&gt;
&lt;span class="c"&gt;# check-missed.sh JOB_NAME MAX_AGE_MINUTES — fail if the job has no finished, successful run recently&lt;/span&gt;
&lt;span class="nb"&gt;set&lt;/span&gt; &lt;span class="nt"&gt;-uo&lt;/span&gt; pipefail
&lt;span class="nv"&gt;job&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$1&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="nv"&gt;max&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$2&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="nv"&gt;log&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;RUN_LOG&lt;/span&gt;&lt;span class="k"&gt;:-&lt;/span&gt;&lt;span class="p"&gt;/var/log/run-records.jsonl&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;
&lt;span class="nv"&gt;last&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;jq &lt;span class="nt"&gt;-r&lt;/span&gt; &lt;span class="nt"&gt;--arg&lt;/span&gt; job &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$job&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="s1"&gt;'select(.job==$job and .status==0) | .end'&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$log&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; 2&amp;gt;/dev/null | &lt;span class="nb"&gt;tail&lt;/span&gt; &lt;span class="nt"&gt;-1&lt;/span&gt;&lt;span class="si"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;[[&lt;/span&gt; &lt;span class="nt"&gt;-z&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$last&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="o"&gt;]]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;then &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"MISSED: &lt;/span&gt;&lt;span class="nv"&gt;$job&lt;/span&gt;&lt;span class="s2"&gt; has no successful run on record"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="nb"&gt;exit &lt;/span&gt;2&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;fi
&lt;/span&gt;&lt;span class="nv"&gt;age&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;$((&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt; &lt;span class="si"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;date&lt;/span&gt; &lt;span class="nt"&gt;-u&lt;/span&gt; +%s&lt;span class="si"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="si"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;date&lt;/span&gt; &lt;span class="nt"&gt;-u&lt;/span&gt; &lt;span class="nt"&gt;-d&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$last&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; +%s&lt;span class="si"&gt;)&lt;/span&gt; &lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="m"&gt;60&lt;/span&gt; &lt;span class="k"&gt;))&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;((&lt;/span&gt; age &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; max &lt;span class="o"&gt;))&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;then &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"MISSED: &lt;/span&gt;&lt;span class="nv"&gt;$job&lt;/span&gt;&lt;span class="s2"&gt; last succeeded &lt;/span&gt;&lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;age&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt;&lt;span class="s2"&gt; min ago (limit &lt;/span&gt;&lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;max&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;)"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="nb"&gt;exit &lt;/span&gt;2&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;fi
&lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"OK: &lt;/span&gt;&lt;span class="nv"&gt;$job&lt;/span&gt;&lt;span class="s2"&gt; succeeded &lt;/span&gt;&lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;age&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt;&lt;span class="s2"&gt; min ago"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;


&lt;p&gt;Schedule it after the job's deadline and send its output wherever alerts go:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;30 7 &lt;span class="k"&gt;*&lt;/span&gt; &lt;span class="k"&gt;*&lt;/span&gt; &lt;span class="k"&gt;*&lt;/span&gt; &lt;span class="nv"&gt;RUN_LOG&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;/var/log/run-records.jsonl /opt/tools/check-missed.sh nightly-report 120 &lt;span class="o"&gt;||&lt;/span&gt; notify-oncall
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It complains in three cases: the job never succeeded, it last succeeded too long ago, or it keeps failing. A failed data&lt;br&gt;
fetch shows up as a failed check instead of a clean report.&lt;/p&gt;

&lt;h3&gt;
  
  
  What this doesn't do
&lt;/h3&gt;

&lt;p&gt;It doesn't make the script correct, and it doesn't tell you &lt;em&gt;why&lt;/em&gt; a run was skipped — only that it was. It's also still a&lt;br&gt;
file on one machine: if several people depend on the job, they need somewhere to see these records that isn't your shell.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;This article was created with the help of AI. The code was run on Linux before publishing.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>bash</category>
      <category>devops</category>
      <category>linux</category>
      <category>monitoring</category>
    </item>
  </channel>
</rss>
