<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: upordown.online</title>
    <description>The latest articles on DEV Community by upordown.online (@upordownonline).</description>
    <link>https://dev.to/upordownonline</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4207346%2F6128a48b-a24a-466f-a304-ab19b1655a1e.png</url>
      <title>DEV Community: upordown.online</title>
      <link>https://dev.to/upordownonline</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/upordownonline"/>
    <language>en</language>
    <item>
      <title>Case study (hypothetical): checkout failed only for cards that needed 3-D Secure, and every monitor stayed green</title>
      <dc:creator>upordown.online</dc:creator>
      <pubDate>Sun, 15 Nov 2026 14:01:07 +0000</pubDate>
      <link>https://dev.to/upordownonline/case-study-hypothetical-checkout-failed-only-for-cards-that-needed-3-d-secure-and-every-monitor-4joi</link>
      <guid>https://dev.to/upordownonline/case-study-hypothetical-checkout-failed-only-for-cards-that-needed-3-d-secure-and-every-monitor-4joi</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;Uptime monitoring from outside your own infrastructure that alerts you the minute your site goes down — &lt;a href="https://upordown.online" rel="noopener noreferrer"&gt;upordown.online&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;This is a hypothetical case study. The shop, the numbers and the timeline are invented, but every piece of it is something that happens to real checkouts.&lt;/p&gt;

&lt;h2&gt;
  
  
  The setup
&lt;/h2&gt;

&lt;p&gt;A mid-sized online shop sells outdoor gear. Its monitoring was better than most:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;An uptime check on the homepage every minute.&lt;/li&gt;
&lt;li&gt;A check on the &lt;code&gt;/checkout&lt;/code&gt; page that looked for the text "Pay now".&lt;/li&gt;
&lt;li&gt;A synthetic test every 15 minutes that added a product to the cart and completed payment with the payment provider's test card in the staging environment.&lt;/li&gt;
&lt;li&gt;An alert if the order rate dropped below 30% of the same hour last week.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;On a Tuesday afternoon, a developer shipped a change to the checkout page: the payment form moved into a modal, to make the page feel faster on mobile. Everything passed review. Every check stayed green.&lt;/p&gt;

&lt;h2&gt;
  
  
  What customers saw
&lt;/h2&gt;

&lt;p&gt;When a customer's bank required Strong Customer Authentication (in Europe, most online card payments above a small amount; elsewhere, whenever the bank's fraud model asks for it), the payment provider needs to show a 3-D Secure challenge: a bank page, usually in an iframe or redirect, where the customer confirms the payment in their banking app.&lt;/p&gt;

&lt;p&gt;The new modal closed itself when it lost focus. On mobile, switching to the banking app to approve the payment counted as losing focus. The customer approved in the app, came back to the browser, and found the checkout page with the modal closed and no confirmation. The payment intent sat in &lt;code&gt;requires_action&lt;/code&gt; until it expired. Some customers tried again; most left.&lt;/p&gt;

&lt;p&gt;Customers whose cards didn't trigger a challenge paid normally. That was most of them outside Europe, and a minority inside it.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;This is why an outside check of the real checkout path matters.&lt;/strong&gt; &lt;a href="https://upordown.online" rel="noopener noreferrer"&gt;upordown.online&lt;/a&gt; monitors the URL that takes payment, not just a health endpoint that stays green.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  Why every check missed it
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;The uptime checks saw a page that loaded.&lt;/strong&gt; The homepage and the checkout page both returned 200 with the expected text. Nothing was down.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The synthetic test used a test card that never asks for 3-D Secure.&lt;/strong&gt; Payment providers publish separate test cards for "succeeds", "requires authentication" and "declined". The test used the first one, so it tested exactly the path that still worked.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The order-rate alert had a threshold built for outages.&lt;/strong&gt; Orders dropped about 22% over the following days. That's alarming once you see it, but it was well above the "30% of last week" floor, which was designed to catch a full checkout outage. A weekday sale the week before also made the comparison noisier.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Nobody owned the &lt;code&gt;requires_action&lt;/code&gt; count.&lt;/strong&gt; The payment provider's dashboard showed a growing pile of payment intents stuck in &lt;code&gt;requires_action&lt;/code&gt;. Nobody looked at that dashboard unless something was already wrong.&lt;/p&gt;

&lt;p&gt;The problem was found on Saturday, when a customer emailed a screenshot from their banking app showing a payment they'd approved for an order that didn't exist.&lt;/p&gt;

&lt;h2&gt;
  
  
  The fix
&lt;/h2&gt;

&lt;p&gt;The immediate fix was small: the modal no longer closes on blur, and the checkout page, on load, checks for any payment intent in the session that's waiting for action and resumes it. The provider's client library already supported resuming; it just needed to be called.&lt;/p&gt;

&lt;p&gt;The monitoring fixes mattered more:&lt;/p&gt;

&lt;h3&gt;
  
  
  1. Test every card outcome, not just success
&lt;/h3&gt;

&lt;p&gt;The synthetic checkout now runs three times per cycle: with the success card, with the "authentication required" card (it completes the test challenge), and with a decline card (it checks the error message is shown). A checkout that only tests the happy path only protects the happy path.&lt;/p&gt;

&lt;h3&gt;
  
  
  2. Alert on payment states, not just order counts
&lt;/h3&gt;

&lt;p&gt;The shop added a scheduled query against the provider's API: how many payment intents are in &lt;code&gt;requires_action&lt;/code&gt; or &lt;code&gt;requires_payment_method&lt;/code&gt; for longer than 30 minutes, compared with the same window last week. A ratio that doubles is a page. This metric would have fired within two hours on Tuesday.&lt;/p&gt;

&lt;h3&gt;
  
  
  3. Split the order-rate alert by segment
&lt;/h3&gt;

&lt;p&gt;Overall order rate hid a collapse in one segment. Splitting by card country and payment method showed it immediately: European card payments were down by more than half while everything else was normal. A drop in a single segment is a much stronger signal than a modest drop overall.&lt;/p&gt;

&lt;h3&gt;
  
  
  4. Test the flow on a real phone
&lt;/h3&gt;

&lt;p&gt;Desktop browsers don't lose focus when you open a banking app. The team added a manual pre-release check on one iOS and one Android device for any change to the checkout UI. It's ten minutes per release and it's the only test that reproduces the actual bug.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I'd take from it
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;"Up" is not the same as "working".&lt;/strong&gt; A 200 with the right text proves the page renders. It proves nothing about whether a payment completes.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Your test data defines what you're monitoring.&lt;/strong&gt; If the synthetic test uses one card, you're monitoring one card.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Look at stuck states.&lt;/strong&gt; Anything that can be "pending", "requires action" or "processing" should have an age alert.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Segment before you average.&lt;/strong&gt; An average across all customers is where partial outages go to hide.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;External checks like the ones &lt;a href="https://upordown.online" rel="noopener noreferrer"&gt;upordown.online&lt;/a&gt; runs are the first layer here: they catch the checkout page going down or losing its "Pay now" button. The second layer, the one this shop was missing, is checking the outcomes behind the page.&lt;/p&gt;

&lt;h2&gt;
  
  
  Related reading
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://dev.to/upordownonline/should-a-failed-health-check-restart-the-service-automatically-or-page-a-human-first-lets-argue-53pl"&gt;Should a failed health check restart the service automatically, or page a human first? Let's argue about it&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/upordownonline/stop-monitoring-your-homepage-monitor-the-path-that-makes-you-money-104o"&gt;Stop monitoring your homepage. Monitor the path that makes you money.&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/upordownonline/our-monitor-said-100-uptime-while-the-cdn-served-a-6-hour-old-homepage-bk9"&gt;Our monitor said 100% uptime while the CDN served a 6-hour-old homepage&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/upordownonline/case-study-hypothetical-the-site-that-was-down-only-for-people-on-ipv6-and-every-monitor-said-14i"&gt;Case study (hypothetical): the site that was down only for people on IPv6, and every monitor said it was fine&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Has a checkout of yours ever failed for only one kind of customer? How long did it take to notice, and what finally gave it away?&lt;/p&gt;

</description>
      <category>webdev</category>
      <category>backend</category>
      <category>testing</category>
      <category>monitoring</category>
    </item>
    <item>
      <title>Should a failed health check restart the service automatically, or page a human first? Let's argue about it</title>
      <dc:creator>upordown.online</dc:creator>
      <pubDate>Sun, 15 Nov 2026 14:00:38 +0000</pubDate>
      <link>https://dev.to/upordownonline/should-a-failed-health-check-restart-the-service-automatically-or-page-a-human-first-lets-argue-53pl</link>
      <guid>https://dev.to/upordownonline/should-a-failed-health-check-restart-the-service-automatically-or-page-a-human-first-lets-argue-53pl</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;Uptime monitoring from outside your own infrastructure that alerts you the minute your site goes down — &lt;a href="https://upordown.online" rel="noopener noreferrer"&gt;upordown.online&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Your health check fails at 03:10. Something is going to happen next. Either the platform restarts the service on its own (Kubernetes liveness probes, systemd &lt;code&gt;Restart=on-failure&lt;/code&gt;, an autoscaling group replacing the instance, a supervisor process), or a person gets paged and decides what to do.&lt;/p&gt;

&lt;p&gt;Most teams never make this choice deliberately. They inherit whatever the platform's defaults are, and find out what they chose during an incident. So let's make the argument out loud.&lt;/p&gt;

&lt;h2&gt;
  
  
  The case for restarting automatically
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Most failures are transient, and a restart fixes them.&lt;/strong&gt; A memory leak that takes six days to exhaust the heap, a connection pool stuck after a database failover, a deadlock that happens once a week. A restart clears all of these in seconds. A human would do the same thing, just 15 minutes later and grumpier.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Humans are slow at 3 a.m.&lt;/strong&gt; Time to acknowledge a page, open a laptop, connect to the VPN and look at a dashboard is rarely under ten minutes. If a restart takes 20 seconds, the automated path has 9.5 minutes of advantage on every incident.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;It scales.&lt;/strong&gt; With 40 services and 200 instances, paging a human for every unhealthy instance means paging constantly. Self-healing at the instance level is the only way to run a fleet without a 24-hour operations room.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;You can still alert on the pattern.&lt;/strong&gt; Restart automatically, and page when restarts happen more than N times per hour. The human gets involved for the recurring problem, not for every blip.&lt;/p&gt;

&lt;h2&gt;
  
  
  The case for paging first
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;A restart destroys evidence.&lt;/strong&gt; The heap state, the stuck threads, the open connections, the exact request that hung the worker: all gone when the process restarts. If you never look at a failing instance before it's recycled, you'll never find out why it failed. You'll restart it every Tuesday for two years.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Restarts can make things worse.&lt;/strong&gt; If the health check fails because a dependency is down, restarting every instance at once means they all come back cold, all reconnect to the struggling dependency at the same moment, and all fail again. That's a restart loop, and it turns a degraded service into a dead one. Health checks that include dependencies are particularly prone to this.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Auto-restart hides slow degradation.&lt;/strong&gt; A service that's restarted 30 times a day by the platform looks green on every dashboard that shows current state. Nobody notices until the restarts happen faster than the startup time and the whole thing falls over.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Some failures need judgement.&lt;/strong&gt; Restarting a stateless web server is safe. Restarting a database primary, a job runner in the middle of a migration, or a process holding a lock can cause data problems that are far worse than the original failure.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Need a second opinion before the restart fires?&lt;/strong&gt; &lt;a href="https://upordown.online" rel="noopener noreferrer"&gt;upordown.online&lt;/a&gt; checks the same endpoint from outside your network, so you can tell a local flake from a real outage.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  Where I've landed
&lt;/h2&gt;

&lt;p&gt;I don't think there's one answer for every service, but there's a reasonable default:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Auto-restart stateless things, with a budget.&lt;/strong&gt; Web servers, API workers and other replaceable processes restart automatically, but with backoff (Kubernetes' CrashLoopBackOff does this; systemd has &lt;code&gt;StartLimitBurst&lt;/code&gt; and &lt;code&gt;RestartSec&lt;/code&gt;) so a restart loop doesn't hammer everything else.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Never restart everything at once.&lt;/strong&gt; Whatever does the restarting should restart one instance at a time, or a small percentage. If all instances fail health checks simultaneously, that's almost never a per-instance problem, and restarting them all is the worst possible response.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Capture evidence before the restart.&lt;/strong&gt; A pre-stop hook that dumps a thread dump, heap summary or the last 500 log lines to somewhere durable costs a few seconds and turns "it was restarted, no idea why" into a real investigation.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Page on rates, not events.&lt;/strong&gt; One restart is a log line. Five restarts of the same service in an hour is a page. Restarts across many instances at once is a page immediately.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Page first for stateful things.&lt;/strong&gt; Databases, queues and anything holding a lock or partway through a long operation get a human before an automated action.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Watch from outside too.&lt;/strong&gt; Self-healing can make internal dashboards look fine while users see errors during each restart. An external check like &lt;a href="https://upordown.online" rel="noopener noreferrer"&gt;upordown.online&lt;/a&gt; shows whether the restarts are actually invisible to users or just invisible to you.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The part I'm least sure about is the threshold. Five restarts an hour? Three? Is one restart of a service that "never restarts" already worth a page?&lt;/p&gt;

&lt;h2&gt;
  
  
  The edge cases that start arguments
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Liveness probes that check dependencies.&lt;/strong&gt; If the database is slow and every pod's liveness probe fails, Kubernetes restarts every pod. Should liveness ever depend on anything outside the process? (I'd say no, but I've heard good arguments for "only the local disk".)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Restarting during a deploy.&lt;/strong&gt; If the new version fails health checks, should the platform restart it, roll it back, or stop and wait for someone?&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Small teams.&lt;/strong&gt; For a team of two with one service, is paging on every failure fine because it happens twice a year?&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  What the default looks like in config
&lt;/h2&gt;

&lt;p&gt;For a systemd service, "restart with a budget" is a few lines:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ini"&gt;&lt;code&gt;&lt;span class="k"&gt;[Unit]&lt;/span&gt;
&lt;span class="na"&gt;StartLimitIntervalSec&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;600&lt;/span&gt;
&lt;span class="na"&gt;StartLimitBurst&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;

&lt;span class="k"&gt;[Service]&lt;/span&gt;
&lt;span class="na"&gt;Restart&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;failure&lt;/span&gt;
&lt;span class="na"&gt;RestartSec&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Five restarts in ten minutes, then systemd gives up and the unit stays failed, which is exactly the moment a human should hear about it. In Kubernetes, keep the liveness probe about the process only (an endpoint that returns 200 if the event loop is responsive), put dependency checks in the readiness probe, and alert on &lt;code&gt;kube_pod_container_status_restarts_total&lt;/code&gt; increasing faster than your threshold.&lt;/p&gt;

&lt;h2&gt;
  
  
  Related reading
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://dev.to/upordownonline/postmortem-hypothetical-our-health-check-was-so-thorough-it-took-every-instance-out-of-the-load-158m"&gt;Postmortem (hypothetical): our health check was so thorough it took every instance out of the load balancer at once&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/upordownonline/alert-after-1-failed-check-or-3-tuning-uptime-alerts-so-on-call-stops-ignoring-them-1od0"&gt;Alert after 1 failed check or 3? Tuning uptime alerts so on-call stops ignoring them&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/upordownonline/your-health-check-returns-200-while-users-get-errors-7-health-check-mistakes-d84"&gt;Your health check returns 200 while users get errors: 7 health check mistakes&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/upordownonline/case-study-hypothetical-checkout-failed-only-for-cards-that-needed-3-d-secure-and-every-monitor-4joi"&gt;Case study (hypothetical): checkout failed only for cards that needed 3-D Secure, and every monitor stayed green&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;So which camp are you in: restart first and investigate the pattern, or page first and keep the evidence? And what's your restart threshold before a human gets woken up?&lt;/p&gt;

</description>
      <category>devops</category>
      <category>sre</category>
      <category>kubernetes</category>
      <category>discuss</category>
    </item>
    <item>
      <title>Adding rate limiting to your API? A 12-point checklist so it doesn't block your own users (or your monitor)</title>
      <dc:creator>upordown.online</dc:creator>
      <pubDate>Sat, 14 Nov 2026 08:04:02 +0000</pubDate>
      <link>https://dev.to/upordownonline/adding-rate-limiting-to-your-api-a-12-point-checklist-so-it-doesnt-block-your-own-users-or-your-f35</link>
      <guid>https://dev.to/upordownonline/adding-rate-limiting-to-your-api-a-12-point-checklist-so-it-doesnt-block-your-own-users-or-your-f35</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;Uptime monitoring from outside your own infrastructure that alerts you the minute your site goes down — &lt;a href="https://upordown.online" rel="noopener noreferrer"&gt;upordown.online&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Rate limiting looks like a ten-minute job. Pick a number, put it in front of the API, done. Then on Monday morning a customer with 300 employees behind one office NAT gets blocked as a single "abusive client", your own mobile app retries every 429 immediately and makes things worse, and your uptime monitor starts reporting an outage that's really your own limiter.&lt;/p&gt;

&lt;p&gt;None of that is a reason not to rate limit. It's a reason to treat the limiter like any other piece of production infrastructure. Here's the checklist I'd go through before turning one on.&lt;/p&gt;

&lt;h2&gt;
  
  
  Before you pick a number
&lt;/h2&gt;

&lt;h3&gt;
  
  
  1. Measure current traffic per client first
&lt;/h3&gt;

&lt;p&gt;Don't guess the limit. Pull a week of access logs and compute requests per minute per key (API key, user ID or IP) at the 99th percentile and the maximum. A limit below your heaviest legitimate client's normal peak is an outage you've scheduled for yourself.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# requests per minute per IP, busiest first (combined log format)&lt;/span&gt;
awk &lt;span class="s1"&gt;'{print $1, substr($4,2,17)}'&lt;/span&gt; access.log &lt;span class="o"&gt;|&lt;/span&gt; sort &lt;span class="o"&gt;|&lt;/span&gt; uniq -c &lt;span class="o"&gt;|&lt;/span&gt; sort -rn &lt;span class="o"&gt;|&lt;/span&gt; head
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  2. Decide what you're limiting by
&lt;/h3&gt;

&lt;p&gt;IP address is the easiest key and the worst one for business customers. Offices, universities, mobile carriers using carrier-grade NAT and corporate VPNs put thousands of real people behind a handful of addresses. Where you can, limit authenticated traffic by API key or user ID and keep IP-based limits for unauthenticated endpoints such as login and signup.&lt;/p&gt;

&lt;h3&gt;
  
  
  3. Make sure you see the real client IP
&lt;/h3&gt;

&lt;p&gt;If you're behind a load balancer or CDN, every request may appear to come from the same few proxy addresses. Rate limiting on those blocks everyone at once. Configure your server to trust &lt;code&gt;X-Forwarded-For&lt;/code&gt; (or the CDN's own header) only from the proxy's known address ranges, and verify the result in your logs before going live.&lt;/p&gt;

&lt;h3&gt;
  
  
  4. Pick limits per endpoint, not one global number
&lt;/h3&gt;

&lt;p&gt;A search endpoint, a login endpoint and a static config endpoint have nothing in common. Login wants a tight limit (credential stuffing). Search might need a moderate one (expensive queries). A cached read endpoint barely needs one. One global number is either too loose for login or too tight for everything else.&lt;/p&gt;

&lt;h2&gt;
  
  
  How the limiter behaves
&lt;/h2&gt;

&lt;h3&gt;
  
  
  5. Return 429 with a Retry-After header
&lt;/h3&gt;

&lt;p&gt;A 429 Too Many Requests with &lt;code&gt;Retry-After: 30&lt;/code&gt; tells well-behaved clients exactly what to do. Adding the remaining-quota headers (&lt;code&gt;RateLimit-Limit&lt;/code&gt;, &lt;code&gt;RateLimit-Remaining&lt;/code&gt;, &lt;code&gt;RateLimit-Reset&lt;/code&gt;, or the older &lt;code&gt;X-RateLimit-*&lt;/code&gt; names) lets clients slow down before they hit the wall. Don't return 403 or 503 for rate limits: clients and monitors treat those as different problems.&lt;/p&gt;

&lt;h3&gt;
  
  
  6. Allow bursts
&lt;/h3&gt;

&lt;p&gt;Real clients are bursty. A page load can fire 15 API calls in 200 milliseconds and then nothing for a minute. A token bucket or a sliding window with a burst allowance handles that; a strict "max 10 per second" fixed window punishes normal page loads. In Nginx that's the difference between &lt;code&gt;limit_req zone=api;&lt;/code&gt; and &lt;code&gt;limit_req zone=api burst=20 nodelay;&lt;/code&gt;.&lt;/p&gt;

&lt;h3&gt;
  
  
  7. Decide what happens when the limiter's store is down
&lt;/h3&gt;

&lt;p&gt;If your limiter counts requests in Redis and Redis becomes unreachable, does every request fail, or does the limiter let traffic through? For most APIs, failing open (allow traffic, log loudly) is better than turning a cache outage into a full API outage. Whatever you choose, choose it on purpose and test it.&lt;/p&gt;

&lt;h3&gt;
  
  
  8. Exempt your own infrastructure deliberately
&lt;/h3&gt;

&lt;p&gt;Health checks from your load balancer, your uptime monitor, internal services and webhooks from partners shouldn't compete with customer traffic for the same quota. Exempt them explicitly, by authenticated identity rather than by "anything from this IP range forever", and write the exemptions down where the next person will find them.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Want to know if the limiter is blocking real users?&lt;/strong&gt; &lt;a href="https://upordown.online" rel="noopener noreferrer"&gt;upordown.online&lt;/a&gt; checks the same paths your customers hit, so a 429 on a critical endpoint shows up as an alert, not a support ticket.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  Rolling it out
&lt;/h2&gt;

&lt;h3&gt;
  
  
  9. Start in log-only mode
&lt;/h3&gt;

&lt;p&gt;Run the limiter for a few days in a mode that records what it would have blocked without blocking anything. Most limiters support this (&lt;code&gt;limit_req_dry_run on;&lt;/code&gt; in Nginx 1.17.1+, "count" or "log" actions in most WAFs). Look at who would have been limited. You'll find at least one legitimate integration you didn't know about.&lt;/p&gt;

&lt;h3&gt;
  
  
  10. Check your own clients' retry logic
&lt;/h3&gt;

&lt;p&gt;Your web app, mobile app and SDKs need to respect &lt;code&gt;Retry-After&lt;/code&gt; and back off with jitter. A client that retries a 429 immediately, in a loop, turns a brief limit into sustained overload. Fix the clients before the limiter ships, especially mobile apps, since old versions stay installed for months.&lt;/p&gt;

&lt;h3&gt;
  
  
  11. Monitor 429s as their own metric
&lt;/h3&gt;

&lt;p&gt;Track the 429 rate per endpoint and per client, and alert on sudden changes. A spike in 429s means either an attack (the limiter working) or a bad limit (the limiter causing an outage). You want to know which within minutes, not when support tickets arrive. Make sure your uptime checks treat 429 distinctly too: a check that gets throttled should tell you "throttled", not "down".&lt;/p&gt;

&lt;h3&gt;
  
  
  12. Document the limits publicly and give people a way to ask for more
&lt;/h3&gt;

&lt;p&gt;Publish the limits in your API docs, include the headers in the examples, and give heavy users a way to request a higher quota. A customer who knows the limit designs around it. A customer who discovers it in production files an angry ticket.&lt;/p&gt;

&lt;h2&gt;
  
  
  The one-sentence version
&lt;/h2&gt;

&lt;p&gt;Measure first, key on identity rather than IP, return honest 429s with Retry-After, roll out in log-only mode, and watch your 429 rate like any other error rate.&lt;/p&gt;

&lt;p&gt;If you're watching your API from the outside, a monitor such as &lt;a href="https://upordown.online" rel="noopener noreferrer"&gt;upordown.online&lt;/a&gt; will show you quickly whether a limit change made the API slower or started turning legitimate checks away.&lt;/p&gt;

&lt;h2&gt;
  
  
  Related reading
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://dev.to/upordownonline/how-to-fix-502-bad-gateway-in-nginx-find-the-error-log-line-that-tells-you-why-1a7h"&gt;How to fix 502 Bad Gateway in Nginx: find the error log line that tells you why&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/upordownonline/rotating-an-api-key-or-database-password-a-12-point-checklist-so-the-rotation-isnt-the-outage-3aep"&gt;Rotating an API key or database password? A 12-point checklist so the rotation isn't the outage&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/upordownonline/putting-a-cdn-in-front-of-your-site-a-12-point-checklist-so-the-cdn-isnt-your-next-outage-4881"&gt;Putting a CDN in front of your site? A 12-point checklist so the CDN isn't your next outage&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/upordownonline/your-health-check-returns-200-while-users-get-errors-7-health-check-mistakes-d84"&gt;Your health check returns 200 while users get errors: 7 health check mistakes&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;What's the worst thing a rate limiter has blocked for you? Mine was a customer's entire head office, on a Monday, behind one IP address.&lt;/p&gt;

</description>
      <category>api</category>
      <category>devops</category>
      <category>backend</category>
      <category>security</category>
    </item>
    <item>
      <title>How to fix 502 Bad Gateway in Nginx: find the error log line that tells you why</title>
      <dc:creator>upordown.online</dc:creator>
      <pubDate>Sat, 14 Nov 2026 08:03:32 +0000</pubDate>
      <link>https://dev.to/upordownonline/how-to-fix-502-bad-gateway-in-nginx-find-the-error-log-line-that-tells-you-why-1a7h</link>
      <guid>https://dev.to/upordownonline/how-to-fix-502-bad-gateway-in-nginx-find-the-error-log-line-that-tells-you-why-1a7h</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;Uptime monitoring from outside your own infrastructure that alerts you the minute your site goes down — &lt;a href="https://upordown.online" rel="noopener noreferrer"&gt;upordown.online&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;A 502 Bad Gateway from Nginx is one of the least informative pages on the internet. The browser shows a grey "502 Bad Gateway / nginx" and nothing else. The good news is that Nginx knows exactly why it returned that page, and it writes the reason to its error log every single time.&lt;/p&gt;

&lt;p&gt;So the whole fix comes down to two steps: find the right log line, then match it to a cause. This post covers both, plus the handful of commands that confirm the diagnosis before you change anything.&lt;/p&gt;

&lt;h2&gt;
  
  
  What 502 actually means
&lt;/h2&gt;

&lt;p&gt;Nginx is acting as a reverse proxy. It accepted the visitor's request, tried to pass it to an upstream (Node, Gunicorn, PHP-FPM, a container, another server), and the upstream either refused the connection, hung up halfway, or sent back something Nginx couldn't parse as HTTP.&lt;/p&gt;

&lt;p&gt;That's different from a 504, where the upstream accepted the request but took too long. If your log says &lt;code&gt;upstream timed out&lt;/code&gt;, you're looking at a timeout problem, not a 502 problem, and the fix is different.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 1: read the error log, not the access log
&lt;/h2&gt;

&lt;p&gt;The access log only tells you that a 502 happened. The error log tells you why.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# Default locations on Debian/Ubuntu and most RHEL-family installs&lt;/span&gt;
sudo tail -f /var/log/nginx/error.log

&lt;span class="c"&gt;# If you're not sure where it is&lt;/span&gt;
sudo nginx -T 2&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;/dev/null &lt;span class="o"&gt;|&lt;/span&gt; grep -E &lt;span class="s1"&gt;'^\s*error_log'&lt;/span&gt;

&lt;span class="c"&gt;# Docker&lt;/span&gt;
docker logs -f &lt;span class="o"&gt;&amp;lt;&lt;/span&gt;nginx-container&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; 2&lt;span class="o"&gt;&amp;gt;&amp;amp;&lt;/span&gt;1 &lt;span class="o"&gt;|&lt;/span&gt; grep -i upstream
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Reload the failing page while the &lt;code&gt;tail&lt;/code&gt; is running. You'll get a line like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;2026/11/14 09:12:44 [error] 1234#1234: *5678 connect() failed (111: Connection refused)
while connecting to upstream, client: 203.0.113.9, server: example.com,
request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:3000/", host: "example.com"
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two parts matter: the message after &lt;code&gt;[error]&lt;/code&gt;, and the &lt;code&gt;upstream:&lt;/code&gt; address. The address tells you exactly where Nginx tried to go, which catches a surprising number of typos on its own.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Need someone else to catch the next 502?&lt;/strong&gt; &lt;a href="https://upordown.online" rel="noopener noreferrer"&gt;upordown.online&lt;/a&gt; watches your site from outside and alerts you the moment Nginx starts returning errors.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  Step 2: match the line to a cause
&lt;/h2&gt;

&lt;h3&gt;
  
  
  &lt;code&gt;connect() failed (111: Connection refused) while connecting to upstream&lt;/code&gt;
&lt;/h3&gt;

&lt;p&gt;Nothing is listening on that address and port. The app crashed, didn't start, or is listening somewhere else.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;ss -ltnp &lt;span class="o"&gt;|&lt;/span&gt; grep 3000          &lt;span class="c"&gt;# is anything listening on the port?&lt;/span&gt;
systemctl status myapp        &lt;span class="c"&gt;# did the service die?&lt;/span&gt;
journalctl -u myapp -n 50     &lt;span class="c"&gt;# why did it die?&lt;/span&gt;
curl -i http://127.0.0.1:3000/  &lt;span class="c"&gt;# does the app answer without Nginx?&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A classic version: the app listens on &lt;code&gt;localhost:3000&lt;/code&gt;, which resolves to &lt;code&gt;::1&lt;/code&gt; (IPv6), while Nginx connects to &lt;code&gt;127.0.0.1&lt;/code&gt; (IPv4). Make both sides use the same explicit address.&lt;/p&gt;

&lt;h3&gt;
  
  
  &lt;code&gt;connect() to unix:/run/php/php8.3-fpm.sock failed (2: No such file or directory)&lt;/code&gt;
&lt;/h3&gt;

&lt;p&gt;The socket path in &lt;code&gt;fastcgi_pass&lt;/code&gt; or &lt;code&gt;proxy_pass&lt;/code&gt; doesn't exist. Usually PHP-FPM was upgraded and the socket is now &lt;code&gt;php8.4-fpm.sock&lt;/code&gt;, or the service isn't running. &lt;code&gt;ls -l /run/php/&lt;/code&gt; shows what's actually there.&lt;/p&gt;

&lt;h3&gt;
  
  
  &lt;code&gt;... failed (13: Permission denied) while connecting to upstream&lt;/code&gt;
&lt;/h3&gt;

&lt;p&gt;Two common causes. With a Unix socket, the Nginx worker user (&lt;code&gt;www-data&lt;/code&gt; or &lt;code&gt;nginx&lt;/code&gt;) can't read or write the socket file: fix &lt;code&gt;listen.owner&lt;/code&gt;, &lt;code&gt;listen.group&lt;/code&gt; and &lt;code&gt;listen.mode&lt;/code&gt; in the FPM pool config. On RHEL, Fedora or CentOS with SELinux enforcing, Nginx isn't allowed to open network connections to upstreams at all until you run:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;sudo setsebool -P httpd_can_network_connect 1
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  &lt;code&gt;upstream prematurely closed connection while reading response header from upstream&lt;/code&gt;
&lt;/h3&gt;

&lt;p&gt;Nginx connected, sent the request, and the upstream hung up before sending a response. Usually the app crashed while handling that request (check its logs for a stack trace at the same timestamp), or a worker was killed for using too much memory (&lt;code&gt;dmesg | grep -i oom&lt;/code&gt;), or Gunicorn's worker timeout killed a slow request.&lt;/p&gt;

&lt;p&gt;If it only happens on some requests under load, look at keepalive: an upstream that closes idle keepalive connections sooner than Nginx expects will produce occasional 502s. Setting the app's keepalive timeout slightly higher than Nginx's &lt;code&gt;keepalive_timeout&lt;/code&gt; in the &lt;code&gt;upstream&lt;/code&gt; block removes the race.&lt;/p&gt;

&lt;h3&gt;
  
  
  &lt;code&gt;upstream sent too big header while reading response header from upstream&lt;/code&gt;
&lt;/h3&gt;

&lt;p&gt;The app's response headers are bigger than Nginx's buffer, usually because of large cookies or a long &lt;code&gt;Set-Cookie&lt;/code&gt; chain after login. Raise the buffers:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight nginx"&gt;&lt;code&gt;&lt;span class="n"&gt;proxy_buffer_size&lt;/span&gt; &lt;span class="mi"&gt;16k&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="n"&gt;proxy_buffers&lt;/span&gt; &lt;span class="mi"&gt;8&lt;/span&gt; &lt;span class="mi"&gt;16k&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="n"&gt;proxy_busy_buffers_size&lt;/span&gt; &lt;span class="mi"&gt;32k&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="c1"&gt;# for PHP-FPM, the fastcgi_ equivalents
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then ask why the headers are that big. A 12 KB session cookie is a bug of its own.&lt;/p&gt;

&lt;h3&gt;
  
  
  &lt;code&gt;no live upstreams while connecting to upstream&lt;/code&gt;
&lt;/h3&gt;

&lt;p&gt;You have an &lt;code&gt;upstream&lt;/code&gt; block with several servers, and Nginx has marked all of them as failed (passive health checks via &lt;code&gt;max_fails&lt;/code&gt; and &lt;code&gt;fail_timeout&lt;/code&gt;). The real error is earlier in the log: scroll up to find what made each server fail the first time.&lt;/p&gt;

&lt;h3&gt;
  
  
  &lt;code&gt;SSL_do_handshake() failed ... while SSL handshaking to upstream&lt;/code&gt;
&lt;/h3&gt;

&lt;p&gt;You're proxying to an HTTPS upstream, and the TLS handshake fails. Very often the upstream needs SNI, which Nginx doesn't send by default:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight nginx"&gt;&lt;code&gt;&lt;span class="n"&gt;proxy_pass&lt;/span&gt; &lt;span class="n"&gt;https&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="c1"&gt;//backend.example.net;&lt;/span&gt;
&lt;span class="n"&gt;proxy_ssl_server_name&lt;/span&gt; &lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Step 3: confirm before you change anything
&lt;/h2&gt;

&lt;p&gt;Whatever the log says, reproduce the problem without Nginx in the way. If &lt;code&gt;curl -i http://127.0.0.1:3000/health&lt;/code&gt; from the Nginx host also fails, Nginx is the messenger and the fix is in the app. If curl works and Nginx still returns 502, compare the address in the log's &lt;code&gt;upstream:&lt;/code&gt; field with what you just curled. They're different more often than you'd expect.&lt;/p&gt;

&lt;p&gt;After any config change:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;sudo nginx -t &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; sudo systemctl reload nginx
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;nginx -t&lt;/code&gt; catches syntax errors before a reload turns one 502 into a site-wide outage.&lt;/p&gt;

&lt;h2&gt;
  
  
  Making the next 502 shorter
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Log upstream details in the access log.&lt;/strong&gt; Adding &lt;code&gt;$upstream_addr $upstream_status $upstream_response_time&lt;/code&gt; to your &lt;code&gt;log_format&lt;/code&gt; shows which backend failed and how long it took, per request.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Give the app a health endpoint&lt;/strong&gt; that doesn't depend on Nginx, so you can curl it directly.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Check from outside.&lt;/strong&gt; A 502 that only appears for some regions or some backends is easy to miss from your own laptop. An external check like &lt;a href="https://upordown.online" rel="noopener noreferrer"&gt;upordown.online&lt;/a&gt; sees what visitors see, which is the version of the error that matters.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Related reading
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://dev.to/upordownonline/vs-504-what-each-error-means-and-who-has-to-fix-it-5g20"&gt;503 vs 504: what each error means and who has to fix it&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/upordownonline/how-to-read-curl-v-output-to-find-where-a-request-fails-49nh"&gt;How to read curl -v output to find where a request fails&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/upordownonline/econnrefused-vs-etimedout-vs-econnreset-what-your-monitors-error-is-actually-telling-you-3pi9"&gt;ECONNREFUSED vs ETIMEDOUT vs ECONNRESET: what your monitor's error is actually telling you&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/upordownonline/is-it-down-or-is-it-me-debugging-a-site-that-wont-load-one-layer-at-a-time-deh"&gt;Is it down, or is it me? Debugging a site that won't load, one layer at a time&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/upordownonline/adding-rate-limiting-to-your-api-a-12-point-checklist-so-it-doesnt-block-your-own-users-or-your-f35"&gt;Adding rate limiting to your API? A 12-point checklist so it doesn't block your own users (or your monitor)&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Which 502 log line have you seen most often? And did anyone else lose an afternoon to the &lt;code&gt;localhost&lt;/code&gt; resolving to IPv6 one?&lt;/p&gt;

</description>
      <category>nginx</category>
      <category>devops</category>
      <category>webdev</category>
      <category>debugging</category>
    </item>
    <item>
      <title>Putting a CDN in front of your site? A 12-point checklist so the CDN isn't your next outage</title>
      <dc:creator>upordown.online</dc:creator>
      <pubDate>Fri, 13 Nov 2026 08:03:50 +0000</pubDate>
      <link>https://dev.to/upordownonline/putting-a-cdn-in-front-of-your-site-a-12-point-checklist-so-the-cdn-isnt-your-next-outage-4881</link>
      <guid>https://dev.to/upordownonline/putting-a-cdn-in-front-of-your-site-a-12-point-checklist-so-the-cdn-isnt-your-next-outage-4881</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;Uptime monitoring from outside your own infrastructure that alerts you the minute your site goes down — &lt;a href="https://upordown.online" rel="noopener noreferrer"&gt;upordown.online&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Adding a CDN is usually sold as a pure win: faster pages, less load on your servers, some protection from traffic spikes. Mostly it is. But it also puts a new system between your users and your code, with its own caching rules, its own certificates, its own DNS and its own idea of what "up" means.&lt;/p&gt;

&lt;p&gt;The outages that follow a CDN rollout tend to look the same: the CDN caches something it shouldn't, or keeps serving a happy cached page while the origin is down, or an error page gets cached and served to everyone for an hour. None of these are exotic. All of them are preventable with a checklist on rollout day.&lt;/p&gt;

&lt;h2&gt;
  
  
  Before you switch DNS
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;1. Decide what must never be cached.&lt;/strong&gt;&lt;br&gt;
Write the list down before you touch any settings: logged-in pages, cart and checkout, API responses that differ per user, anything that sets a cookie. Then make sure your origin sends &lt;code&gt;Cache-Control: private, no-store&lt;/code&gt; (or at least &lt;code&gt;private&lt;/code&gt;) on those responses. Relying on the CDN's defaults to "probably not cache HTML" is how one user's account page ends up served to another.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. Check which headers vary the response.&lt;/strong&gt;&lt;br&gt;
If your origin returns different content based on &lt;code&gt;Accept-Language&lt;/code&gt;, &lt;code&gt;Accept-Encoding&lt;/code&gt;, a device cookie or a geo header, the CDN needs to know, either via &lt;code&gt;Vary&lt;/code&gt; or by including that value in the cache key. Otherwise the first visitor's version becomes everyone's version.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. Lock down the origin.&lt;/strong&gt;&lt;br&gt;
Once traffic flows through the CDN, the origin should only accept connections from the CDN's IP ranges or require a shared secret header. An origin that's still publicly reachable lets attackers skip your CDN entirely, and lets your own monitors give you a different answer from your users.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;4. Get the TLS setup right on both legs.&lt;/strong&gt;&lt;br&gt;
There are two connections now: visitor to CDN and CDN to origin. Use a valid certificate on the origin and set the CDN to verify it ("full, strict" or the equivalent). "Flexible" modes that talk plain HTTP to the origin are a common cause of redirect loops when the origin also forces HTTPS.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;5. Lower your DNS TTL a day ahead.&lt;/strong&gt;&lt;br&gt;
If the switch goes wrong, you want to roll back in minutes, not hours. Drop the TTL on the records you'll change well before the cutover, so resolvers have already forgotten the old long TTL when you need them to.&lt;/p&gt;
&lt;h2&gt;
  
  
  On switch-over day
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;6. Test through the CDN before moving real traffic.&lt;/strong&gt;&lt;br&gt;
Most CDNs give you a test hostname, or you can point a single machine at the CDN with a hosts entry or &lt;code&gt;curl --resolve&lt;/code&gt;. Walk the critical paths: home page, login, checkout, an API call, a file download. Check the response headers to confirm what was cached (&lt;code&gt;Age&lt;/code&gt;, &lt;code&gt;CF-Cache-Status&lt;/code&gt;, &lt;code&gt;X-Cache&lt;/code&gt; and friends).&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;7. Make sure error responses aren't cached for long.&lt;/strong&gt;&lt;br&gt;
Find the CDN setting for caching 4xx and 5xx responses. A cached 502 for ten minutes turns a 30-second origin blip into a ten-minute outage. Short or zero TTLs on errors are usually the right default; some teams cache 404s briefly and 5xx not at all.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;8. Decide what happens when the origin is down.&lt;/strong&gt;&lt;br&gt;
Some CDNs can serve stale content when the origin fails (&lt;code&gt;stale-if-error&lt;/code&gt;). That's great for a blog and dangerous for a stock count or a status page. Choose deliberately per path, and write the decision down, because it changes what "the site is up" means.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;A CDN cutover is only safe if you can see what the public URL returns after it.&lt;/strong&gt; &lt;a href="https://upordown.online" rel="noopener noreferrer"&gt;upordown.online&lt;/a&gt; monitors that URL from outside, so a bad cache rule or origin error doesn't stay hidden behind the CDN.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;h2&gt;
  
  
  After the switch
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;9. Monitor the origin directly, not only through the CDN.&lt;/strong&gt;&lt;br&gt;
A check through the CDN tells you what users see. A check against the origin (on its own hostname or IP, with the CDN's secret header if you added one) tells you whether your servers are actually healthy. With serve-stale enabled, the CDN check can stay green for hours while the origin is down. You want both, and you want to know when they disagree.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;10. Check content freshness, not just status codes.&lt;/strong&gt;&lt;br&gt;
A cached page returns 200 just like a fresh one. Add something that changes on every deploy or every few minutes, such as a build ID or timestamp in a footer or a &lt;code&gt;/version&lt;/code&gt; endpoint, and alert if what the CDN serves is older than you expect. This is the check that catches "the CDN served yesterday's homepage all morning".&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;11. Have a purge procedure that someone has actually run.&lt;/strong&gt;&lt;br&gt;
Know how to purge a single URL, a path prefix and everything. Store the API token where on-call can reach it, and run a purge once during the rollout so the first time isn't during an incident. If deploys should purge automatically, verify that they do.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;12. Plan for the CDN itself having a bad day.&lt;/strong&gt;&lt;br&gt;
CDNs have outages too. Know what you'd do: wait, or switch DNS back to the origin? If the plan is to switch, the origin must be able to take full traffic, its certificate must be valid for the public hostname, and your TTL must be low enough to make the switch useful. Many teams decide to wait, which is a fine decision as long as it's a decision.&lt;/p&gt;
&lt;h2&gt;
  
  
  A small script for the day after
&lt;/h2&gt;

&lt;p&gt;The checks that matter most after rollout fit in a few lines of shell:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;#!/usr/bin/env bash&lt;/span&gt;
&lt;span class="nb"&gt;set&lt;/span&gt; -euo pipefail
&lt;span class="nv"&gt;HOST&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;example.com
&lt;span class="nv"&gt;ORIGIN_IP&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;203.0.113.10

&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"== via CDN"&lt;/span&gt;
curl -sS -o /dev/null -D - &lt;span class="s2"&gt;"https://$HOST/"&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; grep -iE &lt;span class="s1"&gt;'^(HTTP|age|cache-control|x-cache|cf-cache-status)'&lt;/span&gt;

&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"== origin directly"&lt;/span&gt;
curl -sS -o /dev/null -w &lt;span class="s1"&gt;'%{http_code} %{time_total}s\n'&lt;/span&gt; \
  --resolve &lt;span class="s2"&gt;"$HOST:443:$ORIGIN_IP"&lt;/span&gt; &lt;span class="s2"&gt;"https://$HOST/"&lt;/span&gt;

&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"== freshness"&lt;/span&gt;
curl -sS &lt;span class="s2"&gt;"https://$HOST/version"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If the CDN says 200 with a large &lt;code&gt;Age&lt;/code&gt; and the origin says 000 or 502, you're serving stale content over a dead origin, which is fine if you chose that, and an incident if you didn't.&lt;/p&gt;

&lt;h2&gt;
  
  
  The short version
&lt;/h2&gt;

&lt;p&gt;Put the rules about what's cacheable in the origin's headers, lock the origin down, keep error caching short, and monitor both the CDN path and the origin path with a freshness check. Watching both paths from outside, and alerting when they disagree, is the kind of thing we built &lt;a href="https://upordown.online" rel="noopener noreferrer"&gt;upordown.online&lt;/a&gt; for.&lt;/p&gt;

&lt;h2&gt;
  
  
  Related reading
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://dev.to/upordownonline/how-to-fix-your-connection-is-not-private-in-chrome-neterrcertdateinvalid-and-friends-as-l04"&gt;How to fix 'Your connection is not private' in Chrome (NET::ERR_CERT_DATE_INVALID and friends), as a visitor and as the site owner&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/upordownonline/our-monitor-said-100-uptime-while-the-cdn-served-a-6-hour-old-homepage-bk9"&gt;Our monitor said 100% uptime while the CDN served a 6-hour-old homepage&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/upordownonline/your-dns-ttl-is-why-the-outage-lasted-2-hours-set-it-before-you-need-it-meh"&gt;Your DNS TTL is why the outage lasted 2 hours: set it before you need it&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/upordownonline/quick-tip-your-load-balancer-hides-a-dead-server-check-each-backend-with-curl-resolve-2gj6"&gt;Quick tip: your load balancer hides a dead server. Check each backend with curl --resolve&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;What's the worst thing a CDN has cached for you? I'd love to hear the stories, especially the ones that took a while to notice.&lt;/p&gt;

</description>
      <category>devops</category>
      <category>webdev</category>
      <category>networking</category>
      <category>performance</category>
    </item>
    <item>
      <title>How to fix 'Your connection is not private' in Chrome (NET::ERR_CERT_DATE_INVALID and friends), as a visitor and as the site owner</title>
      <dc:creator>upordown.online</dc:creator>
      <pubDate>Fri, 13 Nov 2026 08:03:02 +0000</pubDate>
      <link>https://dev.to/upordownonline/how-to-fix-your-connection-is-not-private-in-chrome-neterrcertdateinvalid-and-friends-as-l04</link>
      <guid>https://dev.to/upordownonline/how-to-fix-your-connection-is-not-private-in-chrome-neterrcertdateinvalid-and-friends-as-l04</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;Uptime monitoring from outside your own infrastructure that alerts you the minute your site goes down — &lt;a href="https://upordown.online" rel="noopener noreferrer"&gt;upordown.online&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Chrome's full-page warning, "Your connection is not private", is one of the few errors that both visitors and developers hit regularly, and both groups tend to react the same way: click "Advanced", look for a "Proceed anyway" link, and hope.&lt;/p&gt;

&lt;p&gt;Sometimes that's harmless. Sometimes it's exactly what an attacker on a café Wi-Fi network wants you to do. The good news is that the small grey code under the warning tells you almost everything about which case you're in. This post walks through the common codes, what a visitor can safely do about each, and how a site owner reproduces and fixes them from a terminal.&lt;/p&gt;

&lt;h2&gt;
  
  
  First, read the code
&lt;/h2&gt;

&lt;p&gt;Under the headline, Chrome shows a code such as &lt;code&gt;NET::ERR_CERT_DATE_INVALID&lt;/code&gt;. Click on it (or on "Advanced") to see more detail. The codes you'll actually meet:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Code&lt;/th&gt;
&lt;th&gt;What it means&lt;/th&gt;
&lt;th&gt;Usually whose problem&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;NET::ERR_CERT_DATE_INVALID&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;The certificate is expired or not yet valid, &lt;strong&gt;or your clock is wrong&lt;/strong&gt;
&lt;/td&gt;
&lt;td&gt;Site owner, or the visitor's clock&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;NET::ERR_CERT_COMMON_NAME_INVALID&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;The certificate doesn't cover this hostname&lt;/td&gt;
&lt;td&gt;Site owner&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;NET::ERR_CERT_AUTHORITY_INVALID&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Chrome doesn't trust whoever issued it (self-signed, private CA, missing intermediate)&lt;/td&gt;
&lt;td&gt;Site owner, or something intercepting traffic&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;ERR_SSL_PROTOCOL_ERROR&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;The TLS handshake failed outright&lt;/td&gt;
&lt;td&gt;Site owner or a middlebox&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;ERR_SSL_VERSION_OR_CIPHER_MISMATCH&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Server only offers protocols or ciphers Chrome refuses&lt;/td&gt;
&lt;td&gt;Site owner&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;NET::ERR_CERT_REVOKED&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;The issuer revoked the certificate&lt;/td&gt;
&lt;td&gt;Site owner&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  If you're the visitor
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;1. Check your computer's clock.&lt;/strong&gt; This is the single most common cause of &lt;code&gt;ERR_CERT_DATE_INVALID&lt;/code&gt; on sites that work for everyone else. A laptop with a dead CMOS battery, a VM restored from an old snapshot, or a phone with automatic time turned off can be days or years out. Chrome compares the certificate's validity window with &lt;em&gt;your&lt;/em&gt; clock. Turn on automatic date and time, then reload.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. Try another network.&lt;/strong&gt; If the error only appears on hotel, airport or office Wi-Fi, something on that network is intercepting HTTPS: a captive portal that hasn't been accepted yet, or a filtering proxy. Open &lt;code&gt;http://neverssl.com&lt;/code&gt; to trigger the captive portal login, or switch to mobile data. If the site works on mobile data, the site is fine and the network is the problem.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. Look at the certificate details.&lt;/strong&gt; Click the "Not secure" label in the address bar, then the certificate info. If the issuer is your antivirus product, your employer, or something you don't recognise, your traffic is being inspected. Antivirus "HTTPS scanning" features are a frequent cause of &lt;code&gt;ERR_CERT_AUTHORITY_INVALID&lt;/code&gt;; turning that feature off often fixes it.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;4. Check whether it's down for everyone.&lt;/strong&gt; If several people or networks see the same error, it's the site, and nothing on your machine will fix it. A quick check from a few outside locations settles it in seconds.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;5. Don't click "Proceed" on anything with a login or payment.&lt;/strong&gt; On a site you'll type a password into, a certificate error means you can't be sure who you're talking to. Wait, or contact the site owner.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;If you're the site owner, you need to know whether the certificate error is showing up for everyone.&lt;/strong&gt; &lt;a href="https://upordown.online" rel="noopener noreferrer"&gt;upordown.online&lt;/a&gt; checks the site from outside and alerts you when TLS starts failing for visitors.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  If you're the site owner
&lt;/h2&gt;

&lt;p&gt;Start by reproducing the error from outside your own network, because your browser may have a cached session, an internal DNS answer, or a corporate root certificate that hides the problem.&lt;/p&gt;

&lt;h3&gt;
  
  
  Expired certificate (&lt;code&gt;ERR_CERT_DATE_INVALID&lt;/code&gt;)
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; openssl s_client -connect example.com:443 -servername example.com 2&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;/dev/null \
  &lt;span class="o"&gt;|&lt;/span&gt; openssl x509 -noout -dates -subject -issuer
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If &lt;code&gt;notAfter&lt;/code&gt; is in the past, the certificate expired. The real question is why auto-renewal didn't work. The usual culprits: the renewal ran but the web server was never reloaded, so it's still serving the old file from memory; the renewal job runs on a server that's no longer the one serving traffic; or the HTTP-01 challenge now fails because a redirect, a CDN rule or a firewall blocks &lt;code&gt;/.well-known/acme-challenge/&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Renew, reload, then rerun the command above against &lt;strong&gt;every&lt;/strong&gt; IP your domain resolves to. One forgotten backend with the old certificate is enough to make the error intermittent.&lt;/p&gt;

&lt;h3&gt;
  
  
  Wrong hostname (&lt;code&gt;ERR_CERT_COMMON_NAME_INVALID&lt;/code&gt;)
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; openssl s_client -connect example.com:443 -servername www.example.com 2&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;/dev/null \
  &lt;span class="o"&gt;|&lt;/span&gt; openssl x509 -noout -ext subjectAltName
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Browsers only look at the Subject Alternative Names list. Common gaps: the certificate covers &lt;code&gt;example.com&lt;/code&gt; but not &lt;code&gt;www.example.com&lt;/code&gt;; a wildcard &lt;code&gt;*.example.com&lt;/code&gt; doesn't cover &lt;code&gt;api.eu.example.com&lt;/code&gt; (wildcards match one label only); or the server is returning its default certificate because the client's SNI name didn't match any configured site. Drop &lt;code&gt;-servername&lt;/code&gt; from the command and compare: if you get a different certificate, your SNI setup is the problem.&lt;/p&gt;

&lt;h3&gt;
  
  
  Untrusted issuer (&lt;code&gt;ERR_CERT_AUTHORITY_INVALID&lt;/code&gt;)
&lt;/h3&gt;

&lt;p&gt;The most common cause on a real public site is a &lt;strong&gt;missing intermediate certificate&lt;/strong&gt;. Your browser at home may have the intermediate cached from another site and show no error, while a fresh phone fails. Check the chain the server actually sends:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; openssl s_client -connect example.com:443 -servername example.com -showcerts 2&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;/dev/null \
  &lt;span class="o"&gt;|&lt;/span&gt; grep -E &lt;span class="s1"&gt;'^( [0-9] s:| *i:)'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;You should see your certificate (depth 0) followed by at least one intermediate. If there's only depth 0, configure the server with the full chain file (with Let's Encrypt, that's &lt;code&gt;fullchain.pem&lt;/code&gt;, not &lt;code&gt;cert.pem&lt;/code&gt;).&lt;/p&gt;

&lt;h3&gt;
  
  
  Handshake failures (&lt;code&gt;ERR_SSL_PROTOCOL_ERROR&lt;/code&gt;, &lt;code&gt;VERSION_OR_CIPHER_MISMATCH&lt;/code&gt;)
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;curl -v https://example.com/ 2&lt;span class="o"&gt;&amp;gt;&amp;amp;&lt;/span&gt;1 &lt;span class="o"&gt;|&lt;/span&gt; grep -E &lt;span class="s1"&gt;'SSL|TLS|ALPN|error'&lt;/span&gt;
openssl s_client -connect example.com:443 -servername example.com -tls1_2 &lt;span class="o"&gt;&amp;lt;&lt;/span&gt;/dev/null
openssl s_client -connect example.com:443 -servername example.com -tls1_3 &lt;span class="o"&gt;&amp;lt;&lt;/span&gt;/dev/null
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If neither TLS 1.2 nor 1.3 connects, the server is misconfigured or something in front of it (a load balancer listener on the wrong port, a proxy speaking plain HTTP on 443) isn't doing TLS at all. A plain-HTTP response on port 443 is a classic &lt;code&gt;ERR_SSL_PROTOCOL_ERROR&lt;/code&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Catching it before visitors do
&lt;/h2&gt;

&lt;p&gt;Every one of these errors is visible from outside before anyone complains:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Check certificate expiry from outside and alert at 21 and 7 days, not on the day.&lt;/li&gt;
&lt;li&gt;Check each hostname you serve, including &lt;code&gt;www&lt;/code&gt; and API subdomains, because they often have separate certificates.&lt;/li&gt;
&lt;li&gt;Check through every backend or region, not only through the load balancer, so a single stale node can't hide.&lt;/li&gt;
&lt;li&gt;After any renewal, verify the served certificate's serial number changed. "The renewal job succeeded" and "the server is serving the new certificate" are two different facts.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;That last check is the one most setups miss, and it's the kind of thing an outside uptime check, such as the ones &lt;a href="https://upordown.online" rel="noopener noreferrer"&gt;upordown.online&lt;/a&gt; runs, is good at noticing.&lt;/p&gt;

&lt;h2&gt;
  
  
  Related reading
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://dev.to/upordownonline/six-tls-certificate-myths-that-cause-outages-and-one-that-just-wastes-money-1b31"&gt;Six TLS certificate myths that cause outages (and one that just wastes money)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/upordownonline/how-to-fix-errconnectiontimedout-in-chrome-on-windows-mac-and-android-16co"&gt;How to fix ERR_CONNECTION_TIMED_OUT in Chrome on Windows, Mac and Android&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/upordownonline/how-to-read-curl-v-output-to-find-where-a-request-fails-49nh"&gt;How to read curl -v output to find where a request fails&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/upordownonline/putting-a-cdn-in-front-of-your-site-a-12-point-checklist-so-the-cdn-isnt-your-next-outage-4881"&gt;Putting a CDN in front of your site? A 12-point checklist so the CDN isn't your next outage&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Which of these codes have you hit most, and was it the server or the network every time? I'm curious whether anyone still sees antivirus interception regularly.&lt;/p&gt;

</description>
      <category>webdev</category>
      <category>security</category>
      <category>networking</category>
      <category>tutorial</category>
    </item>
    <item>
      <title>Six TLS certificate myths that cause outages (and one that just wastes money)</title>
      <dc:creator>upordown.online</dc:creator>
      <pubDate>Thu, 12 Nov 2026 08:07:27 +0000</pubDate>
      <link>https://dev.to/upordownonline/six-tls-certificate-myths-that-cause-outages-and-one-that-just-wastes-money-1b31</link>
      <guid>https://dev.to/upordownonline/six-tls-certificate-myths-that-cause-outages-and-one-that-just-wastes-money-1b31</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;Uptime monitoring from outside your own infrastructure that alerts you the minute your site goes down — &lt;a href="https://upordown.online" rel="noopener noreferrer"&gt;upordown.online&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Certificate problems are one of the most embarrassing kinds of outage. The site is running, the servers are healthy, the code is fine, and every visitor gets a full-page browser warning. Nobody can get past it, and the fix is often five minutes of work that should have happened weeks ago.&lt;/p&gt;

&lt;p&gt;Automated issuance made certificates much easier. It also created a new set of assumptions. Here are the ones I see behind most certificate incidents.&lt;/p&gt;

&lt;h2&gt;
  
  
  Myth 1: "We use automatic renewal, so certificates can't expire"
&lt;/h2&gt;

&lt;p&gt;Automatic renewal is excellent, and it fails silently in all sorts of ways:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The renewal job runs on a server that was decommissioned, or in a container that no longer exists.&lt;/li&gt;
&lt;li&gt;The HTTP challenge fails because someone added a redirect, a WAF rule, or basic auth in front of &lt;code&gt;/.well-known/acme-challenge/&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;A DNS challenge fails because the API token for the DNS provider was rotated or expired.&lt;/li&gt;
&lt;li&gt;Renewal succeeds, but the web server is never reloaded, so it keeps serving the old certificate from memory.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;That last one is particularly sneaky: the file on disk is fresh, so anyone who checks the file thinks everything is fine.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What's true:&lt;/strong&gt; automation reduces the chance of expiry; it doesn't remove the need to check. Monitor the certificate &lt;em&gt;as served&lt;/em&gt;, from outside, and alert well before expiry, at least two to three weeks for automated certificates so you have time to fix a broken renewal.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; openssl s_client -connect example.com:443 -servername example.com 2&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;/dev/null \
  &lt;span class="o"&gt;|&lt;/span&gt; openssl x509 -noout -enddate -issuer -subject
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Myth 2: "If it works in my browser, the certificate is fine"
&lt;/h2&gt;

&lt;p&gt;Browsers are forgiving in ways other clients aren't. The classic example is an &lt;strong&gt;incomplete certificate chain&lt;/strong&gt;: the server sends its own certificate but not the intermediate. Many desktop browsers can fetch or cache the missing intermediate, so the site looks fine. Older Android devices, &lt;code&gt;curl&lt;/code&gt;, many API clients, payment provider webhooks and monitoring tools may fail.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What's true:&lt;/strong&gt; test with a strict client. &lt;code&gt;curl -v https://example.com&lt;/code&gt; will fail on a broken chain where your browser might not. Online TLS testers will also report chain issues explicitly. Always deploy the &lt;em&gt;full chain&lt;/em&gt; file your certificate authority provides, not just the leaf certificate.&lt;/p&gt;

&lt;h2&gt;
  
  
  Myth 3: "One certificate covers the whole domain"
&lt;/h2&gt;

&lt;p&gt;A certificate for &lt;code&gt;example.com&lt;/code&gt; does not cover &lt;code&gt;www.example.com&lt;/code&gt;. A wildcard for &lt;code&gt;*.example.com&lt;/code&gt; covers &lt;code&gt;api.example.com&lt;/code&gt; but not &lt;code&gt;example.com&lt;/code&gt; itself, and not &lt;code&gt;v2.api.example.com&lt;/code&gt;, because wildcards only match one label.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What's true:&lt;/strong&gt; list every hostname you serve, including the bare domain, &lt;code&gt;www&lt;/code&gt;, API hosts, status pages and anything customers might type, and confirm each one is in the certificate's Subject Alternative Names:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; openssl s_client -connect example.com:443 -servername example.com 2&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;/dev/null \
  &lt;span class="o"&gt;|&lt;/span&gt; openssl x509 -noout -ext subjectAltName
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Myth 4: "Our CDN handles certificates, so our origin doesn't matter"
&lt;/h2&gt;

&lt;p&gt;Your CDN probably does manage the certificate visitors see. But depending on your configuration, the CDN also connects to your origin over TLS, and may validate the origin's certificate. When the origin certificate expires, visitors may see a CDN error page instead of a browser warning. Different symptom, same outage.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What's true:&lt;/strong&gt; know which TLS mode your CDN uses to reach the origin. If it validates the origin certificate (which it should for security), monitor that certificate too, using the origin hostname or by connecting directly to the origin IP with the right SNI:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;curl -v --resolve example.com:443:203.0.113.10 https://example.com/
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;A certificate that expires on a Friday still needs to page someone.&lt;/strong&gt; &lt;a href="https://upordown.online" rel="noopener noreferrer"&gt;upordown.online&lt;/a&gt; will see the TLS error visitors see, and alert you the moment the handshake starts failing.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  Myth 5: "Certificate expiry is the only thing that can go wrong"
&lt;/h2&gt;

&lt;p&gt;Expiry is the most common problem, not the only one. Others that cause real outages:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Hostname mismatch&lt;/strong&gt; after adding a new domain or moving a service behind a different load balancer.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Wrong certificate served&lt;/strong&gt; because SNI isn't configured and a default certificate is returned.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Old TLS versions or ciphers&lt;/strong&gt; disabled on the server while some clients (older devices, embedded systems, some corporate proxies) still need them, or the reverse: a client library that doesn't support what the server now requires.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Pinned certificates&lt;/strong&gt; in mobile apps or IoT devices that break when the certificate or its key changes.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Clock problems&lt;/strong&gt; on clients or servers that make a valid certificate look not-yet-valid or expired.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;What's true:&lt;/strong&gt; a useful certificate monitor checks expiry, hostname match and chain validity, and ideally alerts on unexpected changes in issuer or key.&lt;/p&gt;

&lt;h2&gt;
  
  
  Myth 6: "We'll notice quickly if it breaks"
&lt;/h2&gt;

&lt;p&gt;You might not. If the expired certificate is on an API used only by a partner, a webhook endpoint, a staging domain someone demos from, or the mail server, nobody may open it in a browser for days. Meanwhile, integrations fail with errors that look like network problems.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What's true:&lt;/strong&gt; monitor every TLS endpoint that something depends on, not just the main website. Include non-HTTP services that use TLS, like SMTP with STARTTLS, IMAP or database connections, where relevant.&lt;/p&gt;

&lt;h2&gt;
  
  
  Myth 7 (the money one): "Paid certificates are more secure"
&lt;/h2&gt;

&lt;p&gt;For a standard website, a free domain-validated certificate provides the same encryption as a paid one. Paid certificates may come with organisation validation, support, warranties, or fit procurement requirements, and some organisations have good reasons to buy them. But "the padlock is stronger" isn't one of those reasons, and browsers no longer display extended validation details prominently in the address bar.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What's true:&lt;/strong&gt; choose based on your operational needs (automation support, validation type, compliance), not on assumed encryption strength.&lt;/p&gt;

&lt;h2&gt;
  
  
  A minimal certificate checklist
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;[ ] Every hostname you serve is listed somewhere, with an owner.&lt;/li&gt;
&lt;li&gt;[ ] Each is monitored from outside for expiry, hostname match and chain.&lt;/li&gt;
&lt;li&gt;[ ] Alert thresholds give enough time to fix a failed renewal.&lt;/li&gt;
&lt;li&gt;[ ] Renewal is followed by an automatic reload, and that reload is tested.&lt;/li&gt;
&lt;li&gt;[ ] Challenge paths and DNS API credentials are documented and monitored.&lt;/li&gt;
&lt;li&gt;[ ] Origin certificates behind a CDN are covered too.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Certificate expiry and validity checks sit alongside uptime checks in &lt;a href="https://upordown.online" rel="noopener noreferrer"&gt;upordown.online&lt;/a&gt;, because from a visitor's point of view, a browser warning is downtime.&lt;/p&gt;

&lt;h2&gt;
  
  
  Related reading
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://dev.to/upordownonline/blue-green-vs-canary-vs-rolling-deploys-what-each-one-protects-you-from-and-what-it-doesnt-237m"&gt;Blue-green vs canary vs rolling deploys: what each one protects you from (and what it doesn't)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/upordownonline/rotating-an-api-key-or-database-password-a-12-point-checklist-so-the-rotation-isnt-the-outage-3aep"&gt;Rotating an API key or database password? A 12-point checklist so the rotation isn't the outage&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/upordownonline/how-to-read-curl-v-output-to-find-where-a-request-fails-49nh"&gt;How to read curl -v output to find where a request fails&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/upordownonline/econnrefused-vs-etimedout-vs-econnreset-what-your-monitors-error-is-actually-telling-you-3pi9"&gt;ECONNREFUSED vs ETIMEDOUT vs ECONNRESET: what your monitor's error is actually telling you&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;What's the most surprising way a certificate has broken on you: expiry, chain, hostname, or something stranger?&lt;/p&gt;

</description>
      <category>tls</category>
      <category>https</category>
      <category>devops</category>
      <category>security</category>
    </item>
    <item>
      <title>Blue-green vs canary vs rolling deploys: what each one protects you from (and what it doesn't)</title>
      <dc:creator>upordown.online</dc:creator>
      <pubDate>Thu, 12 Nov 2026 08:05:56 +0000</pubDate>
      <link>https://dev.to/upordownonline/blue-green-vs-canary-vs-rolling-deploys-what-each-one-protects-you-from-and-what-it-doesnt-237m</link>
      <guid>https://dev.to/upordownonline/blue-green-vs-canary-vs-rolling-deploys-what-each-one-protects-you-from-and-what-it-doesnt-237m</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;Uptime monitoring from outside your own infrastructure that alerts you the minute your site goes down — &lt;a href="https://upordown.online" rel="noopener noreferrer"&gt;upordown.online&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Every deployment strategy is a bet about what will go wrong. Blue-green bets that you'll want to switch back instantly. Canary bets that a small slice of traffic will reveal problems before everyone sees them. Rolling bets that replacing instances a few at a time keeps enough capacity up to avoid an outage.&lt;/p&gt;

&lt;p&gt;Teams often pick one because it's what their platform does by default. That's fine until the day the default doesn't protect against the failure you actually have. Here's how the three compare, what each needs to work, and how to choose.&lt;/p&gt;

&lt;h2&gt;
  
  
  The quick comparison
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;Rolling&lt;/th&gt;
&lt;th&gt;Blue-green&lt;/th&gt;
&lt;th&gt;Canary&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;How it works&lt;/td&gt;
&lt;td&gt;Replace instances in batches&lt;/td&gt;
&lt;td&gt;Run two full environments, switch traffic&lt;/td&gt;
&lt;td&gt;Send a small % of traffic to the new version, then increase&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Extra infrastructure&lt;/td&gt;
&lt;td&gt;Little (a surge batch)&lt;/td&gt;
&lt;td&gt;Roughly double during the switch&lt;/td&gt;
&lt;td&gt;A little, plus traffic-splitting&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Rollback speed&lt;/td&gt;
&lt;td&gt;Slow-ish (roll back in batches)&lt;/td&gt;
&lt;td&gt;Very fast (switch back)&lt;/td&gt;
&lt;td&gt;Fast (send traffic back)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Blast radius of a bad release&lt;/td&gt;
&lt;td&gt;Grows as the rollout progresses&lt;/td&gt;
&lt;td&gt;All users at once after the switch&lt;/td&gt;
&lt;td&gt;Limited to the canary share&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Old and new versions live together&lt;/td&gt;
&lt;td&gt;Yes, during the rollout&lt;/td&gt;
&lt;td&gt;Briefly, or not at all&lt;/td&gt;
&lt;td&gt;Yes, for the whole canary period&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Depends heavily on&lt;/td&gt;
&lt;td&gt;Good health checks&lt;/td&gt;
&lt;td&gt;Pre-switch testing&lt;/td&gt;
&lt;td&gt;Good metrics and comparison&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  Rolling deploys
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;How it works.&lt;/strong&gt; The orchestrator (Kubernetes Deployments, an autoscaling group's instance refresh, most PaaS platforms) replaces old instances with new ones a few at a time. In Kubernetes, &lt;code&gt;maxSurge&lt;/code&gt; and &lt;code&gt;maxUnavailable&lt;/code&gt; control how many extra and how many missing pods are allowed during the rollout.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What it protects you from.&lt;/strong&gt; Capacity loss during deploys, and releases that fail to start at all. If new pods never become ready, the rollout stalls and old pods keep serving.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What it doesn't.&lt;/strong&gt; A release that starts fine but behaves badly: returns wrong data, slows down under real traffic, errors on one specific endpoint. Readiness probes pass, the rollout continues, and eventually every instance runs the bad version. Rolling back means another rolling deploy, in batches.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Hidden requirement.&lt;/strong&gt; Readiness checks that reflect real ability to serve. A readiness endpoint that returns 200 as soon as the process starts makes rolling deploys much less safe than they look.&lt;/p&gt;

&lt;h2&gt;
  
  
  Blue-green deploys
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;How it works.&lt;/strong&gt; You keep two complete environments. "Blue" serves production; you deploy the new version to "green", test it, then switch traffic, usually at the load balancer, DNS or router level. Blue stays around, ready to take traffic back.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What it protects you from.&lt;/strong&gt; Slow rollbacks. If something's wrong after the switch, switching back is close to instant. It's also clean: users don't see a mix of versions for long.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What it doesn't.&lt;/strong&gt; Problems that only appear under real production traffic. After the switch, &lt;em&gt;all&lt;/em&gt; users hit the new version at once. If your pre-switch tests miss something, everyone finds it together.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Hidden requirements.&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Shared state compatibility.&lt;/strong&gt; Both environments usually share a database. A schema migration that the old version can't handle makes "switch back" impossible. Use expand-and-contract migrations: add new columns and tables first, deploy code that works with both, and remove old structures only later.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Warm-up.&lt;/strong&gt; A green environment with cold caches and no JIT warm-up can be slow for its first minutes of real traffic.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;DNS-based switching is slow to reverse.&lt;/strong&gt; If you switch with DNS, clients cache records, so "instant" rollback isn't. Switching at a load balancer is quicker.&lt;/li&gt;
&lt;/ul&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Whatever the deploy style, you still need a check that the public site works after the cut.&lt;/strong&gt; &lt;a href="https://upordown.online" rel="noopener noreferrer"&gt;upordown.online&lt;/a&gt; watches that URL from outside and alerts you if the new version isn't actually up.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  Canary deploys
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;How it works.&lt;/strong&gt; Deploy the new version alongside the old one and send a small share of traffic to it (say a few percent), compare its behaviour with the stable version, and increase gradually if it looks healthy. Tools like Argo Rollouts, Flagger, service meshes and many managed platforms automate the steps.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What it protects you from.&lt;/strong&gt; Bad releases reaching everyone. If the canary's error rate or latency is worse, you stop and roll back having affected only a small fraction of requests.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What it doesn't.&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Low-traffic problems.&lt;/strong&gt; If the canary gets 2% of a small site's traffic, a bug in a rarely used endpoint may never get exercised during the canary window.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Problems that need time.&lt;/strong&gt; Memory leaks, slow queue build-ups and nightly jobs won't show in a 10-minute canary.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Shared-resource damage.&lt;/strong&gt; A canary that writes corrupt data to a shared database does damage beyond its traffic share.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Hidden requirement.&lt;/strong&gt; Metrics you trust, split by version. Canary analysis is only as good as your ability to compare "new version error rate" against "old version error rate" with enough traffic to be meaningful. Without version-labelled metrics, a canary is just a slow rolling deploy.&lt;/p&gt;

&lt;h2&gt;
  
  
  Which failures does each catch?
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Failure&lt;/th&gt;
&lt;th&gt;Rolling&lt;/th&gt;
&lt;th&gt;Blue-green&lt;/th&gt;
&lt;th&gt;Canary&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;New version won't start&lt;/td&gt;
&lt;td&gt;✅ Stalls safely&lt;/td&gt;
&lt;td&gt;✅ Caught before switch&lt;/td&gt;
&lt;td&gt;✅ Caught early&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Errors under real traffic&lt;/td&gt;
&lt;td&gt;❌ Spreads gradually&lt;/td&gt;
&lt;td&gt;❌ Hits everyone, fast rollback&lt;/td&gt;
&lt;td&gt;✅ Limited exposure&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Slow performance&lt;/td&gt;
&lt;td&gt;⚠️ If probes notice&lt;/td&gt;
&lt;td&gt;❌ After switch&lt;/td&gt;
&lt;td&gt;✅ If latency is compared&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Incompatible DB migration&lt;/td&gt;
&lt;td&gt;❌ Mixed versions break&lt;/td&gt;
&lt;td&gt;❌ Blocks rollback&lt;/td&gt;
&lt;td&gt;❌ Mixed versions break&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Slow leak over hours&lt;/td&gt;
&lt;td&gt;❌&lt;/td&gt;
&lt;td&gt;❌&lt;/td&gt;
&lt;td&gt;❌ (unless canary runs long)&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The bottom rows matter: no strategy fixes incompatible migrations or slow leaks. Those need backwards-compatible schema changes and ongoing monitoring, whatever deploy strategy you choose.&lt;/p&gt;

&lt;h2&gt;
  
  
  How to choose
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Small team, modest traffic, good health checks:&lt;/strong&gt; rolling is usually fine. Invest in readiness checks and an easy rollback command.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;You need fast, clean rollback and can afford temporary double capacity:&lt;/strong&gt; blue-green.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;High traffic, good metrics and real risk from each release:&lt;/strong&gt; canary, ideally automated.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Many teams combine them:&lt;/strong&gt; canary first, then a rolling or blue-green promotion.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  The part every strategy shares
&lt;/h2&gt;

&lt;p&gt;Each strategy assumes someone, or something, notices when the new version is unhealthy. Health checks catch the obvious failures; metrics catch some of the rest; and an outside check of what users actually see catches what's left: the page that loads but is broken, the region that can't reach you, the login that fails.&lt;/p&gt;

&lt;p&gt;That outside view is what &lt;a href="https://upordown.online" rel="noopener noreferrer"&gt;upordown.online&lt;/a&gt; provides: checking your site from beyond your own infrastructure, during deploys and every other minute too.&lt;/p&gt;

&lt;h2&gt;
  
  
  Related reading
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://dev.to/upordownonline/no-deploys-on-friday-wise-rule-or-a-sign-your-monitoring-is-broken-1e4i"&gt;No deploys on Friday: wise rule or a sign your monitoring is broken?&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/upordownonline/postmortem-hypothetical-our-health-check-was-so-thorough-it-took-every-instance-out-of-the-load-158m"&gt;Postmortem (hypothetical): our health check was so thorough it took every instance out of the load balancer at once&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/upordownonline/your-health-check-returns-200-while-users-get-errors-7-health-check-mistakes-d84"&gt;Your health check returns 200 while users get errors: 7 health check mistakes&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/upordownonline/six-tls-certificate-myths-that-cause-outages-and-one-that-just-wastes-money-1b31"&gt;Six TLS certificate myths that cause outages (and one that just wastes money)&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Which deploy strategy does your team use, and has it ever let a bad release through that another strategy would have caught?&lt;/p&gt;

</description>
      <category>devops</category>
      <category>deployment</category>
      <category>sre</category>
      <category>kubernetes</category>
    </item>
    <item>
      <title>Rotating an API key or database password? A 12-point checklist so the rotation isn't the outage</title>
      <dc:creator>upordown.online</dc:creator>
      <pubDate>Wed, 11 Nov 2026 15:07:01 +0000</pubDate>
      <link>https://dev.to/upordownonline/rotating-an-api-key-or-database-password-a-12-point-checklist-so-the-rotation-isnt-the-outage-3aep</link>
      <guid>https://dev.to/upordownonline/rotating-an-api-key-or-database-password-a-12-point-checklist-so-the-rotation-isnt-the-outage-3aep</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;Uptime monitoring from outside your own infrastructure that alerts you the minute your site goes down — &lt;a href="https://upordown.online" rel="noopener noreferrer"&gt;upordown.online&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Rotating secrets is good hygiene. Keys leak, people leave, contractors finish, and a password that's been the same for four years is a liability. Every security checklist says "rotate regularly".&lt;/p&gt;

&lt;p&gt;What the security checklist doesn't say is that rotation is one of the most reliable ways to take your own site down. You change a password, and a service you forgot about (a cron job, a staging box that turns out to be load-bearing, a Lambda nobody's touched since 2023) can no longer connect. Sometimes immediately. Sometimes at 3am, when its connection pool recycles.&lt;/p&gt;

&lt;p&gt;Here's a checklist for making rotation boring. It assumes you have some way of storing secrets (a secrets manager, encrypted environment variables, a vault), but the steps apply regardless of the tool.&lt;/p&gt;

&lt;h2&gt;
  
  
  Before you rotate
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;1. Find every consumer of the secret.&lt;/strong&gt;&lt;br&gt;
This is the step that prevents most rotation outages, and the one most often skipped. Search your infrastructure code, CI configs, deployment manifests and secrets store for references to the secret's name. Then look at the other side: the database or API provider's own logs of which clients, IPs or users are connecting with this credential. Anything in the logs that isn't in your list is a surprise waiting to happen.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. Check whether the provider supports two valid credentials at once.&lt;/strong&gt;&lt;br&gt;
The safest rotation pattern is &lt;em&gt;overlap&lt;/em&gt;: create the new credential, move every consumer to it, then revoke the old one. Many API providers allow multiple active keys; many databases let you create a second user or set a new password while existing sessions continue. If your provider allows only one credential at a time, you need a short, coordinated cutover instead, and the rest of this checklist matters even more.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. Know how each consumer loads the secret.&lt;/strong&gt;&lt;br&gt;
Some services read secrets at startup only. Some re-read on every request. Some cache them in a connection pool that lives for hours. A service that loaded the old password at boot will keep working until it restarts, then fail. Write down, for each consumer, what it takes for it to pick up a new value: a redeploy, a restart, a config reload, or nothing.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;4. Make sure you can roll back.&lt;/strong&gt;&lt;br&gt;
Before revoking anything, know exactly how to restore the old credential if something breaks. With the overlap pattern, rollback is just "don't revoke yet". With a hard cutover, you may need the old value somewhere safe for the duration of the change.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;5. Pick a time when someone is watching.&lt;/strong&gt;&lt;br&gt;
Not Friday at 17:00. Not right before a holiday. Do it when the people who own the consumers are around and when traffic is normal enough that you'll notice errors.&lt;/p&gt;

&lt;h2&gt;
  
  
  During the rotation
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;6. Create the new credential with the same permissions, no more and no less.&lt;/strong&gt;&lt;br&gt;
Rotation is a bad moment to also "tidy up" permissions. If the new key has fewer scopes than the old one, you've combined two changes, and when something fails you won't know which caused it. Change permissions in a separate, later step.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;7. Update consumers one at a time, and verify each.&lt;/strong&gt;&lt;br&gt;
For each consumer: deploy or reload with the new secret, then confirm it's actually using it and working. Confirmation means a real request succeeding, a health check that exercises the dependency, or the provider's logs showing the new credential in use, not just "the deploy went green".&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;8. Watch the old credential's usage drop to zero.&lt;/strong&gt;&lt;br&gt;
If the provider shows per-key or per-user usage, this is the best signal you have. When the old credential has had no traffic for a while (longer than your longest cron interval, ideally a full day for anything batch-related), you've probably found every consumer.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;9. Revoke the old credential deliberately.&lt;/strong&gt;&lt;br&gt;
Revoke it, then immediately watch error rates and logs for anything failing to authenticate. If something breaks now, you've found a consumer that wasn't in your list. Restore if necessary, fix, try again.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;The rotation isn't done until the live site still works for everyone.&lt;/strong&gt; &lt;a href="https://upordown.online" rel="noopener noreferrer"&gt;upordown.online&lt;/a&gt; checks the public URL from outside, so a missed config reload shows up as an alert, not a customer ticket.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  After the rotation
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;10. Monitor the things that run rarely.&lt;/strong&gt;&lt;br&gt;
The consumers most likely to break are the ones that don't run often: weekly reports, monthly billing, quarterly exports, disaster recovery scripts. Make sure each has a heartbeat or a failure alert, so a broken credential on day 30 is noticed on day 30, not on day 90.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;11. Write down what you found.&lt;/strong&gt;&lt;br&gt;
Update your list of consumers for this secret. Note which ones needed a restart, which picked it up automatically, and any surprises. Next rotation starts from this list instead of from scratch.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;12. Schedule the next one.&lt;/strong&gt;&lt;br&gt;
Rotation that happens once every few years is always painful, because nobody remembers how. Rotation that happens every few months becomes routine, and routine is what you want. Some teams automate it entirely; even a calendar reminder is better than nothing.&lt;/p&gt;

&lt;h2&gt;
  
  
  The pattern underneath
&lt;/h2&gt;

&lt;p&gt;Almost every rotation outage comes from the same root cause: &lt;strong&gt;a consumer nobody knew about&lt;/strong&gt;. The checklist is mostly a set of ways to find those consumers before revoking, rather than after.&lt;/p&gt;

&lt;p&gt;Two habits make that easier year-round:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;One secret per consumer&lt;/strong&gt; where practical. If each service has its own key, rotating one can't break the others, and usage logs tell you exactly who's using what.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Monitoring that exercises dependencies.&lt;/strong&gt; A health check that only returns "process is alive" won't notice the database credential stopped working. A check that runs a trivial query will.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  A quick check you can run today
&lt;/h2&gt;

&lt;p&gt;Pick your most important secret, maybe the production database password, and answer three questions:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Can you list every service that uses it?&lt;/li&gt;
&lt;li&gt;If you changed it right now, which of those would fail first, and which would fail only after a restart?&lt;/li&gt;
&lt;li&gt;Would your monitoring tell you within minutes?&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;If the answers are "not really", "no idea" and "probably not", you've found the work to do before the next rotation. External monitoring that notices when your site or API starts failing, whatever the cause, is what we do at &lt;a href="https://upordown.online" rel="noopener noreferrer"&gt;upordown.online&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Related reading
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://dev.to/upordownonline/two-developers-one-outage-no-postmortem-thats-how-the-same-outage-happens-again-1oi8"&gt;Two developers, one outage, no postmortem? That's how the same outage happens again&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/upordownonline/no-deploys-on-friday-wise-rule-or-a-sign-your-monitoring-is-broken-1e4i"&gt;No deploys on Friday: wise rule or a sign your monitoring is broken?&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/upordownonline/your-uptime-alerts-have-never-been-tested-a-12-point-checklist-to-prove-theyll-actually-wake-you-k8o"&gt;Your uptime alerts have never been tested. A 12-point checklist to prove they'll actually wake you&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/upordownonline/your-health-check-returns-200-while-users-get-errors-7-health-check-mistakes-d84"&gt;Your health check returns 200 while users get errors: 7 health check mistakes&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/upordownonline/adding-rate-limiting-to-your-api-a-12-point-checklist-so-it-doesnt-block-your-own-users-or-your-f35"&gt;Adding rate limiting to your API? A 12-point checklist so it doesn't block your own users (or your monitor)&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;What's the most surprising thing that broke the last time you rotated a credential?&lt;/p&gt;

</description>
      <category>security</category>
      <category>devops</category>
      <category>reliability</category>
      <category>checklist</category>
    </item>
    <item>
      <title>Two developers, one outage, no postmortem? That's how the same outage happens again</title>
      <dc:creator>upordown.online</dc:creator>
      <pubDate>Wed, 11 Nov 2026 15:05:01 +0000</pubDate>
      <link>https://dev.to/upordownonline/two-developers-one-outage-no-postmortem-thats-how-the-same-outage-happens-again-1oi8</link>
      <guid>https://dev.to/upordownonline/two-developers-one-outage-no-postmortem-thats-how-the-same-outage-happens-again-1oi8</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;Uptime monitoring from outside your own infrastructure that alerts you the minute your site goes down — &lt;a href="https://upordown.online" rel="noopener noreferrer"&gt;upordown.online&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;In a large company, an outage usually ends with a document. There's a template, a review meeting, action items with owners and a tracker that nags people until they're done.&lt;/p&gt;

&lt;p&gt;In a team of two or three developers, an outage usually ends with someone saying "OK, it's back," a sigh of relief and a return to whatever was interrupted. Maybe a message in chat: "it was the disk again." Then, four months later, it's the disk again, and nobody can remember what they did last time.&lt;/p&gt;

&lt;p&gt;I think small teams need written postmortems &lt;strong&gt;more&lt;/strong&gt; than big ones, not less. Here's why, and what a small-team version looks like.&lt;/p&gt;

&lt;h2&gt;
  
  
  Your memory is the only incident database you have
&lt;/h2&gt;

&lt;p&gt;Big companies have many layers of institutional memory: incident trackers, on-call handbooks, people whose job is reliability. A small team has whatever is in the heads of the people who were there.&lt;/p&gt;

&lt;p&gt;That memory is surprisingly bad. Within a few weeks, details blur. What was the error? Which command fixed it? Did we fix the cause, or just restart the thing? Was it the same problem as March, or a different one that looked similar? Without a written record, every repeat outage starts from zero.&lt;/p&gt;

&lt;p&gt;And small teams change. Someone leaves, someone joins, someone goes on holiday during the next incident. The one person who remembers how to fix it is the one who isn't there.&lt;/p&gt;

&lt;h2&gt;
  
  
  "We don't have time" is the argument for doing it
&lt;/h2&gt;

&lt;p&gt;The usual objection is time. A small team is busy shipping, and writing documents feels like overhead.&lt;/p&gt;

&lt;p&gt;But consider the cost on the other side. An outage you've seen before, without notes, means rediscovering the diagnosis under pressure, often at a bad hour, while customers wait. Twenty minutes of writing after the first incident can save hours during the second.&lt;/p&gt;

&lt;p&gt;And the time doesn't need to be large. The value of a postmortem isn't in its length; it's in existing at all.&lt;/p&gt;

&lt;h2&gt;
  
  
  What a small-team postmortem looks like
&lt;/h2&gt;

&lt;p&gt;Forget the twelve-section enterprise template. For a small team, five headings are plenty:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight markdown"&gt;&lt;code&gt;&lt;span class="gu"&gt;# 2026-11-03: Checkout returned errors for 40 minutes
&lt;/span&gt;
&lt;span class="gu"&gt;## What happened
&lt;/span&gt;Short description in plain words, written so a new teammate understands it.

&lt;span class="gu"&gt;## Timeline
&lt;/span&gt;&lt;span class="k"&gt;- &lt;/span&gt;14:05 First alert (checkout synthetic check failing)
&lt;span class="k"&gt;- &lt;/span&gt;14:12 Started investigating
&lt;span class="k"&gt;- &lt;/span&gt;14:30 Found the cause
&lt;span class="k"&gt;- &lt;/span&gt;14:45 Fixed and confirmed

&lt;span class="gu"&gt;## Why it happened
&lt;/span&gt;The real cause, and the reasons it wasn't caught earlier.

&lt;span class="gu"&gt;## What we did to fix it
&lt;/span&gt;The exact commands, settings or changes. Copy-paste them.

&lt;span class="gu"&gt;## What we'll change
&lt;/span&gt;One to three concrete actions, each with a name next to it.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Write it within a day or two, while details are fresh. Keep it in your repository, in a &lt;code&gt;postmortems/&lt;/code&gt; folder, so it's searchable and lives next to the code.&lt;/p&gt;

&lt;p&gt;That's it. No meeting required, though a short call to read it together is worthwhile.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;The next postmortem is easier if you already have an outside timeline of when the site went down and came back.&lt;/strong&gt; &lt;a href="https://upordown.online" rel="noopener noreferrer"&gt;upordown.online&lt;/a&gt; keeps that record, and alerts you at the start of the incident.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  The rules that matter
&lt;/h2&gt;

&lt;p&gt;A few principles make even a short postmortem useful:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Blameless, genuinely.&lt;/strong&gt; In a team of two, blame is personal, and it kills honesty quickly. Write about what the system allowed to happen, not who made a mistake. "The deploy script didn't check disk space" rather than "Alex forgot to check disk space."&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Include the commands.&lt;/strong&gt; The most valuable part of an old postmortem is often the exact fix. "Cleared the logs" is far less helpful than the actual command you ran and the path.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Few actions, actually done.&lt;/strong&gt; Two actions completed beats ten actions forgotten. If an action isn't going to happen, don't write it down.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Note what went well.&lt;/strong&gt; If an alert fired quickly, or a runbook worked, say so. It tells you what to keep.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Link to it.&lt;/strong&gt; Reference the postmortem in the fix's commit or pull request, and in any related runbook. A document nobody can find is a document nobody reads.&lt;/p&gt;

&lt;h2&gt;
  
  
  The real payoff: patterns
&lt;/h2&gt;

&lt;p&gt;After a year, a small team with written postmortems has something remarkable: a list of every significant outage, why it happened and what was done. Read through it and patterns appear.&lt;/p&gt;

&lt;p&gt;Maybe half your outages trace back to one fragile dependency. Maybe they cluster around deploys, or around a particular time of month. Maybe alerts were consistently slow, or consistently fired for the wrong thing. These patterns are almost impossible to see from memory, and obvious on paper.&lt;/p&gt;

&lt;p&gt;That list is also a powerful tool when you need to argue for time to fix something. "This has caused four outages this year, here are the write-ups" is far more persuasive than "I think this is flaky."&lt;/p&gt;

&lt;h2&gt;
  
  
  When to write one
&lt;/h2&gt;

&lt;p&gt;Not every blip needs a document. A sensible rule for a small team:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Always:&lt;/strong&gt; anything customers noticed, anything that lost or corrupted data, anything involving security.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Usually:&lt;/strong&gt; anything that took longer than half an hour to fix, or woke someone up.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Sometimes:&lt;/strong&gt; a near-miss that could easily have been serious.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If you're unsure, write a short one. The worst outcome is a few paragraphs nobody needed.&lt;/p&gt;

&lt;h2&gt;
  
  
  Start with the last outage
&lt;/h2&gt;

&lt;p&gt;If your team has never written a postmortem, don't wait for the next incident. Write one for the most recent outage you can remember, using the five headings above. You'll be surprised how much you've already forgotten, which is exactly the point.&lt;/p&gt;

&lt;p&gt;Knowing when something broke and how long it lasted, which is the start of every timeline, is what we help with at &lt;a href="https://upordown.online" rel="noopener noreferrer"&gt;upordown.online&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Related reading
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://dev.to/upordownonline/postmortem-hypothetical-our-health-check-was-so-thorough-it-took-every-instance-out-of-the-load-158m"&gt;Postmortem (hypothetical): our health check was so thorough it took every instance out of the load balancer at once&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/upordownonline/our-monitor-said-100-uptime-while-the-cdn-served-a-6-hour-old-homepage-bk9"&gt;Our monitor said 100% uptime while the CDN served a 6-hour-old homepage&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/upordownonline/case-study-hypothetical-the-site-that-was-down-only-for-people-on-ipv6-and-every-monitor-said-14i"&gt;Case study (hypothetical): the site that was down only for people on IPv6, and every monitor said it was fine&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/upordownonline/rotating-an-api-key-or-database-password-a-12-point-checklist-so-the-rotation-isnt-the-outage-3aep"&gt;Rotating an API key or database password? A 12-point checklist so the rotation isn't the outage&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Does your team write anything down after an outage, and if you looked for notes on the last one, would you find them?&lt;/p&gt;

</description>
      <category>sre</category>
      <category>devops</category>
      <category>startup</category>
      <category>incidents</category>
    </item>
    <item>
      <title>How to get alerted when your DNS records change (before a typo takes your site and email down)</title>
      <dc:creator>upordown.online</dc:creator>
      <pubDate>Tue, 10 Nov 2026 08:29:11 +0000</pubDate>
      <link>https://dev.to/upordownonline/how-to-get-alerted-when-your-dns-records-change-before-a-typo-takes-your-site-and-email-down-g8k</link>
      <guid>https://dev.to/upordownonline/how-to-get-alerted-when-your-dns-records-change-before-a-typo-takes-your-site-and-email-down-g8k</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;Uptime monitoring from outside your own infrastructure that alerts you the minute your site goes down — &lt;a href="https://upordown.online" rel="noopener noreferrer"&gt;upordown.online&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Some of the worst outages start with a single edit in a DNS dashboard. Someone deletes an MX record while "cleaning up". A registrar transfer resets the nameservers. A well-meaning colleague points &lt;code&gt;www&lt;/code&gt; at a new server a day early. A domain verification TXT record disappears, and three weeks later a service quietly stops trusting you.&lt;/p&gt;

&lt;p&gt;The website check you already have might catch some of this, eventually. But DNS answers are cached, so a bad change can take hours to show up for some users, and email or verification records aren't covered by HTTP checks at all.&lt;/p&gt;

&lt;p&gt;So here's a simple approach: &lt;strong&gt;write down what your DNS records should be, check them regularly, and alert on any difference.&lt;/strong&gt; It takes about half an hour to set up.&lt;/p&gt;

&lt;h2&gt;
  
  
  What to watch
&lt;/h2&gt;

&lt;p&gt;Not every record matters equally. Start with the ones that break things:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;NS&lt;/strong&gt;: your nameservers. If these change unexpectedly, everything else is in someone else's hands.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A / AAAA&lt;/strong&gt; for the root domain and &lt;code&gt;www&lt;/code&gt;: where your website lives.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;CNAME&lt;/strong&gt; records for important subdomains: &lt;code&gt;app&lt;/code&gt;, &lt;code&gt;api&lt;/code&gt;, &lt;code&gt;shop&lt;/code&gt;, &lt;code&gt;status&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;MX&lt;/strong&gt;: where your email is delivered.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;TXT&lt;/strong&gt; records for SPF, DMARC and domain verifications.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;CAA&lt;/strong&gt; if you use it: which certificate authorities may issue certificates for you.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Step 1: Write down the expected records
&lt;/h2&gt;

&lt;p&gt;Create a plain text file listing each name, type and expected answers. Keep it in version control, so changes to it are reviewed like code.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;# expected-dns.txt  name type
example.com NS
example.com A
www.example.com CNAME
example.com MX
example.com TXT
_dmarc.example.com TXT
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;We'll generate the expected values from your current, known-good setup, rather than typing them by hand.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 2: Snapshot the current answers
&lt;/h2&gt;

&lt;p&gt;This script queries each record with &lt;code&gt;dig&lt;/code&gt; and prints a normalised, sorted result:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;#!/usr/bin/env bash&lt;/span&gt;
&lt;span class="c"&gt;# dns-snapshot.sh: print current answers for every record in expected-dns.txt&lt;/span&gt;
&lt;span class="nb"&gt;set&lt;/span&gt; -euo pipefail
&lt;span class="nv"&gt;RESOLVER&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"${RESOLVER:-1.1.1.1}"&lt;/span&gt;

grep -v &lt;span class="s1"&gt;'^#'&lt;/span&gt; expected-dns.txt &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="k"&gt;while&lt;/span&gt; &lt;span class="nb"&gt;read&lt;/span&gt; -r name &lt;span class="nb"&gt;type&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="k"&gt;do&lt;/span&gt;
  &lt;span class="o"&gt;[&lt;/span&gt; -z &lt;span class="s2"&gt;"$name"&lt;/span&gt; &lt;span class="o"&gt;]&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="nb"&gt;continue&lt;/span&gt;
  &lt;span class="nv"&gt;answers&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;$(&lt;/span&gt;dig +short @&lt;span class="s2"&gt;"$RESOLVER"&lt;/span&gt; &lt;span class="s2"&gt;"$name"&lt;/span&gt; &lt;span class="s2"&gt;"$type"&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; sort &lt;span class="o"&gt;|&lt;/span&gt; tr &lt;span class="s1"&gt;'\n'&lt;/span&gt; &lt;span class="s1"&gt;' '&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;
  &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"$name $type ${answers:-&amp;lt;none&amp;gt;}"&lt;/span&gt;
&lt;span class="k"&gt;done&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Run it once while everything is working, check the output by eye, and save it as your baseline:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;chmod +x dns-snapshot.sh
./dns-snapshot.sh &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; dns-baseline.txt
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Sorting matters. Many records, MX and NS especially, can come back in a different order each time, and you don't want an alert for that.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 3: Compare and alert
&lt;/h2&gt;

&lt;p&gt;Now a second script compares the current state with the baseline:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;#!/usr/bin/env bash&lt;/span&gt;
&lt;span class="c"&gt;# dns-check.sh: exit 1 and print a diff if any record changed&lt;/span&gt;
&lt;span class="nb"&gt;set&lt;/span&gt; -euo pipefail
&lt;span class="nv"&gt;current&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;$(&lt;/span&gt;mktemp&lt;span class="o"&gt;)&lt;/span&gt;
./dns-snapshot.sh &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="s2"&gt;"$current"&lt;/span&gt;

&lt;span class="k"&gt;if&lt;/span&gt; ! diff -u dns-baseline.txt &lt;span class="s2"&gt;"$current"&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="k"&gt;then&lt;/span&gt;
  &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"DNS CHANGED"&lt;/span&gt;
  &lt;span class="nb"&gt;exit&lt;/span&gt; 1
&lt;span class="k"&gt;fi&lt;/span&gt;
&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"DNS unchanged"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A non-zero exit and a readable diff are everything an alerting system needs.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;A record change is one thing; “is the site still up for everyone?” is the question that follows it.&lt;/strong&gt; &lt;a href="https://upordown.online" rel="noopener noreferrer"&gt;upordown.online&lt;/a&gt; answers that from outside your network the moment resolution starts to fail.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  Step 4: Run it on a schedule
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;On a server with cron&lt;/strong&gt;, run it every 10 minutes and send the output somewhere you'll see it. A tiny notifier that posts to a chat webhook, using &lt;code&gt;jq&lt;/code&gt; to build valid JSON:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;#!/usr/bin/env bash&lt;/span&gt;
&lt;span class="c"&gt;# notify.sh: post a file's contents to a chat webhook&lt;/span&gt;
jq -n --rawfile text &lt;span class="s2"&gt;"$1"&lt;/span&gt; &lt;span class="s1"&gt;'{text: $text}'&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt;
  curl -s -X POST -H &lt;span class="s1"&gt;'Content-Type: application/json'&lt;/span&gt; -d @- &lt;span class="s2"&gt;"$WEBHOOK_URL"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;*/10 * * * * &lt;span class="nb"&gt;cd&lt;/span&gt; /opt/dns-watch &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;./dns-check.sh &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; /tmp/dns.out 2&lt;span class="o"&gt;&amp;gt;&amp;amp;&lt;/span&gt;1 &lt;span class="o"&gt;||&lt;/span&gt; ./notify.sh /tmp/dns.out&lt;span class="o"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;On GitHub Actions&lt;/strong&gt;, a scheduled workflow is often easier, because it runs somewhere other than your own infrastructure:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;dns-watch&lt;/span&gt;
&lt;span class="na"&gt;on&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;schedule&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;cron&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"*/30 * * * *"&lt;/span&gt;
  &lt;span class="na"&gt;workflow_dispatch&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="na"&gt;jobs&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;check&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;runs-on&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;ubuntu-latest&lt;/span&gt;
    &lt;span class="na"&gt;steps&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;uses&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;actions/checkout@v4&lt;/span&gt;
      &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;run&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;sudo apt-get install -y dnsutils&lt;/span&gt;
      &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;run&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;./dns-check.sh&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A failing run shows the diff in the log, and you can use the platform's usual failure notifications. Scheduled workflows aren't guaranteed to run exactly on time, which is fine for this purpose.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 5: Decide what a "planned" change looks like
&lt;/h2&gt;

&lt;p&gt;The baseline will need updating whenever you change DNS on purpose. Make that part of the change:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Make the DNS change.&lt;/li&gt;
&lt;li&gt;Run &lt;code&gt;./dns-snapshot.sh&lt;/code&gt;, check the new output.&lt;/li&gt;
&lt;li&gt;Commit the updated &lt;code&gt;dns-baseline.txt&lt;/code&gt; with a message saying why.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Now every DNS change is either in your history with a reason, or it's an alert. That alone makes "who changed the MX record?" much easier to answer.&lt;/p&gt;

&lt;h2&gt;
  
  
  Things to watch out for
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Resolver caching.&lt;/strong&gt; Querying a public resolver gives you a cached answer, which may lag behind a real change. For fastest detection, query your &lt;strong&gt;authoritative nameservers&lt;/strong&gt; directly. Find them with &lt;code&gt;dig NS example.com +short&lt;/code&gt;, then use one as the resolver.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Records that legitimately rotate.&lt;/strong&gt; Some providers return different IP addresses on each query, for load balancing. For those, compare the CNAME target rather than the A records, or check only that answers fall within the provider's published ranges.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;TTL noise.&lt;/strong&gt; &lt;code&gt;dig +short&lt;/code&gt; omits TTLs, which is what you want. TTL values counting down would otherwise create a diff every time.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;One vantage point.&lt;/strong&gt; This checks DNS from one place. Geographic DNS setups can answer differently by location. If you use one, run checks from more than one region.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why this is worth it
&lt;/h2&gt;

&lt;p&gt;DNS changes are rare, which is exactly why they cause big outages: nobody has practised spotting a bad one. A baseline file and a diff turn a silent, slow-spreading failure into an alert within minutes, often before most caches have even picked it up.&lt;/p&gt;

&lt;p&gt;Watching the layers below your website, DNS included, is a big part of what we do at &lt;a href="https://upordown.online" rel="noopener noreferrer"&gt;upordown.online&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Related reading
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://dev.to/upordownonline/postmortem-hypothetical-our-health-check-was-so-thorough-it-took-every-instance-out-of-the-load-158m"&gt;Postmortem (hypothetical): our health check was so thorough it took every instance out of the load balancer at once&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/upordownonline/your-dns-ttl-is-why-the-outage-lasted-2-hours-set-it-before-you-need-it-meh"&gt;Your DNS TTL is why the outage lasted 2 hours: set it before you need it&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/upordownonline/how-to-fix-dnsprobefinishednxdomain-in-chrome-windows-mac-android-iphone-597a"&gt;How to fix DNS_PROBE_FINISHED_NXDOMAIN in Chrome (Windows, Mac, Android, iPhone)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/upordownonline/how-to-check-if-a-website-is-down-for-everyone-or-just-you-4cam"&gt;How to check if a website is down for everyone or just you&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Have you ever been caught out by a DNS change, and how long did it take before someone noticed?&lt;/p&gt;

</description>
      <category>dns</category>
      <category>devops</category>
      <category>monitoring</category>
      <category>bash</category>
    </item>
    <item>
      <title>Postmortem (hypothetical): our health check was so thorough it took every instance out of the load balancer at once</title>
      <dc:creator>upordown.online</dc:creator>
      <pubDate>Tue, 10 Nov 2026 08:20:59 +0000</pubDate>
      <link>https://dev.to/upordownonline/postmortem-hypothetical-our-health-check-was-so-thorough-it-took-every-instance-out-of-the-load-158m</link>
      <guid>https://dev.to/upordownonline/postmortem-hypothetical-our-health-check-was-so-thorough-it-took-every-instance-out-of-the-load-158m</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;Uptime monitoring from outside your own infrastructure that alerts you the minute your site goes down — &lt;a href="https://upordown.online" rel="noopener noreferrer"&gt;upordown.online&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;&lt;em&gt;This is a hypothetical postmortem. The company, timings and details are invented to illustrate a real and common failure pattern. It does not describe a real incident or real data.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Summary
&lt;/h2&gt;

&lt;p&gt;For 38 minutes on a weekday afternoon, our fictional company's API returned errors to every request. The underlying cause was a slowdown in the primary database, which on its own would have made the API slow, not unavailable.&lt;/p&gt;

&lt;p&gt;What turned slow into down was our health check. The &lt;code&gt;/health&lt;/code&gt; endpoint ran a database query. When the database slowed, the query exceeded the health check timeout on every instance at once. The load balancer and the orchestrator both concluded that every instance was unhealthy: the load balancer stopped sending traffic to all of them, and the orchestrator started restarting them. Restarting instances opened new database connections, adding load to the struggling database.&lt;/p&gt;

&lt;h2&gt;
  
  
  Impact
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;38 minutes of near-total API unavailability.&lt;/li&gt;
&lt;li&gt;Mobile and web clients showed errors; background integrations queued and retried.&lt;/li&gt;
&lt;li&gt;No data loss.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Timeline (all times local)
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;14:02&lt;/strong&gt; A large reporting query begins on the primary database, causing lock contention and slower queries across the board.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;14:04&lt;/strong&gt; API latency rises. Requests are slow but succeeding.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;14:05&lt;/strong&gt; &lt;code&gt;/health&lt;/code&gt; responses start exceeding the 2-second timeout. Each instance needs three consecutive failures to be marked unhealthy.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;14:06&lt;/strong&gt; All instances fail their health checks within the same 30-second window. The load balancer has no healthy targets and starts returning 503s.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;14:06 to 14:20&lt;/strong&gt; The orchestrator's liveness probe, which used the same &lt;code&gt;/health&lt;/code&gt; endpoint, restarts instances. Each restart opens a fresh connection pool. The database gets busier.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;14:21&lt;/strong&gt; On-call identifies the reporting query and cancels it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;14:25&lt;/strong&gt; Database latency recovers, but instances are mid-restart and the connection storm continues.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;14:40&lt;/strong&gt; Instances stabilise, pass health checks and return to the load balancer. Service restored.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Root cause
&lt;/h2&gt;

&lt;p&gt;One endpoint was being used to answer three different questions:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Liveness:&lt;/strong&gt; "Is this process stuck and in need of a restart?"&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Readiness:&lt;/strong&gt; "Should this instance receive traffic right now?"&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Dependency health:&lt;/strong&gt; "Is everything this service depends on working?"&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The &lt;code&gt;/health&lt;/code&gt; endpoint answered question 3, and both the load balancer (question 2) and the orchestrator (question 1) acted on it. A shared dependency failing made every instance look broken at the same moment, so the protective mechanisms removed all capacity together.&lt;/p&gt;

&lt;p&gt;Restarting a healthy process because its database is slow can't help; it can only add load.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;A thorough health check can create the outage it's meant to prevent; an outside monitor still has to watch the public URL.&lt;/strong&gt; &lt;a href="https://upordown.online" rel="noopener noreferrer"&gt;upordown.online&lt;/a&gt; tells you what users see, even when the load balancer's own probes are in a bad state.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  What went well
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;On-call found the long-running query quickly.&lt;/li&gt;
&lt;li&gt;External monitoring flagged the outage within a minute, from outside the infrastructure.&lt;/li&gt;
&lt;li&gt;Clients with retries and backoff recovered automatically once service returned.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  What went wrong
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Liveness depended on an external system.&lt;/strong&gt; A process that was perfectly alive got killed.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Readiness failed all-or-nothing.&lt;/strong&gt; No instance could stay in the pool, so a slowdown became an outage.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Restarts amplified the problem.&lt;/strong&gt; Each new instance opened connections to a database already struggling.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The reporting query ran against the primary&lt;/strong&gt; rather than a replica.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Action items
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;1. Split the endpoints.&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;GET /livez   -&amp;gt; 200 if the process can serve a trivial request. No dependencies.
GET /readyz  -&amp;gt; 200 if this instance is ready: started, config loaded, pools initialised.
GET /health/deps -&amp;gt; detailed dependency status, for dashboards and humans. Not for probes.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The liveness check answers only "is this process responsive?" It should almost never fail unless the process is genuinely stuck.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. Keep dependency checks out of liveness entirely&lt;/strong&gt;, and be very careful with them in readiness. If a dependency shared by every instance is down, removing every instance from traffic doesn't help users. Serving degraded responses or clear errors quickly is usually better than serving nothing.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. Fail open when everything fails.&lt;/strong&gt; Many load balancers can be configured so that if every target is unhealthy, traffic is sent to all of them anyway. That turns "every health check failed" into "try anyway", which is often the right call when the cause is shared.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;4. Make health checks cheap and bounded.&lt;/strong&gt; Short internal timeouts, no heavy queries, and results cached for a few seconds so a burst of probes doesn't become a burst of database queries.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;5. Limit restart storms.&lt;/strong&gt; Use startup delays and back-off for restarts, and cap connection pool sizes so a wave of restarts can't overwhelm the database.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;6. Move reporting queries to a replica&lt;/strong&gt;, with a statement timeout on the primary as a backstop.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="no"&gt;ROLE&lt;/span&gt; &lt;span class="n"&gt;api_user&lt;/span&gt; &lt;span class="k"&gt;SET&lt;/span&gt; &lt;span class="n"&gt;statement_timeout&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'5s'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;7. Monitor from outside, as well as inside.&lt;/strong&gt; Internal health checks tell the orchestrator what to do. An external check, from the user's side of the internet, tells you whether customers can actually use the service. They answer different questions; you need both.&lt;/p&gt;

&lt;h2&gt;
  
  
  Lessons
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;A health check is an instruction, not just a measurement.&lt;/strong&gt; Whatever consumes it will act on the answer. Design it around the action it triggers.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Correlated failures defeat redundancy.&lt;/strong&gt; Ten instances that all check the same database will all fail the same check at the same moment.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Automatic remediation needs a brake.&lt;/strong&gt; Restarting and removing capacity are safe when one instance is broken, and dangerous when the cause is shared.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Checking your service the way a customer would, from outside, is exactly what &lt;a href="https://upordown.online" rel="noopener noreferrer"&gt;upordown.online&lt;/a&gt; does. It's the view that stays honest when internal checks get confused.&lt;/p&gt;

&lt;h2&gt;
  
  
  Related reading
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://dev.to/upordownonline/your-health-check-returns-200-while-users-get-errors-7-health-check-mistakes-d84"&gt;Your health check returns 200 while users get errors: 7 health check mistakes&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/upordownonline/quick-tip-your-load-balancer-hides-a-dead-server-check-each-backend-with-curl-resolve-2gj6"&gt;Quick tip: your load balancer hides a dead server. Check each backend with curl --resolve&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/upordownonline/alert-after-1-failed-check-or-3-tuning-uptime-alerts-so-on-call-stops-ignoring-them-1od0"&gt;Alert after 1 failed check or 3? Tuning uptime alerts so on-call stops ignoring them&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/upordownonline/how-to-get-alerted-when-your-dns-records-change-before-a-typo-takes-your-site-and-email-down-g8k"&gt;How to get alerted when your DNS records change (before a typo takes your site and email down)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/upordownonline/two-developers-one-outage-no-postmortem-thats-how-the-same-outage-happens-again-1oi8"&gt;Two developers, one outage, no postmortem? That's how the same outage happens again&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/upordownonline/should-a-failed-health-check-restart-the-service-automatically-or-page-a-human-first-lets-argue-53pl"&gt;Should a failed health check restart the service automatically, or page a human first? Let's argue about it&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;What does your liveness probe actually check today, and what would happen if your database slowed down for ten minutes?&lt;/p&gt;

</description>
      <category>sre</category>
      <category>devops</category>
      <category>kubernetes</category>
      <category>postmortem</category>
    </item>
  </channel>
</rss>
