DEV Community: Kashif Manzer

The API your automation switched on is the one the attacker needs

Kashif Manzer — Wed, 14 Oct 2026 16:20:20 +0000

On October 7, Cisco published a batch of security advisories for NX-OS, the operating system that runs its Nexus data center switches. Buried in the batch were five flaws that let an unauthenticated attacker on the network run code as root on the switch. No login. No credentials. Just crafted network traffic, and the box that carries your whole data center fabric is compromised.

The part that stuck with me is not the severity. A CVSS 9.8 is a number we see every month. It is who turned the attack surface on.

The locked door that ships locked

Take NX-API first. NX-API is a JSON and XML interface over HTTP for managing the switch. Instead of typing commands into the CLI, your automation talks HTTP: Ansible playbooks, Terraform providers, SDN controllers, the CI pipeline pushing config at 2am. It is the switch's management API, built for machines, not humans.

Cisco ships it disabled. On Nexus 3000 and 9000 switches, NX-API sits off by default, and nothing turns it on except a working network: someone enabling it so the tooling can do its job.

CVE-2026-76471 is a heap buffer overflow in NX-API, reached through a crafted HTTP request. An attacker who can reach the management interface sends one bad request, and the parser that was built to serve your automation becomes the path to root on the device. There is no login step in the chain. The API was built for machines, and the attack is a machine sending bytes.

A quick explainer, because the words matter. A buffer overflow means the program reads more data into a fixed-size memory region than the region can hold. The extra data spills into neighbouring memory. A heap buffer overflow means the spilling happens in the heap, the program's dynamic memory area. When that memory belongs to privileged code, the attacker can overwrite the program's own control structures and steer it. Steering privileged code means root.

On the UCS 6300 fabric interconnects, the same CVE rides a different gate. There the vulnerable API is the UCS Manager XML API, enabled by default and impossible to switch off without breaking management. Exploitation there needs a valid low-privileged login, so Cisco rates that variant High rather than Critical. One CVE, two platforms, two different gates.

The monitoring plane is an attack surface too

The second cluster lives in NGOAM, Next Generation Operation, Administration, and Maintenance. NGOAM is the feature that verifies path health on Nexus switches: it checks that the data center fabric is actually carrying traffic the way you think it is. It is monitoring infrastructure, the kind of thing you enable and forget.

Three CVEs, each CVSS 9.8: CVE-2026-76485, CVE-2026-76486, CVE-2026-76501. All three are stack-based buffer overflows from improper input validation, reached with crafted IP packets sent to the switch. CVE-2026-76485 needs only NGOAM enabled. CVE-2026-76486 needs NGOAM plus SRv6 or an NV Overlay. CVE-2026-76501 needs NGOAM plus SRv6. (SRv6 is Segment Routing over IPv6, a way of steering traffic by encoding the path in the packet header. NV Overlay is network virtualization, the VXLAN-based overlay most data center fabrics run on.)

The pattern repeats: a feature that exists to let the network be managed and observed is the one parsing untrusted input in privileged code.

No workarounds

Cisco's position is unusually flat. For these advisories, the workaround section is effectively one sentence: there are none. No configuration change, no ACL, no knob that removes the exposure.

The temporary bridge is a Live Protect shield for CVE-2026-76471, which Cisco describes as a stopgap until upgrades can be scheduled. The permanent fix is the upgrade you keep postponing. The NX-API advisory does not print version strings; it sends operators to the Cisco Software Checker for the first fixed release on each platform and version.

At publication, Cisco's PSIRT said it was not aware of public announcements or malicious use of these flaws. The flaws surfaced during Cisco's internal security testing, which Cisco says combined its existing testing processes with frontier AI models. Make of that what you will. The bugs are real either way.

One line of wider context: the same October 7 publication cycle also carried a License On-Prem advisory with a CVSS 10.0 signature verification flaw, CVE-2026-76482. Separate product, separate advisory, same day, same lesson.

Which side of the line is your fleet on?

The audit is one command. On any Nexus 3000 or 9000 in standalone NX-OS mode:

show feature | include nxapi

If it comes back enabled, that switch has the management-plane exposure until it is upgraded. For NGOAM, the question is whether the monitoring feature is on, and whether SRv6 or NV Overlay rides alongside it.

The uncomfortable takeaway: this exposure is not a misconfiguration in the classic sense. Nobody typed a bad password or left a port open by accident. The features exist to let machines manage the switch, and the machine-centric workflow, your automation, is what enables them. Programmability is the attack surface. The safe default and the working network are the same decision, and right now the decision is an upgrade.

When was the last time you checked which features your fleet has on that you did not personally enable?

Further reading: Cisco's NGOAM advisory cisco-sa-ngoam-rce-LWKQ4BU, the Forkast deep dive on the NX-API flaw, and ThreatAft's full October 7 roundup.

How to stop a cron job from running twice

Kashif Manzer — Mon, 12 Oct 2026 16:11:04 +0000

How to stop a cron job from running twice

You have a cron job that runs every minute. It takes 90 seconds to finish.

That is the whole bug. Read it again. Cron does not check whether your last run finished. It fires on the schedule, blindly, every time. Minute one starts the job. Minute two starts a second copy while the first is still working. Minute three starts a third. They step on each other: double writes, duplicate emails, two workers chewing the same queue.

Everyone learns this the hard way. The fix is one idea: the job must take a lock before it works, and walk away politely if the lock is already taken.

Watch it happen

Here is a tiny Python job. It grabs an exclusive lock, then works for 8 seconds. I ran three copies of it:

import fcntl, time, sys

start = time.time()
with open("/tmp/demo.lock", "w") as f:
    try:
        fcntl.flock(f, fcntl.LOCK_EX | fcntl.LOCK_NB)
    except BlockingIOError:
        print(f"[{time.time()-start:.3f}s] another copy is running, exiting")
        sys.exit(1)
    print(f"[{time.time()-start:.3f}s] lock acquired, doing the work...")
    time.sleep(8)
    print(f"[{time.time()-start:.3f}s] work done")

The real terminal output:

--- instance 1 starts (foreground, holds lock 8s) ---
--- instance 2 starts while instance 1 holds the lock ---
[0.000s] another copy is running, exiting without doing the work
instance 2 exit code: 1
[0.000s] lock acquired, doing the real work...
[8.000s] work done
--- instance 3 starts after instance 1 finished ---
[0.000s] lock acquired, doing the real work...
[8.000s] work done

Instance 2 noticed the lock in under a millisecond and left. Instance 3, starting after instance 1 finished, went straight to work. No queueing, no piling up, no double work.

Why the lock cannot get stuck

This is the part that makes it trustworthy. The lock is not a flag in your program. It lives in the kernel. flock asks the Linux kernel to mark the file "locked by this process", and the kernel releases it the moment the process dies, for any reason.

Kill the job with Ctrl-C? Lock released. The job crashes? Lock released. Someone runs kill -9 and the process never gets to clean up? Lock released. The kernel always holds the other end of the rope, so there is no such thing as a stale lock file to clean up afterwards.

One honest caveat: this holds on a normal local disk. On a network file system, advisory locks can behave strangely depending on the mount and the protocol version. Keep your lock file on local disk and you sidestep the entire class of problems.

The same trick in Go

package main

import (
    "fmt"
    "os"

    "golang.org/x/sys/unix"
)

func main() {
    f, err := os.OpenFile("/tmp/report.lock", os.O_CREATE|os.O_RDWR, 0600)
    if err != nil {
        fmt.Fprintln(os.Stderr, "cannot open lock file:", err)
        os.Exit(1)
    }
    if err := unix.Flock(int(f.Fd()), unix.LOCK_EX|unix.LOCK_NB); err != nil {
        fmt.Fprintln(os.Stderr, "another copy is already running")
        os.Exit(1)
    }
    runTheReport()
}

LOCK_EX means exclusive, LOCK_NB means "do not wait, fail immediately". That second flag is the whole decision: waiting would just recreate the pileup, one job deep.

The laziest version

If the job is a shell script, you do not need code at all. The flock command wraps any program:

* * * * * /usr/bin/flock -n /tmp/report.lock /opt/jobs/report.sh

-n means non-blocking: if the lock is taken, flock exits immediately without running the script. One flag, overlaps gone.

Which pattern fits your job?

Skip the overlap (reports, syncs, backups): non-blocking lock, second copy exits. This is the common case.
Never skip, never overlap (queue drains): blocking lock, second copy waits its turn. Use LOCK_EX without LOCK_NB, but set a timeout so a genuinely stuck job cannot jam the line forever.
Overlap is fine (stateless, idempotent workers): no lock at all. Just be sure the work really is safe to run twice, because one day it will.

Cron fires blindly. That is not a bug in cron; it is the contract. The fix is eight lines and a file the kernel babysits for you.

What is the most surprising thing a double-running cron job ever did to you?

CircleCI was not hacked. Its database filled a disk with its own diary.

Kashif Manzer — Sun, 11 Oct 2026 16:12:46 +0000

CircleCI was not hacked. Its database filled a disk with its own diary.

Tuesday, October 6. You push a commit the way you have a thousand times, and the pipeline does not start. You check your config, because the config is always the first suspect. Nothing changed. You check the status page. It is not you. It is everyone.

Intermittent delays started around 13:30 UTC. From 15:30 to 20:30, pipeline and workflow and job starts were delayed across the platform. From 15:48 to 16:38, nothing started at all. Pipelines triggered from 16:02 onward failed. The UI went stale. Notifications stopped. Commit statuses never reached GitHub. For five hours, CircleCI could not reliably move work forward.

Nobody broke in. No deploy went wrong. No data was lost. The database behind CircleCI's workflow orchestration filled up a disk with its own diary, and then it could not write another word.

The diary every database keeps

Behind every pipeline you trigger sits an orchestration service. It tracks the state of every pipeline, every workflow, every job, and hands ready jobs to the machines that run them. All of that state lives in a database.

Databases keep a diary. Before a database applies any change to its actual data, it writes that change into a transaction log first. This is called write-ahead logging, and the idea is simple: if the power dies halfway through a write, the database replays the diary on restart and recovers every change. The log is the record of what happened, in order. It is what makes your database durable.

CircleCI's database keeps its diary on a dedicated storage volume, a disk reserved just for these logs. The logs are archived continuously: copied somewhere durable, then the space is reused. That only works if archiving keeps up with writing.

On Tuesday, routine maintenance ran against the database's largest tables. Big tables mean big changes, and big changes mean the diary fills up fast. The maintenance wrote transaction logs faster than the archiver could drain them. The dedicated volume filled to the top.

A database with a full transaction-log volume cannot accept writes. There is nowhere to record the next change, so every change waits. And a workflow orchestrator whose database cannot accept writes cannot move any work forward. No new pipeline state, no new job state, nothing handed to the execution fleet.

That is the whole incident in one sentence: the diary filled the disk, and the database went quiet.

The fix that made it worse for 50 minutes

The engineers could see where this was going. A database that cannot write to its log volume will eventually lock itself into read-only mode, and then recovery gets much harder. So they moved the transaction logs to the database's main storage volume, where there was room.

Moving a live database's transaction logs is not a copy and paste. It took the database offline for 50 minutes.

Sit with that for a second, because this is the part that stings. The fix for "the database is about to stop" was "take the database down ourselves, on purpose, for almost an hour." They traded an uncontrolled failure for a controlled one, because the controlled one had a known end. That is the kind of decision that only looks obvious from the outside. From the inside, at 16:00 UTC with every pipeline on the platform stuck, it is a brave call.

The second act

When the database came back, it processed work at about half its normal rate. Pipelines started moving again, but slowly, like traffic after a road reopens with one lane closed. At 18:19 UTC the engineers changed a database setting, and throughput recovered. Job start times were back to normal by 20:30 UTC. Customers whose jobs failed during the window were told to rerun them.

CircleCI published the postmortem on October 7. It is refreshingly plain about the chain: routine maintenance, archive lag, a full volume, a forced move, a slow recovery. No villains. Just a diary that outran its archivist.

The operator's checklist

If you run a database with write-ahead logging, and you almost certainly do, this incident hands you three things to check on Monday:

1. Monitor archive lag, not just disk used. "Disk 70% full" tells you nothing about whether archiving is keeping up. The number that matters is the gap between the newest log written and the newest log archived, and whether that gap is growing. Alert on the lag trend, not the disk gauge. Something shaped like this, with your own metric names:

- alert: TransactionLogArchiveFallingBehind
  # Fire when the archiver has not kept up for 15 minutes straight.
  expr: db_wal_archive_lag_seconds > 300
  for: 15m
  annotations:
    summary: "Log archiver falling behind on {{ $labels.instance }}"

2. Throttle heavy maintenance on your biggest tables. The maintenance that triggered this was routine. That is the point. Routine operations against your largest tables are the ones most likely to outrun your archiving. Run them in smaller batches, or run them while someone is watching the lag.

3. Size and watch the log volume like it is its own blast radius. A dedicated log volume is good design. It is also a single point of failure with a countdown on it. Know how fast it fills at your peak write rate, and make sure the person on call knows where that number lives before Tuesday afternoon.

The diary metaphor is doing real work here: the mechanism that saves your database during a crash is the same mechanism that takes it down when nobody is watching the archivist.

What is the equivalent in your stack? The safety mechanism that quietly becomes the failure mode when it falls behind?

GitHub Is Failing Your macOS Builds on Purpose This Month

Kashif Manzer — Fri, 09 Oct 2026 16:11:19 +0000

GitHub Is Failing Your macOS Builds on Purpose This Month

Monday morning. You open your laptop and your CI is red.

Nobody shipped anything over the weekend. The diff is empty. You re-run the workflow, and it fails again. You re-run it once more for luck, and it fails a third time. Then, late in the afternoon, with no changes at all, it turns green.

Your first instinct is to blame the cache. Your second instinct is to blame yourself. Both are wrong.

GitHub did this on purpose.

What a brownout is

GitHub is retiring the macOS 14 runner image on November 2, 2026. That is the Sonoma-based virtual machine your workflow gets when the job says runs-on: macos-14.

Before the image goes away, GitHub is running brownouts: scheduled windows where every job on the macOS 14 labels fails on purpose. Not because something is broken. To make something visible.

A brownout is the opposite of an outage. An outage is failure you did not plan. A brownout is failure you scheduled, announced, and documented, because you are more scared of the quiet kind. The quiet kind is a workflow that pins an old image and nobody notices until the retirement date, when it fails for real and there is no window left to fix it.

The schedule

There are eight windows, each ten hours, 14:00 UTC to 00:00 UTC:

Oct 5 and 6 (this already happened)
Oct 12 and 13
Oct 16 and 17
Oct 19 and 20
Oct 23 and 24
Oct 26 and 27
Oct 29 and 30
Oct 30 and 31

In Pacific time that is 7:00am to 5:00pm. Most of the American workday falls inside the failure window. The windows cluster toward the end of the month, with two of them back to back, which is the part where patience runs out.

Inside a window, any job on macos-14, macos-14-large, or macos-14-xlarge fails deliberately. It does not matter that your workflow is fine. The failure is the message.

This applies to GitHub Actions and to Azure DevOps, because Microsoft-hosted agents are built from the same images.

Why the scheduled builds get hit first

Here is the cruel part of the design. The builds that turn red first are the ones nobody is watching.

Cron-scheduled workflows, nightly builds, matrix jobs that run across a fleet of repos at midnight: they fire right inside the window, and the first anyone hears about it is a red badge in the morning. A developer pushing at 10am sees the failure immediately and starts debugging. A scheduled job failing at 2am just sits there, red and unexamined, until standup.

That asymmetry is exactly why GitHub does it this way. A brownout that only ever failed while humans were watching would be ignorable. One that fills your mornings with red across repos you forgot about is not.

The quieter killer: the queue

There is a second, sneakier effect, and it runs even outside the windows. Since the deprecation was announced in July, GitHub has been allowed to reduce capacity for the macOS 14 runners. Fewer machines serving the same jobs means longer queue times.

So your macOS 14 jobs might not just fail in bursts. They might also sit waiting in the queue, for no reason your workflow file explains, getting slower and flakier all month. If your builds have been mysteriously slow since summer, this is probably why, and the brownouts are just the part GitHub put on a calendar.

The fix is one search away

Find every workflow that still asks for the old image:

grep -rn "macos-14" .github/workflows/

Then move each one to a supported label:

jobs:
  build:
    runs-on: macos-15   # or macos-latest

GitHub's named migration targets are macos-latest (currently macos-26), macos-15, and their -xlarge counterparts.

One warning before you swap the label and walk away. This is a real OS jump, not a version bump. Sonoma to Sequoia or newer means different Xcode and toolchain versions, different Homebrew and Clang versions, and anything your build quietly pinned to Sonoma behavior. Treat the first green run on the new image as suspicious. Run the full suite, check the artifacts, and re-verify anything that smells like it depended on the old machine.

The pattern worth keeping

Step back from the calendar for a second. What GitHub is doing here is chaos engineering with a courtesy notice. The industry spent a decade learning that the only way to find out what depends on a thing is to take the thing away on a schedule you control, rather than wait for it to disappear on a schedule you do not.

Netflix kills its own servers in the middle of the day for the same reason. GitHub kills your builds for ten hours at a time for the same reason. The lesson generalizes: every dependency you cannot name is a brownout you did not schedule.

November 2 is the real deadline. The brownouts are just the rehearsal, and the next rehearsal starts Monday at 7:00am Pacific.

So here is my question for you: what in your stack is still pinned to macOS 14, and why?

Your plan says (known after apply). OpenTofu 1.13 lets you talk back

Kashif Manzer — Wed, 07 Oct 2026 16:10:02 +0000

Every person who has reviewed an infrastructure plan knows the feeling.

You run tofu plan on a Friday afternoon. The output scrolls past. A third of the values you actually care about say the same thing:

+ id = (known after apply)

So you squint. You reason from experience. You say "yes, that looks right" to a plan that is, in the strictest sense, guessing. Then you apply, and the guessing continues on your side until the resources exist.

OpenTofu 1.13 shipped on September 30 (with a 1.13.1 patch on October 1, security support to August 2027) and it does something about that. It gives you a way to tell the planner what you already know.

Why the plan can't know

First, the mechanism, because it explains everything else.

OpenTofu's plan is a prediction. It walks your configuration, calls the providers, and works out what would change without changing anything. That last part matters: it cannot call the cloud API to actually create things, because creating things is the whole point of apply.

And that is where the unknowns come from. Plenty of values only exist after the remote system acts. An instance's ID, an ARN, the IP a load balancer will be assigned: these are chosen by the cloud API during apply. At plan time, OpenTofu has no way to compute them, so it marks them (known after apply) and does the best it can with the rest.

This is honest engineering. The problem is that unknowns poison everything they touch. If a value is unknown, any expression built on it is usually unknown too. Compare an unknown VPC ID to null and the result is unknown. Feed it into another module and the plan for that module goes murky. One unknown upstream turns into a wall of unknowns downstream, and you are back to squinting.

Tell the planner what you know

Here is the new idea. You, the module author, often know things the provider cannot promise.

Take a VPC ID from the AWS provider. During plan it is unknown. But you know something about it: it will never be null, and every real VPC ID starts with vpc-. The provider's API documentation guarantees this. OpenTofu just had no way to hear it.

The 1.13 assume... family of functions is a megaphone for exactly that:

output "vpc_id" {
  value = assumenotnull(
    assumestringprefix(
      aws_vpc.example.id,
      "vpc-",
    )
  )
}

Read it as a sentence: "I assume this is not null, and I assume it is a string starting with vpc-."

Now any expression that compares this output to null or to the empty string gets a known answer at plan time (false), instead of an unknown. If a downstream module validates that the VPC ID starts with vpc-, that validation now runs during the plan phase, where you can see the mistake before anything is created.

Two more worth knowing: convert tells OpenTofu the expected type of a value it cannot infer (for example, the result of jsondecode on something dynamic), so downstream attribute access gets a known type. And assumeequal declares the expected final value outright.

The catch: your promise gets checked

This is the part that matters most, so say it plainly.

The planner takes your word for it during plan. But at apply time, once the real values arrive, OpenTofu checks. If you promised that an unknown string starts with vpc- and it turns out to start with subnet-, the assumestringprefix call fails during the apply.

That is the right design. An unchecked assumption would turn the plan into fiction. This way the hint is verified by reality, and the cost of lying to the planner is a failed apply rather than silent breakage.

The rule of thumb: only promise what the provider or the vendor API documentation actually guarantees. "The docs say VPC IDs always look like this" is a promise. "In my experience they usually look like this" is not.

A few other things in 1.13

This was a short release cycle. The team cut it short on purpose to line up with the Go release schedule, which stretches the security support window to August 2027. So besides the unknown-value work, there are a couple of early experiments and some housekeeping:

Built-in linting, first pass. tofu plan -lint=all runs four rules: variables missing a type, count used where the newer enabled meta-argument fits, unused variables, unused locals. Very early. Expect it to grow.
Symbol Libraries, experimental. A way to share values, functions, and type aliases across configurations, separate from modules. The shape is still evolving.
Windows on ARM64 is now an official platform.
Before you upgrade: base64gzip output changed slightly, which can make OpenTofu propose diffs on resources that use it. WinRM support for provisioners is gone (use OpenSSH for Windows). This is the last series with official 32-bit builds, and macOS builds now need Ventura or newer.

Why this is bigger than it looks

The quiet problem with (known after apply) was never just aesthetics. Unknown values blind automated checks. Some teams run the machine-readable plan through policy-checking software, and every unknown forces a choice: trust optimistically and risk a violation, or block conservatively and hold up changes that would have been fine.

The assume... functions give module authors a way to sharpen the plan's vision, one honest guarantee at a time. They do not remove uncertainty. They let you say, precisely, where the uncertainty ends.

Have you ever been burned by approving a plan full of unknowns? What was the one that got you?

Codecov went down today over an expired certificate: how TLS renewal actually fails

Kashif Manzer — Mon, 05 Oct 2026 16:14:28 +0000

Codecov went down today over an expired certificate: how TLS renewal actually fails

This morning, Codecov's status page lit up. Frontend and backend, both down. The cause, identified within two minutes of the first update: an expired SSL certificate. The incident timeline, which begins at 06:53 UTC today, goes from "investigating" to "identified: SSL certificate expiration" in two updates.

Yesterday it was Sprinklr. Same failure. An expired certificate knocked part of their production offline for 23 minutes before the team renewed and redeployed it.

Two certificate-expiry outages in 24 hours. And this is not a rare disease. This is the common cold of web operations, and it keeps hospitalizing serious companies.

Your servers are fine. That is the problem.

Here is the strange part of every certificate outage: nothing is actually broken on the server side. The app is running. The database is answering. Every internal health check is green.

The failure happens inside the visitor's browser, in the split second before any page loads. The browser asks for the certificate, sees the expiry date has passed, and refuses to continue. Your monitoring watched the server. The server was never the patient.

This is why the outage always feels so disorienting. The team stares at green dashboards while users stare at a red warning page. Two different realities, and the user's one is the only one that counts.

The hall of fame nobody wants to join

The roll call of serious companies taken down by this is long enough to be embarrassing.

In December 2018, an expired certificate at Ericsson knocked out mobile data for tens of millions of O2 and SoftBank customers across the UK and Japan. O2 users were offline for most of the day, and the damage settlement that followed was reported at around 100 million pounds (source, source).

In February 2020, Microsoft Teams went down for roughly three hours on an expired certificate. Spotify lost about an hour the same year when a TLS certificate lapsed, and the team only found it after a Cloudflare engineer spotted the expired cert (source).

LinkedIn let a certificate on its lnkd.in shortener expire twice, in 2017 and again in 2019. The second time, the renewed certificate had been issued days earlier. It just never got installed on the server (source).

Every one of these teams had smart people and real infrastructure. The certificate still won.

How renewal is supposed to work

A quick primer, because the failure modes only make sense once you see the machinery.

A TLS certificate is a file with an expiry date stamped inside it. Browsers trust it only between its start date and its end date. Public certificates used to last a bit over a year. Now the industry is shrinking that: the CA/Browser Forum cut the maximum validity from 398 days to 200 days starting March 2026, so certificates issued that March started expiring this month (industry reporting). That is the wave we are standing in right now.

Renewal is meant to be automatic. The standard protocol is called ACME. Your server proves it controls the domain, the certificate authority issues a fresh certificate, and a small agent on your machine installs it. Let's Encrypt made this free and ubiquitous. In Kubernetes, a tool called cert-manager does the same job for cluster workloads.

There are two common ways to prove domain control: answer a challenge file over HTTP (called http-01), or create a DNS record (dns-01). Then the agent reloads the web server so the new certificate actually gets served.

That last step is where the bodies are buried.

The five ways renewal silently fails

1. Renewed, but never reloaded. The new certificate lands on disk. The old one stays in the server's memory. Nginx, Apache, or your Go service keeps serving the expired file until someone restarts or reloads the process. This is almost certainly what happened to LinkedIn in 2019: the renewed certificate existed, it just was not the one being served.

2. The copy you forgot. Modern setups terminate TLS in several places: the CDN edge, the load balancer, the origin server. Renewal automation often covers exactly one of these. The edge gets a fresh certificate while the origin's copy quietly ages out, or the other way around. Every layer you did not automate is a timer you are not watching.

3. The DNS token died quietly. DNS-01 challenges need an API token for your DNS provider. Tokens get rotated, scopes get narrowed, the person who created it leaves the company. The renewal job starts failing, and it fails in the one place nobody reads: the output of a cron job. Sixty days of silent failures, then the expiry date arrives.

4. The root you did not know you depended on. Sometimes it is not your certificate that expires but the root it chains to. When Let's Encrypt's old cross-signed root expired, apps like Shopify and Slack saw outages on older devices that did not trust the new root (source). Your certificate can be perfectly fresh and still untrusted, because trust is a chain and chains have more than one link.

5. Monitoring that checks the wrong thing. Most uptime checks hit an HTTP endpoint or a health port. Those pass with an expired certificate. The check that catches this has to complete a TLS handshake from the outside and read the expiry date. Almost nobody's default monitoring does that, so the first alert is always a user.

None of these are exotic. Every one of them is a Tuesday.

A checker that reads the expiry from the outside

The fix is boring, which is why it works. Check every public endpoint from the outside, read the certificate's expiry date, and page someone while there are still weeks left, not minutes.

There is one subtlety that bites people: a normal HTTPS client refuses to talk to a server with an expired certificate, so your checker has to deliberately skip validation. It is not checking identity here, only reading the date. Without that, the handshake fails on an expired cert and you learn nothing.

Here is the whole thing in Go, using only the standard library:

package main

import (
    "crypto/tls"
    "fmt"
    "time"
)

// daysLeft opens a TLS connection the way a browser would,
// reads the certificate's expiry date, and reports how many
// days are left. InsecureSkipVerify is deliberate: we are not
// checking identity here, only reading the date.
func daysLeft(host string) (int, time.Time, error) {
    conn, err := tls.Dial("tcp", host+":443", &tls.Config{
        InsecureSkipVerify: true,
    })
    if err != nil {
        return 0, time.Time{}, err
    }
    defer conn.Close()

    certs := conn.ConnectionState().PeerCertificates
    if len(certs) == 0 {
        return 0, time.Time{}, fmt.Errorf("server presented no certificates")
    }
    expiry := certs[0].NotAfter
    return int(time.Until(expiry).Hours() / 24), expiry, nil
}

func main() {
    domains := []string{
        "example.com",
        "api.example.com",
        "status.example.com",
    }
    for _, d := range domains {
        left, expiry, err := daysLeft(d)
        if err != nil {
            fmt.Printf("%-20s CHECK FAILED: %v\n", d, err)
            continue
        }
        status := "ok"
        if left < 30 {
            status = "RENEW SOON"
        }
        if left < 0 {
            status = "EXPIRED"
        }
        fmt.Printf("%-20s %s: %d days left (expires %s)\n",
            d, status, left, expiry.UTC().Format("2006-01-02"))
    }
}

Put your real domains in the list, not just the marketing site. The API subdomain, the status page, the webhook receiver, the admin panel. The certificate that takes you down is always the one you forgot you had.

Then add the alerting rule that actually works: warn at 30 days, page at 14, treat 7 as an incident. A certificate is free to renew. There is no reason to ever be surprised by its expiry.

The wave is just starting

Back to the timing, because it matters. With maximum validity now at 200 days, every team renews roughly twice as often as they used to. Twice as many renewal windows, twice as many chances for one of the five silent failures above. The teams that treated renewal as "set and forget" are about to learn what "forget" costs, twice a year, on a schedule.

Codecov and Sprinklr, 24 hours apart, might be the first drops of that rain. Or they might be coincidence. Either way, the mechanism does not care about the calendar. It only cares whether someone is watching the expiry date from the outside.

Check yours today. It takes five minutes, and the checker above is most of the work already done.

What is the most embarrassing cause of an outage you have ever seen? I want to hear the one that made the whole team go quiet.

A broken cable on the ocean floor slowed the internet for a whole country

Kashif Manzer — Sun, 04 Oct 2026 16:10:23 +0000

A broken cable on the ocean floor slowed the internet for a whole country

Friday, October 2, 2026. All over the Philippines, people noticed the same thing: the internet was slow. Videos buffered. Pages took their time. Calls stuttered.

It was not their routers. It was not their ISP throttling them. It was a broken cable lying on the bottom of the sea.

By that evening, the advisories were out. Converge, PADECO, even the network help desk at UP Diliman, all warned subscribers about the same thing: the submarine cable segments connecting the Philippines to Singapore and Hong Kong were down. The country's Department of Information and Communications Technology said it was investigating the cause. Sabotage was on the list of possibilities, but officials said it was far too early to draw conclusions.

And then this quiet detail from PLDT, one of the country's biggest carriers: it had lost about 300 gigabits per second of capacity, and it had more than enough spare elsewhere to cover the loss.

That one sentence contains almost the entire story of how the internet actually works. Let me unpack it.

The internet is a maritime business

Here is the fact that surprises almost everyone: when your data travels to another continent, it almost certainly goes through the ocean. Submarine cables carry over 99 percent of all intercontinental data traffic, according to TeleGeography, the research firm that maps them. Satellites handle a tiny fraction. They always have.

There are nearly 700 cable systems down there, with about 1,900 spots where they come ashore. Each cable is roughly as thick as a garden hose. Inside are strands of glass, thinner than a human hair, and data travels through them as pulses of light. Small devices called repeaters, spaced along the cable, boost the signal so it survives the thousands of kilometers between continents.

And these cables break. A lot. Roughly 150 to 200 times a year worldwide. The cause is usually mundane: a ship's anchor dragging across the seabed, a fishing trawler's gear snagging a line, an earthquake, or plain equipment failure. The ocean floor is crowded and violent.

So why is the internet not breaking every week? Because it was built assuming this would happen.

Why these two cables mattered

Singapore and Hong Kong are not random dots on the map. They are two of Asia's biggest internet intersections. Networks from all over the region meet there to exchange traffic, because it is cheaper and faster to hand data to a neighbor than to send it the long way around. Engineers call these meeting points hubs, and the practice of exchanging traffic there is called peering.

A lot of the Philippines' traffic to the rest of the world normally passes through those two hubs. When the segments to both went down at once, it was like losing two major interchanges on the same day.

Why the internet got slow instead of dying

Here is the key question: if the cables broke, why did anything work at all?

Because the internet does not trust any single road. Philippine ISPs lease capacity on many cable systems heading in different directions: toward Hong Kong, Japan, Guam, the US west coast. When two paths died, traffic moved to the rest.

Every network on the internet constantly tells its neighbors which routes are alive. The system they use for this is called BGP, the Border Gateway Protocol. You can think of it as every intersection in the world posting live detour signs. A router in Manila says, "I can reach Singapore through this path." When the path dies, the announcement is withdrawn, the signs update everywhere within minutes, and traffic finds the next best route.

This is why slow, not dead. Picture a highway losing two lanes at rush hour. The same number of cars now squeezes through the lanes that remain. That is congestion. And some of the surviving routes are physically longer: instead of Manila to Singapore to its destination, a packet might travel Manila to Tokyo to the US west coast and back. Light is fast, but distance still counts. The delay has a name every backend engineer knows: latency. A few hundred extra milliseconds is the difference between a snappy call and a buffering one.

Globe, one of the country's big carriers, announced it was rerouting data traffic. That sentence is the press-release version of "we updated the detour signs."

A simplified sketch of what happened:

Manila ====== Singapore ====== rest of the world     X  broken
Manila ====== Hong Kong  ===== rest of the world     X  broken
Manila ====== Tokyo ====== rest of the world            longer, congested

Same cars. Fewer lanes. Longer detours.

How they find the break

This is my favorite part. When a cable snaps, the operator does not send a diver. They send a pulse of light down the fiber and listen for the echo.

It works like sonar. A break in the glass reflects a tiny bit of the light back. By measuring how long the echo takes to return, engineers can pinpoint the break to within meters, from a control room thousands of kilometers away. The tool has a fancy name, an optical time-domain reflectometer, but the idea is ancient: shout into a canyon, time the echo, know the distance.

Then a cable ship sails to the spot. It drags a grappling hook, or sends down an underwater robot, to haul the cable up from the seabed. Technicians splice in a new section of fiber, seal it, and drop it back down. It takes days to weeks, depending on weather and how far the nearest ship is. There are only about two dozen of these ships on standby around the world.

And while the ship is sailing, the internet just routes around the hole. You feel it as slowness. Your video call has no idea why. All it knows is that the road got longer.

What "300 gigabits of capacity" actually means

Back to PLDT's quiet sentence. ISPs do not own most of these cables. They buy capacity: slices of a cable's total throughput, like an airline buying seats on a route it does not own. Three hundred gigabits per second is roughly the room for tens of thousands of simultaneous video streams. Losing it is a real hit.

The second half of the sentence is the interesting part: "we have more than enough to cover it." That is redundancy, and it is the most boring and most important word in infrastructure. PLDT had paid for spare capacity on other routes, every day, for years, for exactly this moment. Redundancy is insurance you buy daily and are grateful for on the worst day.

The takeaway

The internet feels like weather, something in the air. It is not. It is glass tubes on the ocean floor, a small fleet of repair ships, and routers swapping detour signs with each other thousands of times a second.

Every time your connection slows and nobody can tell you why, remember the Philippines on October 2. Somewhere under the sea, possibly, a ship's anchor did it.

Here is a small experiment for this week. Run traceroute (or mtr on Linux, tracert on Windows) against a site you use every day and read the city names in the hops. The journey is more literal than you think. What is the strangest route your packets have ever taken? Tell me in the comments.

Why your retries need jitter: the thundering herd, explained

Kashif Manzer — Fri, 02 Oct 2026 16:10:14 +0000

Picture this. It is 2:14 in the morning and your checkout service starts returning errors. Every request fails for about twenty seconds, then your deploy finishes rolling back and the error rate drops to zero. You exhale.

Then the dashboards spike again. Harder than before. Nothing changed on your end. The service was healthy. So what knocked it over the second time?

Your own services did. Every client that failed during those twenty seconds had been retrying, and the moment the service came back, they all arrived at once. The recovery got flattened by a second wave made entirely of retries. That second wave has a name engineers have used for decades: the thundering herd.

The naive retry

The first retry loop most of us write looks like this: call the service, and if it fails, try again. Immediately. Maybe three times.

For one client, that is harmless. But failures in distributed systems are rarely lonely. A deploy, a network blip, a database failing over: these knock out hundreds or thousands of clients in the same second. Now all of them retry in the same second too. The service comes back to life and instantly receives ten times its normal traffic, all at once, and falls over again. Your retry logic just turned a twenty second outage into a rolling one.

(A quick term check: a distributed system is just a bunch of services talking to each other over a network instead of one big program. The network between them is the part that fails in interesting ways.)

Fixed delays do not break the herd

The obvious fix is to wait between retries. Retry every five seconds, say. That helps the service, but the clients are still marching in lockstep. They all failed together, so they all wait five seconds together, and they all retry together. The herd is still a herd. It is just a slower one.

Exponential backoff spreads them out

The next step is to make each wait longer than the last: one second, then two, then four, then eight. This is exponential backoff, and it does real work. A short blip only costs your clients a second or two, while a long outage backs everyone further and further off, giving the service room to breathe.

But there is still a crack in it. Clients that failed at the same moment are on the same schedule. Their retries still land in the same windows, just further apart. The herd is thinner now, but it is still moving as one.

Jitter scatters the herd

The final ingredient is randomness. Instead of waiting exactly four seconds, wait a random amount of time up to four seconds. One client waits half a second, another waits three, another waits the full four. The synchronized schedule dissolves. The retries that used to arrive as one hammer blow now arrive as a drizzle the service can actually absorb.

This combination has a name too: exponential backoff with jitter. The AWS Architecture Blog wrote the classic post on it ("Exponential Backoff And Jitter"), and the formula most people use is called full jitter:

wait = a random number between 0 and the smaller of (cap, base times 2 to the power of attempt)

The cap keeps the wait from growing forever. The base times 2 to the power of attempt is the exponential backoff. The random part is the jitter. All three matter, and most homegrown retry loops I have seen are missing at least one of them.

The code

Here is a retry helper in Go that does all three. Read it top to bottom. There is nothing clever in it, which is the point.

package main

import (
    "context"
    "math/rand"
    "time"
)

// retryWithBackoff runs fn, retrying failures with exponential
// backoff and full jitter. It gives up after maxAttempts.
func retryWithBackoff(ctx context.Context, maxAttempts int, fn func() error) error {
    const base = 100 * time.Millisecond
    const maxWait = 5 * time.Second

    var err error
    for attempt := 0; attempt < maxAttempts; attempt++ {
        if err = fn(); err == nil {
            return nil
        }
        // Exponential part: 100ms, 200ms, 400ms, ...
        wait := base << attempt
        if wait > maxWait {
            wait = maxWait
        }
        // Jitter part: random between 0 and wait.
        jitter := time.Duration(rand.Int63n(int64(wait)))
        select {
        case <-ctx.Done():
            return ctx.Err()
        case <-time.After(jitter):
        }
    }
    return err
}

A few things worth noticing. The wait doubles every attempt (that << is a bit shift, a cheap way to multiply by powers of two). The jitter picks a random wait anywhere from zero up to that doubled value, so two clients on the same attempt number almost never sleep the same amount. And the whole thing respects the context, so a caller can cancel the retries instead of letting them run to the bitter end. Modern Go seeds its random number generator for you, so there is no seeding ceremony needed.

The same idea in Python is even shorter:

import random
import time

def retry_with_backoff(fn, max_attempts=5, base=0.1, cap=5.0):
    for attempt in range(max_attempts):
        try:
            return fn()
        except Exception:
            if attempt == max_attempts - 1:
                raise
            wait = min(cap, base * (2 ** attempt))
            time.sleep(random.uniform(0, wait))

Same three ingredients. Exponential growth, a cap, and a random pick inside the window.

Three ways people still get this wrong

First, retrying things that should never be retried. Only retry failures that might succeed next time: timeouts, 503s, 429s. Do not retry a 400; the request itself is broken and trying again changes nothing. And only retry requests that are safe to repeat. Engineers call this idempotent: doing it twice has the same effect as doing it once. A retried "charge this card" that is not idempotent can charge it twice.

Second, backoff without a cap. Left alone, base times 2 to the power of attempt grows fast. By attempt ten you are waiting almost two minutes between tries. Pick a ceiling that matches how long your caller can actually afford to wait.

Third, retries with no overall deadline. A retry loop that never gives up will keep a dying request alive long past the point where anyone still wants the answer. Bound the attempts, or bound the total time, and let the caller decide what failure means.

The takeaway

Retries are load. Every retry policy you write is a decision about how much extra traffic your system will generate at the exact moment it is least able to handle it. Backoff and jitter are how you make that extra traffic survivable. The next time you write a retry loop, check for all three: growing waits, a cap, and randomness. If any one is missing, you are one bad twenty seconds away from finding out why.

Have you ever watched a retry storm flatten a service that was already recovering? What did the graphs look like?

How one user's StatefulSet edit could land a pod in someone else's namespace

Kashif Manzer — Wed, 30 Sep 2026 16:09:34 +0000

How one user's StatefulSet edit could land a pod in someone else's namespace

On September 23, Kubernetes shipped fixes for CVE-2026-2270, a confused deputy bug in the StatefulSet controller. The short version: a user with write access to StatefulSets and ControllerRevisions in one namespace could get the cluster itself to create a pod in a different namespace. The cluster did the crossing. The user just pointed.

If you run a shared cluster, this one is worth understanding, because the pattern behind it shows up everywhere.

The wall everyone trusts

Namespaces are the walls of Kubernetes. You give team A their own namespace, team B theirs, and RBAC rules make sure nobody reaches across. A user who can only touch team-a should never be able to create anything in team-b. That is the whole promise.

The bug broke that promise. A namespace-scoped user, someone RBAC confined to a single namespace, could end up with a pod running in another namespace entirely. They did not hack RBAC. They did not escalate their own permissions. They asked a more powerful program to do it for them, and it agreed.

That is what "confused deputy" means. The deputy is the StatefulSet controller, a cluster-wide process with the keys to every namespace. It was confused because it trusted data it should not have trusted.

Suspect one: the ControllerRevision

To follow the trail, you need to know what a ControllerRevision is. It is one of those Kubernetes objects most people never touch directly, but it is doing quiet work in the background.

When you update a StatefulSet, say you change the container image from v1 to v2, the controller does not throw away the old configuration. It saves a snapshot of it in a ControllerRevision object. If the rollout goes wrong, the controller can roll back by restoring from that snapshot. Think of it as the "previous versions" feature of a document editor, but for your stateful workloads.

Here is the part that matters: ControllerRevisions are regular API objects, and RBAC can grant users write access to them. In some clusters, especially multi-tenant ones, developers get broad permissions over the workload objects in their namespace, including StatefulSets and their ControllerRevisions. Nothing about that looks dangerous on its own.

Suspect two: the cluster-wide controller

The StatefulSet controller runs inside kube-controller-manager. It is not scoped to any namespace. It cannot be, because its job is to watch StatefulSets everywhere and reconcile them: compare what exists with what is declared, and fix the difference by creating, updating, or deleting pods.

So you have a process with authority over every namespace, and it reads snapshots that ordinary users are allowed to write. If those snapshots can smuggle instructions the controller will act on, the user has effectively borrowed the controller's authority. That is exactly what happened.

The trick

When the controller restored a StatefulSet from a ControllerRevision, it restored more than it should have. The fix tells the story plainly: patched versions ensure that only the spec field is restored from ControllerRevisions. Which means the old code was restoring additional fields, and those extra fields are what carried the attack across the namespace wall.

An attacker with write access to both objects in their own namespace could craft a ControllerRevision containing a StatefulSet whose metadata pointed at a different namespace. The controller, doing its honest reconciliation work, would read the snapshot, restore the fields, and create the pod where the snapshot said to. The attacker would have full control over the resulting pod's metadata and specification, including the namespace selection.

The user's permissions never changed. They could still only write in their own namespace. But the deputy they tricked could write anywhere.

The twist: the garbage collector was watching

There is a catch that kept this from being a wide-open door. Kubernetes has a garbage collector that cleans up orphaned objects. A pod created in another namespace, with no valid owner there, gets deleted almost immediately.

To keep the pod alive, the attacker needed a valid OwnerReference pointing at a real StatefulSet in the victim namespace. An OwnerReference is how a pod says "I belong to this StatefulSet," and the garbage collector checks it. Forging a convincing one requires the UID of an existing StatefulSet in the target namespace. That is not something you can guess from outside.

So the full attack needed two things: the confused deputy trick to place the pod, and knowledge of a real StatefulSet UID in the victim namespace to keep it there. That second requirement is why the advisory notes this matters most in multi-tenant clusters, where that kind of information leaks across tenants more easily, and where broad write permissions get handed out more casually.

The fix

Kubernetes released patched versions on September 23: v1.34.12, v1.35.9, v1.36.5, and v1.37.1. The fix is surgical. The controller now restores only the spec field from ControllerRevisions, cutting off the path that let attacker-controlled metadata cross the wall.

The vulnerability was reported by ImanOracle, and the fix was coordinated by Maciej Szulik, Filip Křepinský, Verónica López, Jeremy Rickard, and Nathan Herz. No confirmed exploitation has been reported. The CVSS score is 5.9, medium, which reflects the high privileges required and the garbage collector catch, not the seriousness of the boundary it crossed.

If you cannot upgrade immediately, the advisory's guidance is direct: treat StatefulSet and ControllerRevision write access as sensitive. That combination of permissions is more powerful than it looks.

The lesson that outlives the CVE

This bug will be patched and forgotten, but the pattern will not. Every system has deputies: CI runners that deploy with production credentials, webhooks that act on user-supplied payloads, controllers that reconcile user-written objects with cluster-wide authority. The confused deputy shows up whenever a powerful helper acts on data from a less powerful user without checking whether the user was allowed to ask for that outcome.

Namespaces are still the right walls. RBAC is still the right lock. But a lock is only as good as the most trusted person holding a key, and in Kubernetes, controllers hold keys to everything. Audit who can write the objects your controllers read. In this case, that meant one RBAC review:

kubectl auth can-i create statefulsets --as=team-a-dev -n team-a
kubectl auth can-i create controllerrevisions --as=team-a-dev -n team-a

If both of those return yes for users who do not need that combination, you had the same shape of problem, with or without this CVE.

When was the last time you checked which roles in your cluster can write ControllerRevisions?

One bad config file, 8.5 million blue screens

Kashif Manzer — Mon, 28 Sep 2026 16:18:40 +0000

Friday, 19 July 2024. Early morning in Europe, late night in the US.

At airports, departure boards froze mid-update. In hospitals, staff reached for pen and paper. At banks and broadcasters, screens everywhere showed the same thing: a plain blue screen, a frown, and the words "Your PC ran into a problem."

It was not a cyberattack. Nobody had been hacked. Something had been updated, and the update was poison.

The poison

The poison came from CrowdStrike, a security company whose Falcon software runs on millions of corporate Windows machines. Falcon watches for attacks, and to keep up with new ones, CrowdStrike feeds it fresh detection rules constantly. Not once a month. Sometimes several times a day.

These rapid updates are called Rapid Response Content. They travel through numbered channel files. On 19 July at 04:09 UTC, Channel File 291 went out with new rules for spotting attacks that abuse named pipes. (A named pipe is just a way for two programs on the same machine to talk to each other. Attack tools love them, so defenders watch them.)

78 minutes later, at 05:27 UTC, CrowdStrike pulled the file back. But any machine that had grabbed it in that window was already doomed, and the doom had a special cruelty to it. The bad file loaded during boot. So the machine would start, load the file, crash, reboot, and start again. Forever. Fixing it meant visiting each machine by hand, booting into Safe Mode, and deleting the file.

About 8.5 million Windows machines, by Microsoft's count.

Where the poison lived

To understand how one bad file could do this, you need to know where Falcon lives. It does not run like a normal app. Its core, a driver called CSAgent.sys, runs in kernel mode.

A quick explainer, because this is the whole story. Your computer has two neighbourhoods. User mode is where apps live. If an app crashes there, the app dies and the computer carries on. Kernel mode is where the operating system core lives. If code crashes there, the whole machine goes down with it. Security software runs in kernel mode because it needs to see everything. The price is that its bugs become everyone's bugs.

Now the actual defect. Months earlier, CrowdStrike had added a new detection capability: an "IPC Template Type" for watching named pipes, shipped in sensor version 7.11 on 28 February 2024. Think of a template type as a form with blanks. It declares the inputs a detection rule is allowed to use. This one declared 21 input fields.

The sensor code that reads those inputs, the Content Interpreter, had been built to handle 20.

On 19 July, Channel File 291 carried the first rule that actually filled in field number 21. Earlier rules had left that field on a wildcard match, which is why months of testing and four earlier deployments in March and April never tripped the bug. The moment real data landed in field 21, the interpreter read past the end of its list. In plain terms: it reached for item 21 in a list of 20.

In user mode, that is one crashed program. In kernel mode, it is a blue screen of death. And because the file loads at boot, the blue screen arrived before anyone could do anything about it.

A sketch of the shape of the bug (illustration only, not CrowdStrike's real code):

# the template type declares 21 fields
fields = read_template_instance()  # 21 items this time

# but the interpreter was built for 20
for i in range(21):
    value = field_list[i]  # i = 20 reads past the end: boom
    check_against_rule(value)

Why every safety net missed it

This is the part that should make every backend engineer uncomfortable, because none of these failures were exotic.

First, the validator. CrowdStrike has a Content Validator whose entire job is to check new detection content before release. It had a logic error of its own: it never compared the number of fields in the new content against the number the interpreter could handle. The guard dog was checking IDs at the door while the fence had a hole in it.

Second, the tests. The new template type had passed stress tests back in March. But the tests used wildcard matching on the 21st field, so they never walked the exact path that would crash. The tests proved the happy path worked. The bug lived on the path nobody walked.

Third, the rollout. CrowdStrike deploys its sensor software in stages, rings of machines, watching each ring before widening. Sensible. But Rapid Response Content skipped all of that and went to every machine at once. The riskiest updates, the ones shipped several times a day, had the weakest rollout of anything the company shipped. The file was live for 78 minutes. That was enough.

What I take from it

Validate shapes, not just values. The validator checked the content but never asked the simplest question: does this have more fields than the reader expects? Whenever two systems share a format, the count and the shape are the contract. Check the contract.

Test the exact thing you ship. A wildcard in a test is a hole in the test. If your test data cannot trigger the bug, your test is decoration.

"Just a config change" is a story we tell ourselves. Config is code that runs with the privileges of whatever reads it. This config ran in kernel mode.

Roll out everything in stages, especially the stuff you ship most often. Frequency is not safety. The thing you deploy ten times a day deserves the canary more than the thing you deploy twice a year, because it gets ten times as many chances to be wrong.

CrowdStrike's own fix list reads like this postmortem in reverse: stricter content validation, real test cases for the 21st field, error handling in the interpreter so a bad read can never crash the machine again, and staged rollouts for Rapid Response Content.

The scariest bugs are not clever. This one was a list with 21 items and a reader built for 20. The clever part was everything around it that let such a simple bug reach 8.5 million machines in 78 minutes.

What is the scariest "just a config change" you have ever shipped?

Sources

CrowdStrike's root cause analysis, via The Hacker News: https://thehackernews.com/2024/08/crowdstrike-reveals-root-cause-of.html?ref=netmanageit.com
Incident writeup: https://github.com/mahdihedhli/threatpedia/blob/HEAD/site/src/content/incidents/crowdstrike-falcon-global-bsod-outage-2024.md

Two AI APIs die this weekend: Perplexity Sonar (Sep 27) and Appsmith AI (Sep 30)

Kashif Manzer — Fri, 25 Sep 2026 03:05:29 +0000

Two AI APIs Die This Weekend: Perplexity Sonar (Sep 27) and Appsmith AI (Sep 30)

Perplexity retires the Sonar API on September 27, 2026. Appsmith shuts down its managed Appsmith AI plugin on September 30. If you touch either one, this is your last week.

Here is the fast version of what is happening and a 15-minute triage to find out whether you are exposed.

Perplexity Sonar dies Sunday

The announcement came in a Perplexity blog post on August 13: "On September 27, 2026, the Agent API becomes the main surface for this work, and Sonar tiers retire on the same date." Sonar calls will stop working if nothing changes.

Here is the part that should make you check your vendor monitoring: the date is real but badly announced. It appears in the blog post and in the docs warning banner. It does NOT appear in the migration guide, which says Sonar "remains supported." It is not in the changelog either; the Sonar item is filed under July 2026 with no date. Perplexity's own FAQ tells you to watch the changelog for retirements. A team that follows that instruction exactly never finds this one. If your vendor monitoring diffs changelogs, this deadline is invisible to it.

The replacement is the Agent API (canonical endpoint /v1/agent, GA since February 2026). The preset mapping is one to one: sonar to fast, sonar-pro to low, sonar-reasoning-pro to medium, sonar-deep-research to high, with xhigh sitting above all four.

Three things bite people on this migration:

It is not a model-string swap. The endpoint path, the request body, and the response parsing all change. If your UI renders source citations, that is the highest-risk silent break: the UI renders, the sources are just wrong or empty.
The cost model changes shape. Sonar billed tokens. The Agent API bills tokens plus per-tool-invocation billing: web_search $2.50, fetch_url $0.50, finance_search $5, people_search $5 per 1,000 invocations, with step budgets per preset (fast 1, low 5, medium/high 15, xhigh 100). Budget alerts calibrated on Sonar token prices will misfire.
The rate-limit trap. Tiers are cumulative lifetime spend ($0 / $50 / $250 / $500 / $1,000 / $5,000), capped at 1/3/8/17/33/33 queries per second. Tier 5 ($5,000 cumulative) buys exactly the same throughput as Tier 4 ($1,000). If your plan assumes spending more lifts the ceiling, it stops lifting at Tier 4.

Also: sonar-pro has a 200K input window, the largest in the Sonar lineup. Perplexity publishes no context-window figure for the Agent API, only per-preset max output tokens. If you picked sonar-pro for the 200K path, you are migrating on an unknown. Get it in writing from Perplexity before you size the work.

The Search API is not affected. It is a separate product with no deprecation notice. Customers with existing contractual commitments to Sonar are covered through the end of their current term.

Appsmith AI dies three days later

Since release 2.3, you can no longer create new Appsmith AI datasources or new queries against existing ones. The API rejects both with a clear 400 (AE-DTS-4031). Existing datasources and queries keep working until September 30 and now show a non-dismissible deprecation banner with the date. Source: https://github.com/appsmithorg/appsmith/pull/42015

The migration path is BYOK: move to the OpenAI, Anthropic, or Google AI datasources with your own API keys. Managed Appsmith-owned credentials are revoked about 60 days after GA, which is the hard stop even if your instance upgrades late.

Two extra traps:

Uploaded files (file context) will not be available after the shutdown date. There is no transfer path. Re-upload or redesign now.
A recent Appsmith change made the datasource plugins live-fetch current model lists, and saved Anthropic queries on hand-typed modern models (e.g. claude-sonnet-4-*) now get the raw messages-API body instead of the old completion-shaped body. If any widget or JS object binds to a completion field on those queries, update the binding. Source: https://github.com/appsmithorg/appsmith/pull/42114

The 15-minute triage

Run these in order.

1. Am I calling Sonar?

grep -rnE 'sonar(-pro|-reasoning-pro|-deep-research)?' --include='*.py' --include='*.js' --include='*.ts' --include='*.go' --include='*.java' --include='*.rb' --include='*.yml' --include='*.yaml' --include='*.json' --include='*.env*' . 2>/dev/null | grep -viE 'node_modules|dist/|build/|\.git/'

Also grep prompt-template stores, feature-flag configs, and IaC. A hit means Track A applies.

2. Do I parse Sonar responses or citations?

Search for choices[0].message.content and any code reading a citations array from Perplexity responses. Sonar's citation array is what most teams built source-attribution UI on; the Agent API response is a different shape. A hit means your parsing code needs rewriting, not just the model string.

3. Do I depend on sonar-pro's 200K context?

If any request relies on the 200K input path, you have no published equivalent to plan against. Ask Perplexity in writing before you commit a sprint.

4. Am I on Appsmith AI?

In each Appsmith workspace, open Datasources and look for the Appsmith AI tile. If you see the non-dismissible deprecation callout on the datasource editor or any AI query editor, you are on the managed plugin.

5. Do my Appsmith AI queries use uploaded files?

Open each Appsmith AI query and check for file context. A hit means those files do not transfer; you must re-upload them against the new datasource.

Verdict: hits in 1 or 2 mean a standard Sonar migration, roughly a day. A hit in 3 means you have an open question to settle first. A hit in 4 means the Appsmith track applies; a hit in 5 adds file-context rework.

The full kit

This article is the triage. If you have hits, I built the full migration kit for both tracks: the ordered migration checklists with per-step verification, the before/after call-shape patterns, six code-review prompts you can paste into any AI assistant or hand to a reviewer, the testing and verification plan, the honest rollback story (there is no post-Sep-27 rollback for Sonar; Appsmith gets a staged retreat), and a gotchas FAQ covering the badly-announced date, the blog's overstated cost claims, the Tier 4/5 trap, and the Anthropic binding change. Every date, price, and behavior was verified against vendor docs or the deprecation PR on the build date, and anything I could not verify is marked as unverified, not hidden. It is here: https://kmanzer.gumroad.com/l/dsbde

Sunday is three days away. If the triage shows hits, do not wait for the quarter-end freeze to make the decision for you.

GitHub Deleted Node 20 From Actions Runners: The 15-Minute Triage Checklist

Kashif Manzer — Thu, 24 Sep 2026 20:41:04 +0000

On September 23, 2026, GitHub removed the Node.js 20 runtime from Actions runners. Any JavaScript action whose action.yml still declares runs.using: node20 can no longer start: the job fails at action startup, before your tests even run.

Timeline (from the official changelog):

2025-09-19: deprecation announced
2026-06-16: node20 actions force-run on Node 24 with a warning
2026-09-23: Node 20 removed entirely (revised up from Sept 16)

If your CI went red yesterday and you haven't touched anything: this is why. Here's how to triage it in 15 minutes.

Check 1: Confirm you're hit (1 min)

Open your latest workflow run logs and search for: Node.js 20 actions are deprecated

Hit = your pins are stale.

Check 2: Inventory every action pin (5 min)

grep -rhoE 'uses:[[:space:]]*[^[:space:]#]+' .github/workflows/*.yml .github/actions/*/action.yml 2>/dev/null | sort -u

Note the second path: composite actions live in .github/actions/ and are invisible to Dependabot-style tag scans. You must grep them separately.

Check 3: Resolve each pin's actual runtime (5 min)

Do not trust tag names. The node24 boundary is not uniform across majors: upload-artifact only moved to node24 at v6, download-artifact at v7. For each third-party pin:

gh api 'repos/<owner>/<repo>/contents/action.yml?ref=<ref>' --jq '.content' | base64 -d | grep -A1 '^runs:'

using: node20 = broken. using: node24 = safe. using: composite or using: docker = unaffected.

Check 4: Self-hosted runners: check the floor (2 min)

node24 actions require runner >= 2.327.1. Below that, bumped actions fail even with correct pins. Also: Node 24 doesn't run on macOS <= 13.4 or ARM32 at all: those jobs need a different plan.

The bumps everyone hits

Action	Broken pin	Minimal node24 major	Watch out
`actions/checkout`	v4	v5	v6 moves credentials out of `.git/config`; v7 blocks fork-PR checkout by default
`actions/setup-node`	v4	v5	v5 enables package-manager caching by default: disable with `package-manager-cache: false` in privileged workflows
`actions/setup-python`	v5	v6	Clean bump, runtime only
`actions/setup-go`	v5	v6	v6 honors the `toolchain` directive and defaults `GOTOOLCHAIN=local`: use `go-version-file: 'go.mod'` or builds break
`actions/upload-artifact`	v4	v6	Do NOT stop at v5: it still runs node20 by default
`actions/download-artifact`	v4	v7	v5 changed single-artifact download paths; pair with upload v6

Prefer the minimal node24 major: each extra major crossed is unrelated breaking change you absorb for nothing.

The two gotchas that bite backend repos hardest

setup-go@v6 silently changes which Go you build with. v6 honors the toolchain directive in go.mod (v5 ignored it) and defaults GOTOOLCHAIN=local, which forbids the silent toolchain downloads v5 allowed. If your workflow pins go-version: 1.24 but go.mod says go 1.25, builds fail. Fix: go-version-file: 'go.mod'.

There is no rollback to node20. The runtime is gone: FORCE_JAVASCRIPT_ACTIONS_TO_NODE24 and friends do nothing now. "Rollback" means retreating to a different node24 major, never back to the old pin.

I spent today turning this into a complete one-afternoon migration kit: all 19 actions in the pin-bump map with landmines, the ordered 7-step migration, 6 code-review prompts, a testing/verification plan, the honest rollback plan, and every gotcha above sourced. It's $39 here: https://kmanzer.gumroad.com/l/sreej

Every version number in it was verified against upstream release notes or real migration reports on the build date. Good luck out there: and check your composite actions.