DEV Community: DevOps Daily

Log Aggregation Pipeline Explained: Follow One Log Line and Run the Incident in Your Browser

DevOps Daily — Thu, 22 Oct 2026 14:50:09 +0000

Checkout says payments are timing out. You open the log search, filter on ERROR, type payment provider timeout, and get one hit. Is that one timeout, or the one timeout that survived the trip?

A log search shows what reached the index, not what the application wrote. Between the two sit an agent, a parser, a filter, a buffer and a search cluster, and every one of them can delay a line, reject it or drop it without the app knowing. That is the whole point of having a log aggregation pipeline explained stage by stage: when a search comes back empty during an incident, you need to know whether the system is fine or the evidence never arrived.

The Log Aggregation Pipeline Simulator lets you watch that trip one stage at a time and then break it in three different ways. It moves logs in batches and counts them at every stage, rather than tracing one line. Disclosure: I help build it; it is free, runs in the browser, no signup.

Six stages, one line

The simulator draws the pipeline as six cards. Each Step advances one stage; Run flow steps automatically about once a second. Six steps make one cycle.

Sources (Apps & hosts). Raw events, and how many are waiting.
Collect (Fluent Bit). An agent tails files and forwards events. If it falls behind, the source queue grows.
Process (Parse & filter). Extracts fields and removes noise. Its two controls sit in the stage inspector: a Parser picker (JSON parser or Grok pattern) and a "Drop health-check noise" switch. Changing either starts a fresh run.
Buffer (Durable buffer). A bounded queue between parsing and indexing, shown as used over capacity.
Index (Search cluster). Accepted events land on three primary shards, and a "Shard health" tab shows the balance. The simulator deals events to the shards in turn, so the bars stay level in every scenario.
Search (Explore logs). A "Search logs" tab with a text box and level filters. It keeps up to four sample events from each indexing step, and only the newest 24, so judge completeness by the counters, not by the list.

Under the cards, five counters keep the books: Generated, Buffered, Indexed, Rejected and Dropped. The engine behind them conserves every event: anything generated is waiting in a queue, filtered, rejected, indexed or dropped. Filtered noise has no counter of its own; the Process card's "removed" figure adds filtered and rejected together. Carry that habit into real systems: if you cannot account for the difference between "sent" and "searchable", you cannot trust an empty search.

Click any card and the stage inspector shows its "Operator signal", the one thing to watch there. For the Search stage it is the line that matters most in an incident: "Only indexed logs are searchable; filtered, rejected, and dropped logs never arrive here."

Healthy first

Start on Healthy flow and step through a few cycles. Each cycle, the sources emit 24 logs, the noise filter removes 4, and 20 are indexed. Queues drain, the badge says "Pipeline is healthy", and the shard bars grow evenly.

Break it three ways

Traffic spike. Sources emit 84 logs per cycle, but the collector forwards 38. The badge changes to "Backpressure is building" on the very first step. In the third cycle: "Sources emitted 84 logs; 26 were dropped before collection." Look at the buffer while that happens: it is empty. The durable buffer cannot save logs that are lost before they reach it. By the end of cycle five, 420 logs were generated, 140 indexed and 118 dropped. The Collect card's "for parsing" count also climbs by 4 per cycle: the collector sends 38, the processor takes 34.

Parser mismatch. A deployment changes the log format. Of 30 logs per cycle, 4 are filtered as noise, 11 are rejected by the parser and only 15 are indexed. The badge says "Parser rejects detected" and nothing is dropped, so a dashboard that only watches for drops stays green. Try the Grok parser. It does not help: it handles only 27 of the 30 logs per cycle, so 3 more wait in the processor queue every cycle, and of those 27 it filters 4, rejects 10 and indexes 13. A slower parser that still does not match the format makes things worse. The simulator has no parser fix; for real, you update the parser to the new format or roll the change back.

Slow indexing. Switch the parser back to JSON first: the simulator keeps your parser choice when you change scenario. The search cluster indexes 9 logs per cycle while 32 reach the buffer (36 arrive, and the noise filter removes 4). The buffer climbs by 23 each cycle, the warning appears in the third cycle, and in the fifth: "The buffer reached capacity and dropped 19 oldest logs." Now turn off "Drop health-check noise" and run it again. The first drop comes a cycle earlier. Filtering noise before the buffer did not fix the slow indexer, but it bought time.

Now run the incident

Take each scenario as an on-call drill. Pretend you only have the search tab, then use the rest of the screen to answer four questions in order:

Is input normal? Compare Generated per cycle against the healthy 24. A spike is often the first sign of the real incident, and it also strains the pipeline.
Where is a queue growing? A growing source queue points at the collector. A growing buffer points at the indexer. Name the stage before you touch anything.
Is anything rejected? Rejects mean the logs exist but no longer match the parser. If the deploy that changed the format also broke checkout, the error lines you need are among the rejects.
How long until loss? A filling buffer is a countdown. Space left divided by net growth per cycle is a rough, optimistic estimate: here the buffer overflows when a batch lands, before indexing drains it. With the JSON parser and the noise filter on, after cycle four it holds 92 with 13 free, and the next batch of 32 drops 19.

Match the mitigation to the signal: collector capacity for the spike, the parser for the mismatch, index capacity or noise shedding for the slow index.

Try the parser mismatch for real

You can reproduce the parser mismatch with jq and three lines of log:

cat > app.log <<'EOF'
{"level":"INFO","service":"api","msg":"request completed status=200"}
{"level":"ERROR","service":"checkout","msg":"payment provider timeout after=3s"}
ERROR checkout payment provider timeout after=3s
EOF

# what a JSON-only pipeline makes searchable
jq -R -c 'fromjson? | select(.level == "ERROR")' app.log

# what it rejected on the way
jq -R -r '(try fromjson catch null) as $j | select($j == null)' app.log

With jq 1.6, the first command prints only the JSON error line. The second prints the plain-text ERROR checkout payment provider timeout after=3s. Two timeouts were logged, and search finds one. Count the second output and alert on it, and you have a parser-reject metric. Real tools do not always throw a reject away. When parsing fails in Vector's remap transform, "the original, unmodified event is sent downstream" by default, so the line reaches the index as raw text without your fields.

Real agents have their own version of the other two scenarios. The Fluent Bit backpressure docs explain that when an input reaches its mem_buf_limit, Fluent Bit (with memory-only buffering) "will stop buffering data from that source plugin until enough buffered chunks are flushed", and most plugins log paused (mem buf overlimit) when that happens. The same page warns that some inputs can lose data under memory-only buffering and suggests filesystem buffering if you need to avoid that. Search your agent's own logs for that warning before you assume the app went quiet. With filesystem buffering, Fluent Bit drops the oldest data, as the Slow indexing buffer does: at an output's storage.total_limit_size, "the oldest chunk from its queue will be discarded". Vector's buffers block by default.

What to alert on

The simulator's page lists the signals to watch together: input rate, queue age and depth, parser rejection rate, buffer utilization, dropped events, indexing latency and shard balance. In real tools, the loss signals are counters you can scrape: Fluent Bit's fluentbit_output_retries_failed_total and fluentbit_output_dropped_records_total, Vector's component_discarded_events_total, and Loki's loki_discarded_samples_total, labelled by reason.

The simulator is one of 60+ free DevOps games and simulators. Run the three scenarios until you can predict the next step's numbers, then again as drills with only the search tab open.

Where has a log pipeline dropped lines on you without anyone noticing? Tell us in the comments.

A Service Catalog Nobody Updates Can Be Worse Than None: We Checked 217 Entries Against GitHub

DevOps Daily — Thu, 22 Oct 2026 09:00:00 +0000

We took 217 service catalog entries that 30 organizations publish on GitHub, in the Backstage catalog-info.yaml format, and checked each one against what GitHub says today. For one in ten (21 entries), GitHub contradicted a record: in 9 the catalog entry itself was wrong (a repository renamed while the catalog kept the old name, a repository reference without its owner, a dead link, no owner at all), and in 12 the CODEOWNERS file next to it named people or teams GitHub no longer accepts. That is a floor, not a total. For three entries in four, the owner could not be checked at all from outside the organization. And where the catalog and CODEOWNERS both named an owner for the same file, the two agreed, with every owner resolving, only half the time.

That matters most at the worst moment. During an incident, someone opens the catalog to find out who owns the failing service. With no entry, they go and ask. With a wrong entry, they can page a team that was reorganized away, or a person who left, and do it with confidence because the catalog said so. We did not measure that happening; it is the reason the measurement is worth doing. This post shows the checker we built, how we picked the sample, what it found, the bugs two reviews found in our own method, and the habits that keep a catalog true.

TL;DR

217 Component entries from 214 public repositories in 30 GitHub organizations, checked on 2026-10-22: 21 (9.7%) had at least one record GitHub contradicts, 9 of them in the catalog entry itself and 12 only in the CODEOWNERS file next to it. 62 (28.6%) had only warnings, and 134 (61.8%) had nothing we could find.
Ownership is where the records disagree. Of 73 entries whose catalog file is also covered by a CODEOWNERS rule, 37 agree and every owner resolves. 14 name owners that GitHub cannot resolve, and 22 name different owners (in 7 of those, one team name contains the other, which may be the same team).
159 entries (73.3%) have an owner nobody outside the org can check: GitHub teams are only visible to members, and some owners come from other directories. With a token that can read your org, the script checks teams and members directly.
Links barely rotted: 2 of 152 were dead. But only 70 entries had any link, and none was titled as a runbook or a dashboard. We could not test the links that probably matter most during an incident.
We expected older catalog files to drift more. Drift was 8.0% for files changed in the last year and 11.4% for older ones, a gap this sample cannot tell apart from chance.
The fixes are boring and they work: derive owners from the source of truth, check catalog-info.yaml in CI, use GitHub's own CODEOWNERS validation, and retire entries nobody claims.

Prerequisites

Python 3.10 or later (the checker uses only the standard library)
A GitHub token in GITHUB_TOKEN, or a logged-in gh CLI. To check owners inside your own org, the token needs to read the org's teams and members
A catalog in the Backstage catalog-info.yaml format, or the public sample in the repository

The-DevOps-Daily/service-catalog-drift on GitHub

Why a catalog drifts

A catalog entry is usually written once, by a scaffolder template, on the day the repository is created. After that, teams get reorganized and renamed, people move on, repositories are renamed or transferred, and documentation moves from one wiki to the next. None of those changes touch catalog-info.yaml.

None of them break anything either. Backstage's own FAQ, answering whether a plugin should reject entities that fail validation, says: "Never throw errors due to "soft" errors, in particular relations not matching an existing target". So an owner that points at a team that no longer exists loads and shows up like any other. GitHub redirects a renamed repository, so a catalog that still uses the old name keeps working, and nobody has a reason to look. The catalog stays green while it drifts away from the truth.

That is why a stale catalog can be worse than none. An empty owner field sends you looking. A wrong one can send you to the wrong place, quickly.

What the checker compares

drift.py reads Backstage Component entities and checks five things against GitHub, all read-only:

Check	Drift (GitHub contradicts it)	Warning (look at it)	Cannot tell from outside
owner	no `spec.owner`; with org access, a team that does not exist or has no members	a team GitHub cannot resolve in the CODEOWNERS files that name it; a `user:` with no GitHub account of that name	a team no CODEOWNERS file names; a user who is not a public member; a group in another directory
repo	the slug redirects (renamed or transferred); a `github.com/project-slug` that is not `owner/repo`; archived while the catalog does not say `deprecated`	no human commit on the default branch in a year; a root catalog file that describes a different repository	a repository that is not visible (deleted or private)
codeowners	the CODEOWNERS rule for the catalog file names owners GitHub's validation cannot resolve	CODEOWNERS and the catalog name different owners	GitHub's validation not readable
links	404 or 410; a GitHub file missing at a ref that exists		401, 403, 429, 5xx, timeouts, internal hostnames, pages below a GitHub repository
lifecycle		`production` or `experimental` with no human commit in a year; no lifecycle

catalog-info.yaml Component entities
Owner teams API or CODEOWNERS errors
Repository redirects, archive, commits
Links public addresses only
Report reasons and exit code

A drift on the CODEOWNERS check is about the file that routes code reviews, not about the catalog entry itself. We count both because both are ownership records people rely on, but we report them separately below. A warning is a fact worth a look: a library that works may not need a commit for a year. "Cannot tell" is its own status and never counted as either.

Checking an owner from outside the org

Inside your org, checking an owner is a lookup: does the team exist, does it have members, is this user a member. From outside, GitHub hides teams. But GitHub validates CODEOWNERS files, and it publishes the result. For a public repository, anyone can call GET /repos/{owner}/{repo}/codeowners/errors, and it lists every owner GitHub cannot resolve, with messages such as:

Unknown owner on line 3: make sure the team @org/team exists, is publicly visible, and has write access to the repository

So if any CODEOWNERS file in the org names the catalog's owner team, GitHub can tell us whether that team resolves. The checker looks in the entry's own repository first, then runs one code search for the team in the org's other CODEOWNERS files. A team that resolves exists. A team that does not resolve might be deleted, renamed, secret, or only lack write access there, so on the owner check it is a warning, not drift. A team that no CODEOWNERS file names stays "cannot tell". All of this assumes Backstage group names are GitHub team slugs, which is what Backstage's GitHub org provider produces; groups from other namespaces are not mapped.

Links come from strangers

The links in a catalog file are written by whoever wrote the file, so link checks are opt-in (--check-links) and careful. The token only ever goes to api.github.com, and github.com links are checked through the API with every path segment validated and quoted. Everything else gets a HEAD, then a GET, with no credentials and no cookies, at most one request per second per host. The host name is resolved once, every address must be public, and the request goes to that pinned address, so a second DNS answer cannot point it at an internal service. Every redirect hop is checked the same way.

The sample

We wanted catalogs that real organizations maintain, not tutorials. The selection, all recorded in the repository:

GitHub code search for filename:catalog-info.yaml path:/ (files up to 20,000 bytes) on 2026-10-22, between 07:08 and 07:23 UTC. Code search returns at most 1,000 results per query, so sample.py splits the query by file size into 16 ranges. GitHub reported 8,864 matches; we fetched 9,761 unique repositories (code search counts are estimates).
3,579 of those repositories belong to personal accounts and were dropped. The other 6,182 belong to 984 organizations. Code search returned no forks.
53 organizations had a root catalog-info.yaml in 20 or more repositories. Many turned out to be test, demo or QA organizations for developer portals, so we kept the 30 that have a website on their GitHub profile and at least one repository with 10 or more stars. That rule keeps out obvious test orgs; it does not prove that every sampled repository is a real service, and a few are demos or samples. The excluded orgs are stored as hashes, because many are named after a person.
From each of the 30, 8 repositories at random (seed 20261022): 240 files. One was a scaffolder template, one was not valid YAML, and 24 held no Component entity, only other kinds. That leaves 217 Component entries from 214 repositories.

The 30 organizations are a mix of companies, public bodies, research institutes and open-source projects: bcgov, broadinstitute, cds-snc, Clever, contentful, cultureamp, DEFRA, dell, DiamondLightSource, dwp, eduNEXT, elastic, equinor, grafana, hmcts, Invoca, Kajabi, kartverket, open-dxp, openedx, pantheon-systems, prime-radiant-inc, radiorabe, reearth, sendbird, snyk, sourcefuse, spotify, Unity-Technologies and vtex-apps.

Every organization contributed 8 repositories, but a repository can hold several entries or none, so organizations contribute between 1 and 10 entries. The shares in this post are pooled over entries. Averaged per organization instead, the drift share is 10.7% rather than 9.7%.

So the population this describes is narrow: root-level catalog-info.yaml files in public repositories of organizations that use them widely. Catalog files elsewhere in a repository, catalogs kept in a separate repository, and private catalogs are not in it.

The run

One pass makes about 1,700 GitHub API requests, cached on disk per token for 24 hours. --redact keeps people out of the output: user logins become user:<redacted>, and owner and team names become a hash with a key that is random per run, which is why you see group:<56b023> below and different hashes in the committed results. The results name organizations, repositories and services, never a person.

drift.py on the public sample

$ python3 drift.py --sample sample/sample-2026-10-22.json --check-links --redact --quiet > report.txt 2>/dev/null
$ echo $?
1
$ tail -7 report.txt
217 entries from 238 files (1 templates, 1 unparsed, 0 unreadable skipped)
  drift 21 (9.7%), warn 62, ok 134
  owner drift 1, ok 53, unverifiable 159, warn 4
  repo drift 6, ok 168, warn 43
  codeowners drift 14, n/a 144, ok 37, warn 22
  links drift 2, n/a 147, ok 63, unverifiable 5
  lifecycle n/a 38, ok 148, warn 31
$ awk '/^(DRIFT|WARN) /{p=($2=="com.unity.services.samples.relationships"||$2=="changelog-github"||$2=="kaizen-design-system"||$2=="loopback4-microservice-catalog")} p' report.txt
DRIFT com.unity.services.samples.relationships (Unity-Technologies/com.unity.services.samples.friends:catalog-info.yaml) owner group:<56b023>
         repo drift Unity-Technologies/com.unity.services.samples.relationships moved to Unity-Technologies/com.unity.services.samples.friends
         codeowners drift CODEOWNERS line 2 names owners GitHub cannot resolve: @Unity-Technologies/<072a85>
DRIFT changelog-github (cultureamp/changelog-github:catalog-info.yaml) owner group:<ffb463>
         repo drift github.com/project-slug is not a valid owner/repo
DRIFT kaizen-design-system (cultureamp/kaizen-design-system:catalog-info.yaml) owner group:<159434>
         codeowners drift CODEOWNERS line 4 names owners GitHub cannot resolve: @cultureamp/<15544f>
         links drift 1 of 5 links dead: other https://cultureamp.design/<path 642625a6c6> (HTTP 404)
DRIFT loopback4-microservice-catalog (sourcefuse/loopback4-microservice-catalog:catalog-info.yaml) owner group:<046fce>
         owner warn GitHub cannot resolve @sourcefuse/<046fce> in the CODEOWNERS files that name it (deleted, renamed, secret, or without write access)
         codeowners drift .github/CODEOWNERS line 5 names owners GitHub cannot resolve: @sourcefuse/<046fce>

The exit code is 1 because drift was found, so the same command can fail a CI job. The four entries above show the main kinds. The Unity sample was renamed from relationships to friends, its catalog entry still uses the old name, and line 2 of its CODEOWNERS names a team GitHub cannot resolve. cultureamp's changelog tool sets github.com/project-slug to the repository name without its owner, which Backstage's GitHub integrations cannot use. In cultureamp's design system, a CODEOWNERS team does not resolve and the Storybook link answers 404. In sourcefuse's microservice catalog, the catalog's own owner team is the one GitHub cannot resolve in the same repository's CODEOWNERS: drift for the CODEOWNERS file, a warning for the owner.

What it found

What 217 public catalog entries got wrong

	Value	Series
CODEOWNERS names an owner GitHub cannot resolve (14)	6.5%	Drift
Repository renamed, catalog has the old name (4)	1.8%	Drift
project-slug is not owner/repo (2)	0.9%	Drift
Dead link (2)	0.9%	Drift
No owner at all (1)	0.5%	Drift
No human commit in a year (39)	18%	Warning
Says production or experimental, idle a year (28)	12.9%	Warning
Catalog and CODEOWNERS name different owners (22)	10.1%	Warning
Owner team does not resolve where it is named (3)	1.4%	Warning

Share of all 217 entries with each finding; an entry can have more than one. 30 GitHub organizations, 8 random repositories each, checked 2026-10-22. Source: results/summary-20261022T091627Z.json in the demo repository.

Ownership is where the records disagree

Most findings are about who owns what. The clearest view is the 73 entries whose catalog file is also covered by a CODEOWNERS rule, so there are two records of the owner to compare:

summarize.py

$ python3 summarize.py results/run-20261022T091627Z.json | sed -n '/^catalog vs CODEOWNERS/,/Fisher/p'
catalog vs CODEOWNERS: 73 entries with both; agree and resolve 37 (50.7%); unknown owners 14 (teams 8, users 6)
  disagree: 4 catalog names a team, CODEOWNERS people
  disagree: 11 different team names
  disagree: 7 one team name contains the other
drift by file age: under 1 year {'entries': 112, 'drift': 9, 'pct': 8.0}, 1 year or more {'entries': 105, 'drift': 12, 'pct': 11.4}, Fisher p = 0.493

73 entries with both a catalog owner and a CODEOWNERS rule

	Value	Series
Agree, every owner resolves	37	Agree
One team name contains the other	7	Disagree
Different team names	11	Disagree
Catalog names a team, CODEOWNERS people	4	Disagree
CODEOWNERS owner GitHub cannot resolve	14	Cannot resolve

"One name contains the other" is, for example, a short catalog name inside a longer GitHub team slug; it may be the same team, but the data cannot show it. Unresolvable owners are what GitHub's CODEOWNERS validation reports for the rule that covers the catalog file.

37 agree. The other 36, almost half, either name different owners or name owners GitHub cannot resolve. In 7 of them one team name contains the other: in four snyk repositories, for example, the catalog uses a short team name that is part of the longer GitHub team slug in CODEOWNERS. Those may well be the same team. Leave them out and 29 of 73 (40%) still disagree or cannot be resolved. Of the 14 with unresolvable owners, 8 involve a team and 6 involve individual users.

Two caveats keep this from being a verdict on the catalog. CODEOWNERS says who reviews code, which is not always who owns the service, so a disagreement means one record is stale or the two mean different things. And GitHub reports a team as unknown when it was deleted or renamed, is secret, or has no write access to the repository; any of those means the team does not review that code, but only the first two mean it is gone.

Unresolvable owners repeat. In vtex-apps, the same unknown team sits on line 3 of the CODEOWNERS in three of the eight sampled repositories, which looks like a file copied from a template and never updated. Four organizations hold 14 of the 21 drifted entries, and 20 of the 30 had none.

The owner field itself could be checked for only 58 entries (27%): 37 resolved in the entry's own CODEOWNERS, 15 in another repository's, 3 named a team GitHub could not resolve in any CODEOWNERS file that names it, 2 were checked through GitHub's user API, and 1 had no owner at all. Backstage requires spec.owner on a Component, so that entry probably never loaded into anyone's catalog. The 159 we could not check are 142 groups and 17 users: hidden teams, groups that do not look like GitHub teams, users who are not public members of the org, and user IDs that are not GitHub logins.

Repositories that moved

4 entries (1.8%) point at a repository by a name it no longer has. GitHub redirects the old name, so links and API calls keep working. But GitHub's documentation warns that if anyone creates a new repository with the old name, "redirects to the renamed repository will no longer work", and the catalog entry then points at someone else's code. 2 more entries set github.com/project-slug to a bare repository name without the owner, and 2 live in one repository but describe another, which is what a catalog file copied from a sibling repository and never edited looks like.

We found no archived repositories. That is a gap in the sample, not good news: none of the 211 repositories the checker read was archived (each entry records the flag), a code search we tried with archived:true returned nothing (so archived repositories are probably missing from the index), and Backstage's GitHub discovery skips them by default (allowArchived defaults to false).

Idle, but still "production"

39 entries (18.0%) live in repositories with no human commit on the default branch for over a year, and 2 more where the last 30 commits are all from bots. We do not count commits authored by Dependabot, Renovate or other bots as activity. 28 of the 39 still say production (19) or experimental (9). A stable library can go a year without a change, so this is a warning, not drift. An experimental service with no human commit for a year is a decision nobody made.

Links

Only 70 of 217 entries (32%) had any link at all. Of 152 links, 132 answered, 2 were dead, and 18 could not be checked: 12 pointed at hostnames that do not resolve in public DNS (internal tools, most likely), 5 answered 403, and 1 was a page below a GitHub repository that we do not check. The two dead ones were a Storybook page and a vulnerability report page on a vendor's site.

None of the 152 links was titled as a runbook, an on-call schedule or a dashboard. Public catalogs may simply not publish them, or may title them differently; either way we could not test those links, and they are the ones you would want to work during an incident. If you run this on your own catalog, the link check is where we would expect your numbers to differ most from ours.

Does a catalog decay with age?

Drift by when the catalog file last changed

	Value
Under 6 months (81)	9.9%
6 to 12 months (31)	3.2%
1 to 2 years (48)	12.5%
Over 2 years (57)	10.5%

Share of entries with at least one drift, by the date of the last commit that touched catalog-info.yaml. Under one year: 9 of 112. One year or more: 12 of 105. Fisher's exact test p = 0.49, which also treats entries as independent although entries from one organization share habits.

We expected a clear slope and did not find one: 8.0% for files changed in the last year against 11.4% for older files. Fisher's exact test gives p = 0.49, and it assumes the entries are independent, which they are not quite: entries from one organization share templates and habits. So this sample cannot tell the difference from chance. Drift and warnings together are far more common in older files (24.7% and 19.4% under a year, 54.2% and 54.4% over), but mostly by construction: a repository with no commits for a year also has a catalog file nobody touched for a year. The median catalog file in the sample was last changed 352 days before we checked it.

A single snapshot also cannot see the worst decay. Entries that rotted badly enough to be noticed may have been deleted, and deleted entries are not in a sample of files that still exist.

The bugs two reviews found in our checker

The first version of the checker and of this post had problems, found by our own runs and by a second review of the code and the draft. Several changed numbers you just read, and some of the corrections made the result look worse, others better.

An unresolvable team was called a missing owner. The first version reported a catalog owner as drift when GitHub could not resolve the team in a CODEOWNERS file, including one in another repository. But a team can exist and simply lack write access there. That is now a warning on the owner, and only the broken CODEOWNERS file counts as drift. Three entries moved from owner drift to a warning; two of them had no other drift.
An invalid repository reference was quietly replaced. Two entries set github.com/project-slug to a repository name without its owner. The first version fell back to the repository the file lives in and reported nothing. It now reports the annotation as drift, which added two drifted entries just as the first fix removed two. The headline stayed at 21 by coincidence.
A user ID that looks like a GitHub login. One organization names users by an ID from its own directory, three letters and five digits, which is also a valid GitHub login. GitHub had no account by that name, and the first version reported a departed owner. A missing account is now a warning, unless you pass --users-are-github-logins for a catalog where you know users are GitHub logins.
GitHub's two error formats. For a user, the CODEOWNERS error says make sure @user exists; for a team, make sure the team @org/team exists, is publicly visible, .... The first version only parsed the user form, and for teams it fell back to marking every owner on the line as unknown. It changed no result in this sample, but on another catalog it would have.
A YAML reader that was too forgiving. The checker has its own small YAML reader to stay dependency-free. One sampled file had description: at the wrong indentation, which swallowed the annotations: block below it. PyYAML rejects that file, and our reader accepted it. It now raises the same error, and a cross-check against PyYAML on all 240 files agrees on every field the checker reads, with the one invalid file rejected by both. The review also found that it decoded some escapes in quoted strings wrongly; it now handles them or refuses the file.
Too generous with links and CODEOWNERS. Any page below an existing GitHub repository counted as a working link, a redirect loop counted as an answer, docs/* in CODEOWNERS also matched files in subdirectories (GitHub's own docs say it does not), and an unreadable CODEOWNERS validation could still report agreement. Each now reports "cannot tell" or matches GitHub's rules.
People in the output. A bare owner such as owner: someone is treated as a team by Backstage, but it is sometimes a person's login, and one owner was an email address. An early run printed them. That file was deleted, and owner and team names are now hashed with a key that is never written down.

Smaller fixes: a link with a space in it stopped an early run; cached answers now expire after a day and are kept per token; CODEOWNERS lines with invalid syntax are skipped, as GitHub skips them; a file link is not called dead when its ref might contain a slash; and an error in one entry, or a 401 or 5xx from GitHub, makes the run exit with 2 instead of stopping or passing. None of these changed a count in this sample. Every superseded run that left a file is kept in results/ with a note on what changed.

False positives, and what we could not check

A team GitHub cannot resolve is not always deleted. GitHub's message covers deleted, renamed, secret, and teams without write access to that repository.
Different names can be the same people. A parent team and a child team, or a Backstage group name that differs from the GitHub team slug, show up as a disagreement. That is why disagreement is a warning.
A link that answers is not always alive. A 200 can be a login page, or a "page not found" that answers 200. We count both as working. A 403 from a site that blocks scripts is "cannot tell".
Bot detection is by commit author. A merge commit by a person counts as human activity even when it merges a bot's change, so "no human commit in a year" can undercount idle repositories.
Owner coverage is uneven. We could check 27% of owners, mostly those named in some CODEOWNERS file. Those may be better or worse kept than the rest; we cannot tell which.
Private catalogs are not in the sample. Public repositories may be kept better or worse than internal ones, and internal catalogs carry the runbook, on-call and dashboard links we could not test.

How to keep a catalog honest

Derive owners, do not type them

Many of the disagreements above look like two hand-typed names for the same thing. Load groups from the source of truth instead: Backstage's GitHub org provider mirrors GitHub teams as Groups on a schedule, and the catalog products sync from identity providers. Then the owner in catalog-info.yaml and the handle in CODEOWNERS are the same string, and a machine can compare them:

# catalog-info.yaml
apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
  name: payments-api
  annotations:
    github.com/project-slug: acme/payments-api # owner/repo, not just the repository name
spec:
  type: service
  lifecycle: production
  owner: group:default/payments # the GitHub team slug, not a display name


# .github/CODEOWNERS: the same team, so the two can be checked against each other
* @acme/payments

Check the catalog file in CI

Drift arrives two ways: someone edits the file badly, or the world changes while nobody touches it. A pull request check catches the first, a weekly scheduled run catches the second. The repository ships this workflow in examples/:

name: catalog drift

on:
  pull_request:
    paths: ['catalog-info.yaml', '.github/CODEOWNERS', 'CODEOWNERS', 'docs/CODEOWNERS']
  schedule:
    - cron: '17 6 * * 1'
  workflow_dispatch:

permissions:
  contents: read

jobs:
  drift:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v5
      - uses: actions/checkout@v5
        with:
          repository: The-DevOps-Daily/service-catalog-drift
          ref: main # pin this to a commit SHA you have read
          path: .catalog-drift
      - uses: actions/setup-python@v6
        with:
          python-version: '3.13'
      - name: check catalog-info.yaml
        env:
          # The default token cannot read org teams; use a token with Members: read for full owner checks
          GITHUB_TOKEN: ${{ secrets.CATALOG_DRIFT_TOKEN || github.token }}
          REPO: ${{ github.repository }}
          # on a pull request, read CODEOWNERS as the pull request changes it
          REF: ${{ github.event.pull_request.head.sha || github.sha }}
        run: python .catalog-drift/drift.py catalog-info.yaml --repo "$REPO" --ref "$REF" --check-links --no-org-search --cache-hours 0

It exits with 1 on drift or on a catalog file that does not parse, and with 2 when it could not check what it was given, so a broken run never passes quietly.

Use the check GitHub already runs

You do not need our script to find the worst cases. GitHub already validates every CODEOWNERS file, and one call per repository lists what it cannot resolve:

# Owners GitHub cannot resolve in this repository's CODEOWNERS
gh api repos/OWNER/REPO/codeowners/errors \
  --jq '.errors[] | "line \(.line): \(.message | split("\n")[0])"'

Loop it over every repository in your org once a month. Each line it prints is an owner or a rule GitHub cannot use, which can leave reviews without the reviewer you meant, and is often a sign that the catalog owner is stale too.

Give the scorecard consequences

Every catalog product has scorecards, and so does Backstage through the Tech Insights plugin. The useful checks are the ones in this post: the owner resolves, the runbook link answers, the lifecycle matches the activity. Put the result where people already work, such as a comment on the pull request or a weekly list sent to each owning team, and not only on a dashboard.

Retire what nobody claims

An entry whose owner does not resolve and whose repository has been idle for a year may have nobody to fix it. One workflow we would suggest: open an issue for the last known owner, and if nobody confirms ownership within 30 days, mark the entry deprecated; remove it 30 days later unless someone claims it. A missing entry sends people asking. A wrong one can send them to the wrong place.

Build or buy

Every catalog product handles freshness with some mix of the same three tools: sync from a source of truth, automatic discovery, and scorecards. Here is what each one's public documentation says, in alphabetical order, with one documented limit each where we found one:

Backstage (open source) mirrors GitHub teams and users with its org provider, finds catalog-info.yaml files with GitHub discovery, and removes orphaned entities. Checks on catalog data, including the owner field, come from the Tech Insights community plugin. Limit: as quoted above, relations that point at nothing are accepted.
Cortex has scorecard rules such as "has an owner" and "has a runbook", syncs ownership daily from identity providers, Git hosts and HR systems, and auto-archives entities whose repository was deleted or archived. Limit: in GitOps mode, a YAML file that is moved rather than deleted is not archived.
Humanitec says in its own docs that it is not an internal developer portal: the Platform Orchestrator is the backend of a platform and complements portals such as Backstage, Port and Cortex.
OpsLevel has a component ownership check in its rubric, syncs GitHub teams every 12 hours, detects services from repositories and suggests owners from commit authors, and removes repositories deleted at the Git provider. Limit: the team sync page says deleted GitHub teams and users are not synced.
Port has scorecards with a "has a team" rule, syncs users and teams from SSO and Git providers, deletes entities that are gone from the source on a full resync, and has a guide for mapping CODEOWNERS to teams. Limit: entities of a resource type removed from the mapping are not deleted automatically.
Roadie is managed Backstage: catalog-info.yaml is its native format, it discovers files from Git providers, runs Tech Insights scorecards, and its Catalog Completeness beta suggests owners for repositories. Limit: deleting a location does not remove the entities found through it.

If you already run Backstage, the checks in this post fit in a standard-library Python script that runs in CI. What the products add is the plumbing around them: syncing from HR and identity systems, a place where scores are shown, and someone else keeping it running. Whichever way you go, test it the way we tested ours. The repository has a fixture catalog with one of each kind of drift; point your tool at something like it and confirm it fails on each one. A tool you have only seen report "all good" has not shown you that it can find anything.

What we could not conclude

Whether catalogs decay with age. The difference we saw (8.0% against 11.4%) is within chance for this sample, and one snapshot cannot measure a rate.
Whether three owners in four are right. 159 owners could not be checked from outside. Among the 58 we could check, 1 entry had no owner and 4 drew a warning, and that rate does not carry over to the rest.
Whether a disagreement means a wrong owner. CODEOWNERS and the catalog can mean different things by "owner".
Whether runbooks and on-call links rot. None were identified in public catalogs.
How private catalogs compare. Every entry here is in a public repository.
Whether a wrong entry ever caused a bad page."Can be worse than none" is an argument about how people use catalogs during incidents, not something this data measures.

Summary

Check a catalog against the systems it describes, not against itself. Ours found 21 entries out of 217 where GitHub contradicts the entry or its CODEOWNERS, all of which looked fine in their own files.
Ownership is where the records disagree. Where two records of the owner existed, half disagreed or named an owner GitHub could not resolve.
From outside an org, most owners cannot be checked at all. Run the check where the token can see your teams, and map directory groups to GitHub teams explicitly.
Make owners the same string everywhere by deriving them from GitHub teams or your identity provider, then compare catalog and CODEOWNERS in CI.
Run the check on a schedule too: a team deleted in a reorg changes no file.
Retire entries nobody claims. A gap sends people asking; a wrong answer can send them to the wrong place.

The checker, the sample, every result and the tests are in The-DevOps-Daily/service-catalog-drift. Run it against your own catalog, with a token that can see your teams, and see how your numbers compare.

Originally published at devops-daily.com.

PHP2Shell: How to Check That Every PHP You Run Is Actually Patched

DevOps Daily — Thu, 22 Oct 2026 09:00:00 +0000

On 21 October 2026 the PHP project released PHP 8.5.12, 8.4.27, 8.3.36 and 8.2.35 to fix CVE-2026-109618 and CVE-2026-109619, the pair now called PHP2Shell. Both are rated critical, CVSS v4.0 9.2. Unauthenticated remote code execution has been demonstrated against Apache with mod_php before any application code runs, and PHP says every installation that serves web requests is at risk.

So update. The trouble is that "we updated" can be true while the server is still vulnerable. Ubuntu and RHEL fix PHP without changing the version number, so php -v says the same thing before and after. PHP-FPM and Apache keep running the old code until they restart. A fixed base image fixes nothing until your image is rebuilt and the old containers are gone. This post shows how to find every PHP you run, how to prove each one is fixed, and what the interim settings do and do not block. It comes with a read-only check script, and a registry scan of 39 popular PHP images: by their own metadata, 8 still shipped a vulnerable PHP 4.4 hours after the scheduled release, and 3 still did at 5.5 hours.

What to do now

Update PHP everywhere to 8.5.12, 8.4.27, 8.3.36 or 8.2.35, or to your distribution's fixed package. On Ubuntu and RHEL the fixed package keeps the old upstream version, so check the package version, not php -v.
Restart every PHP process : PHP-FPM, Apache with mod_php, php-cgi, LiteSpeed and FrankenPHP. Then confirm nothing still runs the replaced binary.
Rebuild and redeploy containers , then check what the running containers contain, not what the registry has.
If you cannot update today , set session.upload_progress.enabled=0 (and the other settings below) in php.ini or the FPM pool, restart, and treat it as partial. PHP says only the update is a complete fix.
PHP 8.1 and older get no upstream release. Use a vendor backport or move to 8.2 or newer.
Run the check on every host and keep the output as your evidence.

Prerequisites

Shell access to the hosts that run PHP, and root if you want to see processes owned by other users.
Python 3.6 or newer for the check script. It uses only the standard library.
The docker or podman CLI on hosts that run PHP in containers, and kubectl for clusters.

What was fixed

PHP published two advisories:

CVE-2026-109618 , GHSA-w8hw-w9fx-j79w, "RCE via OOM corruption". Out-of-memory handling can interrupt PHP while request data is only partly initialized, and request cleanup can then treat that data as valid.
CVE-2026-109619 , GHSA-6wmf-4rv5-p3rp, "RCE via stale global state". State owned by one request can survive after its memory is freed, and a later request handled by the same worker can use it.

Branch	Fixed in	Note
8.5	8.5.12
8.4	8.4.27
8.3	8.3.36
8.2	8.2.35	End of life on 31 December 2026
8.1 and older	no upstream release	Source backports for 7.4, 8.0 and 8.1 only

The timeline, all on 21 October unless noted (UTC):

20 October : PHP announced the release for 21 October at 14:00 UTC and reserved the right to release earlier if details became public.
02:29 : VirusTotal saw the first submission of a leaked archive with technical details.
13:34 : the GitHub advisories were published. Ubuntu's USN-8948-1 followed at 13:56.
14:00 : the scheduled release time. The php.net announcements carry no time of day; NVD published both CVEs at 14:17.
16:03 : the last of the 16 official php tags we scanned for 8.2 to 8.5 (linux/amd64) was built with a fixed PHP. Docker Hub showed the new version tags as pushed between 16:48 and 17:31.

The first report took about ten weeks to reach a release. Calif, which credits the discovery to Calif, OpenAI and Wiz, dates discovery to 1 August 2026, the first report to PHP to 11 August and the second to 21 August. Calif traces the earliest affected code path to PHP 5.4.0, and says a related out-of-memory issue found in a 2024 audit was left unfixed because exploitation was not proven. In OpenAI's account, Greg Brockman wrote that OpenAI had spent the past two months "leading a coalition to mitigate" the bug, found as part of its Patch the Planet initiative. OpenAI's own write-up is here.

Who is really exposed

PHP's advisory lists "all PHP installations serving web requests, including PHP-FPM, Apache with mod_php, php-cgi, FrankenPHP and LiteSpeed" and puts mod_php at the top. The GitHub advisories go into more detail per server API (SAPI):

SAPI	What the PHP advisories say
Apache mod_php	Highest risk. Unauthenticated RCE validated over HTTP with default limits, before any application code runs.
PHP-FPM	At risk. The CVE-2026-109619 session path was validated against FPM, but it needs a suitable application operation. For CVE-2026-109618, source analysis says FPM is susceptible, but no complete FPM exploit was shown. The advisory also names per-request settings sent by the web server (`PHP_VALUE` or `PHP_ADMIN_VALUE`) that produce a warning as a risk for FPM.
php-cgi	Listed as at risk by php.net. The advisories do not discuss it separately.
FrankenPHP	Listed. Source analysis says that, like mod_php, it is likely susceptible to CVE-2026-109618.
LiteSpeed (LSAPI)	Listed. Application-specific exploits are described as likely but not validated.
CLI	Not named. The bugs are in web request handling. But `php -S` serves HTTP from the CLI binary, and the CLI usually comes from the same source package as FPM and mod_php, so an old CLI is a sign of what else on the host is old.

Red Hat rates both bugs Important, not Critical, because "Standard Red Hat Enterprise Linux deployments use PHP-FPM rather than mod_php." That is a statement about the default, not about your server. PHP names mod_php as the highest-risk setup, so if you run mod_php on RHEL, put it first even though Red Hat's rating is Important.

Note

The sources disagree on the affected range. PHP's advisories and the CVE.org records list only 8.2 to 8.5. NVD marks every version below 8.2.35 as vulnerable, with no lower bound. Google Cloud says "All versions of PHP from 5.4 through 8.5.11", and PHP itself shipped source backports for 7.4, 8.0 and 8.1. Treat every PHP older than the fixed releases as affected, including end-of-life branches.

Why `php -v` does not tell you

Debian ships the new upstream release, so on Debian php -v changes when you are fixed: bookworm gets 8.2.35 and trixie gets 8.4.27. Ubuntu and RHEL backport the patch into the version they already ship. Ubuntu 24.04's fixed php8.3 still reports 8.3.6. So does the vulnerable one.

To show this with real packages, the repo's scripts/ubuntu-backport-demo.sh downloads php8.3-fpm and php8.3-common from the Ubuntu archive, from before the fix (.12) and with the fix (.13). It extracts them without installing or running anything, reads the PHP version compiled into php-fpm8.3, and searches the package changelog for the CVE IDs:

ubuntu-backport-demo.sh

$ scripts/ubuntu-backport-demo.sh
# 2026-10-21T18:26:56Z Ubuntu 24.04 php8.3 packages from http://archive.ubuntu.com/ubuntu/pool
sha256 60187aae56db2056e916d99ba4854415993429f12732ace1b5eb5225bb33f584 php8.3-fpm_8.3.6-0ubuntu0.24.04.12_amd64.deb
sha256 85e4fc03ff84fef7dfed923c99d1a2e0e6f49147a50bcd5eaca4e7cd05a835fa php8.3-common_8.3.6-0ubuntu0.24.04.12_amd64.deb
sha256 ae395e2c15c3b96da56da807a204f771a589eb5bf54a3e5e239b1f50ade2815b php8.3-fpm_8.3.6-0ubuntu0.24.04.13_amd64.deb
sha256 801cae4e842e1b9e9d60a542bf18ba831bf504b2f3070404892bf3f8fb943ea4 php8.3-common_8.3.6-0ubuntu0.24.04.13_amd64.deb

package 8.3.6-0ubuntu0.24.04.12
  php-fpm8.3 embeds: PHP/8.3.6
  changelog top: php8.3 (8.3.6-0ubuntu0.24.04.12) noble-security; urgency=medium
  PHP2Shell IDs: none
package 8.3.6-0ubuntu0.24.04.13
  php-fpm8.3 embeds: PHP/8.3.6
  changelog top: php8.3 (8.3.6-0ubuntu0.24.04.13) noble-security; urgency=medium
  PHP2Shell IDs: CVE-2026-109618 CVE-2026-109619

Both binaries carry the same version. Only the package version and its changelog say which one has the fix. This is what the fixed packages looked like on the evening of 21 October:

System	Fixed package	Does `php -v` change?
Debian 12 bookworm	php8.2 8.2.35-1~deb12u1 (DLA-4848-1)	Yes, to 8.2.35
Debian 13 trixie	php8.4 8.4.27-1~deb13u1 (DSA-6574-1)	Yes, to 8.4.27
Ubuntu 26.04	php8.5 8.5.4-0ubuntu1.5 (USN-8948-1)	No, stays 8.5.4
Ubuntu 24.04	php8.3 8.3.6-0ubuntu0.24.04.13	No, stays 8.3.6
Ubuntu 22.04	php8.1 8.1.2-1ubuntu2.28	No, stays 8.1.2
Ubuntu 14.04 to 20.04 (ESM)	php5, php7.0, php7.2, php7.4 (USN-8948-2)	No
RHEL 8.10 EUS, php:7.4 module	php-7.4.33-8.module+el8.10.0+25117+30337cca (RHSA-2026:87080)	No, stays 7.4.33
RHEL 8 php:8.2, RHEL 9, RHEL 10	No fix listed at 16:49 UTC
packages.sury.org (Debian 12, arm64)	Still at 8.2.34, 8.3.35, 8.4.26 and 8.5.11 at about 18:10 UTC	Yes, when it lands
rpms.remirepo.net	php.net: "The repo will not have updates today"	Yes, when it lands

The same trap works in the other direction. A scanner that compares only the upstream version reports fixed Ubuntu and RHEL packages as vulnerable.

The right check compares the package version with your distribution's fixed version for that package, or finds both CVE IDs in the changelog of the package that owns the binary. The examples use Ubuntu 24.04's php8.3-fpm; take the fixed version for your release and package from its USN, DSA or RHSA:

# Ubuntu 24.04, php8.3-fpm from the Ubuntu archive (fixed in USN-8948-1)
dpkg --compare-versions "$(dpkg-query -W -f='${Version}' php8.3-fpm)" ge 8.3.6-0ubuntu0.24.04.13 \
  && echo fixed || echo NOT fixed

# Which package owns the binary that actually runs, and what its changelog says
dpkg-query -S "$(readlink -f /usr/sbin/php-fpm8.3)"
zcat /usr/share/doc/php8.3-fpm/changelog.Debian.gz | head -1
for id in CVE-2026-109618 CVE-2026-109619; do
  zgrep -q "$id" /usr/share/doc/php8.3-fpm/changelog.Debian.gz && echo "$id listed" || echo "$id NOT listed"
done

# RHEL, AlmaLinux, Rocky Linux: ask about the package that owns the binary
rpm -qf "$(readlink -f /usr/sbin/php-fpm)"
for id in CVE-2026-109618 CVE-2026-109619; do
  rpm -qf --changelog "$(readlink -f /usr/sbin/php-fpm)" | grep -q "$id" && echo "$id listed" || echo "$id NOT listed"
done

# Alpine packages carry the upstream version
apk info -v | grep -E '^php[0-9]+(-fpm|-cgi|-apache2|-embed)?-[0-9]'

Two details make the changelog check less simple than it looks:

On Debian and Ubuntu, /usr/share/doc/php8.3-fpm is usually a symlink to php8.3-common's docs. If an upgrade was partial and php8.3-fpm was held back, you read the newer package's changelog and conclude the old FPM binary is fixed. Check that the first line of the changelog names the version that dpkg-query -W php8.3-fpm reports. The check script refuses a changelog whose top entry does not match the installed package.
Container images often strip /usr/share/doc. Then there is no changelog to read, and the package version against the distribution's tracker is the only evidence.
A changelog line is a mention, not proof. A fix that was later reverted would still be listed. Treat it as supporting evidence next to the package version.

Restart, or the old code keeps running

Upgrading the package replaces the files on disk. The PHP-FPM master and its workers, and every Apache child with mod_php loaded, still have the old binary mapped in memory. Linux marks a mapped file that has been replaced as (deleted), which gives you a direct check:

# Run as root, or processes owned by other users are not visible.
# PHP-FPM, php-cgi, LiteSpeed and FrankenPHP processes on a replaced binary
sudo ls -l /proc/[0-9]*/exe 2>/dev/null | grep -E '(php|frankenphp).*\(deleted\)'

# Apache processes that still map a replaced libphp
sudo grep -l 'libphp.*(deleted)' /proc/[0-9]*/maps 2>/dev/null

Restart the service (systemctl restart php8.3-fpm, systemctl restart apache2), run both commands again until they print nothing, and then check the version or package of the binary the service runs. An empty result only says no replaced file is still mapped; it does not say the file on disk is the fixed one. needrestart on Debian and Ubuntu and dnf needs-restarting on RHEL also list processes that need a restart after an upgrade. PHP's own advice for the interim settings is to "restart every worker" as well, because a setting changed in php.ini does not reach a worker that is already running.

Containers: rebuild, redeploy, then check what is running

A container keeps the PHP it was built with. Google's guidance for Cloud Run says it plainly: services without automatic base image updates "must be rebuilt and redeployed", and "Container images you build yourself are your responsibility."

Rebuild with --pull, so the builder fetches the new base instead of a cached one:

docker build --pull -t registry.example.com/app:2026-10-21 .

If your Dockerfile pins a digest (FROM php:8.4-fpm@sha256:...), update the pin. If it pins a version tag, move to 8.4.27-fpm or later. Then deploy the new tag or digest. On Kubernetes, kubectl rollout restart with a reused tag and imagePullPolicy: IfNotPresent starts the new pods from the image already cached on the node, which is the old one.

Then check the containers that are running, not the image you meant to deploy:

# Docker hosts (use podman the same way): the PHP CLI version in each running container
docker ps --format '{{.ID}} {{.Image}} {{.Names}}' | while read -r id image name; do
  printf '%-30s %-40s ' "$name" "$image"
  docker exec "$id" php -r 'echo PHP_VERSION, PHP_EOL;' 2>&1 || echo "PHP CLI check failed"
done

# Kubernetes: the image digests your pods are actually running
kubectl get pods -A -o jsonpath='{..imageID}' | tr ' ' '\n' | sort -u

The CLI version is a first look, not proof for the web server: a container can have a fixed CLI and an older FPM or mod_php, so check the binary that serves requests (php-fpm -v, or its package) as well. The repo's registry_scan.py takes repository-qualified digests (registry/repository@sha256:...) as arguments and reads the PHP version from each image's metadata without pulling it. Some runtimes report imageID in other forms, such as a bare digest, which you need to map back to a repository first. That works for images built FROM php, which carry ENV PHP_VERSION. For images that install PHP from distribution packages, look inside a running container, because the version is not in the metadata.

How fast the popular images were rebuilt

We wanted to know how long "pull the new image" takes to become true, so registry_scan.py read the PHP version of 39 popular images from registry metadata only: the manifest and the image config, a few KB each, checked against their digests, never a layer. The version is the one the image declares in ENV PHP_VERSION (or Bitnami's APP_VERSION), which images built FROM php inherit. An image could replace PHP after that, so this is what the images report, not a look at the binary. The list is the official php image for 8.1 to 8.5, the official application images built on it, and widely used community images. images.txt in the repo says why each one is there. All results are for linux/amd64.

We ran the scan twice, at 18:21 UTC and at 19:32 UTC, 4.4 and 5.5 hours after the scheduled release:

39 popular PHP images by the PHP version their metadata reports

	Value	Series
18:21 UTC: vulnerable	8	vulnerable
18:21 UTC: unknown	2	unknown
18:21 UTC: patched	29	patched
19:32 UTC: vulnerable	3	vulnerable
19:32 UTC: unknown	2	unknown
19:32 UTC: patched	34	patched

linux/amd64. PHP version from ENV PHP_VERSION (or Bitnami APP_VERSION) in each image config, read without pulling layers. Fixed: 8.5.12, 8.4.27, 8.3.36, 8.2.35; 8.1 and older count as vulnerable. Unknown: no full version in the metadata. Files: results/scan-20261021T182129Z.json, results/scan-20261021T193200Z.json.

All 16 official php tags we scanned for 8.2 to 8.5 reported a fixed PHP in both runs, built 1.8 to 2.1 hours after the scheduled release.
At 18:21, 8 images still shipped a vulnerable PHP : php:8.1-fpm and php:8.1-apache (8.1.34, last built in December 2025; there is no fixed 8.1 release to rebuild with), joomla:latest (8.4.26), nextcloud:latest and nextcloud:fpm (8.5.11), phpmyadmin:latest (8.3.35), yourls:latest (8.5.11) and laravelsail/php84-composer:latest (8.4.1, built in November 2024). The Laravel documentation uses that image to run Composer, not to serve traffic.
At 19:32, 3 were still vulnerable : the two php:8.1 tags and laravelsail/php84-composer:latest. joomla, both nextcloud tags, phpmyadmin and yourls had fixed images by then.
2 images could not be judged from metadata in either run: thecodingmachine/php:8.4-v5-fpm sets PHP_VERSION=8.4 without the patch level, and linuxserver/nextcloud installs PHP from Alpine packages and records no PHP version at all.

Hours from the scheduled 14:00 UTC release to the image build

Series	Samples	Min	Median	p95	Max
Official php 8.2 to 8.5 (16)	16	1.83 h	1.935 h	2.06 h	2.06 h
Official app images (13)	13	2.47 h	2.53 h	4.08 h	4.08 h
Community images (5)	5	2.23 h	3.68 h	4.05 h	4.05 h

Patched images only, by the 'created' time in each linux/amd64 image config, from results/scan-20261021T193200Z.json. This is when the image was built, not when the registry started serving it. Images still on a vulnerable PHP are not on this chart.

One result changes how you should read a registry. Docker Hub listed joomla:latest as pushed at 15:46 UTC, after the release. Its amd64 image was built at 01:52 UTC with PHP 8.4.26. So a fresh "last pushed" time is not evidence that the image you pull has a fixed PHP. The build time is not either: the fixed joomla, phpmyadmin and yourls images that the registry served at 19:32 were built between 16:28 and 16:36 UTC, but at 18:30 the registry still served the old ones for amd64. nextcloud:latest and nextcloud:fpm looked the same at 18:21; by 18:30 their amd64 images had moved to 8.5.12. To check the method, at 18:30 UTC we compared the amd64 digest from the Docker Hub API with the registry's own index for those three images and two others: they agreed for all five (results/method-check-20261021T183037Z.txt).

The interim settings and their limits

If you cannot update today, PHP lists settings that reduce exposure to specific paths. Its own words: they "reduce exposure to specific attack paths but do not make an installation safe."

For CVE-2026-109619, turn off session upload progress, and on PHP 8.5 fatal error backtraces. Set them where they apply before PHP parses the request: php.ini, the FPM pool, or the Apache config. PHP says not to rely on ini_set() or .user.ini, which apply too late.

; php.ini (and the php.ini of every SAPI: cli, fpm, apache2, cgi)
session.upload_progress.enabled = 0
; PHP 8.5 only
fatal_error_backtraces = 0


; PHP-FPM pool, for example /etc/php/8.3/fpm/pool.d/www.conf
php_admin_flag[session.upload_progress.enabled] = off


# Apache with mod_php
php_admin_flag session.upload_progress.enabled off

Then restart every worker. What this does and does not do, per the advisory:

It "prevents the demonstrated early session producing and consuming operations. It does not mitigate the other lifecycle paths." fatal_error_backtraces=0 "blocks the confirmed fatal backtrace path only."
PHP lists it under CVE-2026-109619 only. For CVE-2026-109618 PHP suggests enforcing memory_limit=-1 so applications cannot override it, and warns that an underlying allocation failure can still reach the bug. With no PHP memory limit, use a container or cgroup limit as the availability boundary, and do not use RLIMIT_AS or RLIMIT_DATA, which can cause the failure you are trying to avoid.
Hiding warnings does not help. Changing error_reporting, display_errors or log_errors, or using @, does not protect against CVE-2026-109619. Instead, remove per-request settings that produce warnings or deprecation notices, including PHP_VALUE or PHP_ADMIN_VALUE lines your web server passes to FPM.
If your application does not use multipart form data, PHP suggests blocking it at the frontend. It also suggests a low post_max_size (8M for applications that do not need file uploads, 32M for those that do) and proxy limits that also cover requests without a usable Content-Length.
Edge rules help but do not replace the update. Help Net Security reports that Cloudflare deployed WAF rules for the known exploit vectors to all customers, including Free, and VirusTotal reports its CTO saying that only updating PHP fully protects a site.

PHP also suggests keeping session.upload_progress.enabled=0 after you patch if you do not use upload progress bars, "as more attackers try to find flaws in this hardly-used and hardly-tested feature."

PHP 8.1 and older

The PHP project does not release fixes for 8.1 or older. It published minimal source backports for 7.4, 8.0 and 8.1 for people who build from source, and points everyone else to their vendor. Some vendors have shipped them: Ubuntu for 22.04's PHP 8.1 and, under ESM, for 14.04 to 20.04, and Red Hat for the RHEL 8.10 EUS php:7.4 module.

The official php:8.1 tags we scanned still reported 8.1.34, built in December 2025. Google's guidance says App Engine applications on PHP 5.5, 7.2, 7.3, 7.4 and 8.1 remain vulnerable and must move to 8.2 or newer. VirusTotal counted about 758,000 internet-facing hosts that announce an end-of-life PHP version from 5.4 to 8.1, out of about 1.1 million that announce a version at all. That counts exposed PHP, not confirmed vulnerable hosts, but it shows how much of the long tail has no simple update.

If you are on 8.2, the same applies on 31 December 2026, when it reaches end of life.

Run the check

The-DevOps-Daily/php2shell-check on GitHub

php2shell_check.py is one file of standard-library Python, so you can copy it to a server and run it. For each PHP it finds, it reports the SAPI, the version compiled into the binary or Apache module (so mod_php is checked without running Apache; -v is the fallback for binaries without a readable string), the package that owns the file, and whether that package's changelog names both vulnerabilities by CVE or GHSA ID. A changelog only counts if it belongs to the installed package and dpkg --verify or rpm -V confirms the binary still matches that package. It also reports the interim settings from a fresh php -i, the FPM pool files, or the parsed php.ini files for mod_php, plus host web server lines that set them or pass PHP_VALUE. A running service started with other options can differ. Then it lists running PHP processes and marks any that still run a replaced binary. Through the first container CLI it finds (docker, then podman), it checks the PHP binaries inside each running container that has a PHP process, and adds an unknown result when no checked binary covers the kind of PHP process running there (for example an FPM process but only a CLI binary found). The interim settings are reported only: they never turn a vulnerable result into a pass.

This is the real output on the Raspberry Pi we build on, which has the PHP 8.4.23 CLI from packages.sury.org and no web server:

php2shell-check on the Pi

# 2026-10-21T18:45:30Z Raspberry Pi, Debian 12, PHP CLI from packages.sury.org, no web server, run as a normal user
$ python3 php2shell_check.py
php2shell-check: fixed releases 8.2.35, 8.3.36, 8.4.27, 8.5.12

PHP on disk
  [vulnerable] /usr/bin/php8.4 cli
      version: 8.4.23 (embedded string)
      exposure: CLI: not a web SAPI unless it runs `php -S`; same package line often ships FPM/mod_php
      also as: /bin/php, /bin/php8.4, /usr/bin/php
      package: php8.4-cli 8.4.23-1+0~20260703.52+debian12~1.gbp817a0c (dpkg); CVE IDs in changelog: none
      why: upstream 8.4.23 < 8.4.27
      session.upload_progress.enabled: On [php8.4 -i]
      memory_limit=-1 post_max_size=8M

Running PHP processes
  none

Containers
  none checked
note: no docker or podman CLI found; container processes are judged by the version in /proc

result: VULNERABLE or restart needed (exit 1)
$ echo $?
1

The exit code is 1 when anything is vulnerable or needs a restart. Otherwise it is 2 when anything could not be checked, and 0 when every PHP it found is patched or it found no PHP at all (it says which). Anything left unknown, such as a changelog that lists only one of the two CVEs on an older version, a pre-release build, or an unreadable process, gives 2 rather than 0, unless something else is vulnerable, which gives 1. It only knows the paths and names it searches, so a 0 means "nothing vulnerable found", not proof that no other PHP exists. To run it across hosts without copying it first:

for h in web1 web2 web3; do
  ssh "$h" 'sudo python3 - --json' < php2shell_check.py > "php2shell-$h.json"
  echo "$h exit $?"
done

The test suite builds fake hosts with the cases a version check gets wrong: an Ubuntu backport, the same version without the update, a partial upgrade whose symlinked changelog belongs to another package, an old binary copied over a patched package, a changelog with only one of the two CVEs, a source build where FPM was left on the old version, a process running a deleted binary, and containers on old and new images.

How to prove you are fixed

For each host, keep evidence of all of these, not just the first:

Package : the upstream version is at or above the fixed release, or the package version is at or above your distribution's fixed version.
Processes : no PHP-FPM, php-cgi, LiteSpeed or FrankenPHP process runs a (deleted) binary, and no Apache process maps a (deleted) libphp.
Containers : the binary that serves requests inside each running container (php-fpm, mod_php, FrankenPHP) has a fixed version or a fixed package, not just the CLI. The image digest you deployed is the one running.
Images : every image you build FROM a PHP base has been rebuilt since the fix. The registry scan of its digest is a quick inventory check; the container check above is the proof.
Exposure while vulnerable : if a host was internet-facing and unpatched, check it for compromise. The NCSC asks for this, and Wiz suggests looking for unusual multipart requests followed by memory-limit errors or repeated HTTP 500 responses, unexpectedly large or malformed PHPSESSID response cookies, and unexplained worker churn.

What we could not check

Containers on a real host. The Pi we build on has no Docker, so the container checks are covered by the test fixtures only. The registry scan read real images.
Web servers. The Pi runs PHP from the command line only. mod_php and FPM detection are covered by the tests.
Whether the interim settings block anything. We did not test any part of the bugs, and the tool does not. PHP's own text is the source for what the settings cover.
Later distribution updates. The distribution table is as of the evening of 21 October. At 16:49 UTC Red Hat listed a fix only for the RHEL 8.10 EUS php:7.4 module, the Debian 12 arm64 index on packages.sury.org still had the old versions at about 18:10 UTC, and PHP said Remi would not have updates that day.
Other architectures. The registry scan reads linux/amd64 only. We did not measure arm64 images.
Unusual install paths. The script searches distribution paths, Remi, cPanel, Plesk, CloudLinux alt-php, LiteSpeed, aaPanel and $PATH. PHP elsewhere, .htaccess overrides and stopped containers are not covered.

Checklist

Every host: PHP package updated, checked against the distribution's fixed version, not php -v
Every PHP-FPM pool, Apache, php-cgi, LiteSpeed and FrankenPHP service restarted, with no (deleted) PHP binaries left in /proc
Every image built FROM a PHP base rebuilt with --pull and redeployed by new tag or digest
Running containers checked inside, including the binary that serves requests, and their image digests matched to what you deployed
PHP 8.1 and older: vendor backport installed or upgrade to 8.2+ scheduled, with interim settings in place until then
Interim settings applied where updates wait, in php.ini or the pool, with workers restarted
session.upload_progress.enabled=0 kept where upload progress bars are not used
Internet-facing hosts that were unpatched checked for signs of compromise
Check script output saved per host as evidence

Every fact in this post is listed with its source in the repo's SOURCES.md, together with the points where the sources disagree.

Originally published at devops-daily.com.

Node.js October 2026 Security Release: Finding Every Node You Actually Run

DevOps Daily — Thu, 22 Oct 2026 09:00:00 +0000

On 21 October 2026 the Node.js project released 22.23.4, 24.22.1 and 26.12.1. The headline fix is CVE-2026-111497, an out-of-bounds write in the Brotli decoder that the advisory says "permits remote code execution". The same release fixes two TLS certificate hostname-check bypasses, a DNS bug that truncates names at a NUL character, and updates the bundled CA store. End-of-life lines get no fix.

Patching Node is a one-line version bump. Finding every Node your servers actually run is the work. When we patched a small production box on release day, six Node apps under pm2 installed with nvm, installing the new version was the easy part. Then we found pm2 entries with the old binary written into them, a systemd unit with the old PATH hard-coded, child processes we had not thought about, and cron jobs on two other boxes that pointed at the old install by absolute path. None of them moved when the default version changed.

This post covers what to do now, who is exposed (including services that call fetch() on a URL a user controls), the upgrade steps for nvm, pm2, systemd and containers with the traps we hit, how to verify with /proc, and a read-only audit script with its real output from one of our machines.

What to do now

Upgrade every Node to 22.23.4, 24.22.1 or 26.12.1 or later. Node 20, 25 and older are end-of-life and get no fixed release. Move them to 22, 24 or 26.
List every place Node runs or will start : running processes, pm2 entries (live and saved), the pm2 daemon, systemd units, crontabs and helper scripts, and containers. Changing the nvm default moves none of them.
Move each one with the steps below, then check the real binary of every process in /proc. Process names do not help: on our build machine pgrep node found 0 of 5 Node processes.
Rebuild images with --pull and redeploy. The official node:22, node:24 and node:26 tags have the fix. node:20, node:25 and node:18 will not get it.
If a service cannot be upgraded today and accepts Brotli data from strangers, close that path in front of it (details below). For outbound fetch() the fix is the upgrade; until then, turning off features that fetch user-supplied URLs is the way to close that path.
Run the audit script on every host and keep the output.

Prerequisites

Shell access to the hosts that run Node, as the user that owns the Node processes. Root is not needed.
Python 3.8 or newer for the audit script. It uses only the standard library and only reads.
The docker or podman CLI on hosts that run Node in containers.

What was fixed

The advisory lists ten vulnerabilities. The four rated high:

CVE	What the advisory calls it	Lines
CVE-2026-111497	Brotli decoder out-of-bounds write permits remote code execution	22, 24, 26
CVE-2026-105483	TLS certificate hostname bypass when the IP is written in fullwidth digits	24, 26
CVE-2026-106069	TLS certificate hostname bypass when the host loosely resolves to an IP address	22, 24, 26
CVE-2026-105482	`dns.resolve` truncates names containing a NUL character	22, 24, 26

The medium ones are a keep-alive socket that processes a response written before the request was sent (CVE-2026-105481), DNS SANs that can authenticate IP endpoints written in a non-canonical IPv4 form (CVE-2026-106072), and crypto.randomUUID() and randomInt() caches saved into startup snapshots and single executable applications, so that every process repeats the same values (CVE-2026-106074). The low ones are two Permission Model bypasses and a ZIP header issue on 26.x.

The release also updates the bundled root certificates to NSS 3.131.1 and makes Node reject certificate chains that contain blocked public keys. The advisory lists this as the "ca-cert (eMudhra Intermediate CA Compromise)" update. The bundled undici, c-ares and OpenSSL (22 and 24) are updated too.

Two things the advisory does not give you yet. There is no technical detail on any of the bugs, and on the morning of 22 October all ten CVE IDs were still in the RESERVED state at CVE.org, with no NVD record. So there is no CVSS score; "high" is the Node.js project's rating.

The timeline in UTC, from the release metadata:

21 Oct, 10:32 to 10:36 : the fixed binaries on nodejs.org/dist are dated in this window. The GitHub releases were published at 10:38.
10:48 : the update to the official Docker images was committed.
about 11:35 : the Debian-based official images were built. Docker Hub shows the amd64 and arm64 images as pushed at about 14:40.
19:27 : the Brotli project released 1.2.1, 1.1.1 and 1.0.10, each with "CVE-2026-111497: out-of-bounds write permits remote code execution" in the notes. So the bug is in the Brotli decoder itself, and Node shipped the upstream fix as a cherry-picked commit.
22:58 to 00:08 : the Alpine-based official images were built, about 12 to 13.5 hours after the release.

Who is exposed

Brotli: any Node process that decodes Brotli someone else chose

The decoder is compiled into the node binary and sits behind node:zlib. Every way into it counts, and three are common:

Your code decompresses Brotli directly : zlib.brotliDecompress(), brotliDecompressSync() or createBrotliDecompress() on uploads, archives, cache entries or messages.
Your service fetches URLs. fetch() in Node is the bundled undici, and it decodes responses that say Content-Encoding: br with zlib.createBrotliDecompress. It goes by the response's header, whatever the request asked for, so setting Accept-Encoding: identity does not keep Brotli away from a hostile server. axios in Node does the same unless you pass decompress: false. Link previews, webhook testers, feed readers, "import from URL" features and AI agents that browse are all services that fetch a URL a user chose.
Your service accepts compressed request bodies. body-parser 2 decompresses br request bodies that it parses, and its inflate option defaults to true. Express 5 ships body-parser 2, so express.json() on Express 5 decodes Brotli bodies sent with a JSON content type, from any client. Express 4 uses body-parser 1.x, which handles only gzip and deflate.

Reaching the decoder is not the same as being exploitable, and the advisory does not say what input triggers the bug. These are the paths to upgrade or close first. Because the decoder is in the binary, no npm update fixes it. Upgrading the undici package in your package.json, for example, still decodes through the old Node's zlib.

If you cannot upgrade a service today, the interim options close only the paths you control:

// Express 5: refuse compressed request bodies (415) until Node is upgraded.
// This also refuses gzip and deflate request bodies.
app.use(express.json({ inflate: false }));


# In front of any Node service: refuse Brotli request bodies.
if ($http_content_encoding ~* "br") {
    return 415;
}

Distribution builds are different. Debian's nodejs package (we checked the trixie build of 20.19.2) is built with --shared-brotli against the system libbrotli, so on Debian the Brotli fix comes from the distribution's brotli package, not from the Node version number. At the time we checked, neither Debian's security tracker nor Ubuntu's notices listed CVE-2026-111497 yet. The audit script reports such builds as unknown rather than guessing.

TLS hostname checks and DNS

When Node connects over TLS it checks the server's certificate against the host you asked for. Both high-severity TLS bugs are in that check when the host is, or loosely resolves to, an IP address written in a non-standard form: fullwidth digits (24 and 26 only) or a form that "loosely resolves to an IP address" (all lines). The fix makes Node reject non-canonical IP addresses. The advisory gives no more detail than that. Our reading is that the services to look at first are the ones where someone else chooses the host string: webhook senders, URL fetchers and integrations that connect to customer endpoints.

The DNS bug truncated names at a NUL byte in dns.resolve*(), and the fix rejects such input. Code that resolves names it got from users is where this matters.

End-of-life lines

"End-of-Life versions are always affected when a security release occurs," says the advisory. Node 20 reached end of life on 30 April 2026 and Node 25 on 1 June 2026. Neither gets a fixed release, and the official node:20 and node:25 image tags stay on 20.20.2 and 25.9.0.

Why changing the default moves nothing

nvm installs the new version next to the old one, which is what you want: the old one stays for rollback. nvm alias default 24.22.1 then changes what new shells get. Nothing that is already running reads that alias, and nothing that stored a path to the old install reads it either.

nvm install 24.22.1 next to the old version
nvm alias default new shells get it
What runs the new Node? check /proc, not config

Outcomes:

New shells, and pm2 apps restarted with --update-env from one
pm2 daemon, cluster workers, absolute interpreters, systemd unit, cron, images: still old

On the production box, each of these needed its own step:

pm2 entries with an absolute interpreter. Several entries had the old Node binary written into them as the interpreter, and the ones started as npm or npx had their script path under the old version's bin directory. Changing the default did not move them. They had to be deleted and recreated with the new interpreter.
pm2 entries with a bare node interpreter took the new binary only after pm2 restart <name> --update-env from a shell where the new Node came first in PATH.
The pm2 systemd unit that pm2 startup wrote had the old version's PATH and the old pm2 path hard-coded. After editing it, pm2 update moved the pm2 daemon itself, which restarts every app at once. The sites on that box were down for under a minute while the Next.js apps started.
Child processes. Checking /proc/<pid>/exe for every process turned up chains such as npx to tsx to node, and npm start to next-server. Each process in a chain has its own binary.
Cron jobs and helper scripts on two other boxes had absolute paths to the old Node.

What pm2 really does on an upgrade

To check each of those behaviors on its own, lab/pm2-pin-lab.sh in the repo starts four throwaway apps in a separate PM2_HOME from a shell where Node v24.14.0 comes first in PATH: bare (pm2 start server.js), via-npm (pm2 start npm -- start, where the start script is node server.js), abs (--interpreter set to the old binary) and clustered (-i 1, cluster mode). Then it puts v24.22.1 first in PATH, as a changed nvm default does, and records the real binary of every process after each step. A plain pm2 restart left all four on v24.14.0. This is the state after pm2 restart <name> --update-env for each app:

pm2-pin-lab.sh, step B2

# excerpt of results/pm2-lab-20261022T063002Z.txt (2026-10-22T06:30:02Z, aarch64, pm2 6.0.14)
# old: v24.14.0, new: v24.22.1 first in PATH; a plain restart left all four apps on v24.14.0
$ lab/pm2-pin-lab.sh <v24.14.0 bin> <v24.22.1 bin>
$ pm2 restart bare --update-env
$ pm2 restart via-npm --update-env
$ pm2 restart abs --update-env
$ pm2 restart clustered --update-env

== B2. pm2 restart --update-env from the new shell
daemon pid 1937570 node v24.14.0 <v24.14.0>/bin/node
bare interpreter=node script=<app>/server.js
           saved PATH finds node in <v24.22.1>/bin
  pid 1938284 node v24.22.1 <v24.22.1>/bin/node server.js
via-npm interpreter=<v24.14.0>/bin/node script=<v24.14.0>/bin/npm
           saved PATH finds node in <v24.22.1>/bin
  pid 1938375 node v24.14.0 <v24.14.0>/bin/node npm start
    pid 1938462 dash (not node) /usr/bin/dash sh -c node server.js
      pid 1938468 node v24.22.1 <v24.22.1>/bin/node node server.js
abs interpreter=<v24.14.0>/bin/node script=<app>/server.js
           saved PATH finds node in <v24.22.1>/bin
  pid 1938455 node v24.14.0 <v24.14.0>/bin/node server.js
clustered interpreter=node script=<app>/server.js (cluster mode)
           saved PATH finds node in <v24.22.1>/bin
  pid 1938518 node v24.14.0 <v24.14.0>/bin/node server.js

bare moved. via-npm is split: npm itself still runs on the old Node because pm2 stored the old binary as its interpreter, while its child node server.js got the new one from PATH. abs did not move. clustered did not move either, even though its saved PATH now finds v24.22.1: in cluster mode the daemon forks the workers with its own Node binary, so they move only when the daemon moves. All four steps, from results/pm2-lab-20261022T063002Z.txt:

App	After `restart`	After `restart --update-env`	After delete and recreate	After `pm2 update`
`bare` (bare `node`)	old	new	not recreated: new	new
`via-npm` (npm path stored)	old	npm old, app new	new	new
`abs` (absolute interpreter)	old	old	new	new
`clustered` (cluster mode)	old	old	not recreated: old	new
pm2 daemon	old	old	old	new

pm2 update restarted every app in one command: all four PIDs changed. On the production box that was the step where the sites went down. In the lab the command took 2.3 seconds; that measures the command, not how long the apps were unavailable, and real apps take as long as they take to start.

Upgrade steps: nvm, pm2 and systemd

Run these as the user that owns the pm2 daemon.

# 1. Install next to the old version and bring global packages (pm2) along.
nvm install 24.22.1 --reinstall-packages-from=24.14.0
nvm alias default 24.22.1

# 2. Work from a shell where the new Node comes first in PATH.
nvm use 24.22.1
command -v node && node --version

# 3. See what each pm2 entry is pinned to.
pm2 jlist | python3 -c 'import json, sys
for a in json.load(sys.stdin):
    e = a["pm2_env"]
    print(a["name"], e.get("exec_mode"), e.get("exec_interpreter"), e.get("pm_exec_path"), sep=" ")'

Then move each entry according to what step 3 shows:

# Fork mode, interpreter "node": pick up the new PATH.
pm2 restart api --update-env

# Interpreter is an absolute path to the old node, or the script is
# .../v24.14.0/bin/npm or npx: delete and start it again from this shell
# (or from your ecosystem file with the same name and arguments).
pm2 delete web
pm2 start npm --name web -- start

# Save the list that a reboot brings back.
pm2 save

# Regenerate the startup unit: the old one hard-codes the old PATH and pm2 path.
pm2 unstartup # prints a sudo command; run it
pm2 startup # prints a sudo command; run it

# Move the daemon, and every cluster-mode app with it.
# This restarts ALL apps at once: do it in a quiet window.
pm2 update

The pm2 docs ask for the unstartup and startup pair after every Node upgrade. The new unit is pinned again, this time to the new version. That is fine, as long as something checks it at the next upgrade.

Two details are easy to miss. pm2 save writes the saved list, including each entry's environment, so if you save before --update-env the reboot copy keeps the old PATH. And the daemon only reads the systemd unit when systemd starts it. If someone started pm2 by hand, the fixed unit applies at the next boot, not now.

Cron jobs and helper scripts

Cron runs with a minimal PATH, so jobs that need Node usually name it by absolute path, and that path names one install. The audit script below reads only your own crontab -l, so search the system crontabs and your scripts on every host too:

# Paths inside a versioned nvm, fnm, n, volta or asdf install
sudo grep -rnE '/(\.nvm/versions/node|node-versions|n/versions/node|\.volta/tools/image/node|installs/nodejs)/v?[0-9]+\.[0-9]+\.[0-9]+' \
  /etc/crontab /etc/cron.d /var/spool/cron /etc/systemd/system /usr/local/bin ~/bin 2>/dev/null

Update the path in each match. A job that points at a version you later uninstall stops running, which at least fails loudly; one that points at the old version you kept for rollback keeps running vulnerable and says nothing.

Containers

A container keeps the Node it was built with. Rebuild with --pull so the builder fetches the new base image instead of a cached one, deploy by the new tag or digest, and check what is running:

docker build --pull -t registry.example.com/app:2026-10-22 .

docker ps --format '{{.ID}} {{.Names}}' | while read -r id name; do
  printf '%-30s %s\n' "$name" "$(docker exec -e NODE_OPTIONS= "$id" node --version 2>&1)"
done

node --version inside a container reports the node on the container's PATH. If an image carries more than one Node, docker exec <id> readlink /proc/1/exe shows which binary the container's main process runs (when the image has readlink).

The repo's registry_scan.py read the Node version of 24 official node tags for amd64 and arm64 from registry metadata only (manifest and image config, no layers) at 06:23 UTC on 22 October. Every 22, 24 and 26 tag we sampled, and the latest, lts and alpine aliases, shipped the fixed version. How long they took:

Official node images: hours from the release to build and to push

	Value	Series
Debian-based amd64: built	1 h	built
Debian-based amd64: pushed	4.1 h	pushed
Debian-based arm64: built	1 h	built
Debian-based arm64: pushed	4.1 h	pushed
Alpine amd64: built	12.4 h	built
Alpine amd64: pushed	13.1 h	pushed
Alpine arm64: built	13.5 h	built
Alpine arm64: pushed	16.1 h	pushed

Reference: fixed binaries on nodejs.org/dist by 10:36 UTC on 21 Oct 2026. Built: the image config's created time. Pushed: Docker Hub's last_pushed time for that platform image, which is metadata, not proof of when a tag first served it. Latest value in each group of tags. Source: results/registry-20261022T062338Z-summary.txt.

If you rebuilt an Alpine-based image on the afternoon of 21 October, check it again: by this metadata the fixed Alpine images were built 12 to 13.5 hours after the release. The end-of-life tags will stay where they are: node:18 (18.20.8, built March 2025), node:20 (20.20.2, built April 2026) and node:25 (25.9.0, built April and May 2026).

Verify with /proc, not with names

pm2 renames its daemon to PM2 v6.0.14: God Daemon and Next.js renames its server to next-server, and on our build machine the plain node processes showed up as MainThread. Those names are what pgrep matches and what top shows. On that machine pgrep -c node returned 0 while five processes ran a node binary (results/pgrep-vs-proc-20261022T063334Z.txt). The kernel's link to the executable does not lie:

# Every process whose binary is node, with the binary it really runs (root sees all users)
for d in /proc/[0-9]*; do
  exe=$(readlink "$d/exe" 2>/dev/null) || continue
  case "${exe% (deleted)}" in */node|*/nodejs) ;; *) continue ;; esac
  printf '%-8s %-70s %s\n' "${d#/proc/}" "$exe" "$(tr '\0' ' ' < "$d/cmdline" | cut -c1-50)"
done

# The version of one of them, read from the binary without running it.
# Official builds compile in their headers URL. Do not run another user's
# binary with --version as root: that runs their code as root.
sudo grep -a -o -m1 -E 'node-v[0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z.]+)?-headers\.tar\.gz' /proc/12345/exe
# node-v24.14.0-headers.tar.gz

# Its children, which have their own binaries
ps -o pid,args --forest -g "$(ps -o sid= -p 12345 | tr -d ' ')"

An exe that ends in (deleted) means the file was replaced on disk after the process started, so the process still runs the old code until it restarts. For more on that pattern, and on proving a runtime is patched when the version string can mislead, see PHP2Shell: How to Check That Every PHP You Run Is Actually Patched.

Run the audit script

The-DevOps-Daily/node-runtime-audit on GitHub

node_runtime_audit.py is one file of standard-library Python. Its sources are /proc, the output of pm2 jlist and crontab -l, systemd unit files, and node --version inside running containers; besides those it only runs systemctl show for pm2 units and docker or podman ps and inspect, and looks at file metadata. For every running process it reads the real binary through /proc/<pid>/exe and the version compiled into that file, which works for deleted binaries and for processes inside containers. By default it never runs a binary it finds on the host, and it does not open the Node binaries that pm2 entries, crontabs and units name: their version comes from a running process that uses the same file, or from the path as a hint, where a path that names an unfixed version counts as vulnerable and one that names a fixed version stays unknown. Files are pinned with O_PATH and checked before they are read, so a planted FIFO or device cannot stall it. Command lines and container names are printed with control characters escaped. Run it as the user that owns the Node processes; root is not needed.

This is the real output on the Raspberry Pi we build on, run as the user that owns the Node processes, with --redact, which hides user names and app names. By the time of this run, the fnm default, the crontab and the pm2 systemd unit had already been moved to v24.22.1:

node_runtime_audit.py on the Pi

# 2026-10-22T07:10:33Z Raspberry Pi (Debian 12, aarch64), Node via fnm, pm2 6.0.14, run as the user that owns the Node processes
$ python3 node_runtime_audit.py --redact
node-runtime-audit: Node.js security release of 2026-10-21, fixed releases v22.23.4, v24.22.1, v26.12.1
lines without a fix (end-of-life) count as vulnerable
note: covers the processes, crontab and pm2 of USER; run it as each user that owns Node processes

Running Node processes (real binary from /proc/<pid>/exe)
  [vulnerable] pid 1883303 v24.14.0 USER /home/USER/.local/share/fnm/node-versions/v24.14.0/installation/bin/node
      cmd: PM2 v6.0.14: God Daemon (/home/USER/.pm2)
      parents: systemd(1)
      why: v24.14.0 < v24.22.1
  [vulnerable] pid 1883371 v24.14.0 USER /home/USER/.local/share/fnm/node-versions/v24.14.0/installation/bin/node
      cmd: node /home/USER/projects/app-1/node_modules/.bin/next dev
      parents: bun(1883370) <- node(1883303) <- systemd(1)
      why: v24.14.0 < v24.22.1
  [vulnerable] pid 1883390 v24.14.0 USER /home/USER/.local/share/fnm/node-versions/v24.14.0/installation/bin/node
      cmd: next-server (v16.2.6)
      parents: node(1883371) <- bun(1883370) <- node(1883303) <- systemd(1)
      why: v24.14.0 < v24.22.1
  [vulnerable] pid 1884634 v24.14.0 USER /home/USER/.local/share/fnm/node-versions/v24.14.0/installation/bin/node
      cmd: node /home/USER/projects/app-1/.next/dev/build/postcss.js 38963
      parents: node(1883390) <- node(1883371) <- bun(1883370) <- node(1883303)
      why: v24.14.0 < v24.22.1
  183 processes of other users not inspected: run the script as their owner

pm2 live list (pm2 jlist, PM2_HOME=/home/USER/.pm2)
  [unknown] pm2 CLI /home/USER/.bun/bin/pm2 is writable by group or others (mode 777): not run, live list skipped
  [vulnerable] daemon pid 1883303 v24.14.0 /home/USER/.local/share/fnm/node-versions/v24.14.0/installation/bin/node

Installed Node versions (version managers)
  fnm in /home/USER/.local/share/fnm
      installed: v24.14.0 (running), v24.14.1 (from path), v24.22.1 (from path)
      default alias -> v24.22.1 (from path)
  `node` on this shell's PATH -> v24.22.1 (from path) /home/USER/.local/share/fnm/node-versions/v24.22.1/installation/bin/node
  for information only: these do not change the exit code

Crontab: absolute Node paths
  [unknown] crontab -l (USER):33
      /home/USER/.local/share/fnm/node-versions/v24.22.1/installation/bin/node -> v24.22.1 (from path) (pinned to one install)
      why: from path: v24.22.1 (a hint, not proof: binaries outside /proc are not read)
  [unknown] crontab -l (USER):53
      /home/USER/.local/share/fnm/node-versions/v24.22.1/installation/bin/node -> v24.22.1 (from path) (pinned to one install)
      why: from path: v24.22.1 (a hint, not proof: binaries outside /proc are not read)

systemd units: absolute Node paths
  [unknown] /etc/systemd/system/pm2-USER.service:12
      /home/USER/.local/share/fnm/node-versions/v24.22.1/installation/bin -> v24.22.1 (from path) (pinned to one install)
      why: from path: v24.22.1 (a hint, not proof: binaries outside /proc are not read)
      unit: pm2-USER.service ActiveState=inactive MainPID=0
      note: the unit is not what runs pm2 now; its PATH applies at the next boot

Containers
  no docker or podman CLI found

result: VULNERABLE: something runs or will start an unfixed Node (exit 1)
$ echo $?
1

The fnm default, the shell's node, both cron jobs and the systemd unit point at the v24.22.1 install. The script reports the cron and unit paths as unknown rather than patched, because no running process uses that binary and it does not open it. The four processes that run are v24.14.0. The pm2 daemon is not running under the systemd unit (ActiveState=inactive), so the unit we had fixed is not what runs it now. The one pm2 app is started by bun, and the Node processes under it run the app's node_modules/.bin/next script. A separate check (scripts/pm2-path-evidence.py, results/pm2-path-20261022T070504Z.txt) shows that the process the pm2 daemon started and the saved pm2 list both have the v24.14.0 install first on PATH, which is where those Node processes come from. If the app is resurrected from that saved list after a reboot, it gets the same PATH; we did not reboot to confirm. The fix here is pm2 restart <app> --update-env from a fixed shell, then pm2 save, then moving the daemon.

The run found one more thing: the pm2 CLI on that machine is world-writable (mode 777). The script will not run a CLI that someone else could have changed, so it skipped the live pm2 list and reported it as unknown. Check the permissions of your Node installs and global CLIs while you are in there.

The exit code is 1 when anything runs or will start a vulnerable Node, otherwise 2 when anything could not be checked, and 0 when it found nothing vulnerable or unknown within what it covers, including no Node at all. Unknown never counts as patched: a pre-release build, a distribution build linked to the system Brotli, a binary without an embedded version, a unit it could not read, a failed pm2 jlist, crontab -l or docker ps, a config path that only names a fixed version, or a cron path to a Node that no longer exists all give 2. It only sees what the user running it can see, so run it as each user that owns Node processes. To run it across hosts without copying it first:

for h in deploy@web1 deploy@web2 deploy@web3; do
  ssh "$h" 'python3 - --json' < node_runtime_audit.py > "node-${h#*@}.json"
  echo "$h exit $?"
done

The tests build fake hosts with the cases a version check gets wrong: a fake node that would leave a marker file if the script ever ran it, a binary replaced while it runs, a container binary, a Debian-style build, a pm2 entry whose config is fixed but whose process was never restarted, npm pinned to the old install while its child gets the new one, a cluster-mode app, an old binary in a directory named after a fixed version, end-of-life and pre-release versions, a cron job on a removed version, a shell wrapper that mentions a fixed version, units owned by other users, and FIFOs, devices, hard links and symlinks planted where a file is expected. One test starts a real process, deletes its binary and checks that the script still reads the running file.

Rollback

Keep the old version installed until every check passes. Rolling back is the same work in reverse: nvm alias default 24.14.0, then the same --update-env, delete-and-recreate, pm2 save, startup unit and pm2 update steps from a shell where the old version comes first. For containers, redeploy the previous image digest. A rollback puts the vulnerable version back, so treat it as a few hours to fix a regression, not as a resting state.

Once the audit shows nothing on the old version, nvm uninstall 24.14.0. Anything still pinned to it will then fail to start instead of quietly running the old code, and the audit script reports those paths as missing.

What we could not check

The bugs themselves. The advisory gives no technical detail, and we did not test whether any code path reaches the Brotli flaw. The exposure section is based on which code calls the decoder.
Scores. The CVE records were still reserved when we checked, so there is no CVSS score.
Distribution packages. Debian and Ubuntu had not published anything for CVE-2026-111497 when we checked. Check your distribution's nodejs and brotli advisories.
Containers on a real host. The Pi has no Docker, so the container checks are covered by the test fixtures only. The registry scan read real image metadata.
The downtime number."Under a minute" comes from the production upgrade. The lab's pm2 update took 2.3 seconds because its apps do nothing.
Bare node in cron jobs and units. The script checks absolute paths and PATH= lines; it does not resolve a bare node through the job's PATH. It reads only your own crontab -l, not the system crontabs or the saved pm2 list, and it does not follow EnvironmentFile=.
Other platforms. The registry scan covers linux/amd64 and linux/arm64 only. The script needs Linux and /proc, and does not cover Kubernetes, stopped containers, or Node binaries renamed to something other than node (single executable applications, Electron).

Checklist

Every host: Node 22.23.4, 24.22.1 or 26.12.1 or later installed; Node 20, 25 and older replaced
Every pm2 entry: absolute interpreters and npm or npx script paths recreated, bare node entries restarted with --update-env, then pm2 save
pm2 startup unit regenerated (pm2 unstartup, pm2 startup) and the daemon moved with pm2 update in a quiet window
Cluster-mode apps checked after the daemon moved
Crontabs, systemd units and helper scripts searched for absolute paths to the old install
Images rebuilt with --pull (Alpine-based ones checked after the late rebuild) and redeployed by new tag or digest
Services that fetch user-supplied URLs or accept compressed bodies upgraded first; interim measures where they cannot be
Every Node process checked by /proc/<pid>/exe, not by name, including child processes
Node installs and global CLIs (pm2, npm) writable only by their owner
Audit script output saved per host, and the old version uninstalled once it is clean

Every fact in this post is listed with its source in the repo's SOURCES.md.

Originally published at devops-daily.com.

What Keeps a Neon Compute Awake, and What It Costs

DevOps Daily — Wed, 21 Oct 2026 09:00:00 +0000

We pointed a health check at a Neon compute every 5 minutes, the same as Neon's default scale-to-zero timeout. The compute never suspended. In 40 minutes it took 8 checks, woke once for the first one, and stayed awake the whole time. Over 30 days that is 180 CU-hours at the smallest compute size: about $19 on the Launch plan, $40 on Scale, and on the Free plan it uses up the 100 CU-hour allowance by day 17.

The suspect most people reach for first, an idle connection left open by the app, does not keep a compute awake. We held idle connections, direct and pooled, and the compute suspended every time, a few seconds later than with nothing open. What the suspend did do was cut those connections, including the pooled ones.

This post measures which ordinary clients keep a Neon compute running and which do not, how long a compute takes to suspend, what your app sees when it does, and what each pattern costs per month. Everything runs against real computes in one Neon project, watched through the Neon API, and the scripts are public so you can run them on your own account.

TL;DR

Idle connections do not stop scale to zero. With a 60-second timeout the compute suspended 66 to 75 seconds after the last query with nothing open, and 81 to 83 seconds with an idle direct or pooled connection held. The suspend dropped every held connection, pooled ones included.
An open transaction keeps it awake for as long as it stays open. A session left idle in transaction, direct or through the pooler, kept the compute up for the whole 6-minute window, 4 runs out of 4. Setting idle_in_transaction_session_timeout = '30s' let it suspend again.
A monitoring agent that queries every 20 seconds keeps it awake , even when it only reads pg_stat_database on the postgres database.
A LISTEN connection does not keep it awake. It is dropped on suspend, and a notification sent after the suspend never reached it (2 of 2 runs).
Health checks are the expensive one. With the default 5-minute timeout, a check every 5 minutes kept the compute awake for the whole 40-minute run. In our runs the suspend came 312 to 327 seconds after a wake, so each check arrived before it. A check every 6 minutes still kept it awake 90.6% of the time, and every 10 minutes 52.1%.
A wake adds roughly half a second to a second. End-to-end checks (connect, authenticate, query, close) that woke the compute had medians of 847 to 1,134 ms per run; checks on an awake compute took 308 to 339 ms, from a client 48 ms from the region.

Prerequisites

A Neon account and an API key (Account settings > API keys). The 60-second timeout used in part of the test needs a plan that lets you change it (Scale); everything with the default 5-minute timeout works on any plan.
Node.js 22 or newer.
Basic Postgres knowledge: sessions, transactions, pg_stat_activity.

How Neon decides a compute is idle

Neon's documentation says a compute scales to zero after 5 minutes of inactivity, that you can change the timeout on paid plans, and that a connected logical replication subscriber prevents the suspend. It does not say what counts as activity.

The answer is in the compute's own code. Every compute runs compute_ctl, and its activity monitor checks Postgres every 500 ms. It moves the "last active" time forward when:

The totals in pg_stat_database change: the summed active_time (time spent running queries) or the number of sessions, across all databases except postgres and the templates. This catches a query that started and finished between two checks, and every new connection. The code calls this path experimental but says it is on for everyone.
Any client backend is in a state other than idle. That includes active (a query is running) and idle in transaction (a transaction is open but doing nothing). Backends of Neon's own cloud_admin role are left out here.
An idle backend changed state recently: the latest state_change among idle backends counts as the last activity (the backend check). A connection that ran a query 20 seconds ago and went idle moves the clock to then. Connecting moves it once too, but a connection that then just sits idle does not keep moving it.
A logical replication walsender exists, an active logical replication subscription exists, or an autovacuum worker is running.

The control plane reads the last activity time and suspends the compute once the timeout has passed. This is the open-source code at commit f3a0e4f255; the version running in production may differ, which is one more reason to measure.

So the code predicts that an idle connection should not keep a compute awake, and an open transaction should. The rest of this post checks those predictions and puts numbers on them.

How we measured

The-DevOps-Daily/neon-keep-awake on GitHub

One Neon project in aws-eu-central-1 running Lakebase Postgres (PostgreSQL 17), every compute fixed at 0.25 CU so CU-hours are easy to read off. Each health-check configuration gets its own branch, and each branch has its own compute, so configurations running side by side do not affect each other. The idle-client scenarios ran one after another on a single branch.

The compute's state comes from GET /projects/{id}/endpoints/{id}, which returns current_state and suspended_at. That is a control-plane call: it never touches the compute, so watching cannot keep it awake. The scripts read it every 2 seconds for the idle clients, every 3 seconds for the health checks, and every 15 seconds for the long unattended watch.

Awake time is an estimate. An awake period starts when a check that found the compute suspended began connecting, and ends at the suspended_at that Neon reports, which is in whole seconds. We checked it against Neon's own record: the project's operations log lists every start_compute and suspend_compute. For the health-check runs, the awake time from the log and from our polling differed by at most half a percentage point. That is a cross-check of the timing, not of the bill.

That check caught one bug of ours. A poll taken a moment after a check woke the compute can still show the previous suspend, and our first summary ended the awake period there. The fix is to ignore a poll whose suspended_at is older than the wake, and even that went wrong at first: Neon's timestamps are whole seconds (17:57:46Z) and ours have milliseconds (17:57:46.649Z), and compared as strings 46Z sorts after 46.649Z. The bug cost two runs 3 to 4 seconds of awake time each, small but in the direction that makes the bill look smaller. The summary now compares timestamps as dates. A review of the draft found a second one: the runner could fire one extra check right at the end of a window. The summary now counts only checks scheduled inside the window, which took one check off two of the runs.

The scripts are split by stage, and each writes JSON lines stamped with a hash of the code that wrote it:

neon-keep-awake

$ npm run setup
project odd-poetry-65586042, main branch br-icy-recipe-b91wrumq, compute ep-weathered-shadow-b91petoo
# stage 1: idle clients, 60 s timeout
$ npm run idle-clients
none #1: suspended at 2026-10-18T17:52:15Z (wake 694 ms)
none #2: suspended at 2026-10-18T17:53:30Z (wake 947 ms)
none #3: suspended at 2026-10-18T17:54:46Z (wake 907 ms)
idle-direct #1: suspended at 2026-10-18T17:56:16Z (wake 815 ms)
idle-direct #2: suspended at 2026-10-18T17:57:46Z (wake 805 ms)
idle-direct #3: suspended at 2026-10-18T17:59:16Z (wake 793 ms)
idle-pooled #1: suspended at 2026-10-18T18:00:45Z (wake 774 ms)
idle-pooled #2: suspended at 2026-10-18T18:02:17Z (wake 787 ms)
idle-pooled #3: suspended at 2026-10-18T18:03:46Z (wake 804 ms)
idle-in-transaction-direct #1: still active after 360s (wake 815 ms)
idle-in-transaction-direct #2: still active after 360s (wake 1020 ms)
idle-in-transaction-pooled #1: still active after 360s (wake 940 ms)
idle-in-transaction-pooled #2: still active after 360s (wake 792 ms)
idle-in-transaction-with-timeout #1: suspended at 2026-10-18T18:34:31Z (wake 1080 ms)
idle-in-transaction-with-timeout #2: suspended at 2026-10-18T18:36:16Z (wake 895 ms)
listen-direct #1: suspended at 2026-10-18T18:37:46Z (wake 876 ms)
listen-direct #2: suspended at 2026-10-18T18:39:16Z (wake 296 ms)
monitor-postgres-db-20s #1: still active after 360s (wake 263 ms)

The wake figure in that log is the time to connect and run SELECT 1 at the start of each run. Most runs started right after a suspend, so it is usually a cold start. The last two started on a warm compute: the previous LISTEN run had just woken it to send its test notification, which is why they are near 300 ms.

Idle connections do not keep it awake, and they do not survive the suspend

Stage 1 sets the suspend timeout to 60 seconds, wakes the compute, leaves one kind of client open, and times the suspend from our last query.

What was left open	Runs	Suspended	Seconds from the last query to the suspend
Nothing	3	3 of 3	75, 68, 66
One idle direct connection	3	3 of 3	82, 81, 83
One idle pooled connection (PgBouncer)	3	3 of 3	81, 83, 81
A `LISTEN` connection	2	2 of 2	83, 79

An idle connection did not stop the suspend. The suspend did come later: the median was 82 seconds with an idle direct connection and 81 with a pooled one, against 68 with nothing open. With three runs per row, all on the same compute one after another, that is an observation we cannot explain, not an established effect. Either way, a compute with a 60-second timeout went to sleep in under a minute and a half with connections still open.

Look at what happened to those connections. Five seconds after each suspend, the script ran SELECT 1 on the connection it had held:

idle direct: 'error' event: Connection terminated unexpectedly
              next query: Client has encountered a connection error and is not queryable
idle pooled: 'error' event: Connection terminated unexpectedly
              next query: Client has encountered a connection error and is not queryable

Every held connection was dead, 6 out of 6, pooled ones included, and the LISTEN connections in the last row died the same way. The pooler did not keep its client connections open across the suspend, so a pooled connection string does not protect your app from one.

For most apps the fix is to make sure a dropped connection is handled, not fatal. With node-postgres, use Pool, not a single long-lived Client, and handle the pool's error event. Without a listener, an idle client that loses its socket throws an unhandled error and can crash the process:

import pg from 'pg';

const pool = new pg.Pool({
  connectionString: process.env.DATABASE_URL,
  // Close idle clients before the compute's suspend timeout closes them for you.
  idleTimeoutMillis: 30_000,
});

// An idle client whose socket closed (for example, the compute suspended).
// The pool removes it; this listener stops it from crashing the process.
pool.on('error', (err) => console.warn('idle client dropped:', err.message));

// Retry once on a dropped connection, but only when the caller says the
// statement is safe to run twice.
export async function query(sql, params, { retry = false } = {}) {
  try {
    return await pool.query(sql, params);
  } catch (err) {
    if (retry && /Connection terminated/.test(err.message)) return pool.query(sql, params);
    throw err;
  }
}

The retry is off by default for a reason. When a connection drops, you cannot always tell whether the statement ran before it did, so only retry statements with no side effects or with an idempotency key. A plain read qualifies; an INSERT without a key does not, and neither does a SELECT that calls a function with side effects. The error check also catches only this one message, so treat it as a starting point, not complete coverage.

LISTEN does not keep it awake either

A connection waiting on LISTEN sits in the idle state, so the monitor treats it like any idle connection. Both runs suspended on schedule. After each suspend the script opened a new connection, which woke the compute, and sent NOTIFY keep_awake, 'after-suspend'. The listener never got it, in either run, because its connection had been closed by the suspend.

If you use LISTEN/NOTIFY for cache invalidation or live updates on a compute that scales to zero, the listener must reconnect and run LISTEN again, and it must not assume it saw every notification sent while it was gone. Read the current state after each reconnect.

An open transaction keeps it awake for as long as it stays open

The monitor counts any backend that is not idle as activity, and a session that ran BEGIN and then stopped is idle in transaction. That is the classic leaked transaction: a request handler that checks out a client, starts a transaction, and returns on an error path without COMMIT or ROLLBACK.

What was left open	Runs	Suspended within 6 minutes	After the transaction ended
`BEGIN` on a direct connection, then nothing	2	0 of 2	suspended 65 and 72 s later
`BEGIN` on a pooled connection, then nothing	2	0 of 2	suspended 68 and 72 s later
The same, with `idle_in_transaction_session_timeout = '30s'`	2	2 of 2, 102 and 98 s after the last query

Through the pooler it behaves the same way. In transaction mode, PgBouncer pins a server connection to the client for as long as the transaction is open, so Postgres still sees a backend idle in transaction.

With the timeout set, Postgres ended the leaked session after 30 seconds, and the compute suspended about 70 seconds after that, the same delay as with nothing open. That setting is the one line worth adding to every Neon project that scales to zero, at the database or role level so it applies to every connection:

-- Ends sessions that hold a transaction open with nothing running.
ALTER DATABASE neondb SET idle_in_transaction_session_timeout = '60s';

We checked this form too, with 30s: the database owner role that Neon creates is allowed to run it, a new connection reported 30s without setting anything itself, and a transaction leaked on that connection no longer kept the compute up. It suspended 101 seconds after the last query (scripts/07-database-timeout.mjs).

Pick a value longer than any legitimate pause inside your transactions. A leaked transaction is a bug anyway: it holds locks and stops vacuum from cleaning up. On a compute that scales to zero, it also bills you for every hour it stays open.

A monitoring agent keeps it awake

Monitoring agents such as postgres_exporter keep a connection open and run queries on a timer. We ran the same pattern: one connection to the postgres database, reading pg_stat_database every 20 seconds.

The compute stayed awake for the whole 6-minute window and suspended 63 seconds after the agent stopped. One run only, but it matches the code: each query moves the connection's state_change, and the backend check counts that whichever database the query runs in. Only the pg_stat_database totals leave the postgres database out.

So a database scrape every 15 or 30 seconds keeps the compute awake around the clock. If you want monitoring and scale to zero together, scrape less often than the suspend timeout and accept gaps, or read compute state from the Neon API, which does not wake the compute. That is how we watched compute state for this post without adding database activity.

Health checks: the interval decides everything

Uptime monitors, load balancer probes and Kubernetes readiness probes often run a query to prove the database is reachable. Stage 2 runs exactly that: every N seconds, open a new pooled connection, run SELECT 1, close it. Each configuration gets its own compute, starts from a suspended state, and runs for 12 to 60 minutes while the script records when the compute is awake.

Suspend timeout	Check every	Minutes	Checks	Checks that woke it	Awake	CU-hours per 30 days at 0.25 CU
60 s	30 s	12	24	1	100%	180
60 s	90 s	15	10	10	82.2%	148
60 s	120 s	16	8	8	70.2%	126
60 s	180 s	18	6	6	40.5%	73
300 s	60 s	30	30	1	100%	180
300 s	300 s	40	8	1	100%	180
300 s	360 s	42	7	7	90.6%	163
300 s	420 s	42	6	6	74.7%	135
300 s	600 s	60	6	6	52.1%	94

"Checks that woke it" counts the checks for which the last API reading before the check showed the compute suspended. When that number is 1, only the very first check of the run had to wake the compute, and every later check found it already awake.

Share of time the compute was awake, by health-check interval

	Value	Series
60 s timeout, check every 30 s	100%	60 s suspend timeout
60 s timeout, check every 90 s	82.2%	60 s suspend timeout
60 s timeout, check every 120 s	70.2%	60 s suspend timeout
60 s timeout, check every 180 s	40.5%	60 s suspend timeout
5 min timeout, check every 1 min	100%	5 min suspend timeout (default)
5 min timeout, check every 5 min	100%	5 min suspend timeout (default)
5 min timeout, check every 6 min	90.6%	5 min suspend timeout (default)
5 min timeout, check every 7 min	74.7%	5 min suspend timeout (default)
5 min timeout, check every 10 min	52.1%	5 min suspend timeout (default)

A new pooled connection and SELECT 1 every N seconds, one 0.25 CU compute per row, 12 to 60 minutes each, one run per row. Estimated by polling the Neon API; matches Neon's operations log within half a percentage point. Data: results/summary.json in The-DevOps-Daily/neon-keep-awake.

The reason a 5-minute check beats a 5-minute timeout is the time between a wake and the suspend that follows it. Where the compute suspended between checks, we can estimate that cycle, from the moment the check started connecting to Neon's suspended_at:

With the 60-second timeout, a compute woken by a single check suspended 71 to 87 seconds later (24 cycles, median 74 s).
With the 5-minute timeout, it suspended 312 to 327 seconds later (19 cycles on three computes, median 314 s).

So in our runs, with the default timeout, a check that came within 312 seconds of the previous one always found the compute still up and started the clock again, and a check every 300 seconds kept it up for the whole 40-minute run. That is what we observed, not a guaranteed grace period: the suspend came 12 to 27 seconds after the timeout here, and Neon does not document that gap.

Make the interval a little longer and the compute does get to sleep between checks, but not for long. Every check pays for a full cycle of about 5 minutes and 15 seconds, so every 6 minutes it was awake 90.6% of the time and every 7 minutes 74.7%. A check every 10 minutes woke the compute 6 times out of 6, and each wake kept it up 312 to 313 seconds for a query that took about a second. That is 52% of the hour, for a database that did nothing else.

What a wake costs your users

Letting the compute sleep has a price too: the first request after a quiet stretch waits for the compute to start.

	Median	Max	Checks
Check that found the compute awake	308 to 339 ms per run		59 checks, 3 runs
Check that had to wake it	847 to 1,134 ms per run	1,878 ms	46 wakes, 9 runs
Plain TCP connect to the region (no TLS, no query)	48 ms		10

These are end-to-end times for the whole check: open a new TLS connection, authenticate, run the query and close. That is why even a warm check takes about 300 ms from a client 48 ms away. In three of the runs only the first check woke the compute, so their wake figure is a single sample. Comparing across runs rather than pair by pair, waking the compute added roughly 0.5 to 0.8 seconds, and the slowest wake took 1.9 seconds. None of the 105 checks failed. For context, our Neon free tier benchmarks measured cold starts with a median of 568 ms and a p95 of 1.06 s from a client in the same region, with a different method.

For a health check, an extra second does not matter. For the first user request of the morning it might, and that is the real trade-off you are making when you decide how long the compute may sleep.

What it costs

Neon bills compute in CU-hours: compute size times the hours it runs. A compute kept awake by a light pattern like these sits at its minimum size, so the cost is the minimum CU times the hours awake (real load can scale it up, which costs more). With list prices from Neon's pricing page ($0.106 per CU-hour on Launch, $0.222 on Scale) and the Free plan's 100 CU-hours per project per month, for one 0.25 CU compute over 30 days with nothing else using it:

What keeps it awake	Awake	CU-hours per 30 days at 0.25 CU	Launch	Scale	Free plan
Health check every 1 or 5 min, default timeout	100%	180	$19.08	$39.96	allowance used up after 400 hours, about day 17
Health check every 6 min, default timeout	90.6%	163	$17.28	$36.19	used up around day 19
Health check every 10 min, default timeout	52.1%	94	$9.94	$20.82	fits, with 6 CU-hours left for real traffic
Leaked transaction, monitoring agent every 20 s	100% while it lasts	up to 180	up to $19.08	up to $39.96	as above
Idle connections, `LISTEN`	suspended normally	no sustained cost observed	none seen	none seen	none seen

On the Free plan, Neon's plans page says that when a project runs out of CU-hours, its compute is suspended until the next billing period. A 5-minute health check on a Free project, starting the month with the full allowance and nothing else using it, does not cost money; it takes the database down for the last 13 days of a 30-day month.

Two things make these numbers bigger in practice:

Minimum size. We fixed every compute at 0.25 CU. If your compute's autoscaling minimum is 1 CU, multiply by four: an always-awake 1 CU compute is 720 CU-hours a month, $76 on Launch and $160 on Scale.
Branches. Every branch with a compute has its own suspend timer. A preview environment per pull request, each with its own health check, keeps each preview branch's compute awake. Ten open previews with a 5-minute check are ten always-on computes.

Does Neon wake it on its own?

A May 2026 thread in Neon's GitHub discussions, neondatabase/neon#12900, describes Launch projects at 0.25 CU that each used about 6 CU-hours a day whatever their traffic, which is a compute awake around the clock. One reply blamed check_availability, an operation that Neon's control plane runs to check that a compute can start on a branch. We looked for it in two places.

Our test project. Stage 3 watched the main branch's compute, which nothing ever connected to, for 3.9 hours after it suspended, reading its state every 15 seconds. It never woke. The project's operations log, read at the end, shows no check_availability and no start_compute for it after the first suspend, and Neon's usage counter for that branch read 368 active seconds, about the six minutes the compute ran after we created the project.
An older project of ours. Its main compute suspended on 23 September, and we stopped using the project that evening. Its operations log shows one check_availability for that compute on 24 September, 929 ms long, and no start_compute for it afterwards. The compute's suspended_at still reads 23 September. Two weeks later Neon archived the idle branch.

That matches Neon's own description. The operations page says the check verifies that a compute can start, and a 2023 changelog entry says the availability checker runs on 0.25 CU computes and can use pre-started ones. So in these two records, no availability check restarted a suspended compute. That does not rule out other behavior on other projects, and we cannot tell what kept the computes in that thread awake. The patterns that kept ours awake were open transactions, frequent monitoring queries and frequent health checks.

What to change

Keep the database out of liveness checks. A liveness probe should check that the process is up, not that Postgres answers. If you must check the database, do it well below the suspend timeout's rate: with the default 5-minute timeout, every 10 minutes still kept the compute awake half the time, so think in hours, not minutes.
Check compute state through the API instead of with a query. GET /projects/{project_id}/endpoints/{endpoint_id} returns current_state without waking anything. A monitor that alerts on the API's view of the compute, and only queries the database when a user is actually using it, keeps scale to zero working.
Set idle_in_transaction_session_timeout at the database level. It ends leaked transactions, which otherwise keep the compute awake for as long as they stay open, and they are a bug anyway.
Scrape metrics less often than the timeout, or not from the database. A monitoring agent that queries every 15 to 30 seconds turns scale to zero off.
Expect connections to drop. Idle and LISTEN connections, direct or pooled, were closed when the compute suspended. Use a pool, handle its error event, retry only statements that are safe to repeat, and re-run LISTEN after a reconnect.
If you want it awake, say so. On paid plans you can raise the suspend timeout or turn scale to zero off. For a production database with steady traffic, that is often the right call. The compute bill is the same either way, but a setting you chose is explicit and predictable, and a health check that happens to keep the compute up is neither.

What we could not conclude

Sample sizes are small. Three runs for each idle-connection row, two for the transaction and LISTEN rows, one for the monitoring agent, one for the database-level timeout check, and one run per health-check row. The results agree with the activity-monitor code, which is why we trust them, but the later suspend we saw with idle connections (median 81 to 82 seconds against 68) is not explained, and those scenarios ran one after another on the same compute.
Open transactions were watched for 6 minutes. The code says such a session counts as activity for as long as it stays open, and all four runs stayed awake for the whole window; we did not run one for hours.
One region, one compute size. Everything ran in aws-eu-central-1 at 0.25 CU. The time from a wake to the suspend (71 to 87 seconds with a 60-second timeout, 312 to 327 seconds with 5 minutes) may differ in other regions or under load. It also differed between computes in the same project: with the 5-minute timeout, one compute's cycles took 325 to 327 seconds every time while the others took 312 to 314. We do not know why.
Cost is computed, not billed. Neon's per-branch usage counters (active_time_seconds, compute_time_seconds) read 0 for the short runs when we deleted their branches, and for the longer runs they were well behind our own measurement (for example 988 active seconds against our 1,877), so they seem to update with a delay. We could not check CU-hours against Neon's meter. We checked the awake time against Neon's operations log instead, and converted with list prices.
Health checks were on a precise schedule, and the gap after the timeout is not documented. A real uptime service has some jitter, and Neon may suspend sooner after the timeout on other computes or in other regions. Either could let a 5-minute check miss now and then; we did not test that.
We did not test logical replication or autovacuum. The code says both keep a compute awake, and Neon's docs say so for replication subscribers.

Summary

Scale to zero on Neon worked the way the compute's own activity monitor says it should. Idle connections, pooled or direct, and LISTEN connections did not hold a compute up; they were cut when it suspended. Activity did hold it up: a leaked transaction, a monitoring agent, a health check every 5 minutes with the default timeout. That last one is the expensive surprise. It costs a 0.25 CU compute 180 CU-hours a month, which is about $19 on Launch, $40 on Scale, or almost two weeks of downtime on Free.

Run the scripts against your own project to see which of these you have:

git clone https://github.com/The-DevOps-Daily/neon-keep-awake
cd neon-keep-awake && cp .env.example .env # add a Neon API key
npm install && npm run setup && npm run health-checks

Originally published at devops-daily.com.

SOC 2 Evidence From GitHub and Terraform, and the Part That Still Needs a Person

DevOps Daily — Mon, 19 Oct 2026 09:00:00 +0000

We pointed a small evidence collector at a GitHub repository whose main branch had a strict ruleset: every change through a pull request, one approval, and a required test check. The ruleset control passed. Three of the change-management controls under it failed: none of the seven merged pull requests had an independent approval, one was merged five seconds before its check finished, and one commit reached main without a pull request at all. All three were real, and all three would have looked fine in a screenshot of the settings page.

That gap is the whole job when a customer asks for SOC 2. An auditor does not only want to know what you configured. For a Type II report they want to see what happened under that configuration across the review period, usually by sampling. This post shows which of that evidence you can pull from the GitHub API and your Terraform code, which still needs a person and a signature, and the ten ways the first version of our collector could pass a control without evidence.

TL;DR

Settings are not evidence of operation. Our main branch required approvals and a passing check, and the evidence still showed 0 of 7 pull requests independently approved and 1 merged before its check finished, because admins could bypass the rule.
"The checks are green" is the wrong question. Pull request #3's check passed 5 seconds after the merge. It is green today; it did not gate the merge. Compare the check's completion time with the merge time.
An evidence script must fail closed. Two reviews of our first version found ten ways it could pass a control on missing or partial data, from approvals of an older commit to a branch name that could change the API query. Each now reports fail or incomplete, with a test.
Terraform scanning is evidence of a configuration standard, but a pass is only as good as the rule list. Trivy flagged the public database flag, and its security-group rule passed a Postgres port open to the whole internet, because that rule only covers SSH and RDP.
Access reviews, policy sign-offs, training, risk assessment, vendor reviews and incident-response tests still need a person. Automation can gather the list; it cannot make the judgement.
One run took 23 seconds and 35 API calls for one repository and 90 days, and produced 15 timestamped evidence files with a SHA-256 manifest.

Prerequisites

A GitHub repository you are admin on, and a token in GITHUB_TOKEN or a logged-in gh CLI
Python 3.10 or later (the collector uses only the standard library)
Trivy for the Terraform scan
A rough idea of what SOC 2 is: an AICPA attestation report on your controls, mapped to the Trust Services Criteria

The-DevOps-Daily/soc2-evidence-collector on GitHub

What an auditor actually asks for

SOC 2 reports on the Trust Services Criteria (free download with an AICPA account), a set of criteria with "points of focus" that describe what a control usually covers. You choose the controls; the auditor tests them. A Type I report says the controls were designed and in place on one day. A Type II report says they operated over a period, often 3 to 12 months, and that is where the evidence work goes: the auditor picks a sample of changes, alerts and access changes from the period and asks you to show each one.

Three criteria cover most of what a code repository can prove. In the AICPA's words:

CC8.1 The entity authorizes, designs, develops or acquires, configures, documents, tests, approves, and implements changes to infrastructure, data, software, and procedures to meet its objectives.

CC7.1 To meet its objectives, the entity uses detection and monitoring procedures to identify (1) changes to configurations that result in the introduction of new vulnerabilities, and (2) susceptibilities to newly discovered vulnerabilities.

CC6.1 The entity implements logical access security software, infrastructure, and architectures over protected information assets to protect them from security events to meet the entity's objectives.

For a change, "approves" and "tests" are the parts people get caught on. If your change-management control says that someone other than the author approves each change and that it is tested before it reaches production, the auditor will ask you to show both for each sampled change. For vulnerabilities, they want to see that you detect them and fix the serious ones within the time your own policy promises.

The usual way to produce this is screenshots: the branch protection page, the pull request page for each sampled change, the Dependabot tab, the security settings. A screenshot shows one moment, and nothing in it says where the values came from. An API response saved with the time it was collected is easier to check: the auditor can see the source of each value, and you can run the same query again. A hash manifest lets you show that a file did not change after collection, as long as the manifest is stored where nobody can edit it. And a script can collect the evidence for every change in the period instead of a sample.

The collector

collect.py reads one repository through the GitHub API and evaluates eight controls:

Criterion	Control	Where the evidence comes from
CC8.1	Changes to the default branch go through a pull request	branch rules and classic branch protection
CC8.1	Every merged change passed the required checks before the merge	the last run of each required check on the pull request's head commit that started before the merge
CC8.1	Every merged change was approved by someone other than its author	each reviewer's last decision before the merge, for the merged commit
CC8.1	Nothing reached the default branch without a pull request	the merged pull request behind each commit, and the branch's activity log
CC7.1	Dependency vulnerability alerts are on	the Dependabot setting
CC7.1	No critical or high alert is open past the SLA	open Dependabot alerts and their age
CC6.1	Secret scanning and push protection are on	the repository's security settings
CC6.3	Repository access follows least privilege	collaborator roles, judged by a person

It saves the fields each control uses from each API response, with the collection time, in a timestamped folder with a SHA-256 manifest, and marks each control pass, fail, needs a person or incomplete:

GitHub API rules, PRs, checks, alerts
collect.py 8 controls
raw/*.json timestamped, hashed
summary.json pass, fail, incomplete
A person judgement, sign-off

Three of the controls look back over the window: the merged pull requests with their reviews and checks, the commits on main, and the branch's activity log. The other five read the settings and the open alerts as they are on the day you run it. One run shows the history of changes. For the history of settings across a Type II period, run it on a schedule and keep every folder.

The test subject

We made the collector's own repository its test subject, with a history built on purpose. Its main branch has a ruleset that requires a pull request, one approval and the test check, with a bypass for organization admins. Then:

The first commit went straight to main, before the ruleset existed.
Every pull request was merged by an admin through the bypass. Nobody else could approve: it is a one-person setup, which is exactly the situation many small teams are in when the first enterprise customer asks for SOC 2.
Pull request #3 was merged while its test check was still running.
examples/app/requirements.txt pins PyYAML 5.3 and Jinja2 2.10, which have published advisories, so Dependabot has alerts to report.

The run

collect.py

$ python3 collect.py The-DevOps-Daily/soc2-evidence-collector --days 90
The-DevOps-Daily/soc2-evidence-collector, last 90 days: 15 evidence files, 35 API calls, 22.7 s
  CC8.1 pass Changes to the default branch go through a pull request
         rule on main: pull_request, 1 approval(s) required
  CC8.1 fail Every merged change passed the required checks before the merge
         required: test; 6 of 7 merged PRs passed every required check before the merge (#3 not finished)
  CC8.1 fail Every merged change was approved by someone other than its author
         0 of 7 merged PRs had an independent approval of the merged commit
  CC8.1 fail Nothing reached the default branch without a pull request
         commits not linked to a merged PR: 1 of 8; branch updates that were not PR merges: 1 (branch_creation)
  CC7.1 pass Dependency vulnerability alerts are on
         Dependabot alerts on
  CC7.1 pass No critical or high alert is open past 30 days
         8 open alerts, 0 critical or high open longer than 30 days
  CC6.1 pass Secret scanning and push protection are on
         secret scanning enabled, push protection enabled
  CC6.3 needs a person Repository access follows least privilege
         roles: {'admin': 4}, outside collaborators: 0; the list is evidence, the judgement is not
evidence: evidence/The-DevOps-Daily__soc2-evidence-collector/20261019T131928Z

It found every gap we put in, and it exits non-zero, so it can gate a pipeline or a weekly job.

What the failures mean

The rule was on, and the evidence still failed

The first control passed: a pull request with one approval is required on main. The approval control under it failed for all seven merged pull requests. An admin bypass is not a bug in the ruleset. Many teams keep one for emergencies, and an auditor's sample may well include a change that used it. If your system description says all changes are approved by a second person and your evidence shows admin merges without approval, expect the auditor to report them as exceptions.

For a one-person or two-person team, the honest fix is not to pretend. Common approaches are a reviewer from outside the team for production repositories, or a documented control that someone reviews merged changes after the fact within a fixed time. Which of these your auditor accepts is a conversation to have before the review period starts, not after.

Green now is not the same as tested before the merge

Here is the evidence file for pull request #3 as the collector saved it, without the author's login and with the indentation widened for reading:

{
  "collected_at": "2026-10-19T13:19:28.577807+00:00",
  "source": "GitHub API, The-DevOps-Daily/soc2-evidence-collector",
  "data": {
    "pull": {
      "number": 3,
      "title": "Examples to scan, and the IaC scan script",
      "base": "main",
      "merged_at": "2026-10-19T12:50:03Z",
      "html_url": "https://github.com/The-DevOps-Daily/soc2-evidence-collector/pull/3"
    },
    "head_sha": "d4b17526cfcaf516833023b9ec402f45a70a10ec",
    "reviews": [],
    "check_runs": [
      {
        "name": "test",
        "app_id": 15368,
        "head_sha": "d4b17526cfcaf516833023b9ec402f45a70a10ec",
        "status": "completed",
        "conclusion": "success",
        "started_at": "2026-10-19T12:50:01Z",
        "completed_at": "2026-10-19T12:50:08Z"
      }
    ]
  }
}

The check started at 12:50:01. The pull request was merged at 12:50:03. The check succeeded at 12:50:08. Look at the pull request page today and you see a green check, which is what most screenshots record. The collector compares the timestamps, and so should anything you hand to an auditor.

It asks GitHub for every run of the check, not only the latest, and judges each required check by its last run that started before the merge: that run is the check's state at the merge. If it had not finished by then, the check did not gate the merge, even if an earlier run had passed. A re-run started after the merge does not count. Only success passes: GitHub lets a skipped or neutral required check through, but a skipped check ran no tests. A required check stops a normal merge until it passes, so a merge before completion usually means someone used a bypass. It can also mean the required checks changed after the merge, because the collector compares with today's rules. Either way, it is an exception to explain.

The commit with no pull request

The one commit without a pull request is the repository's first commit, pushed before the ruleset existed. The branch's activity log shows the same event from the other side: one update to main that was not a pull request merge, the creation of the branch. That is a normal, explainable exception, and the evidence lets you explain it with a date.

The two sources catch different things. The commit list only holds commits that are still on main, so a force push that rewrote history leaves nothing in it. The activity log records updates to the branch, including direct pushes and force pushes, so an admin pushing a hotfix straight to main shows up there even when the commit list looks clean. If the log does not reach back to the start of the window, the control reports incomplete instead of pass.

The alerts that passed

Dependabot raised 8 open alerts for the two pinned packages: 2 critical, 1 high and 5 medium. The SLA control passed, correctly, because the alerts were less than an hour old. Our default SLA is 30 days for critical and high; use the number your own vulnerability policy states, since that is what the auditor will hold you to. If those alerts are still open, the same run in five weeks would fail.

This control reads the alerts that are open today. A critical alert that stayed open for two months last spring and was fixed in the summer does not appear in it. To cover a period, keep each run's folder, or extend the collector to fixed alerts and compare their fixed_at with created_at.

Ten ways our first version could pass without evidence

Two reviews of the first version of the collector, one of the code and one of the post against the recorded output, found ten ways it could pass a control, or count something as evidence, when the data did not support it. For a tool whose output goes to an auditor, each one is a real defect: a control must never pass because data was missing.

What went wrong	What the collector does now
An approval of an older commit counted	Only an approval whose `commit_id` is the merged head commit counts; the rest are reported as stale
A withdrawn approval counted	Only each reviewer's last decision before the merge counts, so a later change request or a dismissal replaces an approval
Any passing or skipped check counted as the required one	Each required check is matched by name, and by app when the rule names one, and only `success` passes
An earlier pass could hide a later failure, or a re-run still going at the merge	Each required check is judged by its last run that started before the merge, which must have succeeded before the merge
Unreadable alerts looked like zero alerts	A 403 or 404 makes the SLA control `incomplete`, with the HTTP status
Lists were cut off silently	A list cut at its limit makes its control `incomplete`, and an error on a later page stops the run
Any pull request link counted	The pull request must be merged into the default branch and be one of the merged pull requests evaluated
A force push left no trace	The branch's activity log is read alongside the commit list
The SLA was rounded down to whole days, and alerts without a severity were skipped	The age is compared as a time, and an unknown severity makes the control `incomplete`
Branch names went into URLs unencoded, and answers were not checked	Every value is encoded, and a pull request, activity entry or check run for another branch or commit makes its list `incomplete`

The last one is easy to miss. Git allows & and # in branch names, so a default branch with a crafted name could change the query string, and the collector would judge GitHub's answer to a different question.

None of these changed a result for our demo repository: it has no approvals at all, a readable list of 8 alerts and eight commits. They would change results for a real repository, where a stale approval, a re-run check or a missing permission is common. Each fix has a test in the repository.

Running the collector also found one more problem: the Dependabot alerts API rejects page numbers (Pagination using the page parameter is not supported) and pages with cursors in the Link header instead. The collector now follows the Link header for every list.

Evidence from Terraform

CC7.1's points of focus start with "Uses Defined Configuration Standards" and "Monitors Infrastructure and Software" for noncompliance with them. For infrastructure as code, a misconfiguration scan in CI is evidence of both: the rule set is your standard, and the scan result on each change is the monitoring.

The repository's examples/terraform holds a small AWS stack: a customer-data bucket with a KMS key, versioning and a public access block; an exports bucket added later without any of those; and a Postgres instance, encrypted and backed up, with a security group open to 0.0.0.0/0 on port 5432 and publicly_accessible = true. scan-iac.sh runs Trivy and keeps the report, a summary and a manifest:

scan-iac.sh

$ ./scan-iac.sh examples/terraform
92 checks, 12 failed, 4.6 s
  HIGH AWS-0086 aws_s3_bucket.exports S3 Access block should block public ACL
  HIGH AWS-0087 aws_s3_bucket.exports S3 Access block should block public policy
  HIGH AWS-0091 aws_s3_bucket.exports S3 Access Block should Ignore Public ACL
  HIGH AWS-0093 aws_s3_bucket.exports S3 Access block should restrict public bucket to limit access
  HIGH AWS-0132 aws_s3_bucket.exports S3 encryption should use Customer Managed Keys
  HIGH AWS-0180 aws_db_instance.app RDS Publicly Accessible
  MEDIUM AWS-0090 aws_s3_bucket.exports S3 Data should be versioned
  MEDIUM AWS-0176 aws_db_instance.app RDS IAM Database Authentication Disabled
  LOW AWS-0089 aws_s3_bucket.customer_data S3 Bucket Logging
  LOW AWS-0089 aws_s3_bucket.exports S3 Bucket Logging
  LOW AWS-0094 aws_s3_bucket.exports S3 buckets should each define an aws_s3_bucket_public_access_block
  LOW AWS-0133 aws_db_instance.app Enable Performance Insights to detect potential problems
evidence: evidence/iac/20261019T130914Z

The exports bucket and the public database were both caught. The open security group was not. Trivy's ingress rule AWS-0107, titled "Security groups should not allow unrestricted ingress to SSH or RDP from any IP address", passed: port 5432 is neither SSH nor RDP. The database was flagged only because publicly_accessible was also set. We removed that line, kept the security group and scanned again:

scan-iac.sh, without publicly_accessible

$ cp -r examples/terraform /tmp/sg-only && sed -i '/publicly_accessible/d' /tmp/sg-only/main.tf
$ ./scan-iac.sh /tmp/sg-only
92 checks, 11 failed, 4.2 s
  HIGH AWS-0086 aws_s3_bucket.exports S3 Access block should block public ACL
  HIGH AWS-0087 aws_s3_bucket.exports S3 Access block should block public policy
  HIGH AWS-0091 aws_s3_bucket.exports S3 Access Block should Ignore Public ACL
  HIGH AWS-0093 aws_s3_bucket.exports S3 Access block should restrict public bucket to limit access
  HIGH AWS-0132 aws_s3_bucket.exports S3 encryption should use Customer Managed Keys
  MEDIUM AWS-0090 aws_s3_bucket.exports S3 Data should be versioned
  MEDIUM AWS-0176 aws_db_instance.app RDS IAM Database Authentication Disabled
  LOW AWS-0089 aws_s3_bucket.customer_data S3 Bucket Logging
  LOW AWS-0089 aws_s3_bucket.exports S3 Bucket Logging
  LOW AWS-0094 aws_s3_bucket.exports S3 buckets should each define an aws_s3_bucket_public_access_block
  LOW AWS-0133 aws_db_instance.app Enable Performance Insights to detect potential problems
evidence: evidence/iac/20261019T130925Z

The only database findings left are IAM authentication and Performance Insights. Nothing points at a database port open to the world.

That is the general limit of configuration scanning as evidence. A pass means "none of these rules failed", not "this is safe". Your configuration standard is the rule set you run plus the rules you add for what it misses. Trivy and similar tools accept custom policies, and a rule for "no database port open to 0.0.0.0/0" is a few lines.

The scan reads code, not the running account. For the period itself, the auditor may ask for evidence that production matches the code: drift detection, or reading the live configuration through the cloud provider's API. That is the same pattern as the GitHub collector, with a different API.

What still needs a person

Some controls produce evidence a script can gather but not judge, and some produce no machine evidence at all. These are the ones that usually remain:

Criterion	What the auditor wants	Why a script cannot finish it
CC6.2, CC6.3	Periodic access reviews: someone confirms each account and role is still needed	The collector lists the 4 admins; whether each should be an admin is a decision, with a name and a date on it
CC6.2	Access removed when someone leaves	Needs the HR leaving date to compare with when access ended
CC1.4, CC2.2	Security awareness training completed	The record lives in a training tool or a signed list
CC5.3	Policies written, approved and acknowledged	A policy document and each employee's acknowledgment
CC3.2	A documented risk assessment and its periodic review	A document and a dated review
CC9.2	Vendor and subprocessor reviews	An assessment of each vendor's security, often based on the vendor's own SOC 2 report
CC7.4, CC7.5	Incident response that works	A tabletop exercise or a real incident, with its write-up
A1.3	Recovery tested (if Availability is in scope)	A restore test with its result

Automation helps with all of these by collecting the inputs and reminding people, but the evidence is a person's decision. In our collector, CC6.3 deliberately ends at needs a person: it gives the reviewer the list, and the reviewer's sign-off is the evidence.

Build it or buy it

The compliance automation platforms, such as Vanta, Drata, Secureframe, Sprinto and Thoropass, do the same thing at a much wider scale. They connect to source control, cloud accounts, identity providers and HR systems, map what they collect to controls, track the people-side tasks such as training and policy acknowledgments, and give the auditor a place to review the evidence. For a team going through its first audit, that coverage and the auditor relationships that come with it are the reason to pay for one.

What a small script like ours is good for is understanding what those tools check, and covering what is specific to you: a custom deployment pipeline, a check a platform does not have, or a control you want tested on every change. It can run alongside a platform as well as instead of one. Whichever you use, test it the way we tested ours: give it a repository with known problems and confirm it fails on each of them. A tool you have only seen report pass has not shown you that it can fail.

Time

For one repository and 90 days, the collector made 35 API calls and wrote 15 evidence files in 23 seconds. By hand, the closest you get is a screenshot of the ruleset, one of each merged pull request with its checks and reviews, the Dependabot tab, the security settings and the collaborator list: eleven screenshots here. They still miss the branch's history and the links from each commit to its pull request, which the direct-push control reads, and the pull request page does not put the check's start and completion times next to the merge time. We did not time the manual version, so we will not give a number for it. The difference grows with the sample: an auditor sampling 25 changes across 10 repositories is 25 pull request pages by hand, and one run per repository for the script.

Summary

Collect evidence of what happened, not only of what is configured. Our ruleset passed while three change-management controls failed under it.
Compare timestamps. A check that is green today may have finished after the merge.
Make evidence scripts fail closed: stale or withdrawn approvals, the wrong check run, unreadable data, wrong links and truncated lists must never pass.
Treat a configuration scan as one rule set, not as proof of safety, and add rules for what it misses. Ours passed a database port open to the internet.
Leave the judgement to people, and make their sign-off the evidence. A script can gather the access list; it cannot decide who should be on it.
Agree with your auditor on how exceptions, such as admin bypasses, are handled before the review period starts.

The collector, the Terraform example and the tests are in The-DevOps-Daily/soc2-evidence-collector. Point it at a repository you own and see which controls fail.

Originally published at devops-daily.com.

Neon Realtime, Measured: Live SQL Queries Replace the LISTEN/NOTIFY Layer We Built by Hand

DevOps Daily — Mon, 19 Oct 2026 09:00:00 +0000

In July we built realtime for a Neon app by hand: a Neon Function holding one server-sent events stream per browser, a Postgres LISTEN connection per function isolate, and a pg_notify call after every write. It kept one counter in sync. In August we did it again for a chat, with WebSockets in place of SSE and the same LISTEN/NOTIFY underneath.

Neon Realtime, in public beta since October 15, replaces all of that with a SQL query you subscribe to. We rebuilt the idea as a live order board on Drizzle and measured it on a real Neon project in aws-eu-central-1, from a client about 32 ms away.

A plain live query kept pace with the write itself. Across 16,200 writes at 1, 10 and 100 per second, the update reached the subscriber 35 to 43 ms (p50) after the write was issued, while the write's own commit took 34 to 40 ms to come back. Joins and aggregates were live too. All 48,600 writes in the matrix reached the subscriber, and every check we ran on the updates passed (which rows each one added, how many it removed, the size of every full result, the aggregate's total order count). We did not check every column value.

But in this beta, most of the joins we tried, every aggregate and any ORDER BY were re-sent in full on every write. The three-table join we first wrote for the board pulled 27 KB per write on a 200-row result, 52 times what the plain query used, and the cost grows with the result: 134 KB per write at 1,000 rows. At 100 writes per second, Neon kept up with those full results in our two complete raw-protocol runs. The Realtime SDK on our client needed about 9.4 ms of CPU for each one, and in one of our two SDK runs it fell about two seconds behind within a minute.

An open subscription let the compute scale to zero under it, and kept working when the next write woke it. The board did too, once its API stopped querying Postgres on every renewal. None of 11 tampered, forged or cross-branch attempts disclosed a victim row, an expired capability was refused, and a copied valid one kept working until it expired, which is how Neon documents it.

The-DevOps-Daily/neon-realtime-measured on GitHub

Warning

Realtime is in public beta, and Neon publishes no benchmarks for it yet: its own docs say performance work is still under way. Every number here is a beta number from one project, one region and one client machine, recorded on October 16, 2026, with @neon/realtime 0.1.0.

TLDR

Setup is one deploy. neon deploy with preview: { realtime } in neon.ts turned it on in about 19 seconds, with no waitlist. The primary's wal_level became logical; nothing else changed there that we could see.
A plain live query is about as fast as the write. p50 35 to 43 ms from write issued to update applied, within 4 ms of the write's own acknowledgement, from a client 32 ms away. p99 53 to 85 ms.
All 48,600 writes arrived , and every check on the updates passed: rows added, rows removed, result sizes and aggregate totals.
Most query forms we tried are re-sent whole on every write: a join to customers, EXISTS, IN, aggregates, ORDER BY and LIMIT. That is about 134 bytes per row each time, 27 KB per write at 200 rows. Single-table filters and a membership join pinned to one row stay incremental.
At 100 full results a second, our client was the limit. Two complete raw-protocol runs kept up. The SDK on our client needs about 9.4 ms of CPU per 200-row result, and one of two SDK runs fell two seconds behind.
Sessions end now and then, and a reconnect brings back the current result. The server ended a session 11 times in an hour, and the SDK recovered in about 1.2 s each time. After a 75-second network cut, its backoff took 33 to 48 s to reconnect, and the result then held the same rows as the database.
Open subscriptions let the compute scale to zero , unless the renew path queries Postgres: a membership lookup on every seal kept ours awake for the whole 300 s we watched.
Sealed queries held, and the access rule belongs in the query. No tampered, forged or cross-branch attempt disclosed a victim row. A removed member's result was empty 8 ms after the removal, against 56 s of leaked orders with a check at seal time only.
Budget the bytes. Neon's docs say network transfer is billed as usual, even though the project's meter showed far less than our test received.

Prerequisites

A Neon project in a Realtime region: aws-us-east-1, aws-us-east-2, aws-eu-central-1 or aws-ap-southeast-1 during the beta. There is no waitlist.
The Neon CLI (npm i -g neon) and Node.js 22.18 or later (the repo's scripts run as TypeScript with Node's type stripping).
The companion repository, if you want to rerun anything. Every script writes its raw output to results/.
Our earlier posts for context: Realtime Without a WebSocket Service and Realtime Chat With Auth.

What we are replacing

The July layer had three moving parts, all ours:

A Neon Function holding one server-sent events stream open per browser.
One LISTEN counter_updates connection per function isolate, on the direct connection string, because the transaction pooler cannot hold a session.
A pg_notify call after every write, carrying the new value.

Its own limits section said where it stopped. NOTIFY is fire-and-forget, so a browser that is offline at that moment misses the event. Payloads are capped at 8,000 bytes. And the app decides what each event means for the screen: a notify says "something changed", not "here is the new result".

With Realtime, the subscription is the query the screen shows. There is no function, listener or notify call in the app:

The live part of the app, before and after

July: LISTEN/NOTIFY + SSE

// Function: one LISTEN connection per isolate, on the unpooled URL
const listener = new Client({ connectionString: env.postgres.databaseUrlUnpooled })
await listener.connect()
await listener.query('LISTEN counter_updates')
listener.on('notification', (msg) => {
  for (const c of clients) c.enqueue(encode(`data: ${msg.payload}\n\n`))
})

// Every write path has to remember to notify
await pool.query('SELECT pg_notify($1, $2)', ['counter_updates', String(row.value)])

// Browser
const events = new EventSource('/events')
events.onmessage = (e) => render(Number(e.data))

Now: a sealed live query

// API: seal the query the board shows, scoped by the user id from the verified token
app.post('/api/board/live', async (c) => {
  const userId = await requireUser(c)
  const { storeId } = await c.req.json()
  return c.json(await realtime.seal({ query: openOrdersQuery(db, userId, storeId) }))
})

// Writes are ordinary writes. There is nothing to notify.

// Browser (React)
const { data, status } = useLiveQuery(sealedQuery, { refreshQuery })

The browser never sees SQL. The API builds the query with the user id from a verified Neon Auth token, and realtime.seal() encrypts the SQL, the parameters and the database name into a capability that expires after 60 seconds. The client can subscribe to it but cannot read or change it, and it renews by asking the API again.

Setup

Realtime is enabled per branch. In neon.ts it sits next to Neon Auth, which the board uses for sign-in:

import { defineConfig } from '@neon/config/v1';

export default defineConfig({
  auth: true,
  preview: {
    // Browsers may connect only from these origins. Servers and scripts
    // that send no Origin header are not restricted by this list.
    realtime: { allowedOrigins: ['http://localhost:5173'] },
  },
});

One deploy applied both and wrote the variables the app needs:

one deploy enables Neon Auth and Realtime on the branch

$ npx neon deploy
→ Applying to branch main (br-frosty-queen-baejcesa)
Applied changes
  + Neon Auth
  + Realtime

Utilized services: Postgres, Neon Auth, Realtime
Pulled 7 Neon variables into .env
  Postgres DATABASE_URL, DATABASE_URL_UNPOOLED
  Branch NEON_BRANCH
  Neon Auth NEON_AUTH_BASE_URL, NEON_AUTH_JWKS_URL
  Realtime NEON_REALTIME_URL, NEON_REALTIME_SECRET

We read the primary before and after:

wal_level went from replica to logical, because Realtime consumes logical replication.
In three checks (before, after, and while live queries were open), the primary showed no new replication slot, publication or WAL sender. The Neon API still listed a single read-write compute.

Neon's FAQ says the incremental work "is incurred on a replica". We found no replica we could query, so we could not measure its lag on its own. What we measured is the time from a write to the update arriving, which includes it.

The CLI is the quickest way to see a live query. Here is a join, subscribed in one terminal while another made three transactions:

a join as a live query, from the CLI

# terminal 1: subscribe to a join
$ npx neon realtime subscribe "SELECT o.id, o.status, o.total_cents, c.name AS customer FROM orders o JOIN customers c ON c.id = o.customer_id"
Live query
Project purple-term-17791629
Branch main (br-frosty-queen-baejcesa)
Database neondb
Mode Materialized result
Query SELECT o.id, o.status, o.total_cents, c.name AS customer FROM orders o JOIN customers c ON c.id = o.customer_id
[2026-10-16T14:46:41.545Z] Connecting…

[2026-10-16T14:46:43.904Z] Initial snapshot · 0 rows
Query returned no rows.
[2026-10-16T14:46:43.905Z] Live — waiting for changes.
# terminal 2: seed a store, then three transactions three seconds apart
$ node scripts/smoke-write.ts
[2026-10-16T14:46:48.851Z] insert order { id: 1, txid: '1052' }
[2026-10-16T14:46:51.906Z] update order { id: 1, txid: '1053' }
[2026-10-16T14:46:54.947Z] insert + update in one transaction { new_id: 2, txid: '1054' }
# terminal 1: each transaction arrives as a complete new result (Rows), not as a change
[2026-10-16T14:46:48.844Z] Rows · 0 rows
Query returned no rows.

[2026-10-16T14:46:48.862Z] Rows · 1 row
ID Status Total Cents Customer
1 new 1250 Ada

[2026-10-16T14:46:51.913Z] Rows · 1 row
ID Status Total Cents Customer
1 preparing 1250 Ada

[2026-10-16T14:46:54.952Z] Rows · 2 rows
ID Status Total Cents Customer
1 ready 1250 Ada
2 new 830 Grace

Each transaction came back as a complete new result (Rows), not as a change. In a second run with --changes, the same three kinds of transaction against a plain SELECT ... FROM orders WHERE store_id = 1 arrived as small change sets, Transaction 1056 · 1 upsert, 0 removals and so on. That difference turned out to matter more than anything else in this post.

How we measured

Everything ran from one client machine against the project in aws-eu-central-1. SELECT 1 over a direct connection took 31 to 39 ms (p50 per run, median 32 ms), and a WebSocket ping to the Live SQL API 34 to 45 ms (median 41 ms). That round trip is the floor under every latency below.

The workload. Each run creates a fresh store with 200 orders. The writer is open-loop: it issues writes on a fixed schedule (1, 10 or 100 per second) whether or not earlier ones have finished, over a pool of 12 direct connections. When its timer runs late it catches up, and latency counts from the moment each write was actually issued, including any wait for a free connection. Each write is one transaction that inserts an order and deletes the store's oldest one, so the live result stays at 200 rows for the whole run, and the three shapes are compared at the same size. Each write returns its own transaction id from pg_current_xact_id().

Three query shapes , written with Drizzle and sealed by the SDK's server half:

plain: orders filtered by store_id
join: orders joined to customers for the customer's name and to store_members for access
aggregate: count and revenue per status for the store

Matching updates to writes. The subscriber runs the browser client (createRealtimeClient, with the ws package as its WebSocket) in a separate process on the same machine, and both processes stamp events from the same clock. An incremental update carries the ids of the transactions it covers, so it matches directly. A full result carries no ids, but the wire protocol opens each new result with the Postgres snapshot it was read at (xmin, xmax, xip), and the scorer applies Postgres's own visibility rule to work out which writes it includes. Latency runs from the moment the writer issued a statement to the moment the SDK handed over the first update that included it.

Checking answers, not only timing. Every update is checked against the writes. An incremental batch must add exactly the rows its transactions inserted (by the sequence number each write stamps on its row) and remove one row for each write that deleted one. A full result must hold every stamped row the visible writes leave in the window, and exactly as many rows as the window should have. An aggregate's order counts must add up to the window. These checks do not cover which seed or deleted rows were involved, per-status counts, revenue or other column values. A self-test corrupts copies of recorded runs (a dropped update, a wrong row, a missing row, an aggregate off by one, a stall, a line from another run) and checks that the scorer reports each one.

Separating our own client. In our first recording of the matrix, other jobs on the client machine pushed it into swap, and our recorder processes froze for up to 11 seconds at a time. A frozen recorder looks exactly like a slow server in the timestamps. So the recorders now log every gap of more than 60 ms in their own event loops, the scorer sets apart writes whose window overlaps one, and the first recording stays in the repo marked as superseded. Each configuration ran twice, in two rounds with the shape order rotated.

The latency

Write issued to update applied, p50

	1 write/s	10 writes/s	100 writes/s
plain	41.9ms	42.8ms	34.7ms
join	56.2ms	53.6ms	60.4ms
aggregate	47.2ms	42.4ms	40.7ms
write acknowledged	42.5ms	38.4ms	35ms

Neon Realtime beta, aws-eu-central-1, client about 32 ms away. Each point pools two runs: 600 writes at 1/s, 3,600 at 10/s, 12,000 at 100/s. The live result is held at 200 rows. Dashed: how long the write itself took to be acknowledged, all shapes.

All 18 runs, 48,600 writes:

Shape	Writes/s	Writes	p50	p90	p99	max	After the commit ack, p50
plain	1	600	41.9	44.8	52.6	1,423	3.5
plain	10	3,600	42.8	46.9	58.6	938	2.7
plain	100	12,000	34.7	37.5	85.0	1,011	1.7
join	1	600	56.2	64.9	90.3	105	12.2
join	10	3,600	53.6	59.8	94.4	1,335	17.5
join	100	12,000	60.4	244	2,083	2,320	23.8
aggregate	1	600	47.2	50.7	63.0	176	3.9
aggregate	10	3,600	42.4	44.9	120	1,220	4.0
aggregate	100	12,000	40.7	47.6	648	1,374	7.2

All times are in milliseconds, from the write being issued to the update that included it. The last column is the same, measured from the moment the writer got its commit acknowledged; the update usually arrived within a few milliseconds of the writer hearing back, and sometimes before.

Three things in that table need explaining.

The plain query is about as fast as the network allows. Its p50 is within 4 ms of the write's own acknowledgement at every rate. Both the writes and the updates got faster as the rate went up, by about 7 ms between 1 and 100 writes per second. The likely reason is our client, not Neon: its Wi-Fi saves power when traffic is light (more on that at the end).

The join costs 11 to 26 ms more than the plain query at p50, and the reason is not the join itself. It is that every write re-sends the whole result, which the next section is about. At 100 writes per second its tail is our client falling behind, also below.

The maximums of about one second at most rates are the server ending the session. Eleven times in 59 minutes of subscriptions, a subscription received connection_error: backend_session_lost ("live query session ended; reconnect required"). Every time, the SDK reconnected and was live again 0.95 to 1.51 s later (median 1.2 s), with a complete fresh result. The scorer tied 233 writes slower than 300 ms to these episodes, 66 of them at 1 or 10 writes per second, and they account for every maximum above 900 ms at those rates. All eleven came at the same second past the minute (:42 or :43), as if a task on a 60-second cycle sometimes ends sessions. Our first, superseded recording had nine more. We cannot see the cause from the client.

At 100 writes per second, our own client stalled during part of the plain and aggregate runs (107 and 433 writes overlap a stall); without those writes, the p99 is 46 ms for the plain query and 92 ms for the aggregate. Treat those two as diagnostics, not corrected Neon numbers: a slow write has more chances to overlap a stall, and at high rates the SDK's own work causes some of the stalls.

Most queries are re-sent whole on every write

Neon's CLI docs say "some queries, such as ones with ORDER BY, can get a complete replacement for a change". In our tests, most query forms got one on every write. We subscribed to several forms of the same question at once, on a store with 300 orders, and made three single-row inserts:

Query form (300 orders in the store)	Rows in result	On each insert	Bytes per write
`WHERE store_id = $1`	300	incremental change	479
`... ORDER BY id DESC LIMIT 20`	20	full result again	2,947
`JOIN customers` for the name	300	full result again	37,346
`JOIN store_members`, pinned to one user and store	300	incremental change	472
`EXISTS (SELECT 1 FROM store_members ...)`	300	full result again	33,245
`store_id IN (SELECT ... FROM store_members ...)`	300	full result again	33,245
`count(*) ... GROUP BY status`	3	full result again	977
`count(*)`	1	full result again	760

Of these forms, only two stayed incremental: a single-table filter, and a join to store_members whose WHERE clause pins it to one row. Everything else, including EXISTS and IN, which are the obvious ways to write a membership check, was re-sent in full: 110 to 125 bytes per row on every write, whether the write touched one row or all of them.

A second set, against the board's own queries and three kinds of write:

Query form	Order insert	Order marked done	Customer insert
Board: open orders (`JOIN store_members`, `status <> 'done'`)	change, 396 B	change, 354 B	nothing sent
Board: customers (`JOIN store_members`)	nothing sent	nothing sent	change, 389 B
Board: counts per status (`GROUP BY`)	full result, 885 B	full result, 986 B	nothing sent
`JOIN customers`, both tables filtered to the store	full result, 37,180 B	full result, 37,175 B	full result, 37,175 B
`ORDER BY id`, no `LIMIT`	full result, 33,104 B	full result, 33,099 B	nothing sent
`LIMIT 50`, no `ORDER BY`	full result, 5,978 B	full result, 5,972 B	nothing sent
`SELECT DISTINCT status`	full result, 868 B	full result, 965 B	nothing sent
`WHERE store_id = $1` (control)	change, 387 B	change, 388 B	nothing sent

Two useful details are in there. A write to a table a query does not read sends that subscriber nothing at all. And sorting alone is enough to lose incremental updates: ORDER BY id with no LIMIT re-sent all 300 rows on every write. Sort in the client instead.

The cost of a full result grows with the result. The join, at 1 write per second for 60 seconds, at four result sizes, with a 5,000-row plain query as the control:

Live query	Rows in result	Bytes per write	p50	p90
join	10	1,980	48.1 ms	52.1 ms
join	100	13,953	52.9 ms	62.8 ms
join	1,000	134,225	99.2 ms	111.9 ms
join	5,000	668,496	225.1 ms	273.4 ms
plain	5,000	572	42.6 ms	45.6 ms

That is linear: about 134 bytes per row, re-sent on every write that changes anything in the result. Neon's docs say egress from Realtime is billed as usual (Free includes 5 GB per project per month; Launch and Scale include 500 GB, then $0.10 per GB). One browser tab watching that 1,000-row join while the table takes a steady 10 writes per second receives about 4.8 GB (decimal) an hour. Twenty tabs for an 8-hour day, 22 working days a month, is about 17 TB, or roughly $1,650 a month past the Launch allowance, if nothing else in the project uses that allowance. The plain query in the same setting comes to about 73 GB.

One thing we cannot square with that. When we read Neon's consumption API at 19:26 UTC, it showed 16.2 MB of public network transfer for the project for 15:00 to 18:00 UTC, three hours in which our bench recorders alone received at least 862 MB of Realtime messages. We did not find out why. Realtime is free during the beta, but the docs say the waiver does not cover network transfer, so plan on paying for the bytes above rather than going by the meter's current reading.

At 100 writes per second, the client falls behind first

A full result has to be decoded and applied by the client. For the 200-row join at 100 writes per second, that is 100 full results a second, about 2.7 MB/s.

The 200-row join at 100 writes/s, p50 per 5 seconds

	0s	5s	10s	15s	20s	25s	30s	35s	40s	45s	50s	55s
SDK client, round A	51.6ms	50.4ms	51.7ms	59.2ms	60.5ms	60ms	61.1ms	60.8ms	62.3ms	70.7ms	745.8ms	1787.3ms
SDK client, round B	49.8ms	50.2ms	51.8ms	59.1ms	59.1ms	59.3ms	60ms	53.4ms	61.3ms	61.4ms	64.4ms	64.1ms
raw recorder, run D1	40.8ms	40.2ms	40.2ms	48.9ms	47.8ms	48.4ms	50.7ms	48.8ms	48.4ms	48.7ms	48.9ms	48.7ms
raw recorder, round B	45.6ms	45.3ms	45.8ms	53.4ms	53.4ms	53.2ms	52.2ms	53.7ms	48.7ms	52.5ms	52ms	53.3ms

Every write re-sends the whole 200-row result (about 27 KB). The raw recorder speaks the wire protocol without decoding rows; the SDK client decodes and applies each full result. Same project, same writer.

To find out which side was falling behind, we wrote a second recorder that speaks the wire protocol directly and never decodes the rows. With it, at the same rate, the join held a p50 of 44 to 49 ms and a p99 of 73 to 114 ms over two complete runs of 6,000 writes, with no backlog. A third raw run started badly: 220 writes in its first five seconds were delayed by up to 957 ms, with no recorded client stall overlapping those delays, and a session loss then cut it off 16 seconds in, because the raw recorder does not reconnect. The two complete runs kept up; the third had update-path delays we cannot explain before the server ended its session.

The SDK client is the slow part, on our machine. In a separate benchmark at 20 updates a second, with a materialized subscription, it used 2.1 ms of CPU per update for the plain query, 2.0 ms for the aggregate and 9.4 ms for each 200-row join result. If that cost held at 100 a second, it would take about 94% of one core. In one of our two SDK runs at that rate it fell behind and never caught up (p50 1.8 s in the last five seconds); in the other it held on, with stalls of up to 470 ms. We did not measure other hardware. Either way, the work grows with result size times write rate, on every subscriber.

All four runs in the chart, raw and SDK alike, stepped up by 7 to 9 ms about 15 seconds in and stayed there. We did not find out why.

What a reconnect delivers

Our July layer had no answer for a dropped connection: whatever NOTIFY fired while a browser was away was gone. Realtime's answer is a fresh copy of the result. We watched two live queries (the plain one and the join) while a writer added an order every 500 ms, then cut the network twice: every socket destroyed, and new connections sent to an address that never answers. The second cut lasted 75 seconds, longer than a sealed query lives, and renewal failed during it, as it would for a browser that is offline:

cutting the network for 5 s, then for 75 s

# two live queries, plain and join, on one store; a writer adds an order every 500 ms
$ node scripts/reconnect.ts
   0.57s plain state live 
   0.92s join state live 
   0.92s setup both live: plain 100 rows, join 100 rows 
   8.92s net short: cut for 5 s 
   8.94s join state connecting 
   8.94s plain state connecting 
  13.93s net short: restored; 10 orders were written during the cut 
  16.72s plain state live 
  16.77s join state live 
  20.60s plain short +2 s: 134 rows, database 134, missing 0, extra 0 {"status":"live"}
  20.65s join short +2 s: 134 rows, database 134, missing 0, extra 0 {"status":"live"}
  28.65s net long: cut for 75 s 
  28.65s join state connecting 
  28.65s plain state connecting 
  49.85s join renew failed: offline 
# 107 more renewal failures follow, one a second for each subscription, until the network returns
 103.65s net long: restored; 140 orders were written during the cut 
 136.93s plain renewed 
 137.02s plain state live 
 151.87s join renewed 
 151.97s join state live 
 153.92s plain renewed 
 153.94s join renewed 
 156.06s plain long +2 s: 382 rows, database 382, missing 0, extra 0 {"status":"live"}
 156.14s join long +2 s: 382 rows, database 382, missing 0, extra 0 {"status":"live"}
 159.73s plain final: 382 rows, database 382, missing 0, extra 0 {"status":"live"}
 159.78s join final: 382 rows, database 382, missing 0, extra 0 {"status":"live"}
 159.79s plain state closed 
 159.79s join state closed

After both cuts, the first thing to arrive was a complete result (130 rows, then 351 and 378), not a replay of the changes that were missed, and both live results held exactly the same rows as the database. That is what you want for a screen. It is not a durable event log: if your app needs to see every intermediate state, Realtime will not give it to you after a gap, and neither did NOTIFY.

The wait after the long cut is the part to plan for. The SDK reconnects with jittered exponential backoff (a 1-second base, capped at 60 seconds), and version 0.1.0 does not listen for the browser's online event (we checked the source). So after a long outage the network can be back for half a minute or more before the board is. Show the subscription's status in the UI rather than assuming it is live. The short cut cost 2.8 seconds.

Sealed queries: what we could and could not forge

A sealed query is an AES-256-GCM encrypted capability (a compact JWE) that holds the SQL, its parameters and the database name, and expires after 60 seconds. The browser holds it but cannot read it, and the Live SQL API checks it without calling your server. We tried to get rows we were not sealed for. The attacker holds a valid sealed query for their own store, as any signed-in user of the board would, and wants another store's orders. The script speaks the wire protocol directly, because an attacker is not bound to use the SDK. Each attempt has one expected outcome, and anything else (another error or a timeout) counts as a failure, as does a single row from the victim's store, except in the documented stolen-query case. The control, the stolen-query case and the renewal attack each include a write during the attempt, so incremental changes are checked for leaks as well as first results:

Attempt	Result
Control: the attacker's own sealed query, then an order in their store	delivered: the attacker's 5 rows, then the new order as a change
One character changed in the ciphertext, the tag or the IV	`authorization_failed`
Header and IV from one capability, ciphertext and tag from another	`authorization_failed`
Ciphertext truncated	`authorization_failed`
The victim's query sealed with a guessed secret in the right format	`authorization_failed`
Header rewritten to `alg: none`	`authorization_failed`
A `sql` field added to the subscribe message	`connection_error: invalid_payload`
Renewing a live subscription with a forged capability for the victim's query, then an order in the victim's store	`renewal_failed`, and nothing from the victim's store
A query sealed with a child branch's secret, sent to the parent branch	`authorization_failed`
A query sealed with the parent's secret, sent to the child branch	`authorization_failed`
The attacker's own sealed query, 65 seconds after sealing	`authorization_expired`
The victim's real sealed query, copied, then an order in the victim's store	delivered: all 30 victim rows, then the new order

All 14 outcomes were as expected, in a first run and again in this stricter one after review. No tampered, forged or cross-branch attempt disclosed a victim row or change, and the expired capability was refused. A child branch got its own Realtime URL and its own secret, so a preview branch's capabilities do not open production. To check that the harness could fail, a self-test swaps one attack's payload for the victim's real sealed query; the checks then report that attack as a leak.

The last row is the documented catch: a sealed query is a bearer token. Anyone who holds one can subscribe until it expires. Expiry closes that window within 60 seconds of sealing; it does not take back rows already received. Keep sealed queries out of URLs, logs and persistent browser storage, as Neon's docs say.

Scoping per user without row-level security

Realtime has no row-level security yet. Neon's guidance is to authorize in the request handler and put the authorization inside the query: take the user id from a verified token, and join through the membership table so the query itself only matches rows that user may see. We tested that against two weaker designs, through the board's API, with real Neon Auth users. These runs used a version of the API that also looked the user up in store_members before sealing, which is why renewals were refused after a removal; the board in the repo now skips that lookup (see the next two sections) and seals a query whose membership join simply returns nothing for a non-member:

Design	After the user lost access
Membership join inside the live query (the board)	Their result was empty 8 ms after the removal was acknowledged (one test, checked every 5 ms). 0 of 6 orders added afterwards reached them.
Membership checked when sealing, but the query only filters by `store_id`	12 of 18 new orders still reached them. The last one was added 56 s after the removal. It stopped when the sealed query expired and the API refused to renew it.
A member's sealed query copied to a non-member	12 of 18 new orders reached the non-member. The last one was added 56 s after sealing. The API refused the non-member's renewal.

So the pattern works, and the join is what makes it work. A check at seal time alone leaves a window of up to one sealed-query lifetime, and only because the renewal goes back through your API, which refuses it. Put the access rule in the query, with the user id sealed in from a verified token, and a removal reaches the live result with the next update, without waiting for the sealed query to expire.

One more consequence of the full-result rule: the obvious ways to write that access check, EXISTS and IN (SELECT ...), are re-sent whole on every write. The join to store_members, pinned to one row by the WHERE clause, is not.

The board

The live order board in the repo is a Vite and React app with a Hono API, Drizzle for the schema and the queries, and Neon Auth for sign-in. Its first version used the three-table join from the measurements, which re-sends every order on every write. The version in the repo subscribes to three queries instead, each scoped by a join to store_members on the user id from the verified token:

// Open orders: incremental. The customer name is not joined in here.
export function openOrdersQuery(db: Db, userId: string, storeId: number) {
  return db
    .select({
      id: orders.id,
      customer_id: orders.customer_id,
      status: orders.status,
      total_cents: orders.total_cents,
      created_at: orders.created_at,
    })
    .from(orders)
    .innerJoin(storeMembers, eq(storeMembers.store_id, orders.store_id))
    .where(
      and(eq(storeMembers.user_id, userId), eq(orders.store_id, storeId), ne(orders.status, 'done'))
    );
}

// Customers: incremental, and an order write sends this subscription nothing.
export function storeCustomersQuery(db: Db, userId: string, storeId: number) {
  return db
    .select({ id: customers.id, name: customers.name })
    .from(customers)
    .innerJoin(storeMembers, eq(storeMembers.store_id, customers.store_id))
    .where(and(eq(storeMembers.user_id, userId), eq(customers.store_id, storeId)));
}

// The header: an aggregate, re-sent whole on every write, which is fine at four rows.
export function statusCountsQuery(db: Db, userId: string, storeId: number) {
  /* GROUP BY status */
}

The browser joins the first two with a Map from customer id to name, and sorts on its own side, because ORDER BY in the live query would re-send everything. Status changes are optimistic: the card moves at once with React's useOptimistic, the API's UPDATE ... RETURNING pg_current_xact_id() returns the transaction id, and awaitTxId() on the live query holds the optimistic state until that transaction arrives.

The API's seal endpoint verifies the Neon Auth token and seals, and that is all: sealing is local crypto, so a renewal never touches Postgres. It does not look the user up in store_members first. That lookup would only turn a non-member's empty board into a 403, because the join inside each query is what keeps them out, and it would cost a database query on every renewal of every subscription. The next section shows what that does to scale to zero.

A scripted run through the same API and SDK calls the app makes (scripts/board-check.ts) passed every check. A new order appeared on the board 99 ms after the POST was sent, moving it to preparing showed up in 70 ms, and marking it done took it off the board in 54 ms, each measured to the moment awaitTxId resolved. A user from another store got an empty board from the same endpoint, and a 403 when adding an order to it.

Scale to zero: open subscriptions let the compute sleep

Neon's docs say holding a subscription open does not keep the database awake, but starting one can wake it. We checked both, with the suspend timeout shortened to 60 seconds through the Neon API and the compute's state polled every 5 seconds:

Step	What happened
Control: nothing open	The compute suspended 65 s after we started watching.
A live query started while the compute was suspended	First result in 0.98 s. The compute was active again 0.3 s later.
That subscription held open, renewed every 50 s, no writes	The compute suspended 75 s later, and was still suspended 60 s after that. The subscription stayed `live` the whole time.
A write while suspended, subscription still open	Connecting to the waking compute took 853 ms and the insert 247 ms (1.14 s in all, closing the connection included). The open subscription received the change 1.12 s after the write was issued, with no reconnect.
A second live query started from a suspended compute	First result in 1.84 s, and it woke the compute.

So an open subscription costs no compute while nothing writes, and it survives the compute going to sleep under it. The cost is on the first write or the first new subscriber after a quiet spell: about a second, the same cold start any query pays. We could not see the replica that Neon says does the incremental work, so we cannot say whether it suspends too; Neon does not charge for it during the beta.

The board itself is another matter, because its renewals go through your API. We ran alice's three board queries with the browser's renewal timing (10 seconds before each 60-second expiry), sealed and renewed through the board's API, with the suspend timeout at 60 seconds:

Board API	Compute while the board sat open with no writes
Looks the user up in `store_members` on every seal	Never suspended in 300 s. 18 renewals, each one a database query.
Seals without touching Postgres (the version in the repo)	Suspended 75 s after the board opened.

The membership lookup looks harmless, and it cost scale to zero entirely: with it, each open tab made three database queries roughly every 50 seconds while its renewals succeeded, one every 17 seconds on average, and that kept the compute awake for the whole 300 seconds we watched. Keep the renew path free of database calls. Signing in, and refreshing the Neon Auth token (it lasts about 15 minutes), go through Neon Auth, which keeps its tables in the same database; we did not measure whether a token refresh wakes the compute.

Writes on the primary, with and without 100 live queries

Neon's FAQ says Realtime has "zero impact on the write-latency, write-throughput or write-compute of your primary database". We ran a writer at 20 writes per second for the whole test and switched 100 live queries on and off in alternating 60-second blocks: off, on, off, on, off, on. Each query belonged to a different user and every write touched all 100 of them: nominally 2,000 updates a second (20 writes times 100 queries) to the subscriber process. Writes during each switch and for 20 seconds after it were left out.

	100 live queries off	100 live queries on
Write execution time on the primary (`pg_stat_statements`, mean per block)	0.157 to 0.164 ms	0.150 to 0.166 ms
Write latency on the client, p50 per block	34.3 to 39.4 ms	32.3 to 33.8 ms
Write latency on the client, p99 (all blocks)	47.6 ms	42.5 ms
Writes	3,603	3,603

On the server, the write's own execution time did not move. On the client, the pooled median write was about 2.5 ms faster with the live queries on, and each on block had a lower median than the off block before it, which is the opposite of a slowdown. We think that is our client and not Neon: it is on Wi-Fi with power saving on, and the extra incoming traffic keeps the radio awake. We did not test that explanation. A first version of this test let the CPU clock follow the load, tagged some switching time to the wrong block, and lacked an index that keeps the write's DELETE cheap (its execution time jumped from 0.3 to 3.2 ms partway through). It showed the same direction and is kept in the repo as superseded. Both versions ran with Realtime enabled throughout, so neither measures what turning Realtime on (wal_level = logical) costs.

So under this workload we saw no slowdown of writes on the primary with 100 subscriptions open, which matches Neon's claim. pg_stat_statements does not include commit time, though, and our client's own noise is larger than a millisecond, so "no slowdown we could measure" is as far as this goes.

What we could not conclude

The replica's lag on its own. Neon says the incremental work runs on a replica. We could not see one in the project, the Neon API or the primary's replication views, so every number here is end to end.
Why the server ends sessions. Twenty backend_session_lost errors across both recordings, most of them at the same second past the minute, as if on a 60-second cycle. The SDK handled every one, but we cannot see the cause.
Why the four join runs at 100 writes per second in the chart got 7 to 9 ms slower about 15 seconds in, the raw recorder's included.
What the SDK costs on other hardware. We measured it on one client, a Raspberry Pi on Wi-Fi with power saving on, and nowhere else. The Wi-Fi also shaves a few milliseconds off everything when traffic is heavy, which is the likely reason most numbers improved at higher write rates.
More than 100 subscribers, or more than one client. Neon says thundering-herd protection is built in; we did not test a reconnect storm.
The bill. Realtime is free during the beta and its network transfer is documented as billed, but the project's transfer meter showed far less than our Realtime traffic, so we cannot tell you what this test would have cost.
Whether any of this survives the beta. Neon calls the protocol unstable and the performance work unfinished. The full-result behaviour in particular may change; rerun scripts/explore-incremental.ts before you design around it.

Wrapping up

For a live order board, Neon Realtime replaces everything our July layer did and covers what it could not. There is no function holding connections, no LISTEN session, and no NOTIFY call to forget on a write path. A reconnect brings back the right result instead of losing events, and the board shows a query result rather than an event the app has to interpret. A plain live query arrived as fast as the write's own acknowledgement, and none of the 48,600 writes went missing.

The part to design for is the full re-send. Write the live queries as single-table filters, or join only to a membership table pinned to one row. Do the other joins and all sorting in the client, and keep live aggregates to a handful of groups. Check each query with the CLI's --changes flag: a Rows line on every write means the whole result is coming back each time. Put the access rule inside the query, and budget for the bytes a full result costs on every write, times every subscriber.

The repo has the board, every script, and the raw output of every run, including the ones we superseded.

Originally published at devops-daily.com.

Prompt Caching on DigitalOcean Inference, Measured: What Breaks It and How Long It Lasts

DevOps Daily — Mon, 19 Oct 2026 09:00:00 +0000

Most LLM features send the same long block of text on every request: a system prompt, a policy, a runbook, a list of tools. DigitalOcean Serverless Inference caches that repeated prefix automatically on the models it hosts, bills the cached part at a discount, and starts streaming sooner. You do not turn it on. You can turn it off by accident, and the request itself does not tell you.

We sent 867 requests to Llama 4 Maverick, DeepSeek V4.1 Flash and gpt-oss-120b, with an on-call runbook of up to about 67,000 tokens as the system prompt. With an 8,000-token runbook first and unchanged, the cache served 99.7 percent of the prompt on Llama 4 Maverick after the first request and cut the input cost about 5 times. On DeepSeek V4.1 Flash it cut it more than 12 times. Then we put the current time on the first line of the system prompt, the way a lot of prompt templates do, and every request after the first missed , on all three models.

This post covers what we measured: which prompt layouts keep the cache and which break it, what a hit does to time to first token as the prompt grows, how long a cached prefix lasted on each model, what prompt_cache_key changes, and why gpt-oss-120b was the hardest to keep warm. Every token and cache number comes from the usage block the API returned, timings come from our client, and the harness and raw results are public.

The-DevOps-Daily/do-inference-prompt-cache on GitHub

TL;DR

With a stable 8k-token prefix, input cost per 1,000 requests went from $1.92 to $0.39 on Llama 4 Maverick and from $2.40 to $0.19 on DeepSeek V4.1 Flash. gpt-oss-120b went from $0.78 to $0.35 in one run and to $0.55 in another; the share of its prompt served from cache swung between 35 and 68 percent.
A timestamp on the first line of the system prompt: all 57 requests after the first missed, across the three models. The same timestamp after the runbook: back to 99.5, 93.7 and 63.2 percent of tokens from cache. A user's name a quarter of the way in: only the part before it, 27 to 28 percent of the prompt, was cached.
At 64k tokens, a cache hit cut the time to the first streamed token from 5.0 to 1.4 seconds on Llama 4 Maverick and from 3.1 to 0.9 seconds on DeepSeek V4.1 Flash. gpt-oss-120b never hit at 64k in 5 tries.
How long a prefix stayed usable depended on the model. Llama 4 Maverick hit after a nominal 5 idle minutes and missed after 15. DeepSeek V4.1 Flash missed every prefix it had seen only once, and hit every prefix it had seen twice, up to 30 minutes later. gpt-oss-120b missed all 9 probes after a minute or more.
prompt_cache_key kept caches apart: a second key on the same prefix missed 12 of 12 times. It did not change hit rates in our runs.

Prerequisites

A DigitalOcean account with a Serverless Inference model access key
Node.js 22 or later (the harness has no dependencies)
Optional: an API token with billing read access, for the invoice check in the repo
About $2 of inference credit to rerun everything; our recorded requests cost about $1.06 at list prices

How the cache works on DigitalOcean-hosted models

The prompt caching guide covers three families. Anthropic and OpenAI models have their own controls (cache_control, prompt_cache_retention). The open-weight models DigitalOcean hosts itself, which is what we tested, cache with no code change at all:

The cache matches an exact token prefix, in blocks of up to 256 tokens depending on the model.
Writing to the cache is free. Cached input tokens are billed at the model's cache read rate.
Entries have no fixed expiry. They are evicted least-recently-used first, "and sooner under heavy load".
The cache lives on the replica that served the earlier request. The platform routes matching prefixes back to that replica, but under very high load it may send a request elsewhere, which is a miss.
Every response reports the cached part in usage.prompt_tokens_details.cached_tokens (and cache_read_input_tokens).

The discount differs a lot between models. These are the rates for the three we tested, from the pricing page (last verified 15 October 2026) and our own invoice:

Model	Input, per 1M tokens	Cached input, per 1M	Discount
Llama 4 Maverick	$0.25	$0.05	80%
DeepSeek V4.1 Flash	$0.30	$0.006	98%
`gpt-oss-120b`	$0.10	$0.02	80%

The pricing page lists no cache read rate for gpt-oss-120b; the $0.02 is the rate on the Cache Read Input line of our invoice preview. The caching guide says cached tokens are "discounted by up to 90%", but the pricing page and our invoice both show DeepSeek V4.1 Flash at 98 percent off.

The test

The prompt is an on-call runbook for a made-up platform: 64 services, each with an owner, an SLO, its alerts and numbered first steps with real-looking commands (deployctl rollback checkout-api, featurectl set checkout-api.degraded true). A seeded script cuts it at section boundaries to nominal sizes of 1k, 4k, 8k, 16k and 64k tokens, at about four characters per token. Each model's tokenizer counts the same text a little differently: the 8k runbook is 7,700 to 8,000 tokens, and the 64k one 64,000 to 67,000. The runbook is the system prompt, and the user message is one of 40 on-call questions ("checkout-indexer is returning 5xx after a deploy 10 minutes ago. What is the first command to run?").

Every request is streamed with stream_options: { include_usage: true }, temperature 0 and max_tokens 48. We record the time to the first streamed token, total time and the usage block. For DeepSeek V4.1 Flash and gpt-oss-120b that first token is usually reasoning, not answer text, because both reason before they answer. Each experiment starts from a prefix the cache has never seen: a random revision id on the first line, the same for all requests in that experiment.

Two definitions matter for reading the numbers:

A hit is a request where at least half the prompt came from the cache, and a miss is one where under 10 percent did. gpt-oss-120b reports 64 cached tokens on many requests that share nothing with an earlier one, which looks like its chat template, so "cached tokens above zero" does not mean a hit.
Llama 4 Maverick answered 529 Platform overloaded 55 times during the run. The harness retried each one, counted it and kept it out of the timings. The other two models refused nothing. So the 867 requests are the ones that succeeded: 922 attempts in all, plus a first run of 20 requests we discarded because it did not record its retries.

00-build-prompts runbook 1k to 64k
02-run one experiment, one model
results/raw every usage block
03-score hits, TTFT, cost

A stable prefix: what you get

The first experiment is the best case: the same 8k-token runbook on 20 requests in a row, a different question each time. This chart shows how much of each request's prompt came from the cache.

Share of the prompt served from cache, 20 requests in a row

	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17	18	19	20
Llama 4 Maverick	0%	99.6%	99.7%	99.8%	99.8%	99.8%	99.8%	99.8%	99.6%	99.6%	99.7%	99.8%	99.8%	99.8%	99.8%	99.8%	99.6%	99.6%	99.7%	99.8%
DeepSeek V4.1 Flash	0%	0%	99.2%	99.3%	99.2%	99.2%	99.2%	99.2%	99.1%	99.1%	99.2%	99.2%	99.2%	99.2%	99.2%	99.2%	99.1%	99.1%	99.2%	99.2%
gpt-oss-120b	0.8%	98.9%	0%	99.1%	0.8%	99.1%	99.1%	0%	0.8%	98.9%	99%	99.1%	99.1%	0%	99.1%	99.1%	99%	98.9%	0.8%	99.1%

8k-token runbook as the system prompt, one new question per request, sent one after another. cached_tokens / prompt_tokens from each response. Run on 19 October 2026.

Three different behaviors from one feature:

Llama 4 Maverick hit from the second request on and never missed again.
DeepSeek V4.1 Flash missed the first two requests and hit every one after that. That was common, not universal. Counting every experiment that sent a new prefix twice within seconds, the second request missed 24 times out of 30, and the third request on the same prefix hit all 26 times we sent one.
gpt-oss-120b missed 6 of the 19 requests after the first, scattered through the run. In a later run with the same layout, it hit only 10 of 28. A cached prefix lives on one replica, and the guide says requests can land on one that does not hold it. That fits what we saw, but the response does not say which replica served it, so we cannot confirm it.

Here is what that does to the input bill, per 1,000 requests at list prices:

Input cost per 1,000 requests, 8k-token prompt

	Value	Series
Llama 4 Maverick, no cache	1.92$	Without the cache
Llama 4 Maverick, measured	0.39$	Measured
DeepSeek V4.1 Flash, no cache	2.4$	Without the cache
DeepSeek V4.1 Flash, measured	0.19$	Measured
gpt-oss-120b, no cache	0.78$	Without the cache
gpt-oss-120b, stable run	0.35$	Measured
gpt-oss-120b, later run	0.55$	Measured

Average over requests 2 to 20 of the stable run, scaled to 1,000, so DeepSeek's second-request miss is included. Counting the first request too gives $0.47, $0.30 and $0.37. The gpt-oss-120b later run is the key-ab experiment (both arms, requests 2 to 8 on four prefixes each, about 36% of tokens from cache), priced the same way. Input only; output is capped at 48 tokens and not compared.

On list price, gpt-oss-120b is the cheap one: $0.10 per million input tokens against $0.30 for DeepSeek V4.1 Flash. With a stable prefix the order flips, because DeepSeek's cached tokens cost almost nothing and it hit far more often. If most of your tokens are a repeated prompt, compare models on cached cost and measured hit rate, not on the input price. Output is a separate sum: both models spent most of their 48 output tokens on reasoning in our runs, and DeepSeek's output costs $1.20 per million against $0.70.

What breaks it

The guide lists the patterns that break the prefix match: a timestamp or request ID in the system prompt, per-user data before the shared instructions, tool definitions that change order. We tried three of those layouts on the same runbook, 20 requests each, each from a fresh prefix:

stable [revision][runbook .......................][question]
timestamp-first [time][revision][runbook .................][question]
timestamp-last [revision][runbook .................][time][question]
user-in-middle [revision][runbook ...][name][runbook ....][question]

Share of the prompt from cache after the first request, by layout

	Value	Series
Llama 4 Maverick, stable	99.7%	Llama 4 Maverick
Llama 4 Maverick, time first	0%	Llama 4 Maverick
Llama 4 Maverick, time last	99.5%	Llama 4 Maverick
Llama 4 Maverick, name in middle	27.4%	Llama 4 Maverick
DeepSeek V4.1 Flash, stable	94%	DeepSeek V4.1 Flash
DeepSeek V4.1 Flash, time first	0%	DeepSeek V4.1 Flash
DeepSeek V4.1 Flash, time last	93.7%	DeepSeek V4.1 Flash
DeepSeek V4.1 Flash, name in middle	26.5%	DeepSeek V4.1 Flash
gpt-oss-120b, stable	67.9%	gpt-oss-120b
gpt-oss-120b, time first	0.8%	gpt-oss-120b
gpt-oss-120b, time last	63.2%	gpt-oss-120b
gpt-oss-120b, name in middle	19.5%	gpt-oss-120b

Cached tokens as a share of prompt tokens over requests 2 to 20 of each run. 8k-token runbook, 19 October 2026.

A timestamp on the first line changes the first tokens of every request, so nothing after it can match. All 57 requests after the first missed on all three models (the 0.8 percent on gpt-oss-120b is its 64 template tokens). The requests still succeeded, and the only sign in the response was the usage block. On Llama 4 Maverick that layout cost $1.93 per 1,000 requests instead of $0.39.

Moving the timestamp after the runbook brought the cached share back to the stable run's level on Llama 4 Maverick and DeepSeek V4.1 Flash (99.5 and 93.7 percent), and to 63.2 percent on gpt-oss-120b, inside its usual swing from run to run. The name a quarter of the way in shows how prefix matching degrades. When the cache was used, it served the part before the name, 27 to 28 percent of the prompt, and nothing after it. DeepSeek V4.1 Flash and gpt-oss-120b also missed outright on 1 and 6 of the 19 requests.

The fix is in the template, not the platform:

// Breaks the cache: the first tokens change on every request.
const system = `Current time: ${new Date().toISOString()}\n\n${runbook}`;

// Keeps it: the runbook is a fixed prefix and the time comes after it.
const system = `${runbook}\n\nCurrent time: ${new Date().toISOString()}`;

// Better still: keep the system prompt byte-identical and put
// per-request data in the user message.
const messages = [
  { role: 'system', content: runbook },
  { role: 'user', content: `On-call engineer: ${name}\nTime: ${now}\n\n${question}` },
];

"Byte-identical" includes things that are easy to miss: a JSON blob serialized with keys in a different order, a tool list built from a Set, trailing whitespace from a template engine. If you build the prefix in code, hash it and log the hash. Two hashes for one prompt version is a cache problem waiting to show up on the bill.

Bigger prompts, bigger latency win

We ran the same comparison at four more prompt sizes: 5 requests on a fresh prefix each (all misses), and 6 requests on one prefix per size, where requests 2 to 6 that hit give the hit column. Medians of the time to the first streamed token; each miss cell is 5 requests, and each hit cell is 5 unless marked:

Prompt size	Llama 4 Maverick, miss / hit	DeepSeek V4.1 Flash, miss / hit	`gpt-oss-120b`, miss / hit
~1k tokens	940 / 773 ms	707 / 524 ms (4 hits)	599 / 564 ms (3 hits)
~4k tokens	939 / 807 ms	642 / 546 ms	848 / 519 ms (4 hits)
~16k tokens	1,458 / 878 ms	1,032 / 626 ms	1,169 / 609 ms (2 hits)
~64k tokens	5,015 / 1,444 ms	3,068 / 911 ms	4,770 ms / no hit

Time to first streamed token, nominal 64k-token prompt

	Value	Series
Llama 4 Maverick, miss	5015ms	Miss
Llama 4 Maverick, hit	1444ms	Hit
DeepSeek V4.1 Flash, miss	3068ms	Miss
DeepSeek V4.1 Flash, hit	911ms	Hit
gpt-oss-120b, miss	4770ms	Miss

Median of 5 requests each. gpt-oss-120b did not hit on any of 5 requests at this size. Run on 19 October 2026.

At 1k to 4k tokens the medians differed by 35 to 330 milliseconds. At 64k they differed by 2.2 seconds on DeepSeek V4.1 Flash and 3.6 seconds on Llama 4 Maverick, seconds in which nothing streams back. If you put whole documents or long tool output in the prompt, this can matter more than the discount. The groups are small and were collected a few minutes apart rather than paired, so read the shape, not the exact milliseconds.

How long a cached prefix lasts

The guide says entries have no fixed expiry and are evicted when the space is needed. We wanted to know what that means in minutes. Each prefix was sent once or twice, left alone, and asked again. The minute gaps are nominal: we counted them from the start of priming, so the real idle time was up to 35 seconds shorter (the raw data has the exact times).

Asked again after	Llama 4 Maverick	DeepSeek V4.1 Flash	`gpt-oss-120b`
5 to 45 seconds (sent twice)	8 of 8 hit	8 of 8 hit	4 of 8 hit
1 minute, sent once / twice	hit / hit	miss / hit	miss / miss
5 minutes, sent once / twice	hit / hit	miss / hit	miss / miss
15 minutes, sent once / twice	miss / miss	miss / hit	miss / miss
30 minutes, sent once / twice	miss / miss	miss / hit	miss / miss
60 minutes, sent once	miss	miss	miss

Each cell is one request (two per gap for the seconds row), so this is a sketch of one afternoon, not a guarantee. A miss also does not prove the entry was evicted; the request may have gone to a replica that never had it. It still says three useful things:

Llama 4 Maverick hit at the nominal 5-minute probes and missed at 15 minutes, whether it had seen the prefix once or twice. It was also the model returning 529 overload errors, and the guide says eviction comes sooner under heavy load.
DeepSeek V4.1 Flash missed every prefix it had seen only once, even after one minute, and hit every prefix it had seen twice, up to 30 minutes later. That is the same pattern as the second-request misses above, so it may have nothing to do with time: on this model, the cache paid off from the third request.
gpt-oss-120b hit about half the time within 45 seconds and missed all 9 probes at a minute or more.

For traffic that repeats every few seconds, all three caches work. For a prefix used once every 15 minutes or so, such as a per-tenant prompt on a quiet tenant or a nightly job, plan on paying full price.

Bursts on a cold prefix

What happens when a deploy or a queue drain sends ten requests at once with a prefix nobody has sent before? We sent ten at once on a cold prefix, then ten more:

DeepSeek V4.1 Flash hit on 8 of the first 10 and gpt-oss-120b on 7, even though none of them had seen the prefix before the burst started.
Llama 4 Maverick refused 5 of the first 10 with 529 (9 refusals in all) and the harness retried them a few seconds later, so its number mixes concurrent and later requests. Of the 5 that went through on the first try, 4 hit.
The second wave of ten hit 10 of 10 on all three models.
One request alone and then ten at once on another new prefix: 10, 9 and 9 of the ten hit.

So a cold burst does not pay full price ten times. The responses do not show how the server scheduled the requests, only that most of them reported cached tokens.

What prompt_cache_key does

prompt_cache_key is an optional field on the request. The guide describes it as a way to keep cached context apart inside one account, for different tenants or workloads, and as a cache reset when you roll out a new prompt. Caches are already isolated between accounts.

We checked the separation. On four fresh prefixes per model, we sent key A twice, then key B, then key A again, then no key at all, then key B a second time:

node 03-score.mjs (cache-key rows)

# a-1, a-2: key A twice. b-1: key B. a-3: key A again. none: no key. b-2: key B again.
$ STAMPS_OK=... node 03-score.mjs | grep cache-key
  cache-key a-1 0/4, a-2 4/4, b-1 0/4, a-3 4/4, none 0/4, b-2 4/4
  cache-key a-1 0/4, a-2 3/4, b-1 0/4, a-3 4/4, none 0/4, b-2 3/4
  cache-key a-1 0/4, a-2 2/4, b-1 0/4, a-3 1/4, none 0/4, b-2 1/4

The rows are Llama 4 Maverick, DeepSeek V4.1 Flash and gpt-oss-120b. The first request with key B missed 12 of 12 times across the three models, on a prefix that had just hit under key A in 9 of those 12 trials. The same prefix with no key also missed 12 of 12, so in our runs "no key" behaved as a namespace of its own. Key A kept hitting after B was used on Llama 4 Maverick and DeepSeek V4.1 Flash (8 of 8); gpt-oss-120b hit 1 of 4, in line with its usual misses.

Does a key also improve hit rates, by keeping related requests together? We ran two new prefixes side by side, one with a key and one without, 8 requests each, four times per model. Llama 4 Maverick hit 28 of 28 later requests either way, DeepSeek V4.1 Flash 24 of 28 either way, and gpt-oss-120b 10 of 28 either way. The guide says the key does not pin requests to a replica, and it made no difference here.

Use prompt_cache_key for what it is for: separating tenants that share a prompt template, and forcing a clean start when you ship a new prompt version. Use a value that is stable per tenant or workload. A request ID as the key gives every request its own empty cache.

Watch the cached share in production

The cache never fails loudly. A template change that puts a timestamp first looks exactly like a working deploy, until the invoice arrives. The usage block on every response is enough to catch it:

// After each streamed response, with stream_options: { include_usage: true }
const prompt = usage.prompt_tokens;
const cached = usage.prompt_tokens_details?.cached_tokens ?? usage.cache_read_input_tokens ?? 0;

metrics.counter('llm_prompt_tokens_total', prompt, { feature, model });
metrics.counter('llm_cached_tokens_total', cached, { feature, model });
// Alert on sum(cached) / sum(prompt) per feature and model, over 15 minutes,
// against that feature's normal rate. A drop right after a deploy points at the prompt first.

The DigitalOcean control panel shows the hit rate per model too (Serverless Inference, Analyze tab). Per-feature numbers from your own logs are more useful, because one model usually serves several features and only one of them will break.

What we could not conclude

One afternoon, one account. All 867 recorded requests ran between 12:48 and 13:55 UTC on 19 October 2026. Hit rates and eviction depend on load: the share of gpt-oss-120b's prompt served from cache went from 68 to 36 percent between two runs about an hour apart.
Small samples where it hurts. The idle table is one request per cell, and the latency table is 5 misses and at most 5 hits per cell. They show direction, not exact values.
No view of replicas. The response does not say which replica served it, so the DeepSeek second-request pattern and the gpt-oss-120b misses are described, not explained.
The bill check is not done yet. The repo compares the token counts from the API with the change in the invoice preview. A snapshot taken right after the last request, 68 minutes after the first one, showed no change yet (it is in the repo), so this post uses list prices and the cache read rates from our invoice, not a measured bill.
First streamed token, not first answer word. For the two reasoning models, the latency numbers measure the first reasoning token.
Synthetic prompts, ungraded answers. The runbook is generated, and answers were capped at 48 tokens. This measures input caching, not answer quality.

What to do with this

Put everything static first and keep it byte-identical: system prompt, policies, tool definitions, reference documents. Put the time, request IDs, user names and retrieved passages after it, or in the user message.
Log cached_tokens / prompt_tokens per feature and model, and alert when it drops. The response does not warn you any other way.
Budget at list price and treat the cache as a discount you have to keep earning. On gpt-oss-120b, and on prefixes that repeat less than every few minutes, it may not be there.
When you compare models for a long-prompt workload, compare cached cost and measured hit rate. On our runbook, DeepSeek V4.1 Flash had the higher input price and the lower input bill.
Use prompt_cache_key per tenant or per prompt version, never per request.
If your prompts are tens of thousands of tokens, the latency win is worth more than the discount: on our 64k prompt, a hit reached the first streamed token about 3.5 times faster on Llama 4 Maverick and 3.4 times faster on DeepSeek V4.1 Flash.

The cache is free, automatic and easy to break. Five minutes with the usage block tells you whether yours is working, and the harness runs the same tests on your own prompt if you replace the runbook. If you want the per-feature cost side of this, Knowing What Your AI Feature Costs Before Finance Does shows how to tag usage by feature on one key.

Originally published at devops-daily.com.

Postgres 19 Slipped to November: Every Pulled Feature and Why

DevOps Daily — Sat, 17 Oct 2026 09:00:00 +0000

PostgreSQL 19 was planned for October 2026, and the project's roadmap still says so. General availability is now planned for 19 November, with a second release candidate on 12 November. It will ship without SQL/PGQ graph queries, UPDATE/DELETE ... FOR PORTION OF, online checksum changes, ALTER TABLE ... MERGE/SPLIT PARTITION, and seven smaller features that were in the beta 1 release notes. The short version going around is that AI reviewers found more bugs than the developers could fix. That is part of it. Peter Eisentraut, who committed several of the reverted features, lists several possible causes and says it is probably a different combination for each feature. The mailing-list threads and the commit log fit his account, and which cause matters to you depends on what you planned to use.

This post goes through every feature that was pulled, with the reason given in its revert and what to use instead. It gives you a script that finds beta-era code using them, covers a limit in REPACK CONCURRENTLY (a feature that stayed) that its reference page does not mention, and counts what the commit log actually says about AI.

TLDR

13 release-note items, covering 11 features, are gone since beta 1. Ten features were reverted after beta 1 was stamped on 1 June. The eleventh was reverted in May, but its release-note entry stayed until 13 October.
The five big reverts landed between 27 August and 16 September. Their revert messages list 47 commits (SQL/PGQ), 30 (online checksums, some only partly reverted), 23 (FOR PORTION OF), 14 (MERGE/SPLIT PARTITION) and 13 (pg_get_*_ddl()).
The causes, as Eisentraut describes them: some code was not good enough, some features had design problems that beta could not fix, LLM-assisted review found a long tail of edge-case bugs faster than they could be handled, security work from April to August took the senior reviewers, and "everything works with everything" in Postgres makes some features very hard to finish.
The November date has its own reason: the release team is folding the fixes from the 12 November minor releases into 19 before general availability.
AI in the commit log: 23 of the 937 commits after beta 1 mention AI assistance, an AI tool or an AI lab. That counts only explicit credit, so it is a floor.
For your upgrade: run the check script over migrations and ops code written against a beta, plan REPACK CONCURRENTLY runs one at a time across the whole cluster, and expect the first scheduled minor release on 11 February 2027.

Prerequisites

Code, migrations or runbooks written against a PostgreSQL 19 beta, if you want to check them
bash and GNU grep for the check script
Python 3 and an authenticated gh CLI if you want to rebuild the counts from the source tree

Everything below comes from the REL_19_STABLE branch at commit 221423812d (17 October 2026), the release notes at the REL_19_BETA1 tag, and the threads linked in each section. The scripts and the data snapshot are in the companion repo:

The-DevOps-Daily/pg19-revert-ledger on GitHub

The timeline

Dates are commit and tag dates in the source tree. Public announcements follow a few days after a tag.

Date	Event
April 2026	Feature freeze for 19
11 May	23 security commits for the May minor releases
24 May	Database-specific logical replication snapshots reverted
1 June	Beta 1 stamped
8 June to 17 July	Four smaller reverts: fast defaults for domains, `pg_dumpall` formats, a lower `ALTER DOMAIN` lock, `GROUP BY ALL`
13 July	Beta 2 stamped
10 August	Beta 3 stamped, and 35 security commits for the August minor releases
25 August	Robert Haas posts an LLM ranking of the riskiest PostgreSQL 19 features
27 August to 16 September	MERGE/SPLIT PARTITION, SQL/PGQ, CREATE SCHEMA, `pg_get_*_ddl()`, FOR PORTION OF and online checksums reverted
21 September	Beta 4 stamped
12 October	RC1 stamped
12 November	RC2 planned, the same day as the minor releases
19 November	General availability planned

The cluster from late August to mid September is the part that made news. Five large features came out in three weeks, two of them in the week before the last beta.

What was pulled, and why

The list comes from comparing the release notes at beta 1 with the branch head. An item counts as removed when none of the commits it cites appear in any current item, so items that were only reworded or merged do not count. That gives 13 items. The three pg_get_*_ddl() functions are one feature, so 11 features.

Feature	Reverted	Reason given	What to do instead
SQL/PGQ property graphs (`CREATE PROPERTY GRAPH`, `GRAPH_TABLE`)	7 Sep	Open design questions the release management team listed on 2 September: graph metadata left orphaned by `DROP ... CASCADE` (which could make a graph impossible to dump and restore), how labels and properties are scoped, whether a graph needs a `pg_class` entry, and `pg_dump` dependency loops. Some fixes still needed catalog changes.	Recursive CTEs, or a graph extension such as Apache AGE (check its support for your version)
`UPDATE`/`DELETE ... FOR PORTION OF`	15 Sep	Lost updates under `READ COMMITTED` when concurrent temporal updates and deletes touched the same rows. The documented `SELECT ... FOR UPDATE` workaround did not cover `DELETE`.	Split the ranges in application code or a trigger. The `WITHOUT OVERLAPS` constraints from Postgres 18 still enforce no overlap.
Online data checksums (`pg_enable_data_checksums()`, `pg_disable_data_checksums()`)	16 Sep	The revert message says it "saw a number of postcommit fixes during the beta period" and adds: "Suspicions were raised about the risk of more issues surfacing after GA."	`pg_checksums --enable` on a stopped cluster, as before. Clusters that `initdb` creates on 18 or later have checksums on by default.
`ALTER TABLE ... MERGE PARTITIONS` / `SPLIT PARTITION`	27 Aug	Reverted "due to multiple design issues which are too late to address in this release cycle". The thread lists moved rows decoded as plain inserts (which breaks logical replication), stored generated columns that silently get new values, partition-specific indexes and constraints that are silently lost, and open questions about publication membership.	`DETACH PARTITION`, create the new partition, `INSERT ... SELECT`, `ATTACH PARTITION`
`pg_get_role_ddl()`, `pg_get_tablespace_ddl()`, `pg_get_database_ddl()`	13 Sep	Design review: role output could not recreate password hashes the way `pg_dumpall` does, there was no dependency ordering, and it would be a second, separate implementation of translating catalog state into SQL beside `pg_dump`.	`pg_dumpall --roles-only` or `--tablespaces-only`, and `pg_dump --create` for the `CREATE DATABASE` statement
`pg_dumpall` custom, directory and tar formats, and `pg_restore --no-globals`	18 Jun	A post-commit review recommended reverting and finishing it through a commitfest.	`pg_dumpall --globals-only`, plus `pg_dump -Fc` or `-Fd` per database
`GROUP BY ALL` (with no columns after it)	17 Jul	Wrong results when `ORDER BY` used non-default equality semantics. A clean fix "seems like too much code churn for late beta", and the revert says "We'll revert and try again in v20".	List the grouping columns
Fast default for columns on domains with constraints	8 Jun	Failed with an empty table and an invalid default. The revert says the fix will involve an addition to the table access method API, so it was not ready for 19.	Expect a table rewrite when you add such a column, as in 18
More object types inside `CREATE SCHEMA`	11 Sep	It depended on a change to the order of `CREATE SCHEMA` subcommands, and the compatibility risk of that change was judged larger than the gain. Foreign keys in `CREATE SCHEMA` still run at the end, because the SQL standard requires it.	Create the objects in separate statements after `CREATE SCHEMA`
Lower lock level for `ALTER DOMAIN ... VALIDATE CONSTRAINT`	18 Jun	Unsafe: DML that had already started when a `NOT VALID` constraint was added could still write rows that violate it.	It takes `ShareLock` on tables that use the domain, as before. Run it in a quiet window.
Logical decoding plugins that skip shared catalogs (database-specific snapshots)	24 May	The revert says it "was determined to have fundamental flaws".	Nothing to replace, but it limits REPACK CONCURRENTLY (see below)

Two smaller behavior changes that never made the release notes were reverted too: forcing LC_COLLATE to C in the postmaster (the commit gives the reason as "Out of an abundance of caution") and basing identifier case folding on the default locale, which behaved differently for the builtin provider with locale C and a single-byte encoding.

Two features that stayed were cut back. REPACK CONCURRENTLY lost the snapshot change it depended on, which is the limit covered later in this post. The new fast path for foreign key checks went back to SPI on 15 October for keys whose values need a cast function.

Cause 1: design problems that beta could not fix

Eisentraut starts his own list with the plain possibility that some of the code was not good enough. The threads behind the biggest reverts show something more specific: questions that had no agreed answer before the release.

FOR PORTION OF is the clearest example. On 10 September Andres Freund flagged its concurrency behavior in a code review, and the following day he showed that under READ COMMITTED, concurrent temporal updates and deletes on the same rows can lose all or part of the second operation. The leftover rows one statement inserts are not chained to the original row, and EvalPlanQual does not re-scan for them. With DELETE ... FOR PORTION OF in the mix, even the documented SELECT ... FOR UPDATE workaround did not prevent it. The author proposed raising a serialization failure so the client could retry. Andres replied:

For better or worse, READ COMMITTED is extremely widely used, with one reason for that being not needing to retry, and making FPO not really work that way doesn't seem convincing to me.

When the release management team asked what the committer, Peter Eisentraut, was proposing, he answered: "I was trying to nudge toward reverting without saying it." He reverted it the next day.

SQL/PGQ went the same way. The release management team posted the list of open design issues on 2 September, and Tom Lane had already written on 25 August:

At this point I'd be willing to bet dinner that if we ship it in v19 there will be post-release bug discoveries that are unfixable until v20.

His reason was specific: fixes still under discussion needed parse tree or catalog changes, which bump catversion. Minor releases do not change the catalog, so a bug that needs such a change after release can stay unfixed until 20.

MERGE/SPLIT PARTITION, the database-specific snapshots and the pg_get_*_ddl() functions were reverted for design reasons too. For the DDL functions, Noah Misch's review argued that PostgreSQL should not carry two independent implementations of the logic that translates catalog state into SQL. The fast default for domains needed an addition to the table access method API, and its revert says it was not ready for 19.

Cause 2: a long tail found faster

Eisentraut's post about the reverts is the most direct account from someone on the receiving end. On the long tail:

But with LLM-assisted code review, these kinds of things can get found much faster, and then you’re staring at a list of like forty defect reports. And even if each of them requires only a three-line fix on average, there is a fixed overhead to dealing with each of these, and then you run out of time.

He adds that most of the reverted code was written before LLM-assisted review was this useful, so "the effective quality bar has been raised after the time the feature was written and committed." He also says, as far as he can tell, the problem is not vibe coding: the code predates that.

A mailing-list post from 25 August shows how this fed the decisions. Robert Haas asked Claude to rank the PostgreSQL 19 features by the number and type of bugs already fixed after feature freeze, and posted the top six to ask whether any should be reverted. The counts are the model's tally of fixed bugs, not an audited total, and the ranking found no new bugs itself. Here are its six entries, sorted by fix count, next to what happened to each feature:

Post-freeze fixes per PostgreSQL 19 feature, as tallied by an LLM, and what happened to it

	Value	Series
REPACK / REPACK CONCURRENTLY	28 fixes	Kept in 19
Online data checksums	25 fixes	Reverted
UPDATE/DELETE FOR PORTION OF	17 fixes	Reverted
SQL/PGQ property graphs	17 fixes	Reverted
Foreign key check fast path	16 fixes	Kept in 19
postgres_fdw statistics import	7 fixes	Kept in 19

Counts from the LLM ranking Robert Haas posted to pgsql-hackers on 25 August 2026 ("scary patch contest"). Values marked ~ in the post are shown as the round number. Sorted by count; the post ranked the foreign key fast path first. Outcome from the REL_19_STABLE commit log, 17 October 2026.

The feature with the most fixes, REPACK, is still in 19. The fix count did not decide anything on its own. In the threads, the reverted features had open design questions (FOR PORTION OF, SQL/PGQ) or a failure mode too dangerous to risk a long tail. For online checksums the ranking put it plainly: "A corruption-detection feature producing false positives is exactly the wrong failure mode."

What the commit log says about AI

The commit messages credit AI in a few clear forms: security reports from Anthropic's and OpenAI's research teams, "Reported-by: Claude Code (via Noah Misch)", "Found by Bharath using AI assisted review with Claude", and disclosures that a draft of a patch was written with an LLM. Counting every commit whose message mentions AI assistance, an AI tool or an AI lab:

REL_19_STABLE commits that mention AI assistance, tools or labs, and security commits, per month

	Apr	May	Jun	Jul	Aug	Sep	Oct
AI-credited	0 commits	1 commits	1 commits	2 commits	14 commits	2 commits	4 commits
Security: CVE	0 commits	23 commits	0 commits	0 commits	35 commits	0 commits	0 commits

From the commit messages on REL_19_STABLE, 1 April to 17 October 2026 (1,501 commits). AI-credited = the message names Claude, Codex, an LLM, AI-assisted review, Anthropic, OpenAI or similar. Security = a 'Security: CVE-' trailer. Explicit credit only, so the AI line is a floor.

After beta 1 that is 23 of 937 commits, or 2.5%. It only counts credit in the commit message. Much of the review happens on the mailing list, and a reviewer who used an LLM to find a bug and then reported it in their own words is not counted unless the commit says so. So read it as a floor, not as the share of bugs AI found.

The threads add two details that the headline version leaves out. In the FOR PORTION OF thread, Andres Freund raised the concurrency problem in his own review first. His reproduction of the lost update was then an AI-generated isolation test, but he wrote that he "really needed to force both Opus 5 and Fable 5.1 to get to it, they both swore up and down that this isn't a real issue at first". And in the pg_get_*_ddl() review, Noah Misch ran an Opus 4.8 review as well and wrote that it "found some minor issues that aren't worth addressing before the above". The revert came from his own design objections. In both cases the LLM was a tool in a person's review, and people made the calls.

Cause 3: security work took the reviewers

The third cause is in Eisentraut's post too, and the commit log shows the timing. He calls it the "Vulnpocalypse": from April, the start of feature freeze, to August, the most recent security release, many senior developers were busy fixing security issues. The branch has 58 commits with a Security: CVE trailer since April, covering 40 CVEs, all on two days: 23 on 11 May and 35 on 10 August. Several of the AI-credited commits in August are those security fixes.

The review and testing that normally happens during beta was reduced while that work ran, and it picked up again in August. He offers that as the possible reason so many issues surfaced, and so many reverts happened, so late. Commit counts cannot measure how busy reviewers were, but the two security dates and the late-August cluster of reverts fit it. He thinks that with about two more months, much of the long tail could have been fixed, but the community chose to keep to the schedule instead of delaying for more fixes.

Cause 4: everything works with everything

The last cause is structural. Eisentraut's example is a new SQL feature that has to work with domains, domains over composite types, a domain field with a NOT NULL constraint, a dropped and re-added column, a partition detached and reattached with a different column order, inside a view called from a security-definer function from a trigger. Nobody can test those combinations by hand. He says LLM-assisted fuzzing finds problems in them quickly and probably has to become part of feature development, and that internal interfaces should change so that each new combination does not open so many interactions.

He also points to something the project does not have: a way to commit a feature as experimental and mature it in the main tree. Some of the reverted work suffered from misunderstandings about exactly that.

Why the date is 19 November

The reverts explain what is missing. They do not explain the date. The RC1 announcement does: the minor releases on 12 November "will include a large number of fixes", and "the release team decided to include these fixes in PostgreSQL 19 before its general availability". RC2 ships the same day as those minor releases, and general availability follows a week later.

So the extra weeks are for fixes going into 19, not for features coming back. Some of the reverted features are aimed at 20. The GROUP BY ALL revert says "We'll revert and try again in v20", and Eisentraut writes "I'm confident that many of the reverted features will come back before too long", with a few "80%-ready features" already queued for 20. Nothing promises any particular one.

What this means for your upgrade

Check code written against the betas

If your team wrote migrations, reports or runbooks against a 19 beta, they may use syntax that no longer exists. The companion repo has a check script for eight patterns: SQL/PGQ, FOR PORTION OF, MERGE/SPLIT PARTITION, bare GROUP BY ALL, the online checksum functions, pg_get_*_ddl(), pg_dumpall non-text formats and pg_restore --no-globals. It matches per line and on purpose loosely, so read each hit. GROUP BY ALL region, status has been valid since Postgres 14 (ALL is the default set quantifier) and is not flagged, but a valid GROUP BY ALL whose columns start on the next line is. A clean result means no pattern matched, not that the upgrade is safe. Here it is on the repo's example folders:

check-pulled-syntax.sh

$ scripts/check-pulled-syntax.sh examples/beta-era
FOR PORTION OF examples/beta-era/migrations/20260901_orders.sql:11:UPDATE prices FOR PORTION OF valid_at FROM '2026-10-01' TO '2026-11-01'
MERGE/SPLIT examples/beta-era/migrations/20260901_orders.sql:2:ALTER TABLE orders MERGE PARTITIONS (orders_2026_07, orders_2026_08) INTO orders_2026_q3;
GROUP BY ALL examples/beta-era/migrations/20260901_orders.sql:8:SELECT region, status, count(*) FROM orders GROUP BY ALL
online checksums examples/beta-era/ops/backup.sh:5:psql -c "SELECT pg_enable_data_checksums()"
pg_get_*_ddl() examples/beta-era/ops/backup.sh:4:psql -Atc "SELECT pg_get_role_ddl('app')" > /backups/app-role.sql
pg_dumpall formats examples/beta-era/ops/backup.sh:3:pg_dumpall --format=directory -f /backups/all-$(date +%F)
pg_restore globals examples/beta-era/ops/backup.sh:6:pg_restore --no-globals -d postgres /backups/all-2026-09-01
$ echo "exit $?"
exit 1
$ scripts/check-pulled-syntax.sh examples/postgres-18
no pulled PostgreSQL 19 pattern matched under examples/postgres-18
$ echo "exit $?"
exit 0

It exits 1 on a hit and 2 if the scan itself fails, so you can run it in CI against the repository that holds your migrations:

# Fails the job if a pulled-syntax pattern matches. Pinned to a commit, not main;
# or copy the script into your own repo.
curl -fsSL -o check-pulled-syntax.sh \
  https://raw.githubusercontent.com/The-DevOps-Daily/pg19-revert-ledger/2588e0660651b4b6fd132a85699d12c35d16be62/scripts/check-pulled-syntax.sh
bash check-pulled-syntax.sh db/migrations

Plan REPACK CONCURRENTLY one table at a time

REPACK (CONCURRENTLY) is one of the headline features of 19, and it stayed. But the change it relied on to build logical decoding snapshots per database was the one reverted in May. The revert message says what that means: it "restricts REPACK (CONCURRENTLY) so that only one process can run it concurrently on different tables and even on different databases". The original REPACK commit lists the same limit as a loose end, and Robert Haas's August post describes 19's REPACK CONCURRENTLY as "restricted to one process cluster-wide".

The REPACK reference page lists max_repack_replication_slots (default 5) as a limit, but not this one, and raising the slot setting does not lift it. If your plan was to run REPACK CONCURRENTLY across several bloated tables in parallel during one maintenance window, plan a queue instead. In the source, the decoding worker waits for every transaction that already has a transaction ID before it starts, so a second run most likely waits for the first. We did not run two at once, so treat that as a reading of the code, not a measurement.

Plan the upgrade window

Test on RC1 now. Feature-wise it is close to what ships. RC2 on 12 November adds the minor-release fixes.
Rehearse the upgrade on a copy of your data. The pulled features are easy to find with the script; the changes that did ship are where an upgrade surprises you. If you run on Neon, a branch gives you a copy-on-write copy of production in seconds, so you can clean up beta-era migrations and test the queries the script flags against real data without touching the primary. A major version upgrade on Neon is a move to a new project on the new version, and Neon's major version upgrade guide covers the Import Data Assistant (for databases under 10 GB), dump and restore, and logical replication. Check which major versions Neon offers before you pick a date.
General availability is planned for 19 November. The next scheduled minor release after that is 11 February 2027, per the project's roadmap. If your policy is to wait for the first minor release, that is when 19.1 is due, unless an out-of-cycle release comes first.
Do not plan on a pulled feature returning in a 19.x minor release. Minor releases carry fixes, not new features, so the earliest any of them can return is 20.

Summary

13 release-note items, covering 11 features, left the PostgreSQL 19 release notes after beta 1. Ten of the features were reverted after beta 1, one in May. The big ones are SQL/PGQ, FOR PORTION OF, online data checksums and MERGE/SPLIT PARTITION.
The reasons differ by feature. In the threads, design questions with no agreed answer drove the big reverts. By Eisentraut's account, a long tail of edge-case bugs found faster with LLM-assisted review filled the queue, four months of security work took reviewers away during beta, and the need for every feature to work with every other one makes some of them very hard to finish.
The commit log credits AI in 23 of 937 commits after beta 1. That is a floor, and in the threads people used the tools and people made the decisions.
The 19 November date is about folding the 12 November minor-release fixes into 19, not about features coming back.
Before you upgrade, check beta-era code with the script, and plan REPACK CONCURRENTLY as one run at a time across the cluster.

Sources: Peter Eisentraut, The reverts will continue until morale improves; the PostgreSQL 19 RC 1 announcement; Robert Haas, scary patch contest and Tom Lane's reply; the SQL/PGQ design issues; the FOR PORTION OF thread; pg_get_*_ddl() needs a redesign; MERGE/SPLIT PARTITIONS issues; and The Register's report.

Originally published at devops-daily.com.

How Docker Works Under the Hood: Watch Namespaces Isolate a Container and cgroups Stop a Fork Bomb

DevOps Daily — Thu, 15 Oct 2026 09:02:04 +0000

On a Linux host, start an nginx container, then run ps aux. nginx is right there in the list, with an ordinary host PID. No guest kernel booted and no hypervisor started. That one observation is the key to how Docker works under the hood: a container is a normal Linux process that the kernel agrees to lie to (namespaces) and to ration (cgroups). The rest of docker run is plumbing that sets those two things up.

This post walks the plumbing with two browser simulators, then shows the same kernel features on a real machine, no Docker required. Disclosure: I help build the simulators; they are free, run in the browser, and need no signup.

The path of one docker run

The How Docker Works Under the Hood simulator takes one command, docker run -p 8080:80 nginx, and steps down the stack in eight stages: the Docker CLI, the Docker daemon, the registry pull, containerd, the OCI bundle, runc, the running container, and the Linux kernel. Press Play, or use Prev and Next to go at your own pace. Tick "nginx already pulled" and the registry stage drops out (the counter goes from Step 1 / 8 to Step 1 / 7), the first lesson: a pull only happens on a cache miss, and only missing layers come down.

The stages worth pausing on:

The CLI is a REST client. It sends an HTTP request to dockerd over /var/run/docker.sock. The stage shows a curl --unix-socket request to the daemon's /info endpoint: it starts nothing, but it uses the same transport the CLI does.
dockerd does not start your process. It hands off to containerd, which unpacks the image layers into a snapshot and tracks the container's lifecycle.
The OCI bundle is the whole recipe. It is a config.json plus a root filesystem. The excerpt in the simulator lists the five namespaces to create, with the type names from the OCI runtime spec: pid, network, mount, uts, ipc. The kernel shortens two of them: /proc/<pid>/ns and lsns say net and mnt, as you will see below. The command for this stage is runc spec, which writes a sample config.json so you can read the real shape.
runc does the kernel work and leaves. It creates the namespaces and the cgroup, pivots into the rootfs, drops capabilities, and execs nginx as PID 1 inside the container. Then runc exits and a shim keeps the container attached to containerd.

Every stage has a copy button next to its real command. The last stage ends on four labels: namespaces, cgroups, networking, mounts. Those are the container.

Namespaces: what the process can see

You can look at your own namespaces right now on any Linux box. Every process has a set, and each one is a link in /proc:

$ ls -l /proc/self/ns    # trimmed: link targets only, 6 of 10 entries
cgroup -> cgroup:[4026531835]
ipc -> ipc:[4026531839]
mnt -> mnt:[4026531841]
net -> net:[4026531840]
pid -> pid:[4026531836]
uts -> uts:[4026531838]

Two processes share a namespace when the numbers match; a container's process shows different ones. The unshare tool from util-linux makes new ones by hand. On Debian 12, as a normal user:

$ unshare --user --map-root-user --pid --fork --mount-proc ps -o pid,user,cmd
    PID USER     CMD
      1 root     ps -o pid,user,cmd

ps believes it is PID 1 and that it runs as root. Both are namespace tricks: a new PID namespace starts counting at 1, and the user namespace maps your normal UID to root inside it only. The UTS namespace does the same for the hostname:

$ unshare --user --map-root-user --uts sh -c 'hostname demo-box; hostname'
demo-box
$ hostname
raspberrypi

If your distro restricts unprivileged user namespaces, run them with sudo and drop --user --map-root-user. The namespaces(7) man page lists all eight types.

cgroups: what the process can use

Namespaces hide things. They do not limit anything. A process in its own PID namespace can still create processes until the host runs out. The classic proof is the fork bomb: a tiny shell function named : that calls itself twice, one copy piped into another in the background. Every copy makes two more, the process table fills, and the machine stops responding.

Do not run it. Use the Fork Bomb Simulator instead, which draws the process tree without forking anything real. Hover the code explainer to see what each of its ten pieces does, then press Start. The process count climbs until it hits the PID limit (a slider from 64 to 1024, 512 by default), the log prints FATAL: fork() failed - Resource temporarily unavailable, and the screen reads SYSTEM CRASHED.

Now pick one of the three Protection cards and start again: ulimit -u (a per-user process cap), cgroups (pids.max), or systemd (TasksMax in a unit file). In the simulation, each one caps the run at 100 processes, and it ends with SYSTEM PROTECTED instead. That only works while the PID limit slider is above 100: set it to 64 and the system runs out first, protection or not. Same bomb, same growth, stopped at a line you drew in advance. On a real system they count different things. ulimit -u counts every process and thread of your user ID across the whole machine, and the kernel does not enforce it for root. When I ran ulimit -u 50 in a shell on the Pi, the very first fork() failed, because my user already owned 191 tasks elsewhere. pids.max and TasksMax count tasks inside one cgroup, whoever owns them. (The cgroups card shows the cgroup v2 form: echo 100 > /sys/fs/cgroup/user.slice/user-1000.slice/pids.max.)

Try the limit for real

On a systemd machine with cgroup v2, where the user manager has the pids controller delegated to it, you can put a command in a throwaway cgroup with a task limit, no root needed. To check, cat /sys/fs/cgroup/user.slice/user-$(id -u).slice/user@$(id -u).service/cgroup.controllers must list pids (on the Pi it prints cpu pids). This is not a fork bomb. It is a loop that stops after 50 children, run inside a scope capped at 10 tasks:

$ systemd-run --user --scope -p TasksMax=10 python3 -c '
import os, time
n = 0
try:
    for _ in range(50):
        if os.fork() == 0:
            time.sleep(2)
            os._exit(0)
        n += 1
except OSError as e:
    print("started", n, "children, then:", e)
'
Running scope as unit: run-r01155c5e085f497f94c998714d8323f5.scope
started 9 children, then: [Errno 11] Resource temporarily unavailable

Nine children plus the Python parent is ten tasks, the value systemd wrote to the scope's pids.max. The tenth fork() failed with EAGAIN, which is what the kernel's cgroup v2 docs promise. In the simulator, the same refusal shows as cgroup: fork rejected, pids.max (100) reached on the cgroups card. Nothing else on the machine noticed. A version of the script that also reads its own cgroup files printed pids.events: max 1 at the end: that file counts refused forks, so it is the one to alert on.

Docker exposes the same controller per container: docker run --pids-limit 100 nginx sets that ceiling on the container's cgroup (docs). Without that flag, a runaway container is bounded only by the limits the host already has. On the Pi, systemd's DefaultTasksMax is 3914 (15% of kernel.threads-max), but whether a container's unit inherits a default like that depends on the runtime. Check instead of guessing: on a cgroup v2 host, docker run --rm alpine cat /sys/fs/cgroup/pids.max prints the container's limit, and max means none.

The whole trick

Side by side, the two simulators put how Docker works under the hood in one sentence: runc asks the kernel for new namespaces so the process sees its own small world, and puts it in a cgroup where the limits you set cap what it can use. A cgroup with no limits caps nothing, and a PID cap does not cap CPU or memory. A limit also needs its controller: the Pi boots with cgroup_disable=memory, and a scope with MemoryMax=50M let a process allocate 100 MiB without an error. dockerd, containerd and the registry are how the recipe gets built and delivered.

For the longer tour, with commands to run at each layer of a real Docker host, read How Docker Really Works, From docker run to the Kernel. For the controllers themselves, see cgroups(7).

What surprised you most the first time you looked under the hood of a container? Tell us in the comments.

Let's Encrypt Goes to 64-Day Certificates: When Does Your ACME Client Actually Renew?

DevOps Daily — Thu, 15 Oct 2026 09:00:00 +0000

Let's Encrypt will issue 64-day certificates by default from 10 February 2027 , down from 90, on the way to 45 days in 2028. Its staging environment switched on 14 October 2026, and we had a watcher waiting for it: the first 64-day certificate arrived at 21:35 UTC. We issued 64-day staging certificates with five ACME client installs and recorded when each one renews, and ran the same test at 45 and 90 days for comparison. The clients that use ARI renewed or scheduled a 64-day certificate for day 42 or 43 , at the same two-thirds share of the lifetime as at 45 and 90 days. The one that breaks is old acme.sh: version 3.0.7 scheduled a 64-day certificate for day 59 with its defaults, five days before it expires, and for day 79 with --days 80, fifteen days after it expired. Upgrading to 3.1.6 moved that schedule to day 42 at the next check.

TL;DR

On 64-day staging certificates, Let's Encrypt's ARI endpoint suggested renewing between day 42.04 and 43.32 , 65.7 to 67.7 percent of the lifetime. We measured the same share at 45 and 90 days. That is about 21 days before expiry.
certbot 5.8 renewed inside that window (it picks a random time in it on every run), acme.sh 3.1.6 scheduled day 42.16, and lego 5.5.2 allowed renewal from day 42.67. These configurations are ready for shorter lifetimes.
certbot 2.9, the version apt installed on our Ubuntu 24.04 machine, does not use ARI and became due 30 days before expiry: day 34 of 64. Early, but safe.
acme.sh 3.0.7 schedules the next renewal at issue + --days - 1 day, with no check against the expiry date. Its default of 60 gave day 59.04 of 64, and --days 80 gave day 79.04. Every acme.sh tag we scanned from 3.0.7 to 3.1.2 has the same default of 60.
Search your cron jobs, wrapper scripts and monitoring for hardcoded 60, 80 and 83, as Let's Encrypt itself suggests, and still check your renewal timer, challenge path and deploy hook.

Prerequisites

A server that renews certificates with certbot, acme.sh or lego
Shell access to check the client version and its stored renewal settings
To reproduce the test: a throwaway VM with port 80 reachable, and faketime

Everything we ran, and every output file, is in the repository:

The-DevOps-Daily/le-64-day-renewal-check on GitHub

What changes, and when

From Let's Encrypt's announcement of 7 October 2026:

14 October 2026 : staging switches to 64-day certificates. Our watcher's request at 21:25 UTC still returned a 90-day certificate, and the one at 21:35 UTC returned a 64-day certificate.
10 February 2027 : all subscribers move to 64-day certificates by default. The last 90-day certificates expire around 11 May 2027.
The authorization reuse period drops from 30 days to 10 days, and to seven hours in 2028.
2028 : 45-day certificates by default.

The advice in the announcement is short: if your renewals are automated and your client supports ARI, you should be all set; if not, renew at about two thirds of the lifetime, and look for hardcoded values such as 83, 80 or 60 in cron jobs, wrapper scripts and runbooks. We wanted to see which clients actually follow that, and what happens to the ones that do not.

How we measured it

On a throwaway Ubuntu 24.04 droplet we installed five client versions side by side: certbot 2.9.0 from apt, certbot 5.8.0 from snap, acme.sh 3.0.7 and 3.1.6 from the upstream repository, and lego 5.5.2. They requested real certificates from the Let's Encrypt staging environment, using hostnames under sslip.io that resolved to the droplet, at three lifetimes. Where a client could not request a lifetime, we used copied state or its source:

64-day certificates from the default classic profile, after staging switched. A watcher asked for a classic certificate every 10 minutes from 12:36 UTC and started the suite as soon as the lifetime dropped.
90-day certificates from the same classic profile, earlier that evening.
45-day certificates from the tlsserver profile, the lifetime planned for 2028. acme.sh 3.0.7 and certbot 2.9 cannot request a profile, so certbot 2.9 was tested on a copy of certbot 5.8's 45-day certificate and acme.sh 3.0.7 was read from its source.

Then, for each certificate:

acme.sh stores its planned renewal time in its config, so we read it.
lego prints, at issuance, when a renewal will be allowed; we ran it with ARI on and with --ari-disable.
certbot only decides when certbot renew runs, so we ran it under faketime at chosen hours after issuance, each time on a fresh copy of the state, and recorded whether it renewed.
A small script asked the ARI endpoint (RFC 9773) for the suggested window of every certificate.

Days are counted from each certificate's notBefore time, which Let's Encrypt sets about an hour before issuance. This is the certbot 2.9 probe on the 64-day certificate, as recorded:

certbot 2.9.0 under faketime, 64-day staging certificate

$ probe-certbot.sh certbot $R/cb29 c29n.46-101-102-127.sslip.io 800 812 815 818 830
# certbot 2.9.0 c29n.46-101-102-127.sslip.io notBefore=Oct 14 20:37:00 2026 GMT notAfter=Dec 17 20:36:59 2026 GMT
hour 800 (day 33.33, 2026-11-17 04:37:00 UTC): not due
hour 812 (day 33.83, 2026-11-17 16:37:00 UTC): not due
hour 815 (day 33.96, 2026-11-17 19:37:00 UTC): not due
hour 818 (day 34.08, 2026-11-17 22:37:00 UTC): RENEWED
hour 830 (day 34.58, 2026-11-18 10:37:00 UTC): RENEWED

Not due at day 33.96, renewed at day 34.08: the threshold is between the two, consistent with 30 days before the 64-day expiry.

Results

Client	64-day	45-day	90-day
ARI suggested window	day 42.04 to 43.32	day 29.56 to 30.46	day 59.11 to 60.91
certbot 5.8.0 (snap), ARI	first renewal seen at day 42.50	first renewal seen at day 30.29	first renewal seen at day 59.92
acme.sh 3.1.6, ARI	scheduled day 42.16	scheduled day 29.6 to 30.3	scheduled day 60.7 to 60.9
lego 5.5.2	allowed from day 42.67	allowed from day 30.0	allowed from day 60.0
certbot 2.9.0 (apt), no ARI	due between day 33.96 and 34.08	due between day 14.96 and 15.08	due between day 59.79 and 60.21
acme.sh 3.0.7, default `--days 60`	scheduled day 59.04, 5 days before expiry	day 59, after expiry (source)	scheduled day 59.0
acme.sh 3.0.7, `--days 80`	scheduled day 79.04, 15 days after expiry	day 79, after expiry (source)	scheduled day 79.0

Day each client renews, or plans to renew, a 64-day certificate

	Value	Series
certbot 2.9 (apt)	34 days	safe
certbot 5.8 (ARI)	42.5 days	safe
acme.sh 3.1.6 (ARI)	42.16 days	safe
lego 5.5.2	42.67 days	safe
acme.sh 3.0.7, defaults	59.04 days	tight
acme.sh 3.0.7, --days 80	79.04 days	expired
same, after upgrading to 3.1.6	42.23 days	safe

The certificate expires on day 64. Measured on Let's Encrypt staging on 2026-10-14, one certificate per row. certbot 2.9 became due between day 33.96 and 34.08. certbot 5.8 is its first renewal in our probes; it picks a random time inside the ARI window (day 42.04 to 43.32) on every run. acme.sh shows its stored schedule and lego the day it reported at issuance. The last bar is the --days 80 certificate after upgrading acme.sh to 3.1.6. Source: results/measured-64.txt in the repo.

ARI clients: the window does the work

The ARI window opened at 65.7 percent of the lifetime and closed at 67.7 percent at all three lengths: day 42.04 to 43.32 of 64, day 29.56 to 30.46 of 45, and day 59.11 to 60.91 of 90. On a 64-day certificate, that is 21 to 22 days before expiry. A client that follows the window needs no change for 45 days in 2028 either.

While we waited, we projected the 64-day column from the 45-day and 90-day runs and committed the projection at 21:32 UTC (commit f909cfc in the repository). Our next request, at 21:35 UTC, returned the first 64-day certificate. Every 64-day result in the table matches that projection (results/projection-64.txt and results/measured-64.txt).

certbot 5.8 does not renew at one fixed moment. Each time it runs, it asks for the window, picks a random time inside it, and renews if that time has passed. From its source (acme/client.py):

delta_seconds = (end - start).total_seconds()
random_seconds = random.uniform(0, delta_seconds)
random_time = start + datetime.timedelta(seconds=random_seconds)

So a run a third of the way into the window renews about a third of the time. These are our 64-day probes, each from a fresh copy of the same state:

hour 1010 (day 42.08, 2026-11-25 22:37:12 UTC): not due
hour 1014 (day 42.25, 2026-11-26 02:37:12 UTC): not due
hour 1020 (day 42.50, 2026-11-26 08:37:12 UTC): RENEWED
hour 1030 (day 42.92, 2026-11-26 18:37:12 UTC): not due

The run at day 42.50 had about a 36 percent chance and renewed. The run at day 42.92 had about 68 percent and did not. This spreads renewals out, as RFC 9773 intends, so a fleet does not renew all at once. Once the window has closed, every run renews: on the 45-day and 90-day certificates, every probe after the window renewed. If the ARI request fails, certbot 5.8 catches the error and gets no ARI time; it then renews at two thirds of the lifetime (day 42.67 of 64), or by renew_before_expiry if that is set in the renewal config. We read this in its source (results/certbot-5.8-ari-source.txt) but did not test a failed ARI request.

acme.sh 3.1.6 scheduled its next renewal for day 42.16, with its default and with --days 80. It also stayed inside the window with --days 60 at 45 days and with --days 80 at 90 days. lego 5.5.2, checked at issuance, gave the same answer with ARI on and off: renewal allowed from day 42.67, exactly two thirds of the lifetime.

These are the clients' decisions, not your whole setup. A renewal still needs a timer or cron job that runs often enough, a working challenge path and a deploy hook that reloads the server.

certbot 2.9: early, and fine

certbot added ARI support in version 4.1.0 (June 2025). The certbot that apt installed on our Ubuntu 24.04 machine was 2.9.0, and its probes became due 30 days before expiry at every lifetime: between day 33.96 and 34.08 of 64, between day 14.96 and 15.08 of 45, and between day 59.79 and 60.21 of 90. It renews more often than it needs to, and it leaves about 30 days to notice and fix a failed renewal, if its timer runs normally.

acme.sh 3.0.7: the one that breaks

acme.sh 3.0.7 computes the next renewal when it issues a certificate, and never compares it with the expiry date. From its source:

Le_NextRenewTime=$(_math "$Le_CertCreateTime" + "$Le_RenewalDays" \* 24 \* 60 \* 60)
Le_NextRenewTime=$(_math "$Le_NextRenewTime" - 86400)

Le_RenewalDays is the --days value, 60 by default. The 64-day staging run shows the result:

acme307 a307n notBefore=Oct 14 20:37:26 2026 GMT notAfter=Dec 17 20:37:25 2026 GMT next='2026-12-12T21:36:01Z' days=
acme307 a307n80 notBefore=Oct 14 20:37:43 2026 GMT notAfter=Dec 17 20:37:42 2026 GMT next='2027-01-01T21:36:18Z' days='80'

Both certificates expire on 17 December. With the default, the next renewal is 12 December, day 59.04, so one failed renewal (a DNS hiccup, a firewall change, a rate limit) leaves very little time before expiry. With --days 80, the next renewal is 1 January 2027, day 79.04, fifteen days after the certificate expired, and acme.sh 3.0.7 does not try before then. The 90-day run gave the same days, 59 and 79, which were safe at that lifetime. At 45 days in 2028, the default fails too.

We scanned each acme.sh tag from 3.0.7 to 3.1.6 for the scheduling code (the scan is in the repo):

acme.sh tag	Tag date	Default `--days`	Cap one day before expiry	ARI code
3.0.7 to 3.1.2	Oct 2023 to Nov 2025	60	no	none
3.1.3	Apr 2026	30	no	first ARI function
3.1.4	Jul 2026	30	yes	ARI used when scheduling
3.1.5, 3.1.6	Sep 2026	30	yes	ARI also checked on every renewal run

We only ran 3.0.7 and 3.1.6. The rows for the versions in between come from reading the source, so check your own version's behaviour if it is one of those.

What upgrading does to an existing certificate

acme.sh stores Le_RenewalDays and the next renewal time in each certificate's config, so an upgrade does not by itself change a schedule written by an old version. We tested what happens next: we copied the state of the 64-day certificate that 3.0.7 had scheduled for day 79 and ran acme.sh 3.1.6 --renew on it, without --force:

== before: acme.sh 3.0.7 state
Le_RenewalDays='80'
Le_NextRenewTimeStr='2027-01-01T21:36:18Z'
notAfter=Dec 17 20:37:42 2026 GMT
== acme.sh 3.1.6 --renew on the same state (no --force), 2026-10-14T23:22:34Z
[Wed Oct 14 23:22:34 UTC 2026] Renew: a307n80.46-101-102-127.sslip.io
[Wed Oct 14 23:22:34 UTC 2026] Renewing: 'a307n80.46-101-102-127.sslip.io'
[Wed Oct 14 23:22:34 UTC 2026] Renewing using Le_API=https://acme-staging-v02.api.letsencrypt.org/directory
[Wed Oct 14 23:22:35 UTC 2026] Current renewal time: 2027-01-01T21:36:18Z
[Wed Oct 14 23:22:35 UTC 2026] ARI suggestedWindow: 2026-11-25T21:36:11Z to 2026-11-27T04:18:13Z
[Wed Oct 14 23:22:35 UTC 2026] Updating renewal time picked from ARI window: 2026-11-26T02:10:18Z
[Wed Oct 14 23:22:35 UTC 2026] Skipping. Next renewal time is: 2026-11-26T02:10:18Z
== after
Le_RenewalDays='80'
Le_NextRenewTimeStr='2026-11-26T02:10:18Z'

The stored --days 80 stays, but 3.1.6 asked the ARI endpoint on its first check and moved the renewal from 1 January, after expiry, to 26 November, day 42.23. The same test on the 90-day certificate gave the same result. Its source says it does this on every renewal check unless NO_ARI=1 is set. On a CA without ARI, or with NO_ARI=1, the stored schedule is what counts.

What to check on your servers

Find out which client and version renews each certificate, and what it has stored:

# certbot: version, and any renewal setting that overrides the default
certbot --version
grep -H renew_before_expiry /etc/letsencrypt/renewal/*.conf

# acme.sh: version, the stored --days, and the stored next renewal time
~/.acme.sh/acme.sh --version
grep -H -E '^Le_RenewalDays=|^Le_NextRenewTimeStr=' ~/.acme.sh/*/*.conf

# lego: version (ARI is on unless you pass --ari-disable)
lego --version

Then:

Upgrade acme.sh to 3.1.6 or later wherever it runs (acme.sh --upgrade), and run acme.sh --cron once to see the new schedule. On anything that has to stay on an older version, compare each stored Le_NextRenewTimeStr with the certificate's expiry, and lower --days so the next renewal comes well before it.
Look for 60, 80 and 83 in cron jobs, systemd timers, wrapper scripts and runbooks, as Let's Encrypt suggests. A script that renews "when the certificate is 60 days old" leaves four days of margin at 64 days and none at 45.
Check your expiry alerts. The ARI clients chose renewal times with about 21 days left on our 64-day certificates, so an alert at 30 days left fires on every certificate while renewal is working fine, and people learn to ignore it. Alert on failed renewals, and on expiry with a threshold below the normal renewal point, for example 14 days left.
Check what else issues certificates. The shorter authorization reuse period (10 days instead of 30) does not change a renewal every 42 days, since both periods have passed by then. It matters when you issue again 10 to 30 days after the last validation, for example a manual reissue or a renewal forced early: that order now needs a fresh challenge.
Rehearse on staging before February. Staging's classic profile already issues 64-day certificates, and its tlsserver profile issues 45-day ones. Issue a staging certificate with separate state from production and run your normal renewal and deploy path against it.

What this test does not show

One certificate per configuration. We issued one 64-day certificate for each client setup, on one evening. certbot 5.8 picks a random time inside the window, so our probes show that it renews inside the window, not how its renewals spread across a fleet.
ARI windows can move. Let's Encrypt can change a certificate's window, for example to ask for early renewal before a mass revocation. Clients that check ARI on each run, as certbot 5.8 and acme.sh 3.1.6 do, pick up the change; we did not test that. We measured the normal window. The acme.sh 3.0.7 and certbot 2.9 rows would not change, because neither looks at ARI.
acme.sh 3.0.7 at 45 days comes from its source. It cannot request the 45-day profile, so we could not issue one with it. The 64-day and 90-day runs confirm the formula.
lego was checked at issuance only. We recorded when it said a renewal would be allowed, not a renewal at that time.
Five client installs. We did not test Caddy, Traefik, cert-manager, win-acme or hosting panels that embed their own ACME code. Check those against the same two questions: does it use ARI, and if not, how does it decide?
Staging, not production. Production should behave the same way from February, but we only issued staging certificates.

The short version: in our tests, the clients with ARI enabled renewed on Let's Encrypt's window, and that window was the same share of the lifetime at 45, 64 and 90 days. If it does not, find out what number it renews on, because that number was chosen for 90 days.

Originally published at devops-daily.com.

Kubernetes RBAC Explained by Breaking It: Practice Roles and Bindings in Your Browser

DevOps Daily — Wed, 14 Oct 2026 10:42:14 +0000

Your CI service account needs to read pods. Someone writes a ClusterRole called pod-reader, binds it with a RoleBinding in dev, and it works. A month later the same job runs against prod and gets Forbidden. "But it's a ClusterRole," says everyone in the channel. The fix someone reaches for is a ClusterRoleBinding, and now the CI account can read pods in kube-system too.

Both surprises come from the same rule, and it is the one to learn first if you want Kubernetes RBAC explained in a way that sticks: the role says what, the binding says where. A ClusterRole is only cluster-wide if a ClusterRoleBinding makes it so.

This post walks through RBAC by breaking it on purpose in the Kubernetes RBAC Simulator. Disclosure: I help build it; it is free, runs in the browser, no signup, and needs no cluster.

Four objects, two questions

RBAC has four object kinds:

Role: a list of rules, living in one namespace.
ClusterRole: a list of rules with no namespace. Reusable anywhere, and the only way to describe cluster-scoped resources such as nodes.
RoleBinding: grants a Role or a ClusterRole to subjects (users, groups, service accounts) inside the binding's own namespace.
ClusterRoleBinding: grants a ClusterRole in every namespace and on cluster-scoped resources.

A rule is apiGroups + resources + verbs, optionally narrowed by resourceNames. And the line from the official docs that answers half of all RBAC questions: "Permissions are purely additive (there are no "deny" rules)."

So every "can X do Y?" question comes down to: which bindings name X, and does any of them grant Y in this place?

How the simulator works

The simulator has seven fixed scenarios. Each one shows its Roles and bindings as YAML in one pane. In the other you build a question: pick a subject, one of eight verbs (get to deletecollection), one of seven resources (pods to namespaces), a namespace (dev, staging, prod, kube-system, or cluster scope, which for a namespaced resource like pods means every namespace) and optionally a named object. The answer is Allowed or Denied, with a sentence naming the binding and role that granted it. Every binding that names the subject but did not grant the request is listed with the reason. Under that you get the full list of verbs the subject has on that resource, and a kubectl auth can-i command for the same question.

Break it in seven moves

1. The ordinary case. A Role and a RoleBinding, both in dev. list pods in dev is allowed, "granted by RoleBinding/ci-reads-pods through Role/pod-reader, which applies to namespace dev." This is the model people then over-apply.

2. The one that surprises everyone. Same service account, but the role is now a ClusterRole bound by a RoleBinding in dev. The scenario asks about prod, and the answer is Denied: the binding "binds ClusterRole/pod-reader, but a RoleBinding confines it to dev. The permissions are cluster-wide in definition and namespaced in effect." Switch the namespace to dev and it is allowed. The official sample manifest says it in a comment: "The namespace of the RoleBinding determines where the permissions are granted."

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: ci-reads-pods-in-dev
  namespace: dev            # this line decides where
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: pod-reader
subjects:
  - kind: ServiceAccount
    name: ci
    namespace: dev

3. Nodes are not in a namespace. A ClusterRole that allows get and list on nodes, bound with a RoleBinding. Denied. A namespaced binding can never grant a cluster-scoped resource, however broad the role is.

4. One secret, not the list. A Role with resourceNames: ["db-password"] and only the verb get. get secrets/db-password is allowed. Switch the verb to list and it is denied, because the Role grants only get. Clear the named-object field too and you have the question kubectl get secrets asks: still denied, because a rule with resourceNames does not cover a plain list. The docs add one exception: if a rule grants list or watch with resourceNames, a request with a matching metadata.name field selector is authorized.

5. The silent typo. The binding's roleRef says pod-readr. Denied, with the reason: the role does not exist, and "a dangling roleRef grants nothing and raises no error." Kubernetes skips the lookup for callers that hold the bind verb or are in system:masters, so a cluster admin can apply this binding without an error and no event fires. Anyone else gets rolebindings.rbac.authorization.k8s.io "pod-readr" not found when they create it. The silence ends at request time: on a real API server, the service account's Forbidden error ends with RBAC: role.rbac.authorization.k8s.io "pod-readr" not found, and kubectl auth can-i prints the same reason after its no. Read the whole error. And roleRef is immutable after creation, so the fix is delete and recreate.

6. There is no deny. Two bindings, one to pod-reader and one to an empty Role called read-only-audit. The empty Role takes nothing away. If you are looking for the rule that revoked access, there is not one; look for the grant that is missing.

7. Genuinely cluster-wide. The same ClusterRole, now bound with a ClusterRoleBinding. list pods in kube-system is allowed. This is the one to be careful with, and the one from the opening story.

The permission you did not mean to grant

The simulator answers "who can do what, and why". It does not model escalation paths, and those are where a harmless-looking grant turns into much more. The Kubernetes RBAC good practices page lists them. Three to check in every review:

list or watch on Secrets. The docs say it plainly: these "effectively allow for users to reveal the Secret contents". A list response includes the data. A role that "only lists" secrets can read them all.
Creating Pods, or anything that creates Pods. A Pod can mount Secrets, ConfigMaps and volumes from its namespace and run as any service account there. In the docs' words, permission to create workloads "implicitly grants access to many other resources in that namespace", and that includes create on Deployments or Jobs.
The escalate, bind and impersonate verbs. The first two bypass the checks that stop you granting more than you hold. impersonate lets you act as someone else. None of them appear in the simulator's verb list, so check them by reading your roles, including any wildcard * in verbs or resources.

The built-in edit ClusterRole already holds two of these risks: read on Secrets, and impersonate on serviceaccounts. The docs say it "can be used to gain the API access levels of any ServiceAccount in the namespace." Bind it with a ClusterRoleBinding and that is every namespace.

Ask a real cluster

Every answer in the simulator comes with a can-i command to try. These two are the ones to remember, adapted from the examples in kubectl auth can-i --help:

# one question, asked as the service account
kubectl auth can-i list pods -n prod --as=system:serviceaccount:dev:ci

# everything that account can do in a namespace
kubectl auth can-i --list -n dev --as=system:serviceaccount:dev:ci

--as is impersonation, so you need permission to impersonate, which usually means a cluster admin. Also note the -n: without it, can-i answers for your current namespace, which may be a different question from the one you meant. For a single check across every namespace, use --all-namespaces; --list refuses that flag.

One trap for cluster-scoped resources: -n still goes into the question. I rebuilt move 3 on a real kube-apiserver v1.37.1. kubectl auth can-i list nodes -n dev --as=system:serviceaccount:dev:ci printed Warning: resource 'nodes' is not namespace scoped, then yes. As the same account, kubectl get nodes got Forbidden "at the cluster scope". With --all-namespaces instead of -n dev, can-i says no. For nodes and other cluster-scoped resources, ask with --all-namespaces. On that server, the real requests in all seven moves got the same verdicts as the simulator gives.

Where to go next

Run the seven scenarios in order, and in each one change one field at a time: namespace, verb, then resource. Watching a verdict flip is faster than reading about it. The simulator is one of 50+ free DevOps games and simulators. The primary sources are the Kubernetes RBAC authorization reference and the good practices page linked above.

What is the most surprising permission you have found granted in a cluster? Tell us in the comments.

DEV Community: DevOps Daily

Log Aggregation Pipeline Explained: Follow One Log Line and Run the Incident in Your Browser

Six stages, one line

Healthy first

Break it three ways

Now run the incident

Try the parser mismatch for real

What to alert on

A Service Catalog Nobody Updates Can Be Worse Than None: We Checked 217 Entries Against GitHub

TL;DR

Prerequisites

Why a catalog drifts

What the checker compares

Checking an owner from outside the org

Links come from strangers

The sample

The run

What it found

Ownership is where the records disagree

Repositories that moved

Idle, but still "production"

Links

Does a catalog decay with age?

The bugs two reviews found in our checker

False positives, and what we could not check

How to keep a catalog honest

Derive owners, do not type them

Check the catalog file in CI

Use the check GitHub already runs

Give the scorecard consequences

Retire what nobody claims

Build or buy

What we could not conclude

Summary

PHP2Shell: How to Check That Every PHP You Run Is Actually Patched

What to do now

Prerequisites

What was fixed

Who is really exposed

Why php -v does not tell you

Restart, or the old code keeps running

Containers: rebuild, redeploy, then check what is running

How fast the popular images were rebuilt

The interim settings and their limits

PHP 8.1 and older

Run the check

How to prove you are fixed

What we could not check

Checklist

Node.js October 2026 Security Release: Finding Every Node You Actually Run

What to do now

Prerequisites

What was fixed

Who is exposed

Brotli: any Node process that decodes Brotli someone else chose

TLS hostname checks and DNS

End-of-life lines

Why changing the default moves nothing

What pm2 really does on an upgrade

Upgrade steps: nvm, pm2 and systemd

Cron jobs and helper scripts

Containers

Verify with /proc, not with names

Run the audit script

Rollback

What we could not check

Checklist

What Keeps a Neon Compute Awake, and What It Costs

TL;DR

Prerequisites

How Neon decides a compute is idle

How we measured

Idle connections do not keep it awake, and they do not survive the suspend

LISTEN does not keep it awake either

An open transaction keeps it awake for as long as it stays open

A monitoring agent keeps it awake

Health checks: the interval decides everything

What a wake costs your users

What it costs

Does Neon wake it on its own?

Why `php -v` does not tell you