DEV Community: DevOps Daily

Node.js November 2026 Security Release: How We Tracked and Patched It

DevOps Daily — Wed, 11 Nov 2026 17:50:00 +0000

On 4 November 2026 the Node.js project posted a pre-alert: new releases of the 26.x, 24.x and 22.x lines "on or shortly after, Wednesday, November 11, 2026", and "The highest severity issue fixed in this release is HIGH". No CVE numbers, no components, nothing to triage against. You know a fix is coming, you know it matters, and you do not know when it lands or what it touches.

That gap is where most patch delays come from. The release ships in the afternoon, nobody is watching, and the upgrade happens days later. This time we wrote a small watcher for the release, saw Node.js 24.22.3 appear at 16:58 UTC on release day, and had production on the new version with smoke tests passing 14 minutes later.

This post covers what the release fixes and who should move first, the watcher script, the exact nvm, pm2 and systemd steps we used, and the checklist we will reuse for the next one.

TL;DR

Fixed versions: Node.js 22.23.5 , 24.22.3 and 26.13.1. https://nodejs.org/dist/index.json marks all three as security releases.
What was fixed: 16 CVEs: 7 high, 6 medium, 3 low. All seven high-severity ones are in HTTP/2 and TLS handling: remote crashes, server memory sent to the peer, and a client-certificate bypass. llhttp, undici, sqlite and brotli were updated too.
Move first: any Node process that terminates TLS or serves HTTP/2 to untrusted clients. Everyone on 22, 24 or 26 should upgrade. End-of-life lines get no fix.
Watch for it: poll dist/index.json every 15 minutes, or subscribe to the nodejs-sec list and the vulnerability RSS feed.
Upgrade order under nvm and pm2: install, reinstall pm2, back up dump.pm2, move apps one at a time with a health check, then move the pm2 daemon by restarting its systemd unit. Do not use pm2 update for that last step.

Prerequisites

A Linux host running Node.js apps installed with nvm and supervised by pm2
A systemd unit for the pm2 daemon (the one pm2 startup generates)
Shell access as the app user, plus root for the unit
curl and python3 for the watcher and the checks
A health endpoint per app and a smoke test you trust

The announcement and the release

Node.js security releases follow a fixed pattern. About a week ahead, a pre-alert goes up at nodejs.org/en/blog/vulnerability with the target date, the affected lines, and the highest severity. The details stay private until the fixed builds are out, so nobody can work out the bugs from the patch before you can install it.

For this release:

4 November: pre-alert published. 26.x, 24.x and 22.x, highest severity HIGH.
11 November, afternoon UTC: the builds appeared in dist/index.json with "security": true.
11 November, 17:11 UTC: the advisory page was updated with the full CVE list (the time comes from the nodejs.org repository history).

So the binaries were downloadable before the write-up existed. If you wait for the blog post, you start later than you need to.

You can confirm the fixed versions yourself:

curl -s https://nodejs.org/dist/index.json | python3 -c '
import sys, json
for r in json.load(sys.stdin):
    if r["version"] in ("v26.13.1", "v24.22.3", "v22.23.5"):
        print(r["version"], r["date"], "security:", r["security"])'


v26.13.1 2026-11-10 security: True
v24.22.3 2026-11-10 security: True
v22.23.5 2026-11-11 security: True

The date field is the build date, which is why two of them show the day before release.

What was fixed

The advisory lists 16 CVEs. Severity and affected lines below are as the advisory states them.

CVE	Component	Severity	Lines	What it does
CVE-2026-123863	HTTP/2	High	24, 26	Any unauthenticated client can abort the server through an assertion in session teardown, over TLS or cleartext
CVE-2026-123864	HTTPS	High	22, 24, 26	A zero-length TLS write re-enters the HTTP parser; memory from other connections, including TLS private keys, can be copied into a request body
CVE-2026-123865	HTTP/2	High	22, 24, 26	A protocol error during shutdown leaves the session behind; memory and file descriptors leak with no further cost to the attacker
CVE-2026-123866	HTTP/2	High	22, 24, 26	Stream teardown during a backpressured write sends freed memory to the peer
CVE-2026-126051	TLS	High	24, 26	Remote crash in TLS servers that use `SNICallback`, `OCSPRequest`, `newSession`, `resumeSession` or `server.addContext()`
CVE-2026-123872	TLS	High	22, 24, 26	TLS 1.2 renegotiation can change the servername and carry client-certificate auth to a stricter host
CVE-2026-123874	HTTP/2	High	24, 26	Use-after-free when a `respondWithFile()` or `respondWithFD()` stream is reset during a pending write
CVE-2026-123828	HTTP/2	Medium	24, 26	One-byte heap overflow in `session.altsvc()` with a long origin
CVE-2026-123867	TLS (mTLS)	Medium	22, 24, 26	A CA-only SNI context falls back to the default client CA, bypassing per-host mTLS
CVE-2026-123868	HTTP/2	Medium	24, 26	Queued inbound DATA is not charged to `maxSessionMemory`, so one stream can hold hundreds of megabytes
CVE-2026-123871	HTTP proxy	Medium	22, 24, 26	Proxy credentials appear in `ERR_PROXY_INVALID_CONFIG` error messages
CVE-2026-123877	https.Agent	Medium	22, 24, 26	A TLS session from the default trust store can be reused by a request with a stricter `ca` policy
CVE-2026-123875	HTTP/2	Medium	22, 24, 26	Cancelling a `respondWithFile()` download crashes the server
CVE-2026-123829	Permission Model	Low	22, 24, 26	Recursive `fs.watch` leaks names of denied files (macOS and Windows)
CVE-2026-123862	zlib	Low	22, 24, 26	`params()` plus `destroy()` throws an uncatchable assertion
CVE-2026-106071	Permission Model	Low	22, 24, 26	A wildcard filesystem grant can match paths it should not

The release notes also update llhttp to 9.4.4 on all three lines, sqlite to 3.54.1, undici to 7.30.2 on 24.x and 8.12.4 on 26.x, and brotli to 1.2.1 on 24.x and 26.x.

Note

The advisory and the release notes do not match exactly on the low-severity Permission Model items. The release notes list CVE-2026-123873 ("permission: check heap snapshot destination") and do not mention CVE-2026-106071. If you run with --permission, read both. The action is the same either way: upgrade.

Who should upgrade first

Everyone on 22.x, 24.x or 26.x should upgrade. End-of-life lines such as 20.x and 25.x get no fixed release, and the advisory notes that end-of-life versions are always affected. Use the following order to decide what moves first.

Node terminates TLS or speaks HTTP/2 to the internet. https.createServer, http2.createSecureServer, http2.createServer, tls.createServer, or a framework option that enables them. Most of the high-severity bugs sit here, and several need nothing more than a default server.
mTLS with per-host policies. CVE-2026-123872 and CVE-2026-123867 are authorization bypasses, not just crashes.
Everything else. If a reverse proxy terminates TLS and talks plain HTTP/1.1 to Node, the HTTP/2 and TLS server bugs are not reachable from clients. The client-side fixes (https.Agent session reuse, proxy credentials in error messages, the undici update) still apply.

Two quick checks tell you where you stand. First, which Node processes listen, and on which address:

# Run as root so ss can see every process
ss -ltnpH | while read -r _ _ _ addr _ users; do
  pid=$(echo "$users" | grep -o 'pid=[0-9]*' | head -1 | cut -d= -f2)
  [-n "$pid"] || continue
  exe=$(readlink "/proc/$pid/exe" 2>/dev/null)
  case "$exe" in */node) echo "$addr $exe" ;; esac # match the binary, not the process name
done

Anything bound to 0.0.0.0, [::] or * is reachable from outside the host unless a firewall says otherwise. Second, which code paths you use:

grep -rnE 'https\.createServer|http2\.create(Secure)?Server|tls\.createServer|respondWith(File|FD)|SNICallback|addContext|altsvc|flushHeaders' \
  --include='*.js' --include='*.mjs' --include='*.cjs' --include='*.ts' src/

No hits does not make you safe from the client-side fixes, but it does tell you how urgent the rest is.

How we watched for the release

The pre-alert gives a date, not a time. We did not want someone refreshing a browser tab all day, and we did not want a cron entry to remember to delete afterwards. So we started one background script on release morning that polls dist/index.json every 15 minutes, stops after about 16 hours, and exits as soon as a new 24.x shows up.

#!/bin/bash
# node-release-watch.sh: poll nodejs.org until a Node 24 release newer than
# the current newest 24.x appears. Every 15 minutes, gives up after ~16 h.
LATEST=v24.22.2 # the newest 24.x when you start the watcher
for i in $(seq 1 64); do
  # index.json is sorted newest first, so the first v24.* entry is the latest 24.x
  v=$(curl -s -m 20 https://nodejs.org/dist/index.json | python3 -c 'import sys,json; d=json.load(sys.stdin); print(next(x["version"] for x in d if x["version"].startswith("v24.")))' 2>/dev/null)
  if [-n "$v"] && ["$v" != "$LATEST"]; then echo "NEW NODE 24: $v ($(date -u +%H:%M) UTC)"; exit 0; fi
  sleep 900
done
echo "no new Node 24 release after 16 h"

release day

# start it in the background and forget about it
$ chmod +x node-release-watch.sh
$ nohup ./node-release-watch.sh > node-watch.log 2>&1 &
# hours later
$ cat node-watch.log
NEW NODE 24: v24.22.3 (16:58 UTC)

A few notes if you adapt it:

A failed curl or a bad JSON response leaves v empty, so a network blip does not trigger a false alert.
Swap the echo for a call to your chat webhook if you want a ping instead of a log line.
To wake only for security releases, add and x["security"] to the filter and set LATEST to the newest 24.x security release.
For the long term, subscribe to the low-volume nodejs-sec mailing list and the vulnerability RSS feed. They tell you a release is coming. The watcher tells you the moment it is downloadable.

How we upgraded

Our Node apps run on a Linux VM. Node comes from nvm, pm2 supervises the apps, and the pm2 daemon starts at boot from the systemd unit that pm2 startup generated. The timeline on 11 November, in UTC:

16:58 watcher sees v24.22.3
17:00 nvm install, pm2, dump backup
17:02 apps moved one by one
17:03 move the pm2 daemon
17:12 all on 24.22.3, smoke tests pass

Step 1: install the new version and pm2

As the app user:

nvm install 24.22.3
nvm alias default 24.22.3
nvm use 24.22.3
npm install -g pm2@6.0.13 # same pm2 version you run now; globals are per Node version under nvm
cp ~/.pm2/dump.pm2 ~/.pm2/dump.pm2.bak-$(date -u +%Y%m%dT%H%M)

Under nvm, global packages live inside each Node version, so the new Node has no pm2 until you install it. The new pm2 CLI then talks to the old daemon, which is fine for now.

Step 2: find out how each entry picks its Node

This is the part that decides whether a restart moves an app. pm2 stores an interpreter per entry:

pm2 jlist | python3 -c '
import sys, json
for p in json.load(sys.stdin):
    e = p["pm2_env"]
    print(p["name"], e.get("exec_mode"), e.get("exec_interpreter"), e.get("node_version"), sep="\t")'

Bare node, fork mode: resolved through PATH. A restart with --update-env picks up the new version.
An absolute path such as /home/app/.nvm/versions/node/v24.22.2/bin/node: written in when the app was started with --interpreter, or copied from the daemon's own Node. A restart reuses that path, so the entry has to be deleted and started again. If the ecosystem file hard-codes the path, fix the file first.
Cluster mode (exec_mode is cluster_mode): the daemon forks the workers itself, so they run on the daemon's Node whatever the entry says. They move when the daemon moves in step 4.

Step 3: move apps one at a time

One app, one health check, then the next. If a check fails, you stop with one app affected and the rest still serving. Run it as a script, so exit stops it at the first failure.

health() { # health <name> <port> [path]: retry up to 30 times until a 200
  for i in $(seq 1 30); do
    ["$(curl -s -o /dev/null -w '%{http_code}' -m 5 "http://127.0.0.1:$2${3:-/}")" = 200] && { echo "$1 ok"; return 0; }
    sleep 2
  done
  echo "$1 NOT healthy on :$2"; return 1
}

# Bare "node" interpreter: --update-env passes the new PATH
pm2 restart web --update-env && health web 3000 || exit 1

# Absolute interpreter path: recreate the entry from the ecosystem file
pm2 delete api
pm2 start ecosystem.config.cjs --only api && health api 3001 || exit 1

pm2 save

Then check what each process actually runs. A 200 only tells you the app is up, not which Node it is on:

pm2 jlist | python3 -c 'import sys,json; [print(p["name"], p["pm2_env"].get("node_version"), p["pm2_env"]["status"], sep="\t") for p in json.load(sys.stdin)]'

Two minutes in, every app was on 24.22.3 except one. That entry had no interpreter set, and in that case pm2 starts the script with the daemon's Node. Cluster-mode apps behave the same way, since the daemon forks them itself. Restarting it changed nothing, because the daemon was still on the old version. So the daemon had to move too.

Step 4: move the pm2 daemon through systemd

The unit that pm2 startup writes has the Node version in it twice: in Environment=PATH= and in the path to the pm2 binary. Point both at the new version, as root:

UNIT=/etc/systemd/system/pm2-app.service
cp "$UNIT" "$UNIT.bak"
sed -i 's#/node/v[0-9][0-9.]*/#/node/v24.22.3/#g' "$UNIT"
grep -n '/node/v' "$UNIT" # every match should now say v24.22.3
systemctl daemon-reload

Then restart the daemon through the unit:

# As the app user, with the new pm2 on PATH
pm2 save
cp ~/.pm2/dump.pm2 ~/.pm2/dump.pm2.pre-daemon
pm2 kill # no daemon left for resurrect to attach to

# As root
systemctl restart pm2-app # ExecStart runs "pm2 resurrect" on the new Node

pm2 kill stops every app, and pm2 resurrect starts them all again from the dump, so plan for a short gap. Give it 30 to 60 seconds before you check.

Tip

Don't use pm2 update for this. Restart the systemd unit instead. pm2 update relaunches the daemon from your shell, on the Node that runs the pm2 CLI there. From a login shell or sudo, that can be an older version still installed on the box, and the new daemon then runs outside the unit that starts it at boot. The unit has the version written into it and is the same path a reboot takes, so restarting it gives the same result every time. If the daemon ever comes back without your apps: pm2 kill, copy dump.pm2.pre-daemon back to ~/.pm2/dump.pm2, and restart the unit.

Step 5: verify from /proc, then smoke test

pm2's view is a good start. The kernel's view is the one that counts. As root:

# The real binary of every Node process on the host, grouped
for p in /proc/[0-9]*; do
  exe=$(readlink "$p/exe" 2>/dev/null) || continue
  case "$exe" in */node) echo "$exe" ;; esac
done | sort | uniq -c

# The daemon itself
readlink "/proc/$(cat /home/app/.pm2/pm2.pid)/exe"

Every line should end in v24.22.3/bin/node. Then run the external smoke tests against the public URLs, not localhost. Ours passed at 17:12 UTC, 14 minutes after the watcher fired.

Once the new version has run cleanly for a day, remove the old ones with nvm uninstall. An old Node left on the box is something a shell, a script or a daemon can fall back to without telling you. Until you remove it, check which node in every shell you work from, including root's.

The rest of the fleet

The production host is the urgent part. The rest follows on its own schedule:

CI: actions/setup-node with node-version: 24 picks up 24.22.3 without workflow changes once the runner's tool cache has it. If you want it today rather than when the runner image refreshes, pin node-version: '24.22.3'. check-latest: true helps too, but the version manifest it reads can lag nodejs.org.
Container images: they need a rebuild. At the time of writing, Docker Hub had no 24.22.3-slim tag yet and node:24-slim was last updated on 5 November, so a rebuild right now would still get the old Node. Check before you rebuild, then use docker build --pull and confirm with docker run --rm --pull always node:24-slim node -v.

curl -s https://hub.docker.com/v2/repositories/library/node/tags/24.22.3-slim |
  python3 -c 'import sys,json; print(json.load(sys.stdin).get("last_updated", "not published yet"))'

Dev machines: last, since they do not serve untrusted traffic. nvm install 24.22.3 && nvm alias default 24.22.3.

For a full audit of every place Node hides on a host (cron jobs, helper scripts, child processes, systemd units with old paths), see our October 2026 post on finding every Node you run.

Checklist for the next Node.js security release

When the pre-alert lands

Note the date, the affected lines and the highest severity.
List every host, image and pipeline that runs those lines.
Run the exposure checks above to decide the order.
Check each pm2 entry's interpreter, and fix ecosystem files that hard-code a Node path.

On release day

Start the watcher in the morning, and confirm the fixed versions and "security": true in dist/index.json.
nvm install the new version, nvm alias default, reinstall pm2 under it.
Back up ~/.pm2/dump.pm2.
Move apps one at a time: restart --update-env for bare node, delete and start for absolute paths. Health check each one, then pm2 save.
Check node_version in pm2 jlist for every app.
Point the systemd unit at the new version, daemon-reload, pm2 kill, systemctl restart. Never pm2 update.
Verify from /proc, then run external smoke tests.

After

Rebuild container images once the new tag is on Docker Hub, with --pull.
Make sure CI runs the new version (pin the exact version on release day).
Update dev machines.
After a day on the new version, nvm uninstall the old ones so nothing can fall back to them.

The watcher is what made this fast: it cost nothing to run, and we started within minutes of the release instead of hours. The per-app health checks made every step safe to stop. Keep the checklist somewhere you will find it next time.

Originally published at devops-daily.com.

We Gave an AI Agent a read_only Neon Role. Here Is What It Blocked

DevOps Daily — Wed, 11 Nov 2026 12:00:00 +0000

Teams now connect coding agents to their databases through the Neon MCP Server, and "read-only" is the control they reach for first. We wanted to know what that control stops. We created a read_only role with the Neon CLI and ran 40 one-statement probes against a Lakebase Postgres branch in four setups: a direct connection as that role, and three configurations of the Neon MCP Server. Every INSERT, UPDATE, DELETE and schema change we sent failed in every read-only setup. Over the direct connection, 23 of the 40 probes still ran, and 9 of them left something behind. Four were durable writes from a role named read_only: two rows written through a SECURITY DEFINER function (one of them through EXPLAIN ANALYZE), a large object, and a setting stored on the role itself. The combination in Neon's docs, the MCP Server's read-only mode with the role pinned, stopped all four.

The bigger risk was not a write. A 15-second read from the agent held a routine ALTER TABLE for 11.3 to 12.8 seconds, and all four of the app's writers were stuck behind it, the longest insert for just as long. Sending the same read to a read replica brought the migration down to 0.14 to 0.16 seconds. But nothing in the role or the MCP Server URL puts the agent on that replica: the URL can pin a role, not a compute.

TLDR

The Neon MCP Server with readonly=true and a pinned read_only role stopped every durable write we tried. That is the setup to give an agent.
Almost everything that still ran over a direct connection comes from standard Postgres defaults granted to PUBLIC, not from Neon. The same probes behave the same way on any Postgres, and a few REVOKE statements close most of them.
neon roles create --access read_only runs plain SQL as the database owner: SELECT on the tables that exist, USAGE on the schema, and default privileges for the owner's future tables. It does not touch what Postgres grants to PUBLIC.
Over a direct connection, that role could still create temp tables, call a SECURITY DEFINER function that writes, create a large object, change its own role settings, send NOTIFY, take advisory locks, and read a signing key from a function body.
The Neon MCP Server with readonly=true&roleName=<role> wraps each statement in a READ ONLY transaction and logs in as the pinned role. That blocked every durable write. NOTIFY, advisory locks and reading function source still worked.
readonly=true without a role pin runs as the owner. It still cannot write, but it ran REINDEX, took an ACCESS EXCLUSIVE lock and could read the query text of every other session.
The incident to plan for is lock queueing. Put agent reads on a read replica and give every migration a lock_timeout.

Prerequisites

A Neon account and an API key. The demo creates one throwaway project and deletes it at the end.
Node.js 22 or newer.
Basic Postgres privileges: roles, GRANT, and what PUBLIC means.
A few cents of compute. Neon's consumption history API, which we read after the run, shows 0.39 CU-hours for our whole build, about $0.09 at the Scale list price, including three full runs of the matrix and the runs we threw away. That figure is not one of the repo's stamped results.

Three layers, three different things

A Neon setup for an agent has three controls, and they work at different levels. Mixing them up is where the surprises in this post come from.

Coding agent MCP client or a connection string
Connection string logs in as agent_ro
Neon MCP Server readonly=true&roleName=agent_ro
READ ONLY transaction a Postgres transaction mode
Role privileges what --access read_only granted
Lakebase Postgres branch compute

Connections:

Coding agent -> Connection string
Coding agent -> Neon MCP Server
Neon MCP Server -> READ ONLY transaction (each run_sql)
READ ONLY transaction -> Role privileges
Connection string -> Role privileges (autocommit)
Role privileges -> Lakebase Postgres

The read_only role is SQL that the CLI runs

--access is a feature of the Neon CLI (8.18.0 and newer) and of @neon/sdk, not a field of the REST API, whose role request has only a name and a no_login flag. For a scoped role, the SDK connects as the database owner and creates the role with SQL, in one transaction. The role starts as NOLOGIN with a throwaway password, Neon's API then sets the real password, and only then does the SDK turn on login. This is the list from roleAccessStatements() in @neon/sdk, with our names filled in:

CREATE ROLE "agent_ro" NOLOGIN PASSWORD '<bootstrap>';
GRANT CONNECT ON DATABASE "neondb" TO "agent_ro";
GRANT USAGE ON SCHEMA "app" TO "agent_ro";
GRANT SELECT ON ALL TABLES IN SCHEMA "app" TO "agent_ro";
ALTER DEFAULT PRIVILEGES FOR ROLE "neondb_owner" IN SCHEMA "app"
  GRANT SELECT ON TABLES TO "agent_ro";
-- after Neon replaces the bootstrap password:
ALTER ROLE "agent_ro" LOGIN;

Without --schema, the schema is public. Without --access, the role joins neon_superuser, which is admin on every database. The SDK's own comment states the limit that matters later: default privileges "cover tables the owner creates later, not tables other roles own".

This is the real run. The CLI prints the new role's password, and our script drops that field before anything is written:

neon CLI 9.3.2

$ npx -y neon@9.3.2 roles create --project-id shy-bread-81087327 --branch br-green-rain-bdgi2170 --database neondb --name agent_ro --access read_only --schema app --output json
{
  "branch_id": "br-green-rain-bdgi2170",
  "name": "agent_ro",
  "protected": false,
  "authentication_method": "password",
  "created_at": "2026-11-11T07:44:31Z",
  "updated_at": "2026-11-11T07:44:31Z"
}

The catalog afterwards matches the list exactly: no role memberships, SELECT on every table in app, one default ACL for the owner's future tables. The line that explains several results below is the database ACL, =Tc/neondb_owner. The empty name before = is PUBLIC, and Tc is TEMPORARY and CONNECT. Postgres grants those to every role by default, and it grants EXECUTE on new functions to PUBLIC too. The CLI does not revoke either, and it should not have to: changing database-wide defaults is not what "create a role" means.

readonly=true wraps each statement in a READ ONLY transaction

On the hosted Neon MCP Server, readonly=true (or the older x-read-only header) makes run_sql send the agent's statement through the Neon serverless driver as a one-statement transaction with readOnly: true. Postgres then refuses writes with SQLSTATE 25006. The server also removes run_sql_transaction in this mode, because several caller statements in one shared transaction could end the read-only transaction early. In our runs, the read-only connection listed 59 tools and did not include run_sql_transaction. The read_write connection listed 140.

READ ONLY is a transaction mode, not a privilege boundary. The Postgres docs list what it blocks: data changes, CREATE, ALTER and DROP, COMMENT, GRANT, REVOKE, TRUNCATE, and EXPLAIN ANALYZE or EXECUTE of any of those. They call it "a high-level notion of read-only that does not prevent all writes to disk". The list does not mention NOTIFY, advisory locks, LOCK TABLE, REINDEX or SET.

roleName pins every SQL tool to one role

roleName=agent_ro on the server URL makes every SQL tool log in as that role. There is no fallback: a role_name argument that names another role is refused, and with a pin in place the server also refuses to create scoped roles, because that runs SQL as the owner. The pin covers SQL only, so Neon's docs suggest combining it with readonly=true and category to limit the Management API tools as well. The same page says: "We recommend MCP for development and testing only, not production environments."

One gap in the controls: run_sql also takes a compute_id. In the server source, the URL options are readonly, category, projectId and roleName, so a compute cannot be pinned there. The agent chooses whether to read from the primary or from a replica, and your configuration cannot make it use the replica.

The probe matrix

We seeded a small fake SaaS database (500 customers, 5,000 orders, 10,098 order items, an audit log, a SECURITY DEFINER function that writes to it, and a function with a fake webhook signing key in its body). Then we ran 40 probes in four setups:

S1 : a direct connection as agent_ro, what an agent gets if you hand it a connection string.
S2 : the Neon MCP Server with readonly=true and no role pin, so it runs as the owner.
S3 : readonly=true&roleName=agent_ro, the combination Neon's docs show.
S4 : roleName=agent_rw, a --access read_write role with writes on, as a baseline.

The MCP probes go through the hosted server with the MCP SDK client and an API key, the same calls an agent makes. No model is involved, so every probe is exactly the statement we wrote.

Each probe declared its expected result for each setup before the run: allowed, or denied with an exact SQLSTATE. A result that did not match was a harness failure, not a pass. After each probe, an owner connection checked for a trace: a new row in the audit log, a new large object, a new role setting, or a NOTIFY that a listening session received. Temp tables and session advisory locks were checked inside the probe's own session on the direct connection.

The final run matched its declared expectations in all 160 probe-runs. Two earlier complete runs are kept in results/discarded/, and scripts/compare-matrix-runs.mjs shows that all three agree on every probe's result in all four setups: allowed or refused, the SQLSTATE, and the trace.

40 probes per setup: not blocked, and how many left a trace

	Value	left a trace	Series
S1 direct, read_only role	23 probes	9 probes	direct
S2 MCP readonly, no role pin	21 probes	3 probes	MCP
S3 MCP readonly + roleName	18 probes	3 probes	MCP
S4 MCP read_write role	27 probes	8 probes	write baseline

Bars: probes that ran without an error. Ticks: probes that left a trace (a durable write, a temp table or advisory lock, or a NOTIFY another session received). In S2 to S4, temp-table and advisory-lock traces are inferred from a successful call. Final run, Lakebase Postgres 18, aws-us-east-2.

Here is every probe. ok means it ran, ok* means it ran and left a trace, and a code means it was refused with that SQLSTATE. Durable writes and NOTIFY were checked from an owner session in every setup. Temp tables and advisory locks were checked inside the session only on the direct connection (S1). In S2 to S4 those two marks are inferred from a successful call.

Probe	S1 direct	S2 no pin	S3 pin	S4 rw
`INSERT`, `UPDATE`, `DELETE` a row	42501	25006	25006	ok
`TRUNCATE`	42501	25006	25006	42501
`CREATE TABLE`, `ALTER TABLE`, `DROP INDEX`	42501	25006	25006	42501
`nextval()`	42501	25006	42501	ok
`setval()`	42501	25006	42501	42501
`CREATE TEMP TABLE` (2 probes)	ok*	25006	25006	ok*
`SECURITY DEFINER` function that writes	ok*	25006	25006	ok*
Same, through `EXPLAIN ANALYZE`	ok*	25006	25006	ok*
`lo_from_bytea()` large object	ok*	25006	25006	ok*
`ALTER ROLE agent_ro SET statement_timeout = 0`	ok*	25006	25006	42501
`NOTIFY`, `pg_notify()` (2 probes)	ok*	ok*	ok*	ok*
`pg_advisory_lock()`	ok*	ok*	ok*	ok*
Other advisory lock calls (3 probes)	ok	ok	ok	ok
`LOCK TABLE ... ACCESS SHARE`	25P01	ok	ok	25P01
`LOCK TABLE ... ACCESS EXCLUSIVE`	25P01	ok	42501	25P01
`SELECT ... FOR UPDATE`	42501	25006	25006	ok
Function body with the signing key (`pg_proc.prosrc`)	ok	ok	ok	ok
Customer emails, role list, settings (4 probes)	ok	ok	ok	ok
Other sessions' query text	ok, 0 rows	ok, 11 rows	ok, 0 rows	ok, 0 rows
`SET statement_timeout = 0`, `SET work_mem`, `RESET ALL`, `LISTEN`	ok	ok	ok	ok
`generate_series` sum over 2 million rows	ok	ok	ok	ok
`SET ROLE neondb_owner`	42501	ok	42501	42501
`pg_read_file()`, `pg_ls_dir()`	42501	42501	42501	42501
`CREATE EXTENSION`	42501	25006	25006	42501
`REINDEX TABLE`	42501	ok	42501	42501

What the direct connection left open

S1 is the case to worry about, because a connection string is the easiest thing to give an agent and the hardest to take back. The role blocked what it was built to block: every row write, every schema change, sequences, SELECT ... FOR UPDATE, and every route to more privilege (SET ROLE, server files, extensions). Most of the 23 probes that ran are reads and session settings that a read-only role is expected to allow.

The nine traces are the interesting part:

Four durable writes. app.record_touch() is a SECURITY DEFINER function owned by the owner, and PUBLIC can execute it. Calling it wrote a row to app.audit_log with no table privilege at all. EXPLAIN ANALYZE runs the statement it explains, so the same call through EXPLAIN wrote a second row. lo_from_bytea() created a large object, which needs no grant by default. And ALTER ROLE agent_ro SET statement_timeout = 0 stored a setting on the role in the catalog, where it applies to every future session of that role.
Three session effects. Two temp tables (one filled with 100 order IDs) and a session-level advisory lock. They disappear when the session ends, but a long-lived agent session can hold an advisory lock that blocks your app's own locks for as long as it likes.
Two signals. NOTIFY and pg_notify() reached a session that was listening on the channel. If your app uses LISTEN for job queues or cache invalidation, a read-only agent can trigger it.

There was also one disclosure without a trace: the function body with signing_key text := 'whsec_demo_...' came back from pg_proc.prosrc. Postgres makes the pg_proc catalog readable by every role by default, so a secret in a function body is readable by any role that can connect. Ours came back in all four setups.

None of this is a Neon bug. Each item is standard Postgres behavior: PUBLIC holds TEMP on the database and EXECUTE on functions by default, a large object needs no grant, the NOTIFY statement needs no privilege, and pg_notify() and the advisory-lock functions run on the EXECUTE that PUBLIC holds on built-in functions. It took a SECURITY DEFINER function in our schema to turn EXECUTE into a write, and many real schemas have one. The role does what it says on the label; it just does not say "sandbox".

What the MCP read-only mode closed, and what it did not

S3 refused all four durable writes and both temp tables with 25006. The pinned role kept the privilege checks too: nextval(), ACCESS EXCLUSIVE and SET ROLE failed with 42501 before the read-only check ran. Three traces remained. The two NOTIFY probes reached our listening session. pg_advisory_lock() succeeded, and since that function returns only once it holds the lock, the lock was taken. We did not observe it from another session, because each run_sql call is a separate request to the MCP Server and we had no way to look while it was held.

The function body with the signing key came back in S3 as well.

Without a role pin, readonly=true runs as the owner

S2 shows why Neon pairs readonly=true with roleName. Without a pin, the server logs in as the database owner, and only the transaction mode stands between the agent and your data. It still refused every INSERT, UPDATE, DELETE and schema change we sent, and all four probes that wrote durably over the direct connection. But as the owner, the agent could run REINDEX TABLE, take an ACCESS EXCLUSIVE lock and SET ROLE to the owner. The owner is also a member of pg_read_all_stats, so pg_stat_activity showed it the query text of every other client session, the app's and the read_only agent's included. Our probe counted 11 visible rows for other sessions, some of them background processes with no query. The direct read_only role saw none. Query text often holds literal values: emails in a WHERE clause, tokens in an INSERT.

We got 9 predictions wrong

Our first runs disagreed with our declared expectations in 9 cells. Each disagreement is a small lesson about Postgres:

LOCK TABLE needs a transaction block. As a single autocommit statement, on the direct connection or the read_write MCP connection, it fails with 25P01. The read-only MCP wrapper supplies the transaction, so there it runs, and the privilege decides.
REINDEX is not on the list a READ ONLY transaction blocks, so the owner ran it in S2.
The order of checks differs by statement. With the pinned role, nextval() fails the privilege check (42501) first, while CREATE EXTENSION fails the read-only check (25006) first.
SELECT ... FOR UPDATE takes row locks, so READ ONLY refuses it even for the owner.

We changed the predictions, never the outcomes, and the reason for each is a comment in lib/probes.mjs. Two other mismatches were harness bugs: a DELETE that hit a foreign key, and an INSERT that collided with a row from the previous run. We fixed the probe and added cleanup, and neither one hid a result.

Incident one: a read that stalls a migration

A read takes an ACCESS SHARE lock on every table it touches and holds it until its transaction ends. ALTER TABLE ... ADD COLUMN needs ACCESS EXCLUSIVE, so it waits for the read. Then Postgres queues every new lock request on that table behind the waiting ALTER, including the app's ordinary inserts. The agent never writes anything, and your app still stops writing.

The setup: four app writers insert into app.orders through the pooled connection string, one insert per writer every 50 ms. At 2 seconds the agent runs a 15-second read on app.orders through run_sql (S3). At 4 seconds the owner runs ALTER TABLE app.orders ADD COLUMN note text without a lock_timeout. A direct owner connection samples pg_locks every 250 ms. We ran three variants three times each:

Variant	Migration	Longest app insert	Inserts over 1 s	Backends waiting	Inserts completed
Agent reads on the primary	11.32 to 12.80 s	11.29 to 12.70 s	4 per run	5	104 to 124
Agent reads on a read replica	0.14 to 0.16 s	0.17 to 0.51 s	0	0	350 to 378
Primary, migration with `lock_timeout = '2s'`	gave up at 2.13 s (`55P03`)	2.02 to 2.10 s	4 per run	5	331 to 357

Ranges are over three runs of each variant. With nothing in the way, one insert took 128 to 167 ms at the median, which is the round trip from our test machine to aws-us-east-2 through the pooler. "Backends waiting" is the most lock requests on app.orders that pg_locks showed as not yet granted: the ALTER and all four app writers.

Longest app insert while the agent's 15-second read ran

	Value	migration time	Series
Agent reads on the primary	12.39s	12.43s	primary
Migration with lock_timeout 2s	2.09s	2.13s	lock_timeout
Agent reads on a read replica	0.17s	0.16s	replica

Median of 3 runs per variant. Bars: longest single insert from 4 app writers. Ticks: how long the ALTER TABLE took (or waited before lock_timeout gave up). Lakebase Postgres 18, aws-us-east-2.

The replica fix works because a read replica is a separate compute: a lock taken there is not a lock on the primary. It moves the conflict, it does not remove it. The ALTER still has to be replayed on the replica, where it waits for the agent's read or cancels it. In all 3 replica runs, the agent's read finished without being cancelled.

lock_timeout is the fix on the migration side. The ALTER gave up after 2.1 seconds with 55P03 instead of holding the app for 12.8, and the migration tool can retry. The app still waited about 2 seconds, which is the price of finding out. This is the same advice as in rehearsing migrations on a Neon branch: every migration should set a lock_timeout.

Our first version of this test got the incident wrong. The app writers inserted into a side table, not app.orders, so their latency during the stall matched the baseline and the run seemed to show that only the migration waited. The migration numbers were right, the app numbers measured the wrong table. That run is kept in results/discarded/ in the repo.

Incident two: one heavy read is cheap, and nothing bounds a loop

The role does not limit how much a read may scan. We gave the agent one legal heavy read: a parallel scan of a 2-million-row, 137 MB table with a chain of 20 md5() calls per row. It ran through the MCP Server as agent_ro, first on the primary, allowed to autoscale from 0.25 to 4 CU, and then on a read replica capped at 1 CU. An owner connection on the same compute read num_cpus() from the neon_utils extension about once a second.

Compute	Read time	Peak vCPU	vCPU-seconds, at most	Scale cost, at most	Launch cost, at most
Primary, 0.25 to 4 CU	25.8 s	2	32.84	$0.00203	$0.00097
Read replica, 0.25 to 1 CU	29.2 s	1	31.71	$0.00196	$0.00093

The primary reported 1 vCPU for most of the read and 2 vCPU only in the last four samples, about a second apart. num_cpus() rounds up to a whole vCPU, so a compute at 0.25 CU reports 1. That makes these figures an upper bound: we multiply by Neon's list price of $0.222 per CU-hour on Scale ($0.106 on Launch) and assume every "1" was a full CU. We did not use the consumption API, because it lags a run by minutes.

So one heavy read costs a fraction of a cent, and the capped replica was about 3 seconds slower for about the same cost. That is not the risk. The risk is that nothing in the role bounds how often the agent runs that read or how long a session stays open. The matrix showed that the agent can run SET statement_timeout = 0, and on the direct connection it can store that on its own role. An agent that polls also keeps the compute from suspending; we measured that effect in what keeps a Neon compute awake. A capped replica for agents bounds the size of the compute, and a role-level statement_timeout bounds the common case, as long as you remember that the session can lower it.

Drift: when the role stops matching your schema

A read_only role is a snapshot of the grants at the moment it was created. Two ordinary changes moved the schema away from it. Both failed closed, which is the safe direction, but both are the kind of thing that ends with someone granting more than they meant to.

A table created by another role. Many teams run migrations as a separate role. We created a migrator role and had it create app.migrator_table. agent_ro got 42501 permission denied, because the default privileges from --access read_only are FOR ROLE neondb_owner: they cover the owner's new tables and nobody else's. After ALTER DEFAULT PRIVILEGES FOR ROLE migrator IN SCHEMA app GRANT SELECT ON TABLES TO agent_ro, the migrator's next table was readable. The table it had already created still returned 42501, because default privileges apply only to objects created after them. You need both the default privileges for the migration role and a one-time GRANT for what already exists.
A new schema. app2.secrets returned 42501 permission denied for schema app2, because the role was created with --schema app. An agent that cannot see a schema may report that the data does not exist, which is a wrong answer rather than a leak. Pass --schema once per schema, or re-run the grants when you add one.

Hardening, then the re-run

This is the hardening we applied as the owner, from schema/harden.sql:

-- TEMP and EXECUTE are PUBLIC defaults; take them back for everyone
-- and grant them only to the roles that run the app.
REVOKE TEMPORARY ON DATABASE neondb FROM PUBLIC;
GRANT TEMPORARY ON DATABASE neondb TO neondb_owner, agent_rw;

REVOKE EXECUTE ON ALL FUNCTIONS IN SCHEMA app FROM PUBLIC;
ALTER DEFAULT PRIVILEGES FOR ROLE neondb_owner IN SCHEMA app
  REVOKE EXECUTE ON FUNCTIONS FROM PUBLIC;
GRANT EXECUTE ON FUNCTION app.record_touch(text) TO neondb_owner, agent_rw;

-- Bounds for the agent role. These are defaults: a session can still
-- change statement_timeout with SET.
ALTER ROLE agent_ro CONNECTION LIMIT 5;
ALTER ROLE agent_ro SET statement_timeout = '30s';
ALTER ROLE agent_ro SET lock_timeout = '2s';
ALTER ROLE agent_ro SET idle_in_transaction_session_timeout = '15s';

Revoking TEMPORARY from PUBLIC affects every role that relied on the default, not only the agent, so grant it back to the roles that need it before you run this in production. The same goes for EXECUTE.

Then we ran S1 and S3 again. On the direct connection, four probes flipped from allowed to 42501: both temp tables, the SECURITY DEFINER write and its EXPLAIN ANALYZE twin. That took S1 from 23 probes that ran to 19, and from four durable writes to two. The two that stayed open have no grant to revoke: lo_from_bytea() still created a large object, and ALTER ROLE agent_ro SET statement_timeout = 0 still ran. NOTIFY, the advisory locks and the function body with the signing key were unchanged too: the NOTIFY statement needs no privilege, the hardening revoked EXECUTE only in the app schema and not on built-in functions such as pg_notify() and pg_advisory_lock(), and pg_proc stays readable.

On the pinned MCP connection, nothing opened or closed. The READ ONLY wrapper already refused everything the hardening addresses, and the only difference was the error: the SECURITY DEFINER calls now failed with 42501 (no EXECUTE) instead of 25006, because the privilege check runs first.

So the hardening matters for the connection-string case, and it is not enough on its own there. The two durable writes it cannot close are why a connection string is weaker than the same role behind the read-only MCP Server. One note on the re-run: the probe runner clears the role's statement_timeout before it starts, so the 30-second default was not in place during the re-run. No probe depends on it.

What we could not conclude

One region, one Postgres version, one schema. Lakebase Postgres 18 in aws-us-east-2, a small fake schema. A schema with more functions, extensions or roles has more PUBLIC defaults to check.
The pooled connection string was not probed. Our direct probes used the unpooled host. Through the pooler (PgBouncer in transaction mode), session effects such as SET, temp tables and session advisory locks behave differently, because a session does not keep one backend.
No escape attempts. We probed what single statements can do. We did not try to get around the READ ONLY wrapper or the role pin, beyond checking that run_sql_transaction is gone and that SET ROLE is refused.
No model in the loop. The probes are what a careless or misled agent could send, not what a given model chooses to send.
Small samples for the incidents. Three runs per lock variant, one run per compute for cost. The direction is clear, but the exact seconds are not a benchmark. The cost figures are upper bounds because num_cpus() rounds up.

What to do

Give agents the MCP Server with readonly=true&roleName=<a read_only role>, not a connection string. In our runs that combination stopped every durable write.
If an agent must have a connection string, revoke TEMPORARY and EXECUTE from PUBLIC in the app's schemas first, and keep secrets out of function bodies.
Keep agent reads on a read replica, and set a lock_timeout in every migration. A lock taken on the replica is not a lock on the primary, and lock_timeout makes a migration give up after a set time, so the writes queued behind it wait at most that long. The MCP URL cannot pin a compute, so the replica takes agent instructions that pass its compute_id, or your own layer in front of the database. We did not run the probe matrix on a replica.
Re-check grants after migrations that a different role runs, and pass --schema for every schema the agent should read.

Run it yourself

The-DevOps-Daily/neon-read-only-agent on GitHub

The repository has one script per stage, and each stage writes JSON lines to results/, stamped with a hash of the code that produced them. npm run verify-stamps checks every file against the commit in results/STAMPED_AT.

cp .env.example .env # add a Neon API key
npm install
npm test # offline tests, no database
npm run project # one throwaway project, Postgres 18
npm run seed # fake orders schema and data
npm run roles # neon roles create --access read_only / read_write
npm run matrix # 40 probes x 4 setups
npm run lock-stall # incident one
npm run cost # incident two
npm run drift
npm run harden
npm run rerun # S1 and S3 again after hardening
npm run summarize
npm run verify-stamps
npm run teardown # deletes the project

The run is cheap: the incident stages are a few minutes of a 0.25 CU compute, and the cost stage briefly allows 4 CU. Run teardown when you are done, because the read replica and the primary both count toward CU-hours while they are awake.

Originally published at devops-daily.com.

EC2 Can Now Tell You Where IMDSv1 Is Still Enabled

DevOps Daily — Tue, 10 Nov 2026 09:00:00 +0000

An EC2 instance can make no tokenless metadata requests all week and still allow them tomorrow. That distinction makes an IMDSv2 migration surprisingly easy to declare finished too early. AWS's new ImdsV1Enabled CloudWatch metric gives you a configuration signal to put beside your usage graphs. You can use it to find the remaining exceptions, test one workload at a time, and catch settings that drift back.

AWS announced the metric on November 5, 2026, with availability in every AWS Region and no additional charge for the metric itself. It does not migrate instances for you. The useful change is that your migration dashboard can now answer whether IMDSv1 is still permitted, even when applications happen not to be using it.

Prerequisites

AWS CLI v2 configured for the account and Region you want to inspect.
Permission to describe EC2 instances and read CloudWatch metrics.
A staging instance that represents the application, agents, and scheduled jobs you run in production.
A workload owner available before you change metadata settings. The inventory commands below are read-only; the final change command is not.

Three signals, three different questions

AWS's IMDSv2 transition guide defines the new metric as 1 when HTTP tokens are optional and 0 when they are required. Unlike the request counters, it is emitted from configuration rather than metadata access.

Signal	Question it answers	How to use it
`ImdsV1Enabled`	Does this instance still allow tokenless access?	Find configuration exceptions.
`MetadataNoToken`	Did a tokenless request succeed?	Find callers that need migration.
`MetadataNoTokenRejected`	Was a tokenless request rejected after enforcement?	Investigate compatibility problems.

The two request metrics count successful tokenless calls and rejected attempts respectively; the EC2 metrics reference documents both. A quiet request counter does not tell you which configuration is in place.

Consider a backup task that runs once a month. A five-day observation window might never execute it. My recommendation is to keep an explicit exception list with an owner, the workload's longest relevant execution cycle, and the next test date. That makes a migration gap actionable instead of turning it into another permanently yellow dashboard.

Find exceptions Configuration still permits v1
Exercise callers Boot, jobs, agents, containers
Require tokens Verify health and rejected calls

Build an inventory before changing anything

Start with one account and Region. This example lists running and stopped instances whose metadata options still make tokens optional:

aws ec2 describe-instances \
  --region eu-west-1 \
  --filters \
    Name=instance-state-name,Values=running,stopped \
    Name=metadata-options.http-tokens,Values=optional \
  --query 'Reservations[].Instances[].{Id:InstanceId,State:State.Name,Endpoint:MetadataOptions.HttpEndpoint,Tokens:MetadataOptions.HttpTokens,HopLimit:MetadataOptions.HttpPutResponseHopLimit}' \
  --output table \
  --no-cli-pager

The DescribeInstances API supports these filters and returns the metadata options. Keep the endpoint column: HttpEndpoint=disabled is a separate state, not proof of a working IMDSv2 migration. Record those instances separately rather than enabling metadata access just to make every row look alike.

Repeat the inventory for the accounts and Regions in your actual scope. An empty result in one Region is not a fleet-wide result. Include stopped instances in your review because they can return to service later.

Then discover the metric series available to your caller:

aws cloudwatch list-metrics \
  --region eu-west-1 \
  --metric-name ImdsV1Enabled \
  --output json \
  --no-cli-pager

This command lists series and their dimensions, not current values. Use the returned series in CloudWatch to inspect datapoints. AWS documents that ListMetrics can take up to 15 minutes to show a new metric, and it omits series that have not reported in two weeks. An empty response is therefore a reason to check account, Region, visibility, and instance configuration, not a green migration result.

Test the workload, then require IMDSv2

Choose one representative staging instance and run its real lifecycle: bootstrapping, service restart, deployment, scheduled maintenance, and credentials refresh. Include monitoring and backup agents. For containers, test from inside the container as well as from the host.

AWS notes that a metadata response hop limit of 1 can cause problems in container environments. Do not change hop limits across the fleet just because a host-level test passed; investigate the affected network path separately.

Once the workload is ready, replace the example ID and apply the token requirement to that instance:

aws ec2 modify-instance-metadata-options \
  --region eu-west-1 \
  --instance-id i-0123456789abcdef0 \
  --http-tokens required \
  --no-cli-pager

The modify command can return pending for a running instance. Verify that the setting reaches applied:

aws ec2 describe-instances \
  --region eu-west-1 \
  --instance-ids i-0123456789abcdef0 \
  --query 'Reservations[].Instances[].MetadataOptions' \
  --output json \
  --no-cli-pager

Expect HttpTokens to be required and State to be applied. The commands here are examples, not a transcript from a production account.

Important

Do not pipe every inventory result straight into the modification command. First check application health and rejected tokenless requests on the staging instance. Keep any rollback scoped to the affected workload and record why it was necessary.

Keep replacement instances from undoing the work

Changing one running instance leaves a second question: what happens when Auto Scaling replaces it? Update the launch configuration your fleet actually uses, then launch a replacement in staging and inspect its effective metadata options.

AWS's configuration precedence rules put explicit launch settings ahead of account defaults, then AMI settings. Account defaults also do not rewrite existing instances. Review the launch template version your Auto Scaling group selects rather than assuming a newly edited template is already in use.

For the rollout, keep two views side by side: configuration exceptions and application health. Use MetadataNoToken during preparation and MetadataNoTokenRejected after enforcement. Keep missing telemetry visibly unknown, and assign each remaining exception an owner.

The useful finishing condition is specific: the intended instances require tokens, representative workloads still work, and replacement instances inherit the same setting. The new metric makes the configuration part easier to watch continuously.

Originally published at devops-daily.com.

We Built Opsville, a Free Pixel-Art Town Where You Fix DevOps Incidents

DevOps Daily — Mon, 09 Nov 2026 10:42:36 +0000

Opsville is a small pixel-art town where every district is broken, and you are the one on call. 🏘️ Each district is one DevOps topic: Linux, Docker, Git, CI/CD, Kubernetes, monitoring, security and databases. The people there hand you quests. Most quests open one of our browser simulators, then ask three questions. Fix all eight districts and the town gets its Launch Day.

It is free, it runs in your browser, and you can play it now at devops-daily.com/opsville.

▶️ Watch the full 30-second trailer

Disclosure: we build Opsville at DevOps Daily. It is free and needs no sign-up.

How to play, in 60 seconds ⏱️

Open the game and press Start. Make a character, or skip that and change your look later at the Wardrobe.
Move with the arrow keys or WASD. On a phone you get a joystick and a Talk button. A gamepad works too.
Walk up to Mayor Chris in the town square and press E. He explains the plan: the town goes live soon, and no district works.
Head west to Linux Lodge. Alex says the lodge server keeps saying "No space left on device". Take the quest, and the Linux terminal simulator opens in a panel (on a phone, in a new tab).
Finish it, then answer three questions from our quiz bank. Two right answers pass. You get XP and a badge.
Finish every quest in a district, and the next one opens. Clearing Linux Lodge also gets you a bike (press B) and Pip, a penguin pet who follows you around.

Handy keys: Q for quests, M for the map, P for your profile, H for help, F for photo mode, and 1 to 4 for emotes.

There are 8 districts and 34 quests. Progress saves in your browser, so you can close the tab and come back later.

Three moments we like

1. The boss fight is a live incident 🚨

From Docker Docks on, each district ends with a boss, which opens after the quests before it. The boss quiz plays as an incident: a status board with three systems that go from Down to Fixed as you answer.

In Docker Docks the board shows the container runtime, the host kernel and the image registry. In Git Garden it is the main branch, the release tags and the team clones. In the Observatory it is checkout, the status page and the on-call channel. Get two of the three questions right and the incident closes. Sophia at the Docks sums it up: "Every container stays in its box. Incident closed."

2. The pager 📟

Once you clear Linux Lodge, the pager starts. It stays quiet for the first three minutes, then pages you every 6 to 10 minutes, never while a panel is open. Each page comes from a district you have already opened:

Disk full on the Lodge server!

Press On my way and you have two minutes to get there and 30 seconds to fix it. You see the symptom (df -h shows / at 100%, and /var/log/app.log is 38 GB and growing) and three possible fixes. Pick "Reboot the server" and the game tells you why that fails: the disk is still full after a reboot, and the log keeps growing. The right fix is to truncate the log and set up logrotate.

There are ten pages across the eight districts, from a container stuck in a restart loop to an expired TLS certificate. If the pager gets too much, snooze it for 15 minutes or turn it off in Help.

3. The secret server room 🗄️

Twelve lost packets are hidden around town, and each one has a short story. "A DNS query from 2009" is still waiting for an answer. "A retry with no backoff" tried 9,000 times a second and is tired now.

Find all twelve, and an old hatch by the lake in the town square opens. Below it is a basement full of blinking racks and one terminal:

$ uptime
 09:00:00 up 3652 days,  1 user,  load average: 0.01, 0.01, 0.00
$ cat /var/log/packets.log
delivered: A lonely TCP SYN
delivered: A DNS query from 2009
...
$ echo "12 of 12 packets home"
12 of 12 packets home

It is the oldest server in town, ten years without a reboot. Finding every packet also gets you the golden packet cap.

How we built it 🛠️

Vite, TypeScript and Phaser 3. Phaser draws the world: the map, the people, the weather and the effects. The panels, quizzes and HUD are plain DOM on top of the canvas.

Maps from text. The whole world is one text file of 72 x 50 characters, where one character is one 16 x 16 tile. A build script turns it into Tiled JSON with ground, decor, collision and object layers. Three rows from the top of the map:

P..*.:::.......::::.*..P................ggggg...P..%%%%........%%%%....P
P.v........)..:....v...P................SSDSS...P..HHHH........HHHH.T..P
P..*..T.:::::::.*..T.*.P........[.........:.....P..HHHH....]...HHHH....P

P is a pine, : a stone path, g a grey roof, S a stone wall, D a door and H a castle wall. Moving a building is a diff you can review.

Static files on Cloudflare Pages. The game builds with the base path /opsville/. A release script copies the build into our main site, and Cloudflare Pages deploys it with everything else. The game needs no server of its own.

XP that the server decides. Sign-in is optional, with a free DevOps Daily account. When you are signed in, the game only tells our learning platform's API "I finished quest X". The API looks the quest up in its own list and decides the XP. Each quest counts once per player, two quests must be at least 30 seconds apart, and one player can count at most 24 quests in 24 hours. A test in the game fails when its quest list drifts from the server's list, so the weekly leaderboard stays fair.

The launch post has the rest of the build story: how a simulator in a frame tells the game you finished, and how we made two art sources look like one game.

Your turn

Start in Linux Lodge and see how far you get before the pager goes off: play Opsville free.

If you could add a ninth district, what would it be, and what would its boss incident look like? Tell us in the comments. 👇

Opsville: A Free Pixel-Art DevOps Town in Your Browser

DevOps Daily — Mon, 09 Nov 2026 09:00:00 +0000

Opsville is a small pixel-art town where every district is broken, and you are the one on call. It is free, it runs in your browser, and it is live today at devops-daily.com/opsville. Most quests open one of the simulators from our simulators page, then ask you three questions from our quiz bank. Fix all eight districts and the town gets its Launch Day.

This post covers the game first, then how we built it.

Prerequisites

A modern browser on a desktop or a phone.
A keyboard, a gamepad or a touch screen.
No account. Your progress saves in your browser.

Walk into town

You start in the town square. Mayor Chris has a problem: the town goes live soon, and none of the districts work. That is the Launch Day story. Each district is a chapter, with a closing scene once you fix it. The Agent, the robot from our book The DevOps Survival Guide, waits in the square and comments when you walk into a new district.

There are 8 districts and 34 quests. The districts open one after another:

Linux Lodge
Docker Docks
Git Garden
CI/CD Railway
Kubernetes Castle
Observatory
Security Fort
Neon Data Depot

A broken district looks broken. Smoke rises from its landmark, its lamps flicker at night, and its people grumble. When you fix it, lights go up over its sign and its music opens up.

How a quest works

Walk up to someone and press E, or tap Talk on a phone. They give you a task, such as "The disk is full" in Linux Lodge. The quest opens a simulator from our site in a panel: the Linux terminal, the Fork Bomb Simulator, the Container Security Challenge, the Kubernetes Scheduler Challenge and more. When you finish, you answer three questions from our quiz bank. Two right answers pass the quest.

From Docker Docks on, each district ends with a boss. A boss quiz plays as a live incident : a status board with three systems that go from "Down" to "Fixed" as you answer.

Each quest gives you XP and a badge. Each cleared district gives you something new: a bike after Linux Lodge, a pet for each of the first five districts (a penguin, a little whale, a cat, a dog and a mini Agent), and new hats.

Things to do between quests

The pager. After you fix Linux Lodge, an incident in an open district pages you now and then. You get two minutes to get there and 30 seconds to fix it. You can snooze it, or turn it off in Help.
The daily quest. One question a day from the quiz bank. Seven days in a row earns the Streak beanie.
Your house. You get a log cottage in Linux Lodge with your name on the sign. Inside, there is a cup for every cleared district and a medal for every badge. The garden next to it grows a plant for each day you played in the last four weeks.
The newsstand. The latest DevOps Daily posts, in the town square.
Photo mode. Press F, pick a frame and a filter, and save a PNG.
Your look. A character creator on first start, and a wardrobe in the square for later changes.
The Advent board. From 1 to 24 December, the town gets snow, lights and a board with one door a day.

Twelve lost packets are hidden around town. Find all of them, and an old hatch by the lake opens. Below it is a server room that has not been rebooted in a long time.

Accounts are optional

You can play the whole game without an account. If you sign in with a free DevOps Daily account, your save follows you to other devices. You can also join the weekly leaderboard (opt-in, with a board name only), help fill a weekly town goal with everyone else's XP, and race district speedruns.

The Help panel has a larger text switch. You can play with a keyboard, a gamepad or touch controls. If your system asks for reduced motion, the game turns off the moving effects. Sound is off until you turn it on.

How we built it

Vite, TypeScript and Phaser 3

Phaser draws the world: the map, the people, the weather and the effects. The panels, quizzes and HUD are plain HTML on top of the canvas.

The map is a text file

The whole world is one text file: 72 x 50 characters, where one character is one 16 x 16 tile. A build script turns it into Tiled JSON with ground, decor, collision and object layers. Moving a building is a diff you can review. Here are four rows from the top of the map:

P..*.:::.......::::.*..P................ggggg...P..%%%%........%%%%....P
P.v........)..:....v...P................SSDSS...P..HHHH........HHHH.T..P
P..*..T.:::::::.*..T.*.P........[.........:.....P..HHHH....]...HHHH....P
P.*.......:......*.....P########################P..........:...........P

P is a pine, : a stone path, # a rail track, g a grey roof, S a stone wall and D a door. Characters such as [ and ) are objects, defined in a list below the grid.

Static files inside the main site

The game builds with the base path /opsville/. A release script copies the build into the site's repo, adds the game's response headers, and opens one pull request. Cloudflare Pages deploys it with the rest of the static site. The game needs no server of its own.

Simulators in a frame

A quest loads /games/<slug>/embed?embed=opsville&run=<id> in a frame. When you finish, the simulator sends a message on the same origin:

// Sent by the simulator when the player finishes it
{ type: 'devops-daily:game-complete', game: '<slug>', run: '<id>' }

On phones the simulator opens in a new tab, and the same message goes over a BroadcastChannel. If no message arrives, a "Done, take the quiz" button covers it. Either way, the quiz is the real gate.

Quest-giver talk to a local
Simulator /games embed
game-complete postMessage
Quiz gate 2 of 3 to pass
XP and badge server-checked

XP that the server decides

When you are signed in, the game only says "I finished quest X". The learning platform's API looks the quest up in its own list and decides the XP. Each quest counts once per player, two quests must be at least 30 seconds apart, and a player can count at most 24 quests in 24 hours. The rows live in the learning platform's existing Neon database, so the game did not need a new one. A test in the game fails when its quest list drifts from the server's list.

Art from two sources

The tiles and characters come from Kenney's CC0 packs: Tiny Town, Tiny Dungeon and RPG Urban Pack. The landmarks, signs and portraits are generated pixel art. A Python script cuts out their background, scales them to the pixel grid, and maps every colour to one warm palette shared with the tiles. That way both sources look like one game. The music is ours too: chords and patterns played with WebAudio, with no sound files.

Play it and tell us what breaks

Play Opsville free at devops-daily.com/opsville. Start in Linux Lodge, and see how far you get before the pager goes off.

Found a bug, or have an idea for a quest or a district? Open an issue on our GitHub repo and tell us.

Originally published at devops-daily.com.

Build an Everyday AI Assistant with DigitalOcean Inference

DevOps Daily — Fri, 06 Nov 2026 09:00:00 +0000

You finish work with groceries to buy, a kitchen to clean, and a workout you meant to do. A friend wants to meet, too. Writing those things into a to-do list takes a minute. Deciding what fits into an evening when you are already tired is the harder part.

That is the problem we are building for. Daylight is a small personal AI assistant that helps you make tradeoffs, proposes a manageable plan, and adapts when your circumstances change. Work is one part of the picture; dinner, hobbies, friends, chores, and rest belong in the same conversation.

You can run the app locally with Python and a DigitalOcean Inference key. After getting it working, we will look at how a model proposes structured changes without getting direct control of your saved data. The same request, validation, and persistence pattern is useful in plenty of internal tools.

Prerequisites

Python 3.11 or newer, Git, and a terminal with make.
A DigitalOcean account with Serverless Inference billing configured.
A model access key scoped to a text model with function calling support.
Node.js 22 or newer only if you want to run the optional browser-rendering tests.

The example uses mimo-v2.5-pro. DigitalOcean lists Chat Completions, tool calling, and structured outputs for it in the model catalog. Availability and supported parameters vary by model, so check the catalog when selecting another one.

Each chat submission makes a billable inference request. Completing tasks, accepting plans, deleting tasks, and starting a new day are local application actions.

Get Daylight running

The complete app is in the companion repository:

The-DevOps-Daily/digitalocean-inference-daylight on GitHub

Clone it, install the Python dependencies, and copy the configuration template:

git clone https://github.com/The-DevOps-Daily/digitalocean-inference-daylight.git
cd digitalocean-inference-daylight
make install
cp .env.example .env

There is no frontend build or separate database service. FastAPI serves the browser interface and API; Python's built-in SQLite support handles persistence.

Create a model access key

Open DigitalOcean's Model Access Keys page and create a key for Daylight. Select the model you intend to use. For a request from your laptop, choose No VPC network ; a VPC-restricted key requires requests from that network. Copy the secret when it is shown.

The key-management documentation explains model scoping and VPC restrictions. Use a key dedicated to this app rather than sharing credentials between experiments.

Set these values in .env:

DIGITALOCEAN_INFERENCE_KEY=your-model-access-key
DIGITALOCEAN_INFERENCE_MODEL=mimo-v2.5-pro
DAYLIGHT_DATABASE=data/daylight.sqlite3
DIGITALOCEAN_REASONING_EFFORT=none
INFERENCE_TIMEOUT_SECONDS=120

DIGITALOCEAN_REASONING_EFFORT is optional. Remove it if your selected model does not support that setting. DigitalOcean documents reasoning controls in Using Reasoning with the Chat Completions and Responses API.

Start the server:

make run

Open http://127.0.0.1:8082. Keep the terminal running while you use the app. After changing .env, stop the server with Ctrl+C and run make run again.

The access key stays in the backend process. The browser sends requests to your local FastAPI server, which calls DigitalOcean's Chat Completions endpoint at https://inference.do-ai.run/v1/chat/completions.

Note

You control the application code and its local database. Model inference is hosted: chat submissions send recent conversation, your saved plan, preferences, pending proposals, and recently removed task IDs and titles to DigitalOcean and the selected model service. This is not an offline assistant.

Give it a day that needs a little rearranging

Set the available time to 90 minutes and energy to Running low. Then try:

I need groceries for dinner, the kitchen needs cleaning, and I wanted to exercise. A friend also wants to meet tonight. Nothing has a hard deadline except eating dinner. I prefer relaxed evenings. Help me make a realistic plan.

A useful response does more than repeat your list. It considers which activities can shrink, what depends on something else, and what can wait. It might suggest a focused grocery trip, a simple meal, a short kitchen reset, and lighter movement instead of a full workout. It should explain the tradeoffs so you can disagree with them.

In our live test, the first proposal allocated 85 minutes to groceries, dinner, cleanup, and a short walk or stretch. It deferred the full workout and meeting a friend. That is one possible answer, not the correct answer for everyone's evening. If seeing your friend matters more, tell the assistant that and revise the proposal.

The right-hand panel labels the result Proposed changes. Select This works for me to save it. Until then, the accepted plan remains unchanged.

Now mark groceries complete, change the time control to 20 minutes , and send:

The groceries are done. My evening changed and I only have 20 minutes now. Keep dinner as the priority. Suggest a simple meal and defer the cleanup and exercise if necessary. Preserve everything else, including the completed grocery task.

In our test, Daylight reduced dinner to 20 minutes and moved the remaining activities to Later. Groceries stayed completed. Accepting the revision and reloading preserved that state.

The screenshot recreates the 20-minute dinner scenario above. Your model's wording and task estimates will vary.

This is the contribution we want from the AI: help someone decide what is doable when time, energy, and priorities collide. The task list gives that reasoning somewhere to land.

How the assistant proposes changes

The implementation is small enough to follow without an agent framework:

app/
├── agent.py # Prompt, inference request, response validation
├── config.py # Inference timeout validation
├── main.py # API routes and user actions
├── models.py # Request and plan schemas
├── store.py # SQLite state and revision checks
└── static/ # HTML, CSS, JavaScript, local Markdown renderer

Each chat submission includes the current date from the browser, available minutes, energy, saved plan, pending proposal, removed tasks, and the latest 16 conversation messages. Accepted preferences live inside the saved plan rather than depending on an ever-growing transcript.

Your situation message, time, energy
FastAPI + context saved plan and history
DO Inference question or tool arguments
Validated proposal review the tradeoffs
Accept and save SQLite transaction

If an important detail is missing, the model can ask a clarifying question instead of proposing a plan. Otherwise, it can return a function tool call named propose_plan.

This is a constrained planning assistant, not a background worker that runs arbitrary tools. There is one model request per submitted message. The tool call is a structured proposal; the app does not dispatch model-generated function names to Python functions or let the model write to SQLite.

Define what a plan contains

In app/models.py, a task has a stable ID, title, time estimate, group, and reason:

class Task(StrictModel):
    id: str = Field(pattern=r"^[A-Za-z0-9_-]{1,80}$")
    title: str = Field(min_length=1, max_length=160)
    minutes: int = Field(ge=1, le=1440)
    when: Literal["today", "later", "done"]
    reason: str = Field(min_length=1, max_length=500)

The full plan also contains a headline, explanation, task array, and preferences:

class Plan(StrictModel):
    headline: str = Field(min_length=1, max_length=180)
    explanation: str = Field(min_length=1, max_length=1500)
    tasks: list[Task] = Field(max_length=40)
    memory: str = Field(max_length=2000)

StrictModel rejects unknown fields and strips surrounding whitespace. The memory field is a visible note of preferences you have explicitly shared, such as preferring relaxed evenings. The prompt tells the model to preserve those preferences unless corrected and to avoid inventing sensitive traits. You review proposed preferences with the plan before acceptance.

Put that schema in the inference request

The request in app/agent.py uses the Pydantic schema as the function's parameter definition. This excerpt shows the relevant fields; the full implementation also handles configuration and provider errors:

payload = {
    "model": os.getenv("DIGITALOCEAN_INFERENCE_MODEL", "mimo-v2.5-pro"),
    "messages": messages,
    "max_tokens": 3000,
    "stream": False,
    "tools": [{
        "type": "function",
        "function": {
            "name": "propose_plan",
            "description": "Propose a complete realistic plan for user review; does not save it.",
            "parameters": Plan.model_json_schema(),
        },
    }],
    "tool_choice": "auto",
}

The model sees the request's context and decides whether to ask a question or supply tool arguments. It does not see an API key in the message content; httpx adds authentication as a server-side authorization header.

Validate the proposal before showing an accept button

A function schema tells the model what to produce. The server still needs to check what it actually returned:

proposal = Plan.model_validate_json(
    calls[0]["function"]["arguments"]
).model_dump()

ids = [task["id"] for task in proposal["tasks"]]
if len(ids) != len(set(ids)):
    raise ValueError("Duplicate task IDs")

planned_minutes = sum(
    task["minutes"]
    for task in proposal["tasks"]
    if task["when"] == "today"
)
if planned_minutes > request.minutes:
    raise ValueError("Plan exceeds available time")

The code also checks that saved commitments are not silently omitted, completed tasks remain completed, and deleted task IDs do not reappear. If validation fails, it rejects the response without replacing the saved plan.

Accepting arbitrary JSON because the HTTP request returned 200 would skip all of those checks. Passing the arguments through Pydantic is the first step; checking the application's own rules is the second.

These checks do not prove that grocery shopping really takes 25 minutes, that a suggested dinner suits you, or that two tasks represent the same commitment under different IDs. You still review the proposal. The prompt also tells the assistant not to claim it checked a calendar, contacted your friend, or scheduled a reminder: none of those integrations exists in this app.

Keep control of the plan between conversations

The accepted plan stays in data/daylight.sqlite3 across page reloads, browser sessions, and server restarts. It has no expiry date and does not reset at midnight.

The usual task actions are ordinary application behavior:

Action	What happens
Complete a task	Moves it to Done and invalidates an older pending proposal.
Reopen a task	Restores its previous Today or Later group.
Delete a task	Removes it immediately; the previous plan is available through Undo task deletion until another plan change replaces that snapshot.
Accept a proposal	Saves the reviewed replacement plan and keeps one previous plan for undo.
Dismiss a proposal	Keeps the accepted plan.

A trash button appears beside each saved task's time estimate. Proposed tasks must be accepted before those controls become active.

Deleting a task also adds its ID and title to bounded removed-task context, so the assistant is told not to recreate it from an old conversation. Reusing the deleted ID is rejected. This is not semantic duplicate detection: a model could still invent a different ID for the same activity, so inspect the next proposal. You can explicitly ask to add an activity again with a new ID.

Clear the conversation or start a new day

Both actions are in the Chat options menu at the top-right of the conversation panel:

Clear conversation removes the chat and pending proposal while keeping the accepted plan, task progress, and preferences.
Start a new day clears the chat, removes completed tasks from the current plan, and carries unfinished tasks into Later for review. Preferences remain saved.

Both clear the old undo snapshot and ask for confirmation. Starting a new day prepares an editable message; it does not submit an inference request until you press Send. Check your time and energy controls before sending it.

Saved chat is bounded to 60 messages. Inference reads only the latest 16, together with the explicit plan context. That makes the accepted plan the durable record of commitments, rather than hoping that a model can recover them from an unlimited chat log.

SQLite also protects against a less visible problem: stale updates. Every state change increments a revision. A transaction reads the current revision under BEGIN IMMEDIATE, compares it with the request's revision, applies the change, and commits. If a task changes or you clear the conversation while an inference request is running, the old response cannot overwrite the newer state. The API returns a conflict instead.

Test the behavior, then make it your own

Run the backend checks and optional Markdown checks:

make check
node --test tests/markdown.test.cjs

The reviewed version has 39 backend tests and four Markdown tests. They cover plan validation, saved state, completion, deletion and undo, new-day behavior, stale updates, provider failures, host validation, and rendering untrusted model text. They use mocked inference responses and do not incur model charges. The repository also includes GitHub Actions checks for Python 3.11 and 3.14.

Live testing answers a different question. Try your own mix of work and personal activities, accept a plan, reduce the time available, and check that the new proposal both fits the budget and preserves unfinished commitments. Reload after completing an item to verify persistence. A passing schema test alone cannot tell you whether the advice is useful.

Expect a few operational details

We observed planning requests near a minute with MiMo V2.5 Pro, including one diagnostic at 58.7 seconds. Earlier requests exceeded the original 60-second timeout. That is a development observation, not a benchmark or a latency promise. The app now defaults to a configurable 120-second response timeout and a separate 10-second connection timeout. On timeout, it keeps the saved plan and leaves your message in the composer.

If a request fails, identify which layer failed:

Symptom	First thing to check
Local server offline / Failed to fetch	Is `make run` still running? Start it, then select Reconnect.
HTTP 401 from DigitalOcean	The key may be invalid or revoked. Replace it in `.env` and restart.
HTTP 403	Check model access and any VPC restriction on the key.
HTTP 429	Wait before retrying; check the account's rate limits.
Invalid plan	The model returned malformed data or broke an application rule. Clarify the request and retry.
State conflict	Another action changed the plan while the request was running. Review the latest state before resubmitting.

Reconnect reloads local state; it does not resend a potentially billable request. Assistant Markdown is rendered with a locally bundled parser, with raw HTML and remote images disabled. User messages remain plain text, and provider error bodies are not echoed into the interface.

Keep this first version on localhost. It has no authentication or separation between users. A shared deployment needs those features, a deliberate host policy, and durable database storage. Notifications, calendars, and unattended actions are separate projects, not hidden capabilities of the model.

A useful next experiment is a small set of scenarios you can rerun when changing the prompt or model: low energy, an urgent errand, a missed day, a completed task, and a new activity you genuinely want to make room for. Judge whether the assistant makes sensible tradeoffs as well as whether its JSON passes validation. If you want a fuller evaluation workflow, our DigitalOcean Inference error-explainer guide walks through one.

You now have a local app whose AI helps turn competing intentions into a plan you can inspect and change. Try it with a real evening, tell it when its priorities are wrong, and see whether the revised day feels more doable.

Originally published at devops-daily.com.

cgroup v2 memory.high Throttles, memory.max Kills: Measured on a Host Without Swap

DevOps Daily — Fri, 06 Nov 2026 07:00:00 +0000

We ran one Python workload that allocates memory in 256 KiB steps, on a Linux 6.8 host with no swap, under cgroup v2 memory.max, memory.high, both and neither. When it asked for 320 MiB with only memory.max at 256 MiB, it ran at full speed and the kernel killed it 0.28 to 0.39 seconds after it started. With only memory.high at 230.4 MiB, the kernel did not kill it. It made new allocations wait instead, and after 150 seconds the workload held 11 to 12 MiB more than memory.high.

The case to worry about is the smaller one. A version of the workload that needs 239 MiB fits under memory.max and runs at full speed there. With memory.high at 90% of that limit, which is what Kubernetes MemoryQoS writes for a 256Mi limit and a zero memory request with the 1.36 default factor, its median allocation took 191.8 ms on a CPU of its own and 6.2 seconds on a CPU it shared with our 100 ms sampler, against 0.26 ms with no limit. It reported no error.

The kernel's throttle formula predicted the length of each sleep to within about 10%. The number of sleeps is what changed: once per 64-page batch, as the code suggests, or once per 4 KiB page. We found in review that our own sampler, on the same CPU, kept the task at once per page. Without it, the count still switched to once per page in phases, while other work ran on the host. And one condition changed both limits: when the program's own code was charged to its cgroup, the kernel reclaimed that code and read it back from disk over and over, and memory.max did not kill it within the 20 seconds we allowed.

This post has the numbers, the kernel formula, the method problems we found on the way, and a check of what systemd, Docker and Kubernetes write to these files.

TL;DR

memory.max alone killed at full speed. The 320 MiB workload ran at normal latency (p99 of the last 10% of operations 0.29 to 0.41 ms) and the kernel OOM-killed it 0.28 to 0.39 seconds after it started, in 3 of 3 runs.
memory.high alone did not kill within the 150 second cap. It made new allocations wait. Each sleep was 1.06 to 1.10 times the kernel's formula at the median, 64000 x o^2 ms with o the share over memory.high. At the 2 s cap the median sleep was 2,047.9 ms, which fits the timer wheel's 64 ms step for a 2,000 ms timeout.
The number of sleeps decides the cost, and a neighbour on the same CPU decides the number. The kernel's comment says one sleep per 64-page batch. With our sampler on the same CPU, the task slept once per 4 KiB page in 95% of the operations whose sleep was 50 ms or more. Without that neighbour (the workload on another CPU, or the sampler once a second), it switched between about 1 and 64 sleeps per batch in phases of tens of seconds, and spent 46% to 72% of that time in the slow phases.
A program that fits under memory.max but not under memory.high ran much slower and reported nothing. With memory.high at 90% of a 256 MiB limit (the kubelet's value for a zero memory request and factor 0.9), the median allocation took 191.8 ms on its own CPU and 6,216 ms next to the busy sampler (15 to 17 of its 200 churn operations in 150 s), against 0.257 ms with no limit.
Both limits only delayed the kill. With memory.high at 90% of memory.max, a leak was OOM-killed after 29.1 minutes next to the busy sampler. With memory.max alone the kill took 0.3 seconds. On a CPU of its own, it was not killed within the 120 second cap in 3 of 3 runs.
Code pages in the cgroup turn both limits into thrashing. When the program's own binary was charged to its cgroup, the kernel reclaimed it and read it back from disk, 2.4 to 2.6 GB in 25 seconds at our 100 MiB/s disk cap. memory.max did not kill it within 20 seconds in 3 of 3 runs, and memory PSI showed only 8% while io PSI showed 85%.
Page cache is not throttled or killed. Neither limit killed the file-reading runs or put them to sleep. They capped the cache, so later reads went to disk more often (p99 1.0 to 2.6 ms against 0.13 ms with no limit).
The tools: systemd writes MemoryHigh= and MemoryMax= to memory.high and memory.max as given. Docker has no flag for memory.high, and on cgroup v2 --memory-reservation becomes memory.low. Kubernetes MemoryQoS writes memory.high; it is alpha and off in 1.36, and beta and on in 1.37 but with no throttling factor set, so no memory.high unless you set one.

Prerequisites

A Linux machine with cgroup v2 (stat -fc %T /sys/fs/cgroup prints cgroup2fs) and systemd
Root, or a cgroup subtree delegated to your user
Python 3 for the workload and the analysis scripts
Check for swap first: swapon --show. Our results are for a host with no swap. With swap, the kernel can move anonymous memory out, and both limits behave differently

Two limits, two code paths

Both files live in a cgroup v2 directory. Both act when the kernel charges a new page to the cgroup. They do different things when the cgroup is over its value.

memory.max is the hard limit. The kernel docs for 6.8 say: "If a cgroup's memory usage reaches this limit and can't be reduced, the OOM killer is invoked in the cgroup." The charge path tries to reclaim first, retries, and calls the OOM killer only when reclaim makes no progress.
memory.high is the throttle. The same docs say: "Going over the high limit never invokes the OOM killer and under extreme conditions the limit may be breached." The charge always succeeds. On the way back to user space, mem_cgroup_handle_over_high() tries to reclaim, and if usage is still over the value it puts the task to sleep.

The sleep length comes from calculate_high_delay() in mm/memcontrol.c. On the Ubuntu 6.8.0-147 kernel we ran, with HZ=1000, it works out to:

o = (usage - memory.high) / memory.high
penalty = 64000 * o^2 ms per charge batch of 64 pages (MEMCG_CHARGE_BATCH)
          no sleep if penalty <= 10 ms; at most 2000 ms (MEMCG_MAX_HIGH_DELAY_JIFFIES = 2 * HZ)

So the sleep is 0 until usage is 1.25% over memory.high, 160 ms at 5% over, 640 ms at 10% over, and it hits the 2 s cap at 17.7% over. The comment above the cap reads "Clamp the maximum sleep time per allocation batch to 2 seconds." Each reclaim_high() call adds 1 to the high counter in memory.events. When reclaim frees nothing, the kernel calls it 17 times before it sleeps (once, then MAX_RECLAIM_RETRIES = 16 retries). When reclaim frees something, it keeps retrying and does not sleep. So in a cgroup with nothing to reclaim, the counter rises by 17 per sleep, and that is how we count sleeps from outside the kernel.

page fault charge to the cgroup
over memory.max before the charge
over memory.high after the charge
reclaim direct, in the fault
reclaim_high x17 +17 to memory.events high
OOM kill nothing reclaimable
sleep 64000 x o^2 ms, at most 2 s
thrash reclaim frees the task's own code

Connections:

page fault -> over memory.max
page fault -> over memory.high
over memory.max -> reclaim
over memory.high -> reclaim_high x17
reclaim -> OOM kill (freed nothing)
reclaim -> thrash (freed file pages)
reclaim_high x17 -> sleep (freed nothing)
reclaim_high x17 -> thrash (freed file pages)

Without swap, the kernel has nowhere to put anonymous memory: heaps, stacks, anything a process allocated and wrote. In our runs that memory was never reclaimed (pgsteal stayed 0 in every warm anonymous run). What reclaim could take were file pages: page cache and the mapped code of the program in the cgroup. That decides most of what follows.

The lab

One host: Ubuntu 24.04, kernel 6.8.0-147-generic, systemd 255, 4 vCPU, 7.75 GiB of RAM, no swap device and zswap off, CONFIG_HZ=1000, transparent huge pages on madvise (results/00-host.txt). It is a shared development machine. So every run happened inside one transient systemd scope with Delegate=yes and MemoryMax=1G, every run got its own child cgroup, and no limit went above 1 GiB.

The workload is one script, scripts/lib/workload.py. Each operation maps a 256 KiB private anonymous chunk and writes all of it. That is 64 pages, which is also the kernel's charge batch. Each operation also copies one random chunk it already holds, so the working set stays in use. The phases:

Grow until the workload holds its target.
Churn 200 times: free one random chunk and allocate a new one, so usage stays flat while the program keeps allocating, like a busy service.

The random choices are seeded (mulberry32). Before and after every operation, the workload reads memory.current and the high and max counters in memory.events from open file descriptors, and writes one JSON line with the latency. A driver outside the cgroup samples memory.current, memory.events, memory.pressure, io.pressure, memory.stat, cpu.stat and each process's /proc/<pid>/wchan and /proc/<pid>/stat every 100 ms. The driver pinned itself to CPU 0, and the workload inherited that, so in the main matrix the two shared one CPU. We only saw that in review, and it changed one of the results a lot. The section on how many sleeps a task pays has the runs that measure it.

The two limits are fixed for every run:

memory.max = 268435456 bytes (256 MiB)
memory.high = 241590272 bytes (230.4 MiB). That is the value the kubelet computes with MemoryQoS for limits.memory: 256Mi, requests.memory: 0 and the 1.36 default throttling factor of 0.9. It is the lowest memory.high that formula gives for a 256Mi limit. More on that below.

Run	Workload	What it shows
`anon-fit-*`	grows to 232 MiB (about 239 MiB in `memory.current`), then churns	a program that fits under `memory.max` but not under `memory.high`
`anon-over-*`	asks for 320 MiB	a program that needs more than the limit, like a leak
`anon-over-4p-high`	the same 320 MiB split over 4 processes	whether the throttle is per process
`anon-fit-high-populate`	`mmap` with `MAP_POPULATE`	the same bytes, charged in one system call per chunk
`anon-clamp-high`	grows to 300 MiB, then sets its own `memory.high` so its usage is 33% over it	the 2 s cap
`*-owncode`	a fresh copy of the interpreter binary	a program whose code pages are charged to its own cgroup
`file-over-*`	reads 320 MiB of a 512 MiB file	page cache instead of anonymous memory
`anon-over-both-long`	`anon-over` with both limits, 45 minute cap	how long until the OOM kill

Each of none, max, high and both ran 3 times, in a fixed order per repetition, with a 150 second cap. The owncode runs thrashed the disk in the smoke tests, so we capped them at 20 seconds. We also capped every run's disk reads and writes at 100 MiB/s with io.max on its own cgroup, so that a thrashing run would not take the shared disk from other work. One run, anon-over-max-owncode-fastdisk, keeps the uncapped disk.

The-DevOps-Daily/cgroup-memory-high-vs-max on GitHub

We wrote the claim and what would disprove it before the first run and committed it first (CLAIM.md). The smoke runs then changed the design twice, and we added those changes to the same file before the main matrix. Every run directory carries a hash of the run code. The summary script refuses a batch with mixed hashes, or a hash that matches no commit in the repository.

memory.max: full speed, then the kill

With only memory.max, the workload never slowed down. In the three anon-over-max runs the median operation took 0.20 to 0.27 ms, the p99 of the last 10% of operations before the kill was 0.29 to 0.41 ms, and the kernel killed the process 0.28 to 0.39 seconds after it started (results/summary.md). In each run memory.events counted 35 times that a charge hit the limit, then one OOM kill. The kernel log names the cgroup:

oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=ddmemhigh-20261105T112606Z.scope,mems_allowed=0,oom_memcg=/ddmemhigh.slice/ddmemhigh-20261105T112606Z.scope/anon-over-max-r1,task_memcg=/ddmemhigh.slice/ddmemhigh-20261105T112606Z.scope/anon-over-max-r1,task=python3.12,pid=142949,uid=0
Memory cgroup out of memory: Killed process 142949 (python3.12) total-vm:276772kB, anon-rss:261504kB, file-rss:7296kB, shmem-rss:0kB, UID:0 pgtables:580kB oom_score_adj:0

Nothing in the cgroup could be reclaimed. The cgroup held only the program's anonymous pages, and without swap the kernel cannot free those. So reclaim found nothing, the charge path ran out of retries, and the OOM killer ran. Memory pressure stayed at 0: the task never waited for memory. It was killed.

The anon-fit-max runs are the other half of the same result. That workload needs 239.2 MiB, which fits under 256 MiB. It finished all 200 churn operations with a median of 0.277 ms per churn operation and a max counter of 0. With no limit, the same median was 0.257 ms. If a program fits under memory.max, the limit costs nothing.

memory.high: slower and slower, not killed

With only memory.high, the same 320 MiB workload was not killed in any of the 3 runs within the 150 second cap. The kernel docs say memory.high never invokes the OOM killer, and our driver stopped each run at the cap. Once it was a few MiB over the limit, it moved at the speed of the kernel's sleep timer, not at the speed of a program:

Latency of one 256 KiB allocation vs memory.current

	226	227	228	229	230	231	232	233	234	235	236	237	238	239	240	241	242	243	244	245	246	247	248	249	250	251	252	253	254	255	256
no limit	0.248ms	0.252ms	0.249ms	0.251ms	0.246ms	0.253ms	0.242ms	0.26ms	0.24ms	0.252ms	0.25ms	0.256ms	0.243ms	0.24ms	0.213ms	0.226ms	0.238ms	0.242ms	0.235ms	0.215ms	0.223ms	0.224ms	0.225ms	0.214ms	0.253ms	0.234ms	0.235ms	0.215ms	0.239ms	0.218ms	0.222ms
memory.max only	0.236ms	0.224ms	0.242ms	0.195ms	0.242ms	0.223ms	0.238ms	0.221ms	0.211ms	0.216ms	0.224ms	0.203ms	0.215ms	0.199ms	0.207ms	0.19ms	0.202ms	0.204ms	0.246ms	0.199ms	0.233ms	0.232ms	0.191ms	0.209ms	0.229ms	0.243ms	0.216ms	0.225ms	0.233ms	0.238ms
memory.high only, sampler on the same CPU	0.226ms	0.216ms	0.2ms	0.206ms	0.235ms	0.228ms	0.22ms	25.621ms	39.782ms	209.811ms	1525.021ms	4583.747ms	5839.866ms	7303.861ms	8839.941ms	10295.852ms
memory.high only, own CPU, sampler at 1 s	0.22ms	0.259ms	0.258ms	0.272ms	0.248ms	0.27ms	0.262ms	14.873ms	106.819ms	519.352ms	52.8ms	71.752ms	95.774ms	119.723ms	135.811ms	171.815ms	191.793ms	323.75ms	383.851ms	851.785ms	11251.414ms	379.811ms	1847.768ms	1759.826ms	1135.82ms	1727.849ms	3199.782ms
kernel formula, 1 sleep per 64 pages	0.243ms	0.243ms	0.243ms	0.243ms	0.243ms	0.243ms	0.243ms	11.243ms	20.243ms	31.243ms	44.243ms	60.243ms	79.243ms	99.243ms	123.243ms	148.243ms	176.243ms	206.243ms	239.243ms	274.243ms	312.243ms	352.243ms	395.243ms	439.243ms	487.243ms	536.243ms	588.243ms	643.243ms	700.243ms	759.243ms	821.243ms
kernel formula, 1 sleep per page	0.243ms	0.243ms	0.243ms	0.243ms	0.243ms	0.243ms	0.243ms	704.243ms	1280.243ms	1984.243ms	2816.243ms	3840.243ms	5056.243ms	6336.243ms	7872.243ms	9472.243ms	11264.243ms	13184.243ms	15296.243ms	17536.243ms	19968.243ms	22528.243ms	25280.243ms	28096.243ms	31168.243ms	34304.243ms	37632.243ms	41152.243ms	44800.243ms	48576.243ms	52544.243ms

1 ms: 1ms

x axis: memory.current in MiB when the op started (memory.high = 230.4 MiB, memory.max = 256 MiB). Median per 1 MiB over 3 repetitions, warm code, Ubuntu 6.8.0-147, no swap. 'Sampler on the same CPU' is the main matrix (anon-over-high: the workload inherited the driver's CPU 0, sampled every 100 ms); 'own CPU' is diag-over-high-cpu2-1s (workload on CPU 2, sampled every 1 s). memory.max only stays flat up to 256 MiB, then the OOM kill. The formula lines add the no-limit median to the kernel's sleep: once per 64-page op, or once per 4 KiB page. Source: results/runs/anon-over-/ops.jsonl, results/diag/diag-over-high-cpu2-1s-/ops.jsonl, scripts/50-charts.py.

In the matrix runs (the orange line), up to 232 MiB an operation took about a quarter of a millisecond, the same as with no limit. Within the next 5 MiB the median went from 0.22 ms to 4.6 seconds, and at 241 MiB it was 10.3 seconds per 256 KiB. After 150 seconds memory.current was 241.5 to 242.2 MiB, 11 to 12 MiB over memory.high, and still growing by 25 to 30 KiB per second. The process used 0.59 to 0.72 seconds of CPU in those 150 seconds. In 98% to 99% of the samples it was in state D, in mem_cgroup_handle_over_high. The purple line is the same workload on a CPU of its own. It stays near the one-sleep-per-batch line for most of the range, with jumps when it falls into a slow phase. The section on how many sleeps a task pays explains both lines.

Here is the same workload through systemd's MemoryMax= and MemoryHigh=, without our driver. The progress line shows the slowest operation in each second:

results/25-demo.txt (no driver, no CPU pinning)

# commands as scripts/25-demo.sh prints them (it also passes --quiet --slice=ddmemhigh.slice); 320 MiB target
$ systemd-run --scope -p MemoryMax=256M python3 -I scripts/lib/workload.py --kind anon --target-mib 320 --progress
pid 173416 in /sys/fs/cgroup/ddmemhigh.slice/run-r765e1a07143e4aa1be21a9eb2292aaff.scope, memory.high=max memory.max=268435456
   0.0s held 0 MiB memory.current 7.0 MiB ops 1 slowest op 0.33 ms
scripts/25-demo.sh: line 17: 173416 Killed systemd-run --scope --quiet --slice=ddmemhigh.slice -p MemoryMax=256M $w
exit status 137
$ systemd-run --scope -p MemoryHigh=241590272 -p MemoryMax=256M python3 -I scripts/lib/workload.py --kind anon --target-mib 320 --progress --max-seconds 30
pid 173417 in /sys/fs/cgroup/ddmemhigh.slice/run-rbd28bd3d399c47a9ab104c3b6d4b6eaf.scope, memory.high=241590272 memory.max=268435456
   0.0s held 0 MiB memory.current 6.7 MiB ops 1 slowest op 0.30 ms
   1.0s held 231 MiB memory.current 238.2 MiB ops 924 slowest op 79.86 ms
   7.1s held 231 MiB memory.current 238.5 MiB ops 926 slowest op 5488.05 ms
  12.8s held 231 MiB memory.current 238.8 MiB ops 927 slowest op 5759.67 ms
  19.0s held 232 MiB memory.current 239.0 MiB ops 928 slowest op 6135.79 ms
  25.5s held 232 MiB memory.current 239.3 MiB ops 929 slowest op 6511.84 ms
  27.5s held 232 MiB memory.current 239.8 MiB ops 930 slowest op 1992.06 ms
  28.5s held 234 MiB memory.current 241.1 MiB ops 936 slowest op 359.79 ms
  29.6s held 234 MiB memory.current 241.9 MiB ops 939 slowest op 759.73 ms
end: time after 30.4s, 940 ops, held 235 MiB, memory.current 242.1 MiB
exit status 0

The first process was killed before its second progress line, so in less than a second, with exit status 137. The second got 11.7 MiB past memory.high in 30 seconds, and its operations took 5.5 to 6.5 seconds each until they sped up near the end. The next section explains that change.

The anon-fit runs matter more, because nothing in them is wrong. The program needs 239 MiB, fits under its 256 MiB memory.max, and runs at full speed there. With memory.high at 230.4 MiB added, it held about 239 MiB and every churn operation slept. In the matrix, where it shared its CPU with our sampler, the median churn operation took 6,216 ms instead of 0.257 ms, and it finished 15 to 17 of its 200 churn operations before the 150 second cap. That is about 24,000 times slower per operation, with the same memory use and no error anywhere. On a CPU of its own it was about 750 times slower. The next section explains the difference. memory.max was never reached and its counter stayed at 0.

How many sleeps: it depends on who else runs on the CPU

To compare with the kernel, we count sleeps. In the warm runs reclaim never freed anything, so each sleep raised the high counter by 17, and an operation's sleep count is its rise divided by 17. We skip operations whose rise is not a multiple of 17 (11 to 13 per run, all at the edge of the limit, where the formula gives 10 ms or less and the kernel does not sleep). Then the measured latency divided by (sleeps x formula) should be 1.

It was 1.06 to 1.10 at the median in every single-process matrix run, and 97% to 100% of the throttled operations were within 30% of the formula (results/predict.md, section 1). The formula is right about how long each sleep is. The section on the 2 second cap shows where part of the extra 6% to 10% comes from.

The number of sleeps was the surprise. The kernel charges memory in batches of 64 pages and keeps the rest of a batch as a per-CPU precharge (consume_stock()), so the code comment on the cap says "per allocation batch". A 64-page operation should sleep about once. In the matrix, once the sleeps were 50 ms or longer, almost every operation slept 64 times, once per page:

Formula sleep length	Operations	Median sleeps per 64 pages	Operations with 64 or more
11 to 25 ms	83	2	24%
25 to 50 ms	95	7	23%
50 to 100 ms	228	64	92%
100 to 200 ms	73	64	100%
200 to 400 ms	22	64	100%
400 to 2000 ms	29	64	100%

A new charge, and so a new sleep, only happens when the precharge is empty. __refill_stock() empties a CPU's precharge when a different cgroup charges memory on that CPU. In review we found a cgroup that did exactly that in every run: our own driver. It pinned itself to CPU 0 before it started the workload, so every workload in the matrix inherited CPU 0 (the cpu field of every operation says 0). The driver woke up every 100 ms to sample and parsed the files into Python objects, which charges memory to the driver's own cgroup on that CPU. We did not trace each drain, but the runs below show the effect.

So we ran the 320 MiB memory.high workload again, 3 times each, with the workload on CPU 0 or CPU 2 and the sampler every 100 ms or every second (results/diag, 120 second cap):

Workload CPU	Sampler	Median sleeps per op (formula sleep 50 ms or more)	Share of that time in ops with 32 or more sleeps	`memory.current` at the cap
0, with the sampler	every 100 ms	64	100%	241.3 MiB
0, with the sampler	every 1 s	2	46%	248.8 to 249.8 MiB
2, alone	every 100 ms	1	53%	250.0 to 252.0 MiB
2, alone	every 1 s	1	72%	243.5 to 252.4 MiB

With the sampler on the same CPU every 100 ms, the task slept once per page for the whole run, as in the matrix: a median of 64 sleeps per operation, and 100% of the throttled time in operations with 32 or more sleeps. In the other three settings the median operation slept once or twice. But within each run the count switched between phases of about 1 sleep per operation and phases of 64, each lasting tens of seconds, and the slow phases still took 46% to 72% of the throttled time (results/predict.md, section 1c). That is consistent with other memory activity on the host emptying the precharge in those phases, but we did not trace it. The demo session further down ran with no driver and no pinning at all, and it also paid 5.5 to 6.5 seconds per 256 KiB operation for 25 seconds before it sped up.

So our sampler was not the only cause of the per-page cost. With it on the same CPU, the cost was constant. Read the matrix numbers as a task whose CPU also runs another cgroup that allocates memory every 100 ms. On its own CPU, the same anon-fit workload with memory.high took a median of 191.8 ms per churn operation in all 3 runs (diag-fit-high-cpu2-1s), and finished all 200 churn operations in 2 of the 3 runs within 120 seconds. That is about 750 times the no-limit median, instead of about 24,000 times.

MAP_POPULATE is a second control. It faults in the whole chunk inside the mmap() call, so the task returns to user space once per chunk. On CPU 0 with the 100 ms sampler, it slept 2 times in 592 of 600 churn operations and 3 times in the other 8, a median churn operation took 191.8 ms, and all 200 churn operations were done in about 41 seconds (anon-fit-high-populate). How a program touches its memory changed the cost of the same memory.high by a factor of 32 here, with the same neighbour on the CPU.

The 2 second cap

None of the runs above got far enough over memory.high to reach the cap, because the throttle slowed them first. So anon-clamp-high gets there another way: it grows to 300 MiB with no limit, then writes 241590272 to its own memory.high. Its usage of 307.2 MiB is then 33.3% over the limit. The formula then asks for a sleep of about 7,100 ms, and the kernel caps it. Each churn operation then allocates one 4 KiB page.

Run	Over `memory.high`	Formula without the cap	Sleeps per operation	Time per sleep, median	min	max
`anon-clamp-high-r1`	33.3%	7,106 ms	1, 2	2,047.9 ms	2,047.3 ms	2,064.9 ms
`anon-clamp-high-r2`	33.3%	7,108 ms	1, 2	2,047.9 ms	2,021.8 ms	2,048.0 ms
`anon-clamp-high-r3`	33.3%	7,107 ms	1, 2	2,047.9 ms	2,011.7 ms	2,047.9 ms

The median sleep took 2,047.9 ms in all three runs, not 2,000 ms. The cap is 2000 jiffies, and at HZ=1000 the kernel's timer wheel keeps timeouts from 512 ms to 4,095 ms in 64 ms steps (kernel/time/timer.c), so a 2,000 ms timeout fires at one of those steps. The median of 2,047.9 ms fits that. Single sleeps ranged from 2,011.7 to 2,064.9 ms, depending on where in a step the timer started. The same rounding, at 8 ms steps from 64 to 511 ms, explains part of the 6% to 10% we saw on the shorter sleeps: with half a step added, the median ratio for sleeps of 64 ms or more drops to 1.030 to 1.055 in the matrix runs. Under 64 ms, where the step is 1 ms, it stays at 1.054 to 1.091, and we did not find the rest (results/predict.md, section 1).

So the cap holds, but it is per sleep, not per allocation. In these runs each 4 KiB operation slept twice after the first one, so it took 4,096 ms. A 256 KiB allocation that sleeps once per page at the cap would take more than two minutes.

Both limits: high at 90% of max

systemd's man page recommends both: "use MemoryHigh= as the main control mechanism and use MemoryMax= as the last line of defense". The idea is that the program slows down before it dies. The anon-over-both runs show what that looks like with our 230.4 MiB and 256 MiB:

memory.current from 10 s to 150 s, the workload asks for 320 MiB

	10s	20s	30s	40s	50s	60s	70s	80s	90s	100s	110s	120s	130s	140s	150s
memory.high only, 1 process	237.3 MiB	237.9 MiB	238.4 MiB	238.8 MiB	239.2 MiB	239.6 MiB	239.9 MiB	240.2 MiB	240.5 MiB	240.8 MiB	241.1 MiB	241.4 MiB	241.6 MiB	241.8 MiB	242.1 MiB
memory.high only, 4 processes	242.3 MiB	243.4 MiB	244.5 MiB	245.6 MiB	246.1 MiB	246.6 MiB	247.1 MiB	247.5 MiB	247.9 MiB	248.3 MiB	248.7 MiB	249.1 MiB	249.4 MiB	249.8 MiB	250.1 MiB
memory.high + memory.max	236.8 MiB	237.5 MiB	238 MiB	238.5 MiB	238.9 MiB	239.3 MiB	239.7 MiB	240 MiB	240.3 MiB	240.6 MiB	240.9 MiB	241.2 MiB	241.4 MiB	241.7 MiB	241.9 MiB

memory.high: 230.4 MiB

memory.max: 256 MiB

Median of 3 repetitions at each 10 s mark, sampled from outside the cgroup every 100 ms. memory.max only reaches 256 MiB and is killed in under 2 s, so it is not drawn. Source: results/runs/anon-over-/samples.jsonl.*

Within the 150 second cap, memory.high plus memory.max behaved like memory.high alone, because the process did not get near memory.max. After 150 seconds all three runs were at 241.7 to 242.0 MiB, 14 MiB under the kill. The growth rate fits the model of one sleep per page: 4 KiB per sleep, at the formula's sleep length for the current usage, predicts 24.7 to 26.1 KiB/s at the end of the runs, and we measured 26.6 to 28.0 KiB/s over the last 60 seconds (results/predict.md, section 5).

To see when the kill comes, we ran the same workload once more with a 45 minute cap (anon-over-both-long, one run, the same setup as the matrix, so on CPU 0 next to the sampler). The kernel killed it after 1,747.7 seconds, 29.1 minutes:

memory.high at 90% of memory.max: the climb to the OOM kill

	1m	2m	3m	4m	5m	6m	7m	8m	9m	10m	11m	12m	13m	14m	15m	16m	17m	18m	19m	20m	21m	22m	23m	24m	25m	26m	27m	28m	29m
memory.current	239.7 MiB	241.4 MiB	242.8 MiB	243.9 MiB	244.8 MiB	245.7 MiB	246.4 MiB	247.1 MiB	247.8 MiB	248.4 MiB	248.9 MiB	249.5 MiB	250 MiB	250.5 MiB	250.9 MiB	251.3 MiB	251.7 MiB	252.1 MiB	252.5 MiB	252.8 MiB	253.2 MiB	253.5 MiB	253.9 MiB	254.2 MiB	254.5 MiB	254.8 MiB	255.1 MiB	255.4 MiB	255.7 MiB

memory.high: 230.4 MiB

memory.max: 256 MiB

One run, sampled every 100 ms, one point a minute. Source: results/runs/anon-over-both-long-r1/samples.jsonl.

In those 29 minutes the process used 1.19 seconds of CPU, its memory pressure some covered 1,742 seconds, and its slowest 256 KiB allocation took 53.2 seconds. From the point where the sleep starts (233.4 MiB, 0.3 seconds in), the formula with one sleep per page predicts 1,720 seconds to reach memory.max, 1.6% less than we measured. With one sleep per 64 pages it would have been 27 seconds (results/predict.md, section 3). Then the OOM kill came with 35 max events, as in the memory.max runs. memory.high did not save the program. It delayed the same OOM kill by half an hour.

On a CPU of its own (diag-over-both-cpu2-1s, 3 runs, 120 second cap), the same workload was not killed within the cap either. It reached 240.8 to 254.4 MiB, depending on how much of the run fell into the one-sleep-per-page phases. So we have two reference points, not a proven range: the formula's 27 seconds if every sleep covers 64 pages, and the 29.1 minutes we measured once next to the sampler.

The throttle is per process

memory.high puts each allocating task to sleep on its own. In the matrix, the four processes of anon-over-4p-high all ran on CPU 0 next to the sampler. We cannot split memory.events per process, but their combined growth matches the model of one sleep per page for each process: about four times as fast as one process at the same overage. In anon-over-4p-high the four processes reached 249.1 to 250.8 MiB in 150 seconds, against 241.5 to 242.2 MiB for one process. Over the last 60 seconds they grew 34.0 to 42.3 KiB/s, where the one-sleep-per-page model predicts 31.9 to 38.0 KiB/s for four processes at that usage. A program with many allocating worker processes gets to memory.max sooner than a single process.

The four throttled processes also changed a number most dashboards show. While they slept, the host's load average was 5.33 on 4 vCPUs, and vmstat showed the CPUs 96% and 99% idle in the two one-second samples:

results/observed/loadavg-during-4p-high.txt

# recorded during anon-over-4p-high-r1: ps, vmstat 1 3, /proc/loadavg, nproc
$ cat results/observed/loadavg-during-4p-high.txt
# 2026-11-05T11:39:12Z during anon-over-4p-high-r1 (ps, vmstat 1 3, /proc/loadavg)
 146187 D mem_cgroup_handle_over_high 0.2 /usr/bin/python3.12 -I /root/dd-memory-high/scripts/lib/wor
 146188 D mem_cgroup_handle_over_high 0.2 /usr/bin/python3.12 -I /root/dd-memory-high/scripts/lib/wor
 146189 D mem_cgroup_handle_over_high 0.2 /usr/bin/python3.12 -I /root/dd-memory-high/scripts/lib/wor
 146190 D mem_cgroup_handle_over_high 0.2 /usr/bin/python3.12 -I /root/dd-memory-high/scripts/lib/wor
procs -----------memory---------- ---swap-- -----io---- -system-- -------cpu-------
 r b swpd free buff cache si so bi bo in cs us sy id wa st gu
 0 0 0 1407320 32980 5042700 0 0 835 650 876 2 4 1 94 0 1 0
 0 0 0 1410512 32980 5042720 0 0 0 0 1627 1417 2 3 96 0 0 0
 0 0 0 1413104 32980 5042736 0 0 0 28 621 787 1 0 99 0 0 0
5.33 3.56 2.27 1/347 147320
4

The kernel puts the task to sleep with schedule_timeout_killable(). That is a killable sleep: only a fatal signal ends it early. ps shows it as state D, and Linux counts it in the load average like other tasks in state D. A high load average with idle CPUs can mean throttled memory, not only slow disks.

When the cgroup holds its own code, both limits thrash

So far the interpreter's code pages were in the page cache before each run started, charged to a different cgroup. That is the warm setup, and it is the reason the runs above had nothing to reclaim. A container on its first start is different: its binary and libraries are read from its image after it starts, so the page cache for its own code is charged to its own cgroup. The owncode runs copy the interpreter binary to a new file, drop it from the page cache (mincore() found 0 of its pages cached before each run), and run that copy, so the binary is read inside the run cgroup.

That gives the kernel something to reclaim, and it changes both limits:

Run (20 s cap)	Outcome	`memory.current`	Pages reclaimed	Read from disk	PSI memory `some`	PSI io `some`
`memory.high` only, 232 MiB target	not killed in 20 s (3 of 3)	230.4 MiB	591,486 to 617,906	2,426 to 2,535 MB	11.4 to 11.9 s	19.9 to 21.2 s
both, 320 MiB target	not killed in 20 s (3 of 3)	230.2 to 230.4 MiB	633,090 to 634,192	2,593 to 2,598 MB	12.4 to 14.3 s	21.3 to 21.5 s
`memory.max` only, 320 MiB target	not killed in 20 s (3 of 3)	255.9 MiB	631,197 to 633,258	2,585 to 2,594 MB	1.9 to 2.1 s	21.3 to 21.7 s
`memory.max` only, uncapped disk, 1 run	not killed in 20 s (1 of 1)	256.0 MiB	2,559,211	10,483 MB	6.2 s	13.6 s

Under memory.high, reclaim kept succeeding, and what it took was the program's own code. The task faulted it back in from disk, reclaim took it again, and so on. Because reclaim kept freeing pages, the kernel's retry loop never got to the sleep (if (nr_reclaimed || nr_retries--) retries as long as anything was freed). The sampler never found the process in mem_cgroup_handle_over_high, and only 1 of the 108 operations that raised the high counter raised it by a multiple of 17. The process sat at 230.2 to 230.4 MiB, right at memory.high. In 54% to 90% of the samples it was in folio_wait_bit_common, waiting for its code to come back from disk, and in the rest it was running. It never finished its grow phase.

Under memory.max alone, the same thing replaced the OOM kill. The charge path only calls the OOM killer after 16 tries in which reclaim freed nothing, and here reclaim kept finding code pages to free. The process held 255.9 MiB and was not killed in any of the three 20 second runs. memory.events counted 19,140 to 20,019 max events per run, and no OOM. On the uncapped disk, it read 10,483 MB in 25 seconds and was not killed either. In an earlier smoke run with the same setup (older code, in results/_smoke5), the kernel did kill it, 21.7 seconds after it started. So the kill does come in some runs, after a period of thrashing we could not predict. In another smoke run on the uncapped disk (results/_smoke3) it was still thrashing at 70 seconds when our driver stopped it.

Memory PSI under-counts this. In the memory.max runs with the 100 MiB/s cap, memory some was 1.9 to 2.1 seconds out of 25, about 8%, while io some was 21.3 to 21.7 seconds. The kernel counts a wait for a page read as a memory stall only when the page carries the workingset flag (folio_wait_bit_common() in mm/filemap.c), and most of this time went to the disk. A pressure-based killer that watches memory pressure alone would not see this as clearly as the warm memory.high runs, where memory some was 99%.

Page cache: neither limit throttles or kills

The file-over runs read 320 MiB of a file in 256 KiB reads, then read random chunks of it. That is page cache, which the kernel can drop and read again. None of the 12 runs was killed or throttled:

Run	Outcome	Peak MiB	Pages reclaimed	PSI memory `some`	p99 per read
no limit	not killed (3 of 3)	327.7	0	0.00 s	82.1 to 86.5 ms
`memory.max` only	not killed (3 of 3)	256.0	20,608 to 21,440	0.02 s	83.4 to 88.6 ms
`memory.high` only	not killed (3 of 3)	230.9	28,352 to 28,864	0.03 s	85.0 to 87.5 ms
both	not killed (3 of 3)	230.9	28,352 to 28,864	0.02 to 0.03 s	82.9 to 89.9 ms

Both limits did what the docs say: memory.current stopped at the limit and the kernel reclaimed file pages to stay there. memory.events counted 416 to 430 high events and 322 to 335 max events per run. Every high event was a single reclaim pass with no sleep: the rise per operation was 1 or 2, never 17, because each pass freed enough to get back under the limit. The p99 of 82 to 90 ms is the same with no limit, so it is not the memory limits. The slow reads are in the first pass over the file, read from disk under our 100 MiB/s cap. The limits did cost something later: in the last 10% of operations, random reads that had to go back to disk gave a p99 of 1.0 to 2.6 ms, against 0.13 ms with no limit, where the whole file stayed cached. For a cache-heavy program without swap, both limits are a cache size, not a throttle and not a kill.

What PSI and the counters show

PSI was a reliable witness for the warm throttle. In every warm single-process matrix run with memory.high, the cgroup's memory.pressure some total matched the time the operations spent over their normal latency, at a ratio of 0.998 to 1.068 (results/predict.md, section 4). In the diagnostic runs it was 0.997 to 1.286. PSI also counts the last operation, which was still asleep when the driver stopped the run, and our operation log does not, so the ratio goes above 1 in runs that ended in the middle of a long operation. With one task in the cgroup, full was the same as some, and avg10 reached 97% to 99%. The host saw it too. During the long run, the host-wide /proc/pressure/memory read some avg10=96.53 and full avg10=95.09 (results/observed/host-psi-during-long.txt). Our test cgroup was the one we know was stalled, and the host-wide number was almost as high, so an alert on host memory pressure would fire.

The counters to watch, from these runs:

memory.events high rising: the cgroup is over memory.high. 17 per sleep when nothing is reclaimable.
memory.events max rising without oom_kill: the cgroup is at memory.max and reclaim keeps finding something. In the own-code runs that meant thrashing.
memory.stat pgsteal and workingset_refault_file rising together: the cgroup reclaims pages and reads them straight back.
memory.pressure some near 100% with almost no CPU use and state D: throttled by memory.high.
io.pressure high while memory.pressure is low, in a cgroup at its memory limit: thrashing on file pages.

What systemd, Docker and Kubernetes write

systemd

systemd maps its properties one to one. We started transient scopes with each property and read the files back from inside (results/10-systemd-map.txt, systemd 255):

Property	File	Value we set	Value in the file
`MemoryMin=16M`	`memory.min`	16M	16777216
`MemoryLow=64M`	`memory.low`	64M	67108864
`MemoryHigh=241590272`	`memory.high`	241590272	241590272
`MemoryMax=256M`	`memory.max`	256M	268435456
`MemoryHigh=3%`	`memory.high`	3% of RAM	249806848

A property you do not set stays at the kernel default: max for memory.high and memory.max, 0 for memory.min and memory.low. memory.oom.group stayed 0 in every scope.

The systemd 255 man page says MemoryHigh= is "the main mechanism to control memory usage of a unit", and it recommends MemoryMax= "as the last line of defense". On a host with swap, the kernel can move cold anonymous pages out and hold the unit near the limit. We did not test that. Without swap, our runs show what "main mechanism" means: the unit does not stop at the limit, it slows down, and with MemoryHigh= alone the kernel never kills it.

MemoryMin= and MemoryLow= are the other direction. They protect a unit's memory from reclaim when something else needs it. They do not limit or slow the unit itself.

Docker

Docker 29.9.3 has no flag for memory.high. docker run --help on the test host lists --memory, --memory-reservation, --memory-swap, --memory-swappiness, --oom-kill-disable and --oom-score-adj (results/00-host.txt). We followed the two memory flags through the source (results/01-source-check.txt):

Docker puts --memory into the OCI spec as memory.limit and --memory-reservation as memory.reservation (getMemoryResources in moby daemon/daemon_unix.go, docker-v29.9.3).
runc 1.5.2, the host's runtime, writes the limit to memory.max and the reservation to memory.low on cgroup v2 (fs2/memory.go). With the systemd cgroup driver it sets the MemoryMax and MemoryLow properties (systemd/v2.go).

So on cgroup v2, --memory is a hard limit and --memory-reservation is reclaim protection. The Docker docs call --memory-reservation "a soft limit smaller than --memory which is activated when Docker detects contention or low memory on the host machine." That wording comes from cgroup v1's soft limit. On v2 it is memory.low: it does not slow the container down at any size.

runc does accept memory.high through the OCI spec's unified map, and passes it on as the systemd MemoryHigh property. That is the path the kubelet uses. The Docker CLI does not expose it, so with plain docker run you only get the memory.max behaviour.

Kubernetes MemoryQoS

The kubelet sets memory.max from limits.memory on cgroup v2 in every version. memory.high is a separate feature, MemoryQoS (KEP-2570), and its status changed in the last two releases. All of this is from results/01-source-check.txt:

	Kubernetes 1.36 (v1.36.6)	Kubernetes 1.37 (v1.37.2)
`MemoryQoS` feature gate	alpha, off by default	beta, on by default
`memoryThrottlingFactor` default	0.9	nil: no `memory.high` unless you set it
`memoryReservationPolicy` default	`None`: `memory.min` and `memory.low` are 0	`None`

With the gate on and a factor set, the kubelet computes, for each container whose memory request differs from its limit:

memory.high = floor((requests.memory + factor * (limits.memory - requests.memory)) / pageSize) * pageSize

With no limit, it uses node allocatable memory instead of limits.memory. It puts the value in the CRI container config's unified map under the key memory.high (kuberuntime_container_linux.go), and the runtime writes it to the container's cgroup. runc passes a unified memory.high on as the systemd MemoryHigh property when it uses the systemd driver. In 1.36, a container whose memory request equals its limit gets no memory.high. 1.37 makes one exception: a container with no request and no limit can still get one when a factor is set.

Our memory.high value is this formula for limits.memory: 256Mi, requests.memory: 0 and factor 0.9: 241590272 bytes. Two details matter here. If a pod sets a limit and leaves the request out, the API server sets the request equal to the limit (SetDefaults_Pod in pkg/apis/core/v1/defaults.go), and the kubelet then writes no memory.high at all. And with a request R between 0 and the limit, memory.high is R + 0.9 x (256Mi - R), so it sits between 230.4 MiB and 256 MiB. Our runs are the strictest case: a Burstable container with an explicit zero memory request on a 1.36 node with the MemoryQoS gate turned on. The anon-fit rows are the ones to read for that.

The KEP explains why it took so long. The 1.28 beta was cancelled because a workload "stays stuck indefinitely" at memory.high, and the KEP says kernel 5.9 fixed that livelock. On 6.8 our warm runs did make progress, slowly. The own-code runs barely did. The 1.37 beta turns the gate on but leaves memoryThrottlingFactor unset, so upgrading does not add memory.high to any container. If you set the factor to 0.9, you opt into the both rows above.

The two Kubernetes posts this week cover the other side of the same cgroups: an empty resources: {} makes pods BestEffort on 1.34 to 1.36 and changes their oom_score_adj and cgroup slice, and the 1.37 review lists what else moved in that release.

What we could not conclude

One host, one kernel, no swap. Everything here is Ubuntu 6.8.0-147 on a 4 vCPU machine with no swap. With swap or zswap, reclaim can move anonymous pages out, so memory.high can hold a cgroup near the limit instead of only slowing it, and memory.max reaches the OOM killer later. We did not test that. Newer kernels have changed the memcg charge code, and we did not test them either.
What empties the precharge. We measured how many sleeps a task paid, and we read in the source what makes a new charge necessary. We did not trace which event emptied the per-CPU precharge each time. Our sampler on the same CPU made it happen on almost every long sleep. Without it, the phases came and went with the host's other work, which we did not control. An idle machine, or one with more CPUs, could pay fewer sleeps.
The rest of the extra per sleep. The timer wheel's rounding explains about half of the 6% to 10%. We did not split the rest into reclaim time, wake-up delay and our own probes.
When the own-code thrash ends. Under memory.max it was not killed in 20 seconds in the matrix (one smoke run was killed at 21.7 seconds, another was still running at 70). We stopped these runs early because they read about 100 MB/s from a shared disk, so we cannot say how long a real container would thrash before the kill. The result also depends on disk speed, which we changed with io.max.
Run order and a shared host. Every repetition ran the configurations in the same order, on a machine with other services. Background load can move a single run either way. The three repetitions agree closely, but we did not randomize the order.
Real programs. Our workload is a Python loop that touches memory in a fixed pattern. A JVM, a Go program or a database touches memory differently, and the MAP_POPULATE run shows that the touch pattern alone changed the cost 32 times.
The 4-process numbers per operation. memory.events counts the whole cgroup, so in the 4-process run one operation's rise in high includes the other processes' sleeps. We used that run only for growth, not for sleeps per operation.
Kubernetes on a real node. We wrote the same memory.high value the kubelet would write, but we did not run a kubelet. The Kubernetes section is from the source at v1.36.6 and v1.37.2.

What to do with this

Without swap, do not use memory.high as the only limit. It does not stop growth. It slows new allocations, and a program that keeps allocating stays slow as long as it stays over the limit. The kernel never kills it for memory.high, so add memory.max or a pressure-based killer.
If you set both, leave room. With memory.high at 90% of memory.max, a program that fits between the two ran about 750 times slower per allocation on a CPU apart from our sampler, and about 24,000 times slower next to it. Apart from the sampler it finished in 2 of 3 runs within 120 seconds; next to it, none finished within 150 seconds. Measure your program's peak first. Put memory.high above the normal working set, not at a fixed fraction of the limit.
Do not plan with the formula alone. It gives the length of one sleep. How many sleeps a task pays depends on what else allocates memory on the same CPU, by up to 64 times, so test on a host that is as busy as production.
Read memory.events and memory.pressure, not only memory.current. A throttled cgroup looks healthy by memory use: it sits a few MiB over its limit and uses no CPU. The high counter and memory some near 100% show it at once.
Watch io.pressure and pgsteal on cgroups at their limit. Thrashing on a program's own code showed up there, not in memory pressure.
On Kubernetes 1.37, MemoryQoS is on but does not throttle until you set memoryThrottlingFactor. If you set it, test your Burstable pods whose usage sits between the factor and the limit.

The scripts, the raw output of every run, and the claim we wrote before the first run are in the repository above.

Originally published at devops-daily.com.

Alpine OpenSSH Password Bypass (CVE-2026-123158): Check and Fix

DevOps Daily — Thu, 05 Nov 2026 21:00:00 +0000

If you run sshd on Alpine Linux, upgrade openssh and restart sshd now:

apk upgrade -U -a && rc-service sshd restart

On 2026-11-05, Alpine published fixed openssh packages for every branch from 3.16 to edge, and OpenSSH released 10.6p2. The bug is CVE-2026-123158. On an affected host, a person who can reach the SSH port can log in to an account that has no password, without a password. The cause, in one sentence: when sshd is built without PAM, an error value from musl's crypt() could be treated as a match for accounts that have no password.

This post tells you how to find out if a host or a container is affected, how to fix it, and what to look for in your logs. It does not show how to trigger the bug.

TLDR

You are affected only if four conditions are all true. In our tests, a fresh Alpine container with openssh-server failed two of them.
The fix is apk upgrade -U -a, then a restart of sshd. The upgrade alone does not replace the running daemon.
If you cannot upgrade, set PasswordAuthentication no and restart sshd.
Our read-only script checks the four conditions, the package version and the restart for a host or for every running container.

Prerequisites

Root on the Alpine host, or access to docker exec for containers
A second SSH session open before you change sshd settings

Who is affected

The Alpine advisory says you are affected only if all of these are true:

sshd is built without PAM. The openssh-server package is. The openssh-server-pam package is affected too if UsePAM is no, which is the OpenSSH default when the config does not set it.
PasswordAuthentication yes , which is the OpenSSH default.
An account has * in its password field. On Alpine base images root is root:*, but the default PermitRootLogin prohibit-password blocks password logins for root. Key-only users, the alpine user on tiny-cloud images, and directory users through musl-nscd (they have no shadow entry) also count.
At least one local account has a $5$ or $6$ hash. One user with a password is enough.

You are not affected if PasswordAuthentication is no, if you use openssh-server-pam with UsePAM yes, if no account has a $-style hash, or if your containers do not run sshd. Locked accounts (! in the password field) are not affected. Instances where cloud-init sets PasswordAuthentication no are not affected.

These are the fixed openssh versions from the advisory:

Branch	Fixed version	Branch	Fixed version
edge	10.6_p2-r0	3.20	9.7_p1-r6
3.24	10.3_p1-r2	3.19	9.6_p1-r3
3.23	10.2_p1-r1	3.18	9.3_p2-r4
3.22	10.0_p1-r11	3.17	9.1_p1-r7
3.21	9.9_p2-r1	3.16	9.0_p1-r6

Check one host

We wrote a small POSIX sh script that reads these facts and changes nothing:

The-DevOps-Daily/alpine-sshd-check on GitHub

It reads the branch from /etc/alpine-release, the installed package version from apk, the effective settings from sshd -T, and whether the running sshd is still the sshd binary on disk. From /etc/shadow it reads only the type of each password field: it counts * fields and $5$ / $6$ fields. It never prints a hash or a user name, and it does not connect to sshd. Because it runs as root, it executes only the packaged sshd binaries in /usr/sbin, and only if root owns them and no one else can write to them. If openssh can come from edge on a stable host (an untagged edge repository, or a pin such as openssh-server@edge), it compares against the edge fix, and it says "cannot tell" when it cannot find where the package came from. Exit codes are 0 not affected, 1 affected, 2 fixed but restart needed, 3 cannot tell. Add --json for machine-readable output.

sudo sh check.sh

We ran it in throwaway containers on Alpine 3.21, 3.22, 3.23, 3.24 and edge after apk add openssh-server. sshd ran with --network none, so no port was open to anyone. Every branch installed the fixed version, and every run exited 0. The run below is the closest one to an affected host: Alpine 3.24 with a key-only user set to * and one user with a random password, so all four conditions are true. The version is the only thing that keeps it safe:

check.sh in the Alpine 3.24 test container

# all four conditions true, fixed package installed
$ docker exec -i asc-users sh -s < check.sh
alpine-sshd-check: CVE-2026-123158
Alpine release: 3.24.4 (branch 3.24)
sshd binary: /usr/sbin/sshd (package openssh-server)
Installed openssh: 10.3_p1-r2
Fixed in: 10.3_p1-r2 (installed is fixed: yes)
[1] non-PAM sshd: yes
[2] PasswordAuthentication yes: yes
[3] account with '*' that can use a password login: yes (2 '*' accounts, 1 usable; root is '*': yes, PermitRootLogin prohibit-password)
[4] local $5$/$6$ hash: yes (1 found)
sshd running: yes, started 2026-11-05T21:49:52Z, binary on disk changed 2026-11-05T21:49:35Z, restart needed: no
note: root is '*' but PermitRootLogin is 'prohibit-password'; root counts only with PermitRootLogin yes (an account that can doas to root still matters)
Result: NOT AFFECTED: fixed openssh installed (exit 0)

Two things we found while we ran it:

On 3.24, adduser -D deploy wrote ! (locked), not *. A locked account is not affected, but it cannot log in with a key either. If you changed key-only users to * so that key login works, those are the accounts that matter.
OpenSSH 10.6 prints sshd -T keywords in mixed case (PasswordAuthentication yes). Older versions print lower case. The first version of our script matched only lower case and reported "unknown" on edge. Use grep -i in your own checks.

Check containers

Containers that do not run sshd are not affected. To find the ones that do, scan-containers.sh lists the running containers, skips those that are not Alpine or have no sshd process, and pipes check.sh into the rest with docker exec -i ... sh -s. Nothing is copied into the container. Because it runs each container's own sh as root inside that container, scan only containers whose images you trust.

scan-containers.sh

$ sh scan-containers.sh
CONTAINER IMAGE ALPINE INSTALLED FIXED EXIT RESULT
asc-users asc-demo:3.24 3.24.4 10.3_p1-r2 10.3_p1-r2 0 not affected
asc-pa-no asc-demo:3.24 3.24.4 10.3_p1-r2 10.3_p1-r2 0 not affected
asc-edge asc-demo:edge 3.25.0_alpha20260805 10.6_p2-r0 10.6_p2-r0 0 not affected
asc-3.24 asc-demo:3.24 3.24.4 10.3_p1-r2 10.3_p1-r2 0 not affected
asc-3.23 asc-demo:3.23 3.23.8 10.2_p1-r1 10.2_p1-r1 0 not affected
asc-3.22 asc-demo:3.22 3.22.8 10.0_p1-r11 10.0_p1-r11 0 not affected
asc-3.21 asc-demo:3.21 3.21.10 9.9_p2-r1 9.9_p2-r1 0 not affected

If you build images that install openssh-server, a rebuild can reuse a cached apk add layer with the old package. Rebuild with docker build --pull --no-cache, then replace the running containers. A restart of the old container does not change its package.

Fix it, and restart sshd

apk upgrade -U -a
rc-service sshd restart

The restart is not optional. The package upgrade replaces the files on disk, but it does not restart the sshd that is already running, and the advisory says to restart it after the upgrade. check.sh checks whether the running sshd is still the same file as the sshd binary on disk. An upgrade writes a new file, so a daemon that was not restarted is a different file, and the script exits 2. Run it again after the restart.

On edge, the fix comes with OpenSSH 10.6p2. If you jump from an older 10.x, read our post on what OpenSSH 10.6 changed first: it refuses some usernames and changes ssh -C.

Workaround: turn off password logins

If you cannot upgrade now, turn off password authentication. Either step closes the hole.

Warning

Make sure that key login works first. Keep your current session open, and test a new key login in a second session before you close it.

mkdir -p /etc/ssh/sshd_config.d
echo 'PasswordAuthentication no' > /etc/ssh/sshd_config.d/00-no-passwords.conf
sshd -t && rc-service sshd restart
sshd -T | grep -i passwordauthentication # must print: passwordauthentication no

Alpine's default sshd_config reads /etc/ssh/sshd_config.d/ from 3.19 onward. On 3.18 and older, change the line in /etc/ssh/sshd_config itself; the advisory has a sed command for that. A Match block can turn password logins back on for some users or addresses, and sshd -T without -C does not show that, so read your Match blocks too. In our 3.24 test container, the drop-in changed condition 2 to no.

The tiny-cloud alpine user

The official tiny-cloud images (qcow2, VHD and AMI) from 3.18.11 to 3.24.3 have an alpine user with * in the password field and passwordless doas. They are safe on first boot. They become exposed as soon as any $6$ hash is on the machine, for example when an admin sets a console password or user-data adds a user with hashed_passwd. Then an attacker can get a shell as alpine and use doas to become root.

Images from 3.21.10, 3.22.8, 3.23.8 and 3.24.4 contain the fixed package. Instances that started from older images do not get the fix from a new image: upgrade openssh on each one, or turn off password logins.

What to look for in logs

Look for password logins to accounts that have no password. On a host with syslog, that is /var/log/messages. For a container that runs sshd -e, it is docker logs.

# accounts with '*' in the password field (names only)
awk -F: '$2 == "*" { print $1 }' /etc/shadow

# password logins; compare the user names with the list above
grep 'sshd.*Accepted password for' /var/log/messages
docker logs <container> 2>&1 | grep 'Accepted password for'

Any Accepted password for alpine, for root, or for a key-only user is a bad sign. On exposed machines, also review doas use, new users, new files, and changes to authorized_keys files. If you cannot rule out such a login, treat the machine as compromised and rebuild it.

What we could not check

A vulnerable package or a real upgrade. Alpine replaced the packages on every branch the same day, so each fresh install got the fixed version. Exit codes 1 and 2 are simulations in the repo's test-exit-codes.sh: a copy of the script with a raised fixed version, and the same sshd binary renamed over /usr/sbin/sshd while sshd runs. Both gave the expected code.
openssh-server-pam, musl-nscd users, tiny-cloud images and branches 3.16 to 3.20. We did not measure them.
Other musl systems. The CVE record covers sshd before 10.6p2 on Linux with musl and without PAM, and names Alpine as the example. We did not check other musl-based distributions or firmware builds.

Sources: the Alpine advisory, the OpenSSH 10.6p2 release notes and the CVE record.

Originally published at devops-daily.com.

OpenTelemetry Collector Custom Builds: 6.5x Smaller, and What Resource Detectors Cost at Startup

DevOps Daily — Thu, 05 Nov 2026 15:00:00 +0000

The contrib release of the OpenTelemetry Collector 0.164.0 is a 420.3 MB binary on linux/amd64. We built a Collector with the OpenTelemetry Collector Builder (ocb) that has only the 10 components a VM agent uses and only the env and system resource detectors. It is 64.3 MB, so contrib is 6.5 times larger. After 10 seconds idle, the custom build held 45.1 MiB of resident memory and contrib held 199.8 MiB. The anonymous part (heap, stacks and runtime data) was 8.0 MiB against 34.8 MiB.

The binary adds a little to startup: contrib needed 110 to 160 ms more than the small build on the droplet, and 300 to 500 ms more on the Pi. The detector list in your config can add seconds. We started both builds with cloud detectors for clouds the host was not on, in four network setups. In every list without akamai, each start waited for the slowest detector. When that detector got no answer, it waited for the whole timeout, which is 5 s by default. That happened on a Raspberry Pi with no metadata service. With a list of 13 cloud detectors it also happened on a DigitalOcean droplet, because two of them ask addresses that DigitalOcean does not answer. The detectors run in parallel, so where nothing answered, the list with 3 cloud detectors and the list with 13 cost the same 5 s.

timeout: 1s cut the wait to about 1 s in most cases. Two detectors do not obey it. digitalocean ignores the deadline, so where nothing answers it waits its own 2 s. akamai makes its network call while the processor is created. In all four setups we tested, none of them on Akamai, the Collector did not start. It exited at once where the request got an answer or a fast error, and after 30 s where it got no answer.

On 29 October the OpenTelemetry blog showed how to compile detectors out with ocb and reported a Kubernetes distribution about 20% smaller. This post measures three things that page does not cover. What does the same filter do to a small build? What does the build do to memory at idle? And what does the detector list cost at startup on hosts that are not on the clouds the config names? Every number below comes from a file in the companion repo.

TL;DR

Size. On amd64, contrib is 420.3 MB, the core otelcol release is 183.8 MB, our agent with every detector is 124.1 MB, and the same agent with only env and system is 64.3 MB. On this small build, the detector filter alone removed 48% of the binary and 64 Go modules: the AWS SDK, the Google Cloud clients, the Docker client, the Consul API and four more cloud metadata clients. arm64 gives the same picture.
Memory. After 10 s idle: contrib 199.8 MiB RSS, core 98.4 MiB, our agent 61.5 MiB, the agent without cloud detectors 45.1 MiB. About four fifths of each number is file-backed memory (RssFile), which grows with the binary. The anonymous part (RssAnon) was 34.8 MiB for contrib and 8.0 MiB for the small build.
Startup with no cloud detectors: about 190 to 220 ms for contrib and 45 to 85 ms for the small agents on the droplet, and 480 to 610 ms against 110 to 270 ms on the Pi.
Startup with cloud detectors for other clouds: [env, system, ec2, gcp, azure] held startup for the full 5 s timeout where 169.254.169.254 got no answer. Where the connect failed at once, it held startup for 4.1 s, because the AWS SDK retries the connect. On the droplet those three cost under 4 ms each, but the 13-cloud list still took 5.1 s there, because alibaba_ecs and scaleway ask other addresses.
The timeout setting bounds the wait: 1 s, 5 s or 10 s plus normal startup. The exceptions are digitalocean, which waits its own 2 s whatever you set when nothing answers, and akamai.
No start at all. akamai in the list: exit 1 in all four setups, after 30 s in the two with no answer. fail_on_missing_metadata: true: exit 1 after the 5 s timeout. A detector that ocb compiled out but the config still names: validate passes, then a nil pointer panic at start. The processor's name for a detector in the ocb manifest (ec2 instead of aws_ec2): the build stops.
What to do. List only the detectors for the platform the host runs on, and build with the same list in dist.resource.detection.detectors. If one config must cover several clouds, keep the list short, set timeout to 1 or 2 s, and leave out akamai.

Prerequisites

An x86_64 Linux host for the builds, with about 3 GB of free disk. ocb cross-compiles the arm64 binaries there.
curl, python3 and ip (iproute2). Root only for the two network namespaces.
An arm64 Linux host if you want the second architecture. We used a Raspberry Pi with 4 cores and less than 4 GB of memory.
About an hour for a full run. The cases that fail after 30 s take most of it.

What we built and where we ran it

We downloaded the official 0.164.0 releases of otelcol-contrib and otelcol for both architectures and checked each tarball against its published .sha256. Then ocb 0.164.0 built two versions of one small agent. It has the components a VM agent uses:

receivers: OTLP, hostmetrics and filelog
processors: batch, memory_limiter and resource_detection
exporters: debug, OTLP and OTLP/HTTP
extension: health_check

otelcol-agent uses the ocb default, so every detector is compiled in. otelcol-agent-envsys adds the ocb setting that compiles out every detector you do not list:

# manifests/agent-envsys.yaml (excerpt, comment on detectors added)
dist:
  name: otelcol-agent-envsys
  version: 0.164.0
  build_tags: 'grpcnotrace' # same tag as the contrib release manifest
  resource:
    detection:
      detectors: # ocb 0.163.1 and later: every detector not listed gets an omit_detector_<name> build tag
        - env:
        - system:

go version -m on the release binaries reports go1.26.10, so we pinned the same Go for our builds. go version -m and file show -trimpath=true, CGO_ENABLED=0 and stripped symbols for every binary (results/buildinfo/).

Every start ran the same pipeline: OTLP in, batch, debug out. The harness gave each start a clean environment with only PATH and an empty HOME. No OTEL_* or AWS_* variable and no ~/.aws file could change what a detector found. The Pi runs also set TZ=UTC, so their logs use UTC like the droplet's; the droplet runs used an earlier revision of the harness without it. Only the resource_detection block changed between configs. Each case ran 5 times in a shuffled order, after one warm-up start per binary that we left out of the numbers.

DigitalOcean droplet 4 vCPU, 8 GB, amd64
- host network metadata server answers
- blackhole namespace packets dropped
- noroute namespace connect fails at once
Raspberry Pi 4 cores, under 4 GB, arm64
- home network router drops 169.254.169.254

The four network setups matter more than the two architectures:

Setup	What a request to 169.254.169.254 gets
Droplet host	DigitalOcean's metadata server answers. The EC2 and Azure paths return 404 in about 1 ms. This is the "host on another cloud" case.
Blackhole namespace	The default route goes to a dummy device, which discards every packet, so a connect gets no answer. DNS fails at once, because nothing listens on 127.0.0.53 inside the namespace.
Noroute namespace	Loopback only. A connect fails at once with "network is unreachable".
Raspberry Pi	The home router drops the packets. DNS works. This is the plain "not on any cloud" host.

The-DevOps-Daily/otelcol-ocb-vs-contrib on GitHub

Size: what ocb removes

Binary size, linux/amd64, OpenTelemetry Collector 0.164.0

	Value	Series
otelcol-contrib (release)	420.3 MB	release
otelcol core (release)	183.8 MB	release
agent, all detectors	124.1 MB	custom
agent, env + system only	64.3 MB	custom

results/sizes.tsv. Stripped, -trimpath, CGO_ENABLED=0, go1.26.10. The two agents have the same 10 components; only the compiled-in detectors differ.

The contrib manifest lists 246 components and 9 config providers, and the binary links 1,088 Go modules. The core release is not small either. Its 30 components include the Kafka and Prometheus receivers and exporters, and it also ships resource_detection with every detector. Our agent with every detector links 289 modules, and the detector filter takes it to 225.

The filter saves more on a small build than on a big one. The OpenTelemetry blog measured the Kubernetes distribution and got 182 MB to 147 MB with 13 fewer dependencies. On our agent it removed 59.8 MB of 124.1 MB (48%) on amd64, 52.9 MB of 112.7 MB (47%) on arm64, and 64 modules. A big distribution needs most of those SDKs for other components, so removing the detectors does not remove the SDKs. In a small agent, nothing else needs them. results/deps/removed-by-detector-filter-amd64.txt lists the 64: 15 aws-sdk-go-v2 and smithy-go modules, the Google Cloud compute, auth and API clients, the Docker (moby) client, the Consul API with its HashiCorp libraries, and the Hetzner, Scaleway, Linode and DigitalOcean clients.

For containers, the official otel/opentelemetry-collector-contrib:0.164.0 image is 116.1 MB compressed on amd64 (results/images.tsv). We did not build an image of the small agent. Its binary compresses to 20.3 MB with gzip -6, so a FROM scratch image would be near that, but that is an estimate, not a measurement.

One build difference favours our side. The contrib manifest asks for the grpcnotrace build tag and our builds use it, but the release binaries record no build tags in go version -m. We built otelcol-agent-notag to put a size on it: 124.23 MB against 124.11 MB, so the tag is worth 0.1%.

Memory at idle

Resident memory after 10 s idle, droplet, amd64

	Value	Series
contrib: RssAnon	34.8 MiB	RssAnon
contrib: RssFile	165.2 MiB	RssFile
core: RssAnon	18.7 MiB	RssAnon
core: RssFile	79.5 MiB	RssFile
agent, all detectors: RssAnon	11.2 MiB	RssAnon
agent, all detectors: RssFile	50.2 MiB	RssFile
agent, env + system: RssAnon	8 MiB	RssAnon
agent, env + system: RssFile	37 MiB	RssFile

Median of 5 starts, /proc//status, config without resource_detection, results/summary.md. RssFile is resident file-backed memory; RssAnon is resident anonymous memory (heap, stacks, runtime).

Resident memory follows the binary size, but not one to one. Contrib held 199.8 MiB, 4.4 times the 45.1 MiB of the small agent. About four fifths of every number is RssFile, resident memory backed by files. It grows with the binary: 165.2 MiB for contrib, 37.0 MiB for the small agent. That points at the binary's own pages, but we did not map them one by one. The kernel can drop clean file-backed pages under memory pressure and read them again later. RssAnon is anonymous memory, which it cannot drop without swap. There contrib held 34.8 MiB against 8.0 MiB. A Go program runs the init code of every package it links before main, so a likely part of that difference is setup work for code that this pipeline never uses. We did not profile the heap to prove it.

Detection with [env, system] added little: contrib went to 205.4 MiB and the small agent to 47.8 MiB. The Raspberry Pi gave the same order on arm64: contrib 183.8 MiB with 34.7 MiB anonymous, the small agent 40.6 MiB with 8.0 MiB.

We did not test what happens to RssFile under memory pressure, or how a cgroup memory limit counts it. Read the RssAnon column as the floor and VmRSS as what top shows you.

Startup: the slowest detector sets the time

Median time from exec to the log line "Everything is ready", 5 starts each, from results/summary.md. Each cell is the small agent with every detector, then contrib in brackets:

Config	Droplet host	Blackhole namespace	Noroute namespace	Raspberry Pi
`[env, system]`	79 ms (189 ms)	77 ms (210 ms)	83 ms (221 ms)	192 ms (481 ms)
`+ ec2, gcp, azure`	81 ms (196 ms)	5,088 ms (5,203 ms)	4,098 ms (4,242 ms)	5,172 ms (5,486 ms)
`+ 13 VM clouds, no akamai`	5,090 ms (5,213 ms)	5,086 ms (5,203 ms)	4,087 ms (4,227 ms)	5,167 ms (5,393 ms)
`+ akamai`	exit 1 after 0.1 s (0.2 s)	exit 1 after 30.1 s (30.2 s)	exit 1 after 0.1 s (0.2 s)	exit 1 after 30.2 s (30.5 s)

Without a network wait, contrib needs about 110 to 160 ms more than the small agent to start on the droplet, and about 300 to 500 ms more on the Pi. That is real but small, and it matters only if you start collectors very often, for example one per short CI job.

The wait comes from three things in the processor source (processor/resourcedetectionprocessor at v0.164.1):

Detectors run in parallel. detectResource starts one goroutine per detector and then reads their results in order (internal/resourcedetection.go, lines 203 to 222). The wait is the slowest detector, not the sum.
The whole detection has one deadline: timeout. The default is 5 s (factory.go, lines 153 to 157). With the default retry.max_elapsed_time: 0, Refresh puts that timeout on the context that every detector gets (internal/resourcedetection.go, lines 159 to 168).
Startup waits for it. The processor's Start runs the first detection and returns its error (resourcedetection_processor.go, lines 36 to 40). The Collector logs "Everything is ready" only after every processor has started.

The processor also records how long each detector took, in its own otelcol_resourcedetection_detector_duration histogram. We read it from the Collector's Prometheus endpoint after each start. This is the 13-cloud list in the blackhole namespace:

Time per detector, 13 VM cloud detectors, blackhole namespace, timeout 5s

	Value	Series
ec2	5000.7 ms	waits for timeout
alibaba_ecs	4999.8 ms	waits for timeout
azure	4999.7 ms	waits for timeout
hetzner	5000.2 ms	waits for timeout
scaleway	4999.6 ms	waits for timeout
digitalocean	2000.6 ms	own 2 s timeout
nova	2000.4 ms	own 2 s timeout
upcloud	2000.7 ms	own 2 s timeout
vultr	2000.8 ms	own 2 s timeout
oraclecloud	200.7 ms	fast
ibmcloud_vpc	1.2 ms	fast
gcp	0.9 ms	fast
tencent_cvm	0.7 ms	fast

Median of 5 starts of the small agent with all detectors, from otelcol_resourcedetection_detector_duration in results/runs/droplet-blackhole-extra.jsonl. Every detector reported outcome=success: "not on this cloud" is a success with no attributes.

The same histogram tells you which detector held each of the other setups:

Setup	Detectors that took the longest
Droplet host	`alibaba_ecs` and `scaleway`, 5.0 s each. They ask `100.100.100.200` and `169.254.42.42`, and nothing on DigitalOcean answers those. The other 11 finished in 2 to 20 ms (medians).
Blackhole namespace	Five detectors at 5.0 s and four at 2.0 s (see the chart).
Noroute namespace	`ec2` alone, 4.0 s. Every other detector finished in under 2 ms.
Raspberry Pi	Six detectors at 5.0 s: the five from the blackhole namespace and `ibmcloud_vpc`, which took 1.2 ms in the namespace. We did not trace why. The same four at 2.0 s. With working DNS, `gcp` took 13.5 ms and `tencent_cvm` 15.9 ms.

How each detector probes decides how long it waits:

azure sends one GET with a plain &http.Client{} that has no timeout (internal/metadataproviders/azure/metadata.go, lines 33 to 38). When the packets get no answer, it waits for the processor's deadline. Then it returns an empty resource and no error (internal/azure/azure.go, lines 55 to 63).
ec2 uses the AWS SDK's IMDS client: a 250 ms dial timeout, a 500 ms response header timeout (aws-sdk-go-v2 feature/ec2/imds v1.20.1, api_client.go, lines 277 and 278), and up to 3 attempts per request (internal/aws/ec2/config.go, lines 16 to 19). A connect that fails at once is retried with backoff, which is why ec2 took 4.0 s in the noroute namespace. With no answer at all and a 10 s timeout, it gave up by itself after 5.5 s.
gcp calls metadata.OnGCEWithContext, which races a DNS lookup of metadata.google.internal against an HTTP probe. On a host that does not look like a Google VM, it trusts the first answer (cloud.google.com/go/compute/metadata v0.10.0, metadata.go, lines 456 to 480). A fast NXDOMAIN or a fast DNS failure ends it.
digitalocean ignores the context: its Detect takes _ context.Context (internal/digitalocean/digitalocean.go, line 51), and the DigitalOcean metadata client has a fixed 2 s timeout (digitalocean/go-metadata, client.go, lines 25 and 64).

We started from this claim: on a host that is not on EC2, Google Cloud or Azure, each cloud detector adds a wait at startup. Two results would prove it wrong: no wait at all, or the same wait for 3 detectors and for 13. We got the second, so it was half wrong. The detectors do not each add a wait. During detection they overlap, and the slowest one sets the time. But a host on a cloud with a metadata server is not safe either, because not every cloud uses 169.254.169.254.

The timeout setting

Start to "Everything is ready" with [env, system, ec2, gcp, azure]

	timeout: 1s	timeout: 5s (default)	timeout: 10s
contrib, blackhole namespace	1238.8 ms	5203.1 ms	10235.4 ms
agent, blackhole namespace	1077.1 ms	5087.6 ms	10085.4 ms
contrib, Raspberry Pi	1526.7 ms	5485.5 ms	10710.8 ms
contrib, noroute namespace	1240.1 ms	4241.6 ms	4229.5 ms
contrib, droplet host	209.3 ms	196 ms	216 ms

Median of 5 starts per point, results/summary.md. The droplet host answers on 169.254.169.254; the blackhole namespace and the Pi do not. The noroute namespace fails the connect at once.

Where nothing answers, the wait is the timeout plus normal startup, on both architectures. In the noroute namespace it stops at 4.1 to 4.2 s, where ec2 gives up. timeout: 1s brought every three-cloud case that waited down to 1.1 to 1.2 s.

The 13-cloud list with timeout: 1s did not get there. In the blackhole namespace it started in 2,097 ms (contrib 2,234 ms), and on the Pi in 2,175 ms (contrib 2,410 ms), because digitalocean took its 2 s anyway.

A short timeout has a cost that we could not measure. On a real EC2, Google Cloud or Azure VM, a detector that needs more than 1 s gets cut off. It returns an empty resource with no error, and your telemetry has no cloud attributes. We did not run on those clouds, so we cannot say whether 1 s is enough there. Most examples in the processor README use timeout: 2s.

Four ways the Collector does not start

ocb build unknown detector name: stops
otelcol validate passes a compiled-out detector
create processor akamai fetches a token
Start: detection parallel, one deadline
Everything is ready

1. The akamai detector

Of the 14 VM cloud detectors we ran, akamai is the only one that waited or failed while the processor was created. Its NewDetector calls newAkamaiClient(context.Background()) (internal/akamai/akamai.go, lines 43 to 57), and the Linode metadata client fetches a token inside NewClient (linode/go-metadata v0.3.1, client.go, lines 88 to 91). This happens before detection, so timeout does not apply, and a failure there is a failure to build the pipeline.

On the droplet, DigitalOcean's metadata server answers the token request, but not with a token, and the Collector exits at once:

results/runs/droplet-host-extra.jsonl, otelcol-contrib, configs/akamai.yaml

$ bin/linux-amd64/otelcol-contrib --config configs/akamai.yaml
Error: failed to build pipelines: failed to create "resource_detection" processor, in pipeline "traces": failed creating detector type "akamai": failed to refresh metadata token: failed to generate metadata token: no token returned from API
# exit code 1, 0.2 s after exec

In the noroute namespace it exited at once with connect: network is unreachable. In the blackhole namespace and on the Pi, the token request first waited for a 30 s dial timeout. The Collector exited after 30.1 s in the namespace and 30.2 s on the Pi (contrib: 30.2 s and 30.5 s). With timeout: 1s it still took 30 s.

2. fail_on_missing_metadata: true

This setting makes a cloud detector return an error when its metadata service is missing, instead of an empty resource. Retry is on by default (factory.go, lines 142 to 151), so the processor retries the detector with backoff until timeout, and then Start returns the error. With [env, system, ec2, gcp, azure], the Collector exited with code 1 after 5.1 to 5.3 s in all three droplet setups. The config comment says the setting is for nodes that boot before their metadata service is ready (config.go, lines 63 to 70). It also turns "not on EC2" into "does not start".

3. A detector that ocb compiled out

This is the trap in the new ocb feature. otelcol-agent-envsys has only env and system. We gave it the three-cloud config:

results/break/amd64-validate.txt and results/break/amd64-start.txt

$ bin/linux-amd64/otelcol-agent-envsys validate --config configs/clouds3.yaml
exit code: 0
$ bin/linux-amd64/otelcol-agent-envsys --config configs/clouds3.yaml
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x130a757]

goroutine 57 [running]:
github.com/open-telemetry/opentelemetry-collector-contrib/processor/resourcedetectionprocessor/internal.(*ResourceProvider).detectWithRetry.func2()
    github.com/open-telemetry/opentelemetry-collector-contrib/processor/resourcedetectionprocessor@v0.164.1/internal/resourcedetection.go:314 +0xb7
...
exit code: 2

In v0.164.1 each *_omit.go stub returns nil, nil from NewDetector, for example internal/aws/ec2/ec2_omit.go. The processor keeps the nil detector and calls it. This is issue #52519, open when we wrote this. On main, #52063 changes the stubs so they register no factory. It was merged after v0.164.1. With it, the processor returns detector "ec2" is not compiled into this binary with the list of compiled detectors. Until a release has that change, validate does not catch the mismatch, and the first sign is a crash.

4. The processor's names in the ocb manifest

ocb and the processor do not use the same detector names. The processor config says ec2, nova and aks. The ocb manifest wants the build tag names aws_ec2, openstack_nova and azure_aks (cmd/builder/internal/builder/config.go, lines 117 to 151). This one fails early, which is the good kind of failure:

results/build/agent-badname-amd64.log

$ ocb --config manifests/agent-badname.yaml
Error: invalid configuration: ec2: unknown resource detector

What to put in the manifest and the config

Make the two lists match, each in its own names:

# ocb manifest: build tag names
dist:
  resource:
    detection:
      detectors:
        - env:
        - system:
        - aws_ec2: # the processor calls this one "ec2"


# Collector config: processor names, the same detectors
processors:
  resource_detection:
    detectors: [env, system, ec2]
    timeout: 2s # the most a missing metadata service can add to startup

Use one list per platform. A fleet on EC2 does not need azure, and a fleet on bare metal needs neither.
If one config must cover several clouds, keep the list short and set timeout to 1 or 2 s. Expect digitalocean to add 2 s, whatever the timeout, on hosts where its request gets no answer.
Leave akamai out unless the hosts run on Akamai Cloud.
Keep fail_on_missing_metadata off unless "no metadata" should mean "do not start".
Use the name resource_detection. In 0.164 the old name resourcedetection still works, with a deprecation warning (results/break/amd64-legacy-name.txt).
Until the #52063 change is in a release, start each new custom build once with its real config before you ship it. validate is not enough.

What we could not conclude

On a real AWS, Google Cloud or Azure VM. We had no instances there, so we did not measure how long a detector takes when it succeeds, or whether a 1 s timeout cuts it off.
The container image of the small agent. We report the compressed size of the official image and the gzip size of our binary. We did not build or push an image.
Memory under pressure. RssFile can be reclaimed. We did not measure how much the kernel takes back, or how a cgroup limit counts it.
Small startup differences. Each case has 5 starts. The 110 to 160 ms between contrib and the small agent repeats in every droplet setup. The Pi also ran other work, and three of its starts took 4.6 to 22 s. In all three, the time went before the Collector wrote its first log line, which points at reading the binary from the SD card, not at detection. We did not confirm that. They move the medians little, but they are in the min and max columns of results/summary.md.
Slow DNS. DNS failed at once in the namespaces and answered on the Pi, so gcp and tencent_cvm were fast everywhere. A host whose DNS times out would make them wait too. We did not test that.
Other versions. Everything here is 0.164.0 and 0.164.1. The akamai and omit behaviour may change in the next release.

Summary

A custom Collector with only the components you use is 6.5 times smaller than contrib and holds under a quarter of its memory. On a small build, compiling out the unused detectors is half of the size saving. The binary costs tens to hundreds of milliseconds at startup. The detector list can cost seconds. Apart from akamai, the cloud detectors you list for clouds the host is not on share one wait: the slowest probe. Where that probe gets no answer, it usually runs to the full timeout, even on a host on another cloud. digitalocean waits its own 2 s instead, even past a shorter timeout. Keep the list to the platforms you run on, and build with the same list. Set a short timeout if the list must stay long. Start each new binary once with its real config. Leave akamai out unless you run on Akamai.

Originally published at devops-daily.com.

An Empty resources: {} Makes Kubernetes 1.34 to 1.36 Pods BestEffort

DevOps Daily — Wed, 04 Nov 2026 15:00:00 +0000

On Kubernetes 1.34, 1.35 and 1.36, one empty block in a pod spec changes how the node treats the pod. If the pod sets spec.resources: {} at pod level, the kubelet gives it QoS class BestEffort. This happens even when every container sets requests and limits, whenever the PodLevelResources feature gate is on, which is the default from 1.34. GitLab Runner's Kubernetes executor builds its job pods with that empty block. Its source does this from 18.10.0 to 19.4.3, unless the runner config sets pod-level values.

We ran the same pods on k3s v1.34.12, v1.35.9, v1.36.5 and v1.37.1. On the three older versions, the empty block moved the pod into the BestEffort cgroup slice. The kubelet set its oom_score_adj to 1000, the highest value the kernel accepts. Next to a busy neighbour, a pod that requested half a CPU got 0.06 to 0.08 cores instead of 0.57 to 0.58. A namespace quota that blocks BestEffort pods rejected it. On 1.37.1 the same pods kept their Burstable and Guaranteed classes.

One thing did not change, and the research we started from got it wrong. It said these pods are first in line for kubelet eviction under memory pressure. They are not. The kubelet's eviction ranking compares memory usage with the memory request. For an empty block, that request still comes from the containers. The QoS class is not part of the ranking, and the kubelet evicted our pods in the same order on every version.

TL;DR

On k3s v1.34.12, v1.35.9 and v1.36.5, a pod with container requests and limits plus spec.resources: {} reported qosClass: BestEffort. The Guaranteed variant did too. On v1.37.1 they reported Burstable and Guaranteed. With the new PodLevelResourcesFixKubeletQOSClass feature gate turned off, 1.37.1 went back to BestEffort.
The node settings followed the class. oom_score_adj went from 935 (Burstable) and -997 (Guaranteed) to 1000, and the pod cgroup moved to kubepods-besteffort.slice with cpu.weight 1 and no pod-level memory.max. The container-level limits stayed in place.
A neighbour pod kept both vCPUs busy. Next to it, the BestEffort-classified pod got a median of 0.061 to 0.083 cores on the three affected versions. The same pod without the empty block got 0.573 to 0.580 (so 10.6% to 14.3%). On 1.37.1 the two were equal.
A ResourceQuota with scope BestEffort and pods: "0" rejected the pod on 1.34 to 1.36 and admitted it on 1.37.
The kubelet's memory-pressure eviction order was the same on all four versions, in all 12 repetitions: the true BestEffort pod first, then by memory usage relative to the container request. The empty block did not move a pod forward.
If you run GitLab Runner on Kubernetes, upgrade to 19.5.0 or later. As of 2 November 2026 the latest Kubernetes patch releases for 1.34, 1.35 and 1.36 still have the bug.

Prerequisites

A cluster on Kubernetes 1.34, 1.35 or 1.36 (kubectl version), or a machine where you can run the lab
kubectl and jq
If you run GitLab Runner with the Kubernetes executor: its version (gitlab-runner --version)
To run the lab: a DigitalOcean token (referral link); one s-2vcpu-4gb droplet for about an hour costs a few cents

Where the empty block comes from

Pod-level resources (KEP-2837) let you set CPU and memory for the pod as a whole, not only per container. The feature gate PodLevelResources went alpha in 1.32 and beta, on by default, in 1.34. Our Kubernetes 1.36 post covers the in-place resize of those values.

The QoS calculation has a shortcut for them. This is ComputePodQOS at v1.34.0, v1.35.0 and v1.36.0, and still at the latest patches, v1.34.13, v1.35.10 and v1.36.6:

// pkg/apis/core/v1/helper/qos/qos.go, v1.36.6
// When pod-level resources are specified, we use them to determine QoS class.
if utilfeature.DefaultFeatureGate.Enabled(features.PodLevelResources) &&
    pod.Spec.Resources != nil {

An empty block is not nil. It has no requests and no limits, so the function returns BestEffort. The loop over the containers never runs. The issue that reported this, #135082 from November 2025, says that almost everywhere else Kubernetes does not differentiate between nil and empty pod-level resources. The helper for that, IsPodLevelResourcesSet, treats {} as not set.

The fix is PR #137150, merged on 15 July 2026 for 1.37. It adds the condition IsPodLevelResourcesSet(pod) behind a new gate, PodLevelResourcesFixKubeletQOSClass, beta and on by default from 1.37. It was not backported. The patch releases of 27 October 2026, v1.34.13, v1.35.10 and v1.36.6, still have the old condition. The rest of that release is in our Kubernetes 1.37 review; PodLevelResources itself stays beta in 1.37.

GitLab Runner added pod-level resources to its job pods in 18.10.0 (MR !5922). At every tag we read (18.10.0, 19.3.5 and 19.4.3), the helper that builds the block returns a non-nil object:

// executors/kubernetes/kubernetes.go, GitLab Runner v19.4.3
func (s *executor) podResourcesReference() *api.ResourceRequirements {
    resources := api.ResourceRequirements{}

    if s.configurationOverwrites != nil {
        resources.Limits = s.configurationOverwrites.podLimits
        resources.Requests = s.configurationOverwrites.podRequests
    }

    return &resources
}

If the runner config sets none of pod_cpu_request, pod_memory_request, pod_cpu_limit and pod_memory_limit, both maps are empty. The pod then goes to the API server with resources: {}. The MR that fixed it, !7346, says the same: job pods "were showing qosClass: BestEffort instead of the expected Burstable", and after the fix .spec.resources "should now be empty/absent (was {})". It merged on 1 October 2026 and shipped in 19.5.0. The 19.4.3 and 19.3.5 tags of 28 October 2026 still have the old function. The class matters only when the runner config gives the job containers requests or limits. Without them, the pods were BestEffort anyway.

The lab

The lab runs on one DigitalOcean droplet: s-2vcpu-4gb in fra1, Ubuntu 24.04, cgroup v2. A cloud firewall allows SSH from one address and nothing else. The Kubernetes API port is never exposed, and kubectl runs on the node over SSH. For each version, a script removes k3s, installs the pinned release, and runs five stages. The kubelet gets an explicit hard eviction threshold, memory.available<2Gi. It also gets a 30 second pressure transition period instead of the default 5 minutes, so repetitions do not wait for the node condition to clear.

pod spec resources: {} + container requests
ComputePodQOS sees a block: BestEffort
IsPodLevelResourcesSet sees {}: not set
oom_score_adj 1000
cgroup slice besteffort, cpu.weight 1
ResourceQuota scope BestEffort
eviction ranking container requests

Connections:

pod spec -> ComputePodQOS
pod spec -> IsPodLevelResourcesSet
ComputePodQOS -> oom_score_adj
ComputePodQOS -> cgroup slice
ComputePodQOS -> ResourceQuota scope
IsPodLevelResourcesSet -> eviction ranking

The pods:

Pod	Pod-level block	Container requests	Container limits
`be`	none	none	none
`ctl-burstable`	none	100m CPU, 256Mi	500m CPU, 512Mi
`emp-burstable`	`{}`	100m CPU, 256Mi	500m CPU, 512Mi
`ctl-guaranteed`	none	200m CPU, 256Mi	200m CPU, 256Mi
`emp-guaranteed`	`{}`	200m CPU, 256Mi	200m CPU, 256Mi
`plr-burstable`	requests 100m/256Mi, limits 500m/512Mi	none	none

plr-burstable is the control for the feature itself: a real pod-level block works as documented on every version. The empty block is the only problem.

We wrote down the claim and what would disprove it before the droplet existed, and committed it first (CLAIM.md). Every recorded file carries a hash of the manifests and node scripts, and the summary script refuses to mix files with different hashes.

The-DevOps-Daily/k8s-empty-pod-resources on GitHub

Finding 1: the class flips on 1.34 to 1.36

The API server stored the empty block exactly as written. On v1.34.12, v1.35.9 and v1.36.5 the kubelet then reported BestEffort for both emp-* pods, while their controls kept Burstable and Guaranteed. On v1.37.1 the emp-* pods got the same class as their controls:

results/demo-k3s-v1.36.5.txt and results/demo-k3s-v1.37.1.txt

# k3s v1.36.5+k3s1
# ... kubectl apply and wait omitted; the whole session is in the file
$ kubectl get pods -n demo -o custom-columns='NAME:.metadata.name,POD-LEVEL:.spec.resources,QOS:.status.qosClass'
NAME POD-LEVEL QOS
ctl-burstable <none> Burstable
ctl-guaranteed <none> Guaranteed
emp-burstable map[] BestEffort
emp-guaranteed map[] BestEffort
# k3s v1.37.1+k3s1, same manifests
# ... kubectl apply and wait omitted
$ kubectl get pods -n demo -o custom-columns='NAME:.metadata.name,POD-LEVEL:.spec.resources,QOS:.status.qosClass'
NAME POD-LEVEL QOS
ctl-burstable <none> Burstable
ctl-guaranteed <none> Guaranteed
emp-burstable map[] Burstable
emp-guaranteed map[] Guaranteed

The same result on every version, from results/summary.md:

Pod	v1.34.12	v1.35.9	v1.36.5	v1.37.1	v1.37.1, gate off
`ctl-burstable`	Burstable	Burstable	Burstable	Burstable	Burstable
`emp-burstable`	BestEffort	BestEffort	BestEffort	Burstable	BestEffort
`ctl-guaranteed`	Guaranteed	Guaranteed	Guaranteed	Guaranteed	Guaranteed
`emp-guaranteed`	BestEffort	BestEffort	BestEffort	Guaranteed	BestEffort
`plr-burstable`	Burstable	Burstable	Burstable	Burstable	Burstable
`be`	BestEffort	BestEffort	BestEffort	BestEffort	BestEffort

plr-burstable, with real pod-level requests and limits, was Burstable everywhere. Pod-level resources work. Only the empty block breaks the class.

To check that the 1.37 gate is the fix and not something else in the release, we reinstalled v1.37.1 with PodLevelResourcesFixKubeletQOSClass=false on both the API server and the kubelet. Both emp-* pods went back to BestEffort, with oom_score_adj 1000. The API server and kubelet metrics reported the gate as BETA, on by default and off in that run.

Finding 2: the kernel sees a BestEffort pod

QoS is not only a label. The kubelet turns it into numbers that the kernel uses. We read them from /proc and /sys/fs/cgroup on the node, for the main process of each container:

oom_score_adj of the main container process

	Value	Series
ctl-burstable, every version	935	no pod-level block
emp-burstable, 1.34 to 1.36	1000	resources: {}, 1.34 to 1.36
emp-burstable, 1.37.1	935	resources: {}, 1.37.1
ctl-guaranteed, every version	-997	no pod-level block
emp-guaranteed, 1.34 to 1.36	1000	resources: {}, 1.34 to 1.36
emp-guaranteed, 1.37.1	-997	resources: {}, 1.37.1

Read from /proc//oom_score_adj on the node. The 1.34.12, 1.35.9 and 1.36.5 values are identical. The kernel adds this value to a process's badness score when it picks one to kill. Source: results/summary.md.

The kernel adds oom_score_adj to each process's badness score before it picks a process to kill, and it accepts values from -1000 to +1000 (proc.rst). A Burstable container gets 1000 - 1000 * request / node memory, which is 935 for a 256Mi request on this node. A Guaranteed container gets -997. The empty block gave both 1000, the same as a pod with no resources at all. oom_score, the current OOM killer score the kernel shows for the process, went from 1290 (Burstable) and 4 (Guaranteed) to 1334.

The cgroup changed too. On v1.36.5:

	`ctl-burstable`	`emp-burstable`	`ctl-guaranteed`	`emp-guaranteed`
QoS slice	`kubepods-burstable.slice`	`kubepods-besteffort.slice`	`kubepods.slice`	`kubepods-besteffort.slice`
pod `cpu.weight`	4	1	8	1
pod `cpu.max`	0.5 CPU	max (none)	0.2 CPU	max (none)
pod `memory.max`	512 MiB	max (none)	256 MiB	max (none)
container `cpu.max`	0.5 CPU	0.5 CPU	0.2 CPU	0.2 CPU
container `memory.max`	512 MiB	512 MiB	256 MiB	256 MiB
`oom_score`	1290	1334	4	1334

The container cgroup still has its own limits, so emp-burstable cannot use more than 512 MiB or half a CPU. What it loses is everything the kubelet sets at pod and QoS level. The pod cgroup has no memory limit and no CPU quota of its own. It sits in kubepods-besteffort.slice, where the kubelet sets cpu.weight to 1 for the whole slice.

Finding 3: half a CPU requested, 0.06 to 0.08 cores received

cpu.weight 1 matters only when the CPU is busy. To see how much, we ran cpu-neighbour, a Burstable pod that requests 1 CPU and runs two busy processes, so both vCPUs are always taken. Next to it, cpu-ctl and cpu-emp took turns, three times each, alternating which went first. Both request 500m CPU with a 1 CPU limit and run one busy process; cpu-emp adds the empty block. Each pod reads its own cgroup's usage_usec over 20 seconds after a 5 second warm-up.

CPU used next to a busy neighbour, median of 3 runs

	Value	Series
v1.34.12 cpu-ctl	0.579 cores	cpu-ctl (no block)
v1.34.12 cpu-emp	0.083 cores	cpu-emp (resources: {})
v1.35.9 cpu-ctl	0.58 cores	cpu-ctl (no block)
v1.35.9 cpu-emp	0.07 cores	cpu-emp (resources: {})
v1.36.5 cpu-ctl	0.573 cores	cpu-ctl (no block)
v1.36.5 cpu-emp	0.061 cores	cpu-emp (resources: {})
v1.37.1 cpu-ctl	0.576 cores	cpu-ctl (no block)
v1.37.1 cpu-emp	0.577 cores	cpu-emp (resources: {})

2 vCPU droplet. Both pods request 500m with a 1 CPU limit and run one busy process; the neighbour requests 1 CPU and runs two. Each pod measured its own cgroup usage_usec over 20 s. Source: results/summary.md.

On the three affected versions, cpu-emp got a median of 0.083, 0.070 and 0.061 cores, against 0.579, 0.580 and 0.573 for cpu-ctl. That is 14.3%, 12.1% and 10.6% of the control's CPU, or 7.0 to 9.4 times less. On v1.37.1 the two were the same: 0.576 and 0.577 cores.

The weights explain the direction. cpu-ctl's pod cgroup has cpu.weight 20 for its 500m request and sits in the burstable slice next to the neighbour. The kubelet sets the burstable slice's weight from the sum of the CPU requests of the Burstable pods in it, which gave 63 here. cpu-emp's pod cgroup has cpu.weight 1 and sits in the besteffort slice. The kubelet always sets that slice to the minimum, cpu.weight 1. The 500m request of cpu-emp still sets a weight on its container cgroup, but that only counts against other containers in the same pod. On a busy node like this one, a CI build in the BestEffort slice competes with cpu.weight 1 at that level and can get far less CPU than Burstable or Guaranteed pods. We did not run a build.

The runs were short and sequential. Each pod ran three times in a fixed alternating order, and we did not record the neighbour's own usage or other load on the host. On the affected versions, each of the nine cpu-emp runs got less CPU than every cpu-ctl run, including the runs where cpu-emp went first. The exact percentages still belong to this setup.

Finding 4: a "no BestEffort" quota rejects the pod

Some clusters forbid BestEffort pods with a quota like this one, so every workload must declare resources:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: no-besteffort
spec:
  hard:
    pods: '0'
  scopes: ['BestEffort']

The API server applies the scope at create time, before any kubelet sees the pod, and it calls the same GetPodQOS function. With no status yet, that function computes the class from the spec. On 1.34 to 1.36 it rejected emp-burstable, a pod whose container sets requests and limits:

quota.py on k3s v1.36.5

$ grep -A3 ' quota$' results/k3s-v1.36.5/run.log
2026-11-02T10:33:20Z quota
be admitted=False Error from server (Forbidden): error when creating "/root/k8s-empty-pod-resources/manifests/quota/be.yaml": pods "be" is forbidden: exceeded quota: no-besteffort, requested: pods=1, used: pods=0, limited: pods=0
ctl-burstable admitted=True pod/ctl-burstable created
emp-burstable admitted=False Error from server (Forbidden): error when creating "/root/k8s-empty-pod-resources/manifests/quota/emp-burstable.yaml": pods "emp-burstable" is forbidden: exceeded quota: no-besteffort, requested: pods=1, used: pods=0, limited: pods=0

On v1.37.1 the API server admitted emp-burstable, and still rejected be. We did not run a runner. But a job pod from 18.10.0 to 19.4.x carries the same empty block, so in a namespace with this quota it should fail the same way on 1.34 to 1.36. The fix discussion names the reverse risk: a pod "could theoretically bypass restricted BestEffort quotas by being classified as Guaranteed during admission, while still behaving as BestEffort at the node level". That can happen when the API server has the fix and the kubelet does not.

Finding 5: the eviction order did not change

The five eviction pods each allocate a fixed amount of memory and then sleep. They are sized so that the two possible rules predict different orders:

Pod	Memory request	Allocates	Working set before the hog	Usage against request
`be`	none	50 MiB	54 MiB	above (request is 0)
`ctl`	256Mi	200 MiB	204 MiB	about 52 MiB below
`gua-ctl`	256Mi (= limit)	150 MiB	154 MiB	about 102 MiB below
`emp` + `{}`	256Mi	100 MiB	104 MiB	about 152 MiB below
`gua-emp` + `{}`	256Mi (= limit)	50 MiB	54 MiB	about 202 MiB below

If the kubelet ranked by QoS class, emp and gua-emp would go out together with be, before ctl and gua-ctl. If it ranks by container requests, be goes first because any usage exceeds a request of zero, and the rest follow in order of usage minus request: ctl, gua-ctl, emp, gua-emp.

A host process outside every pod, hog.py, then holds MemAvailable at 1000 MiB. That is below the 2Gi threshold even after all five pods are gone, so the kubelet keeps evicting until nothing evictable is left. The kubelet cannot evict the hog. The pods differ in size on purpose, to separate the two models, so this is not a matched pair. CoreDNS also ran on the node. The kubelet ranked it last in every ranking it logged. In 5 of the 12 logs the window ran long enough to show the next step: the kubelet logged cannot evict a critical pod once only CoreDNS was left.

Every repetition on every version gave the request order. This is the kubelet's own ranking on v1.36.5, while it reported emp and gua-emp as BestEffort:

k3s v1.36.5, repetition 1

# the class the kubelet reported for each pod
$ jq -c 'select(.rep == 1) | .qos' results/k3s-v1.36.5/40-pressure.jsonl
{"be":"BestEffort","ctl":"Burstable","emp":"BestEffort","gua-ctl":"Guaranteed","gua-emp":"BestEffort"}
# the kubelet's ranking before each eviction; sed keeps the journald timestamp and the message, and drops the host name, the service PID, the severity letter, the kubelet's own timestamp, the second PID and the source file
$ grep -E 'ranked for eviction|evicted successfully' results/k3s-v1.36.5/40-kubelet-rep1.log | sed -E 's/^([^]+) .*eviction_manager.go:[0-9]+\] /\1 /'
2026-11-02T10:38:17.816980+00:00 "Eviction manager: pods ranked for eviction" pods=["evict-check/be","evict-check/ctl","evict-check/gua-ctl","evict-check/emp","evict-check/gua-emp","kube-system/coredns-7cfb7bc9c7-svb26"]
2026-11-02T10:38:20.108268+00:00 "Eviction manager: pod is evicted successfully" pod="evict-check/be"
2026-11-02T10:38:21.127785+00:00 "Eviction manager: pods ranked for eviction" pods=["evict-check/ctl","evict-check/gua-ctl","evict-check/emp","evict-check/gua-emp","kube-system/coredns-7cfb7bc9c7-svb26"]
2026-11-02T10:38:23.555889+00:00 "Eviction manager: pod is evicted successfully" pod="evict-check/ctl"
2026-11-02T10:38:24.579473+00:00 "Eviction manager: pods ranked for eviction" pods=["evict-check/gua-ctl","evict-check/emp","evict-check/gua-emp","kube-system/coredns-7cfb7bc9c7-svb26"]
2026-11-02T10:38:27.242735+00:00 "Eviction manager: pod is evicted successfully" pod="evict-check/gua-ctl"
2026-11-02T10:38:28.259762+00:00 "Eviction manager: pods ranked for eviction" pods=["evict-check/emp","evict-check/gua-emp","kube-system/coredns-7cfb7bc9c7-svb26"]
2026-11-02T10:38:30.555840+00:00 "Eviction manager: pod is evicted successfully" pod="evict-check/emp"
2026-11-02T10:38:31.572048+00:00 "Eviction manager: pods ranked for eviction" pods=["evict-check/gua-emp","kube-system/coredns-7cfb7bc9c7-svb26"]
2026-11-02T10:38:34.078890+00:00 "Eviction manager: pod is evicted successfully" pod="evict-check/gua-emp"

The source explains it. rankMemoryPressure sorts by three keys. First, whether usage exceeds the memory request. Then pod priority. Then usage minus the memory request. The first and third keys get the request from GetResourceRequestQuantity. For an empty block, that helper asks IsPodLevelResourcesSet, gets false, and uses the container requests. Priority does not read requests, and the QoS class is not an input at all. If the kubelet had used a request of zero for emp, emp would have been in the first group with be. It came fourth.

When each pod was evicted, 12 repetitions

Series	Samples	Min	Median	p95	Max
be	12	5.6s	8.350000000000001s	11.5s	11.5s
ctl	12	9.1s	11.95s	15.3s	15.3s
gua-ctl	12	12.7s	16s	20.9s	20.9s
emp	12	16.1s	20.15s	24.7s	24.7s
gua-emp	12	19.5s	23.6s	29.9s	29.9s

Seconds after the host hog started, from the kubelet's log. 3 repetitions each on k3s v1.34.12, v1.35.9, v1.36.5 and v1.37.1. Every repetition evicted the pods in the same order. Source: results/summary.md.

In all 12 repetitions the kubelet started to reclaim 3.0 to 9.2 s after the hog started, and evicted the first pod 2.3 to 3.3 s later. After that it evicted one pod at a time. 41 of the 48 gaps between evictions were 4.0 s or less (median 3.6 s, longest 8.2 s). The last pod was gone 19.5 to 29.9 s after the hog started. It does not wait 10 seconds between evictions: after each one it waits for the pod to be cleaned up and then checks again. Its own available figure at the first eviction was 931 to 1212 MiB, well under the 2Gi threshold and far from empty. The kernel logged no OOM kill in any repetition, so the order is the kubelet's alone.

The two mechanisms are easy to mix up. The kubelet's eviction manager checks every 10 seconds by default when it has nothing to evict, and it ignores the class. The kernel OOM killer acts when memory runs out before the kubelet frees any. It adds oom_score_adj to each process's badness score, and the empty block gives the pod the same 1000 as a pod with no resources at all (Finding 2). We did not trigger a kernel OOM, so we did not watch the kernel make that choice.

What we got wrong before measuring

The research note we started from had the core right and two details wrong.

Right: Kubernetes 1.34 to 1.36 compute BestEffort for an empty pod-level block. GitLab Runner 18.10 to 19.4 build their job pods with that block (from the source and the fix MR). On 1.34 to 1.36 the pods get oom_score_adj 1000. 1.37 fixes the class while its new gate is on, which is the default.
Wrong: it called #137150 the upstream issue. That is the fix PR. The issue is #135082.
Wrong, and it matters: it said these pods are first in line for kubelet eviction under memory pressure. Finding 5 shows they are not.
Missing: the 1.37 fix is a beta feature gate, not a plain code change, and nothing was backported to the 1.34, 1.35 or 1.36 patch releases.

We got one prediction wrong ourselves. CLAIM.md said the BestEffort-classified pod would get under 10% of the control's CPU. It got 10.6% to 14.3%. Our falsification line was 50%, so the claim stands, but the number we wrote down was too pessimistic, and we report it as a miss.

We also kept a one-repetition smoke run on v1.36.5 (results/_smoke-k3s-v1.36.5/). It used the same code stamp and gave the same results, but the summary does not count it.

What to do

Find the pods. This query lists every pod with an empty pod-level block and the class it got:
GitLab Runner 18.10.0 to 19.4.x: upgrade to 19.5.0 or later. The job pods then go out without a pod-level block, and the class comes from the containers again.
If you cannot upgrade the runner yet, give the block real values. Set pod_cpu_request and pod_memory_request (and the _limit keys if you want limits) under [runners.kubernetes]. The QoS class then comes from those values, like plr-burstable above. This is a behaviour change, not only a workaround: the pod cgroup then takes its limits from the pod-level values too, as it did for plr-burstable.
Kubernetes 1.37 fixes it by default. The fix sits behind a beta gate, so check that nobody turned PodLevelResourcesFixKubeletQOSClass off.
Check your own templates. A Helm chart that renders a pod-level block from a value that defaults to {} produces the same bug. Render the key only when it has content:

An empty resources: {} inside a container is harmless. Only the pod-level one under spec changes the class.

What we could not conclude

k3s is one distribution. We ran upstream Kubernetes code as k3s packages it. A managed service can turn the PodLevelResources gate off or carry its own patches. Run the jq query on your own cluster.
No GitLab instance and no runner ran here. The runner part rests on its source at each tag and on the MR description. The emp-* pods reproduce the field the runner sends, not a full job pod with helper and service containers.
We did not trigger a kernel OOM kill. The oom_score values are the numbers the kernel uses to choose a victim, read from /proc. We did not watch it choose.
The eviction timing is ours. It depends on the 2Gi threshold, the hog's 1000 MiB target and how fast it allocates. The order is the finding, not the seconds.
The CPU gap depends on the neighbours. One neighbour with a 1 CPU request on a 2 vCPU node gave the numbers above. More Burstable or Guaranteed load should make the BestEffort slice's share smaller. On an idle node the weights do not matter, because they only divide contended CPU time; we did not measure that case.
One node, one admission check. We did not test scheduling across nodes, or any admission rule other than the one quota. The gate-off run on 1.37.1 covered only the class, the kernel-facing numbers and the quota, not CPU or eviction.
Other consumers of the class were not tested. From the source, the kubelet also rejects new BestEffort pods while the node has the MemoryPressure condition. The static CPU manager gives exclusive CPUs only to Guaranteed pods. We did not measure either.

Summary

On Kubernetes 1.34, 1.35 and 1.36, with the PodLevelResources gate at its default (on), spec.resources: {} makes a pod BestEffort no matter what its containers request. On our node that meant oom_score_adj 1000 instead of 935 or -997, and a place in the BestEffort slice with cpu.weight 1 and no pod-level memory or CPU limit. Next to a busy neighbour the pod got about 7 to 9 times less CPU, and a quota that forbids BestEffort pods rejected it. The empty block did not change the kubelet's eviction order. The ranking uses memory requests, and for an empty block those still come from the containers: 12 of 12 repetitions evicted our pods in exactly the order the requests predict. Kubernetes 1.37 fixes the class behind a beta gate that is on by default, and GitLab Runner 19.5.0 stops sending the block. Run the jq query above on your clusters, and upgrade whichever side you can first.

Originally published at devops-daily.com.

Can a Vision Model Read Your Grafana Panel? 100 Panels on DigitalOcean Inference

DevOps Daily — Wed, 04 Nov 2026 07:00:00 +0000

Grafana can attach a screenshot of the alerting panel to a notification. A bot can send that image to a vision model and post one line next to the page. NVIDIA Nemotron 3.5 Super VL came to DigitalOcean Serverless Inference on 15 October 2026 (release notes), so we tested it on that job.

We drew 100 dashboard panels with known answers and asked four questions about each: the peak value, the time of the peak, whether the data crossed the threshold, and which series is highest. Our claim, written before we measured: Nemotron 3.5 Super VL gets all four right on at least 90 of 100 panels, for under $0.001 per screenshot. It got all four right on 48 panels with reasoning off and 54 with reasoning on. The claim failed. The cost limit held only with reasoning off: $0.00026 per screenshot, against $0.0024 with reasoning on.

None of the four models we tested came close to 90. Qwen3.8-Max got 65 and Gemma 4 got 64, at $0.11 per 1,000 screenshots for Gemma. Log-scale axes broke every model.

So the numbers in an alert message should come from the alert payload, not the model. The more useful result is a small bot that lets the model describe the screenshot and lets rules decide whether to trust it. On Nemotron's 300 reads, the rules caught 63 percent of the wrong reads and blocked no correct read. With the wrong screenshot attached on purpose, the bot refused it 99 times in 100.

TL;DR

All four facts right, of 100 Grafana-style panels: Qwen3.8-Max 65, Gemma 4 64, Nemotron 3.5 Super VL 54 with reasoning and 48 without, Llama 4 Maverick 21. Our bar was 90.
Peak time and threshold were easy (86 to 99 percent for every model). Peak value and top series were hard. On log axes, the best model read all four right on 8 of 25 panels.
Reasoning made Nemotron 13 times slower (19 seconds at the median, 82 at worst) and 9 times more expensive. The gain from 48 to 54 is inside the noise.
At temperature 0, Nemotron gave the same four answers in all three runs on 13 of 100 panels. Qwen3.8-Max did on 76.
A cross-check against the alert caught 63 percent of Nemotron's wrong reads (81 percent at a 5 percent tolerance). It cannot check the peak time, the series on a stacked panel, or the words in the summary.

Prerequisites

A DigitalOcean account (referral link) with a Serverless Inference model access key
Node.js 22 or later
About $1 of inference if you rerun every model; the reasoning-mode run is most of it

The-DevOps-Daily/grafana-panel-vision on GitHub

The job and the claim

A bot that reads the attached panel could save the on-call engineer one click: "p99 went to 1.8 s at 13:55 UTC and stayed under the 2.5 s threshold". The question is whether you can trust that sentence during a night page.

Each model got the same prompt for every panel, with the panel as a PNG, and had to answer in JSON. This is a real answer from Nemotron 3.5 Super VL with reasoning off:

{
  "peak": { "value": 1.8, "unit": "s" },
  "peakTime": "13:45",
  "thresholdCrossed": "no",
  "topSeries": "search",
  "summary": "Upstream response time stays under 1 second until midday, then spikes sharply to about 1.8 seconds in the early afternoon while remaining below the 2.5 s warning threshold."
}

The truth for that panel is 1.85 s at 13:55 UTC, not crossed, search, so this read passes. The tolerances, set before the first run:

Peak value : within 5 percent of the true peak, after unit conversion. A unit from the wrong family, such as MiB on a latency panel, is wrong. "MB" counts as "MiB", because the axis says MiB.
Peak time : within 5 percent of the panel's time range. That is 3 minutes on a 1-hour panel, 18 on 6 hours and 72 on 24 hours.
Threshold crossed : exact. The answer is "yes", "no" or "no threshold".
Top series : the answer is matched to the nearest legend name, with at most 2 typing errors (1 for short names). A tie, or an answer that names two legend entries, counts as wrong. On a stacked panel, the top series is the one with the thickest band at the peak, not the band drawn on top, and the prompt says so.

A panel counts as read only when all four facts are right.

100 panels with known answers

Warning

These are Grafana-style panels drawn by our own renderer, not screenshots from Grafana. Real data has plateaus and near-equal peaks, which our generator rejects so that each question has one answer. Many real dashboards show a legend with Max or Last columns, which would turn the value question into text reading and help the models. A model trained on real Grafana screenshots could read real panels better or worse than ours. We do not know which.

The renderer writes SVG and converts it to PNG with @resvg/resvg-js. It copies the Grafana dark theme: panel background, Inter font, classic palette, grid, axis labels with units, a list legend, and thresholds as a dashed line, a shaded region or both. There are no tooltips and no legend values. A seeded generator (mulberry32) draws the data, and the truth comes from the same numbers.

What varies	Panels
Unit	ms 29, percent 22, req/s 23, bytes 26
Series per panel	1: 17, 2: 25, 3: 22, 4: 19, 5: 17
Log y-axis	25
Stacked areas	19
Similar colours (shades of one hue)	27
Small panel (420x230 or 360x200)	19
Threshold shown (line, region, both)	65 (28, 17, 20), crossed on 44
Peak at the right edge	28
Time range 1 h, 6 h, 24 h (UTC clock)	33, 36, 31
Noise low, medium, high	32, 42, 26

In an earlier project, a bug in our own test data made strong models look wrong, so we checked the truth first. A second script reads every panel back from the SVG alone, the way a reader would: the peak from the tick labels, the time from the clock labels, and the top series from the line colours and the legend (on the 19 stacked panels, from the band heights at the peak). It agreed with the truth on all 100 panels, with a largest peak error of 0.011 percent. To prove the check can fail, a third script breaks the renderer in a copy: wrong byte labels gave 38 problems, lines drawn 3 percent too high gave 204, and stacked bands drawn in the wrong colours failed the band check on all 19 stacked panels. We also viewed 14 panels next to their truth records.

Which models we tested

A probe sent the same question to 15 models, with no image and then with the smallest and the largest panel. A model passed if it answered and its prompt token count rose with the image, which proves the image arrived. 11 passed. Mistral 3 14B returned "model does not support image input", and Kimi K2.6 rejected temperature 0. Our account gets HTTP 403, "not available for your subscription tier", for the GPT models, so this test has open-weight models only.

Several vision models reason before they answer, which is too slow for a page. On the two probe panels, Kimi K3 took 28 and 165 seconds, DeepSeek V4.1 Flash 45 and 13, and Qwen 3.5 397B 71 (with no answer) and 42. We kept Nemotron 3.5 Super VL (nemotron-3.5-super-vl), with reasoning off and on, and three models that answered in 1.5 to 3 seconds: Qwen3.8-Max (qwen3.8-max), Gemma 4 (gemma-4-31B-it) and Llama 4 Maverick (llama-4-maverick). Nemotron Nano 12B v2 VL was also fast; we left it out to keep the test to four models.

Every call in the comparison used temperature 0 and at most 1,024 output tokens. Reasoning mode got 16,384, because the reasoning counts against that limit. Without reasoning, each model read every panel three times. Reasoning calls took about 20 seconds each, so that mode read all 100 panels once and a fixed 34-panel subset two more times.

Nemotron reasons by default. Two request settings turn it off. The /no_think system prompt does not:

stage 0b: turning reasoning off on one 1000x500 panel

$ node scripts/00b-reasoning-switch.mjs nemotron-3.5-super-vl
nemotron-3.5-super-vl | default | HTTP 200 | 22795 ms | completion 4096 | reasoning chars 13191 | $0.00291
nemotron-3.5-super-vl | reasoning_effort=low | HTTP 200 | 22271 ms | completion 4096 | reasoning chars 14191 | $0.00291
nemotron-3.5-super-vl | reasoning_effort=none | HTTP 200 | 1207 ms | completion 99 | reasoning chars 0 | $0.00031
nemotron-3.5-super-vl | chat_template_kwargs.enable_thinking=false | HTTP 200 | 1268 ms | completion 92 | reasoning chars 0 | $0.00031
nemotron-3.5-super-vl | system prompt /no_think | HTTP 200 | 21921 ms | completion 4096 | reasoning chars 14294 | $0.00291

That probe had a 4,096-token limit, and the reasoning runs used all of it without an answer. In the main run, none of the 168 reasoning calls hit the 16,384 limit. The longest used 11,979 tokens.

Results

Model	All four right (95% CI)	Peak value	Peak time	Threshold	Top series	$ per 1,000	Latency p50 / p95
Qwen3.8-Max	65 (55 to 74)	77	97	97	91	1.47	1.5 s / 2.2 s
Gemma 4	64 (54 to 73)	77	99	96	86	0.11	2.7 s / 3.4 s
Nemotron 3.5 Super VL, reasoning on	54 (44 to 63)	71	94	96	80	2.37	19.0 s / 58.5 s
Nemotron 3.5 Super VL, reasoning off	48 (39 to 58)	69	96	91	75	0.26	1.4 s / 1.8 s
Llama 4 Maverick	21 (14 to 30)	34	89	86	81	0.28	2.7 s / 9.1 s

Accuracy and cost are from run 1. Latency covers every successful call in all runs, and it is the time of the attempt that succeeded. The parallel runs hit HTTP 429 often (retried attempts: 88 for Nemotron with reasoning off, 7 with reasoning on, 99 for Qwen3.8-Max, 75 for Gemma 4), and those waits are not in the latency figures.

Panels with all four facts right, by panel feature (run 1)

	Value	Series
1 series	88.2%	Nemotron, reasoning off
1 series	88.2%	Qwen3.8-Max
1 series	76.5%	Gemma 4
stacked	31.6%	Nemotron, reasoning off
stacked	63.2%	Qwen3.8-Max
stacked	68.4%	Gemma 4
small panel	42.1%	Nemotron, reasoning off
small panel	63.2%	Qwen3.8-Max
small panel	57.9%	Gemma 4
similar colours	25.9%	Nemotron, reasoning off
similar colours	55.6%	Qwen3.8-Max
similar colours	48.1%	Gemma 4
4-5 series	30.6%	Nemotron, reasoning off
4-5 series	58.3%	Qwen3.8-Max
4-5 series	55.6%	Gemma 4
log y-axis	4%	Nemotron, reasoning off
log y-axis	8%	Qwen3.8-Max
log y-axis	32%	Gemma 4

100 Grafana-style panels drawn by our renderer, 31 October 2026. Panels per row: 1 series 17, stacked 19, small 19, similar colours 27, 4-5 series 36, log y-axis 25. Rows this small give wide intervals; read them as directions, not rates.

What stands out:

Time and threshold are easy. Every model got the peak time right on 89 to 99 panels and the threshold on 86 to 97.
A simple panel is almost solved. With one series, Nemotron and Qwen3.8-Max both got 15 of 17 right. Every added feature cost accuracy. Similar colours cost Nemotron the most: 16 of its 25 wrong series names were on the 27 similar-colour panels.
Qwen3.8-Max and Gemma 4 cannot be told apart. Each got 14 or 13 panels right that the other got wrong. Both beat Nemotron with reasoning off: Qwen3.8-Max on 20 panels against 3, Gemma 4 on 24 against 8 (sign test p = 0.007).
The 5 percent bar hurt Gemma 4 most. It rounds: 20,000 req/s for a peak of 21,330, 300 ms for 331. Within 10 percent, Gemma read the value right on 93 panels, Qwen3.8-Max on 87 and Nemotron on 82.

Repeat runs at temperature 0 showed another difference. Qwen3.8-Max and Gemma 4 gave the same verdict in all three runs on 97 and 98 panels. Nemotron with reasoning off did on 71, and gave the same four answers on only 13. Llama 4 Maverick gave the same four answers on 92 panels and was right in all three runs on only 21. Consistent is not the same as correct.

Where the reads go wrong

Log axes. The models read a log axis as if it were linear, or round to a tick. Nemotron read the value within 5 percent on 1 of 25 log panels, Qwen3.8-Max on 4 and Gemma 4 on 10.

The spike peaks at 21.3K req/s at 14:28 UTC. Every model found the series and the time. Qwen3.8-Max and Gemma 4 said 20,000 every time, 6.2 percent low. Nemotron's three runs said 30 req/s, 45,000 req/s and 30K req/s.

The threshold label as the answer. When a spike crosses a dashed line, the label of that line is the easiest number on the panel to read.

The rise at the right edge goes to 246 ms, above the 200 ms line. Qwen3.8-Max, Gemma 4 and Llama 4 Maverick answered 200 ms in all three runs: the threshold, not the peak. Nemotron answered 500 ms, 0.3 s and 300 ms. Only Llama 4 Maverick named inventory in every run, and it said the threshold was not crossed.

The band on top is not the biggest. On a stacked panel, the series drawn last sits on top, even when its own values are small.

node-c3 is the thin band on top. Qwen3.8-Max and Llama 4 Maverick named it in all three runs and Nemotron in two. Gemma 4 named node-a1 every time. This panel also shows the hardest threshold style, a shaded region with no line. The models got a region-only threshold right on 12 to 15 of 17 panels, and a region with a line on 19 or 20 of 20.

Reasoning on: slower and more expensive, and not clearly better

With reasoning on, Nemotron got 54 panels right against 48. Only reasoning got 14 panels right, and only no-reasoning got 8. A sign test gives p = 0.29, so 100 panels cannot say whether reasoning helps. The cost is clear: 3,349 output tokens per call on average in run 1 (8,163 at the 95th percentile), and 19.0 seconds at the median over all 168 calls, 81.8 at worst.

The usage block hides the reasoning. Every reasoning call returned reasoning_tokens: 0, while completion_tokens included the reasoning and the reasoning text came back in the reply. If you cap or alert on reasoning_tokens, you see nothing.
Temperature 0 does not explain it. Reasoning models are often tuned for a higher temperature, so we ran the 34-panel subset again at 0.6. It got 19 right, against 20 at temperature 0 and 20 with reasoning off.
It is not more repeatable. On the subset, reasoning gave the same four answers in all three runs on 3 panels, against 7 with reasoning off.

What a screenshot costs

DigitalOcean's pricing page ("Last verified 30 Oct 2026", read on 31 October) lists no separate image rate for these models. We priced each call from the token counts the API reported, at list input and output prices. That assumes the image tokens inside prompt_tokens are billed at the input rate. The closed October invoice agrees with this. For each of the four models, the change in billed input, cached input and output tokens is equal to the totals the API reported for our 1,522 recorded calls. Each rate on the invoice is the list price, and the invoice has no separate charge for images.

Model	Image tokens, 360x200	Image tokens, 1000x500	Prompt tokens (cached)	Output tokens	$ per screenshot
Nemotron 3.5 Super VL, reasoning off	288	514	652 (0)	97	0.00026
Nemotron 3.5 Super VL, reasoning on	288	514	652 (0)	3,349	0.0024
Qwen3.8-Max	68	498	594 (262)	126	0.0015
Gemma 4	266	255	604 (325)	102	0.00011
Llama 4 Maverick	437	1,017	1,096 (271)	72	0.00028

Image tokens are the prompt tokens with the panel minus those of the same question with no image. Gemma 4 uses about 260 at any size, Qwen3.8-Max scales with the pixels, and Llama 4 Maverick jumps from 437 to 1,017 above 600 pixels wide.

Our first cost numbers were wrong in our favour. Runs 2 and 3 send the same image again, and Qwen3.8-Max, Gemma 4 and Llama 4 Maverick served most of it from the prompt cache: Llama's cached tokens went from 271 in run 1 to 1,044. A bot sees each screenshot once, so we now price run 1 only. That raised Qwen3.8-Max from $1.11 to $1.47 per 1,000 and Gemma 4 from $0.097 to $0.113. Nemotron reported no cached tokens on any call, even for the prompt text that every call shares.

Rules decide, the model explains

The alert already knows the numbers. In our test, each alert comes from a rule on the panel's query, reduced with max over the panel's range, so the payload has the peak value, the series label (unless the rule is on a sum), the threshold and the firing state. If your rule reduces with last or a shorter window, compare against that value instead. The model reads the image blind, with no numbers from the alert, and three rules compare its read with the payload:

// lib/bot.mjs (shortened). tol is 10% by default.
export function crossCheck(alert, answer, { tol = 0.1 } = {}) {
  if (!answer) return { verdict: 'could not confirm', checks: [{ fact: 'read', ok: false }] };
  const checks = [];
  // 1. Peak value against the alert's reduced value.
  const p = readPeak(alert.unit, answer.peak);
  checks.push({
    fact: 'peak value',
    ok: p.ok && Math.abs(p.base - alert.values.B) / alert.values.B <= tol,
  });
  // 2. Series, when the alert is for one series. A rule on a sum has none.
  if (alert.labels.series) {
    checks.push({
      fact: 'series',
      ok: matchSeries(answer.topSeries, [alert.labels.series]) === alert.labels.series,
    });
  }
  // 3. Threshold state, when the model saw a threshold on the panel.
  const th = normThreshold(answer.thresholdCrossed);
  if (th === 'yes' || th === 'no') {
    checks.push({ fact: 'threshold', ok: th === (alert.status === 'firing' ? 'yes' : 'no') });
  }
  return { verdict: checks.every((c) => c.ok) ? 'confirmed' : 'could not confirm', checks };
}

The message always starts with the alert's own facts. The model's sentence is added only when its read agrees. Here is the bot on three panels, with Nemotron and reasoning off:

the bot on three alerts

$ node scripts/05b-bot-demo.mjs p034 p090 p003
--- p034 (data/panels/p034.png) -> confirmed, 2033 ms
Container RSS: firing, max 3.19 GiB over the last 6h, threshold 2.51 GiB.
Screenshot (checked against the alert): Container RSS memory usage spikes sharply around 08:15 UTC to over 3 GiB, driven primarily by worker-5c6b-h3t7, then settles back to roughly 1-2 GiB.
--- p090 (data/panels/p090.png) -> confirmed, 1380 ms
Checkout latency p95: firing, max 106.72 ms over the last 24h, threshold 73 ms, series search.
Screenshot (checked against the alert): Checkout latency p95 is stable around 30-40 ms until a spike to about 100 ms near 22:00 UTC, crossing the red 100 ms threshold.
--- p003 (data/panels/p003.png) -> could not confirm, 1311 ms
HTTP requests: firing, max 21.33K req/s over the last 6h, threshold 17K req/s, series POST /login.
Could not confirm the screenshot (the screenshot reads 25K req/s, the alert says 21.33K req/s). Open the dashboard before you trust the image.

The p090 message shows the limit. The value, series and threshold state agree, so the bot says "confirmed", but the summary says "the red 100 ms threshold" and the threshold is 73 ms. The rules check the structured answer, not every word. That is why the bot posts the alert's numbers itself.

Over all recorded reads:

Model	Wrong reads	Caught at 10%	Caught at 5%
Nemotron 3.5 Super VL, reasoning off	155 of 300	97 (63%)	125 (81%)
Qwen3.8-Max	109 of 300	58 (53%)	90 (83%)
Gemma 4	111 of 300	54 (49%)	98 (88%)
Llama 4 Maverick	236 of 300	143 (61%)	214 (91%)

"Correct reads blocked: 0" is by construction. Our alert value is the true peak and the bot's checks are looser than the scorer's, so a correct read cannot fail them. A real alert that reduces differently from the panel would add false flags. What the 10 percent check let through for Nemotron:

24 reads with only a value error between 5 and 10 percent, such as 300 ms for 331 ms
21 wrong series names, all on stacked panels, where the alert is on the sum and has no series to compare
6 threshold errors where the model said "no threshold" on a panel that shows one; the bot cannot tell that apart from a rule threshold that is not drawn
7 wrong peak times, alone or with a small value error; the payload has no peak time

Then we broke it on purpose: each read was checked against the alert of a different panel with the same unit, like a wrong image on the notification. The bot said "could not confirm" for 99 of 100 reads from Nemotron and from Qwen3.8-Max, and for 100 of 100 from Gemma 4 and Llama 4 Maverick. The pair that passed had a wrong alert on a sum (no series check) whose value was 8.6 percent from the read.

We expected the model to copy the alert's numbers if the prompt contained them, which is why the bot reads blind. For Nemotron that was wrong. With the wrong alert in the prompt, its peak matched the image on 81 of 100 panels and the alert on 1, and the cross-check still refused 98. We keep the blind read because it costs nothing, but this test does not show that it is necessary.

What 100 panels cannot tell you

The claim failed by a wide margin. The 95 percent intervals for Nemotron end at 58 and 63 panels, far below 90.
We cannot rank Qwen3.8-Max against Gemma 4, or Nemotron with reasoning against without. Both gaps are too small for 100 panels.
Feature rows are directions, not rates. Each covers 17 to 36 panels. Gemma 4's 8 of 25 on log axes has a 95 percent interval of 17 to 52 percent.
The panels are ours. Real Grafana screenshots are a different input, easier in some ways and harder in others.
One prompt. We did not try a prompt that warns about log axes. We did not use a JSON response mode, and Qwen3.8-Max wrote invalid JSON twice in 300 calls ("thresholdCrossed": no).
One day of one service. The models DigitalOcean served on 31 October 2026, at temperature 0. Prices are list prices times reported tokens. The closed October invoice billed the same token totals at the list prices.

Run it yourself

Each stage writes a file that the next one reads. Every recorded call is stamped with a hash of the runner code and the inputs. The scorer refuses unstamped, mixed or stale files, and records filed under the wrong configuration. The runner stops before its own count of spend passes a cap ($3.50 by default).

git clone https://github.com/The-DevOps-Daily/grafana-panel-vision
cd grafana-panel-vision
npm ci
cp .env.example .env # add your model access key

node scripts/01-render-panels.mjs # 100 PNGs + data/truth.jsonl + data/alerts.jsonl
node scripts/01b-check-panels.mjs # read every panel back from the SVG
mv results/raw results/raw-ours # keep our run; yours goes into results/raw
node scripts/02-run-models.mjs --configs nemotron-3.5-super-vl,gemma-4-31B-it --runs 3
node scripts/03-score.mjs # results/summary.md
node scripts/05-crosscheck.mjs # how often the bot catches a wrong read

To test your own dashboards, replace the renderer with exports of your real panels and write the truth for each by hand. It is slower, and it is the only way to know how these models read your panels.

Summary

A vision model on DigitalOcean can describe a Grafana panel in about 1.5 seconds for a fraction of a cent. On our 100 panels, no model read the four key facts reliably enough to be the source of truth in a page. Log axes, stacked areas and similar colours caused most errors, and reasoning did not fix them.

What we would ship:

Take the numbers from the alert, not from the image. The payload already has the value, the threshold and the series.
Let the model narrate, and check it first. Compare its value, series and threshold state with the payload, and say "could not confirm" when they disagree.
Use a fast model with reasoning off. Gemma 4 and Qwen3.8-Max were the most accurate, and Gemma 4 cost $0.11 per 1,000 screenshots.
Make attached panels easy to read. Use a linear axis, distinct colours, few series, and a threshold drawn as a line and a region.

Originally published at devops-daily.com.

Neon Anonymized Branches: Mask the Columns, Then Scan the Rest

DevOps Daily — Tue, 03 Nov 2026 15:00:00 +0000

We created an anonymized branch of a 20,000-user SaaS database on Neon. It had masking rules for the six PII columns of the users table: email, name, phone, street address, date of birth and signup IP. The masking run took 9 seconds. Afterwards, no row in users held its original email, name, phone, address or IP (3 of the 20,000 random dates of birth matched the original by chance). Then we read every text and JSON column on the branch. All 20,000 original email addresses were still there , and so was every original name, phone number, street address and IP address.

Most of them were in places nobody writes a rule for: the email copied onto each order at checkout, the JSON payloads of login and checkout events, the text of support tickets. The other 20,000 copies came from the masking run itself. The masking updates each row of users, so the audit trigger on that table fired 20,000 times and saved every original row into audit_log.

Anonymized branches mask the columns you list, and in our runs they did. This post measures what is left in the rest of a realistic schema, and shows the scanner we run as a CI gate before preview branches are made from the masked branch. Everything ran on a real Neon project through the Neon API, and the scripts are public.

TL;DR

Masking the listed columns works. No masked email, name, phone, address or IP kept its original value.
The copies are not masked. After round 1, 18,776 of the 20,000 original emails (93.9%) were still readable outside the audit table: in orders, event payloads, support tickets and invitations.
The masking run fires your triggers. audit_log grew from 1,888 to 21,888 rows during masking, and each new row held one user's full original row. The branch refuses connections until the run ends, so you cannot switch the trigger off first.
Free text and JSON need a decision. We found no rule that edits JSON in place, and regexp_replace removed emails and phone numbers from ticket bodies but not names. Only replacing free text and JSON passed the gate.
Pick generators with care. anon.dummy_free_email() and anon.dummy_safe_email() made the run fail on a unique column at 20,000 rows, and random IP addresses and phone numbers can be equal to real ones.
Previews from the passing template were ready in 2.3 to 2.5 seconds. The same rules failed the scan after a migration copied emails into a new table.

Prerequisites

A Neon account and an API key (Account settings > API keys). Everything here ran on a Scale plan project. Anonymized branches are a beta feature.
Node.js 22 or newer.
Basic Postgres knowledge: triggers, jsonb, materialized views.

What an anonymized branch does

Neon's data anonymization creates a branch of a Lakebase Postgres database from a parent branch, then masks it with PostgreSQL Anonymizer (the anon extension, version 2.5.1 on our branches). The masking is static: the run rewrites the rows on the new branch, and the parent does not change. You send the rules with the create call:

curl -X POST "https://console.neon.tech/api/v2/projects/$PROJECT_ID/branch_anonymized" \
  -H "Authorization: Bearer $NEON_API_KEY" -H "Content-Type: application/json" \
  -d '{
    "branch_create": { "branch": { "name": "template", "parent_id": "'"$MAIN_ID"'" } },
    "masking_rules": [
      { "database_name": "neondb", "schema_name": "public", "table_name": "users",
        "column_name": "full_name", "masking_function": "anon.dummy_name()" }
    ],
    "start_anonymization": true
  }'

Three details matter later:

The branch refuses connections until the run ends (Access to this endpoint is temporarily disabled). We saw the same error after each of our failed runs, so in our tests a failed run did not hand out unmasked data.
GET /branches/{id}/anonymized_status reports anonymizing, then anonymized or error.
Each rule names one column. Nothing in the API looks at the columns you did not name.

That is by design, and Neon's pg_redact post makes the same point: column rules cannot reach PII inside free text. We wanted to know how much PII that is in an ordinary schema, where it sits, and how to find it before someone gets a connection string.

The test database

The schema is a small invoicing SaaS. users holds the PII on purpose. Everything else holds it by accident, the way real apps do:

users email, full_name, phone, address, signup_ip
orders contact_email, shipping_*
support_tickets.body free text
events.payload jsonb
invitations invitee email, name, message
audit_log trigger: old and new row
customer_summary materialized view
users.search generated tsvector

Connections:

users -> orders (copied)
users -> support_tickets.body (quoted)
users -> events.payload (logged)
users -> invitations (named)
users -> audit_log (every UPDATE)
users -> customer_summary (refresh)
users -> users.search (generated)

A seeded generator writes 20,000 users, 40,191 orders, 6,558 tickets, 59,738 events and 2,035 invitations (47.6 MB). Then it changes the email or the address of 1,888 users, so the audit trigger has history with former values. All names are made up, emails use the reserved .example and .test domains, and phone numbers use the fictional 555-01xx range. The generator records every PII value it writes into each row, so we know what each column holds, and it mixes in decoys such as invoice numbers, tracking numbers and the app's own support address.

The scanner

The-DevOps-Daily/neon-anonymized-branches on GitHub

scan.mjs takes two branches: the anonymized branch and its parent. It reads the PII columns you list in pii-sources.json on the parent (here five columns of users), then reads every text, JSON, tsvector, inet and xml column of the user tables and materialized views on the anonymized branch (the anon extension's own tables are left out). Per column it reports:

known : rows that still hold a value from the parent, also inside free text and JSON. Phone numbers match on their last 10 digits, and a full name matches as two neighbouring words, so /team/tarek-stratkett in a URL matches "Tarek Stratkett".
pattern : values shaped like an email, phone number, IP address or street address that are not known values, such as former emails and people who are not users.
row counts of every table that changed during masking, and a fingerprint of the data on the branch (per table, the row count and a sha256 over the sorted md5 of each row).

It fails when a column holds a known value, when a column that no rule replaces holds PII-shaped values, when a rule changed no row, when the branch has a database it did not read, when the branch holds data it cannot read (such as a bytea column, large objects or foreign tables), or when the data changed during the scan. Two columns here are computed only from masked columns, a materialized view and a generated search column. They are on a review list: their PII-shaped values go to a person instead of failing the gate, and known values in them still fail.

A pass means the scan found no known value anywhere, and no PII-shaped value outside replaced and review columns. It does not mean the branch is clean. The scanner does not know phone numbers outside the US format, nicknames or encoded values, and it does not read numeric columns.

Against the unmasked parent, the rows it found matched the planted ground truth for all 33 combinations of column and kind. That is not an independent test of recall, because we planted the data in formats the scanner knows. It shows that the scanner counts what it should on this data. A second check, 08-predict.mjs, predicts each column from the parent's data and the rules, without Neon. Where the real scan differs from the prediction, the masking run did something on its own.

The first gate was too clever

The first version of the gate raised four false alarms on two templates. Three were values that the fake data generator made by chance and that belong to other users: 2 masked IP addresses and 1 phone number. The fourth was our bug in the tsvector name check. We first excused any known value that also appeared in a masked column. A review showed that this fails open: a real leak with the same value would pass too.

So the gate has no exception for chance matches. Every known value fails it, and a person reads the report. On real data this happens more often than in ours, because anon.dummy_name() returns real first and last names. A false alarm costs a few minutes. A leak costs much more. The old scans are in results/first-gate/.

Round 1: the columns everyone lists

The first rule set masks the PII columns of users, the way the docs and the GitHub Action examples do. For the unique email column we used the Console's Random Unique Email, pg_catalog.concat(anon.dummy_uuidv4(), '@example.com'), which Neon recommends for unique columns.

scan the round 1 template

$ node scripts/scan.mjs --target template-v1 --known main --evidence --out results/scan-template-v1.json
scanned 26 columns in 7 tables of template-v1 (68.8 s)
  audit_log.old_row rule=none known: email 20938, name 21888, phone 18552, address 20950, ip 21888 pattern: email 950, address 938
  audit_log.new_row rule=none known: email 1888, name 1888, phone 1585, address 1888, ip 1888 pattern: email 20000, phone 17807, address 954, ip 20000
  customer_summary.email rule=review known: - pattern: email 20000, phone 227
  events.payload rule=none known: email 19931, name 12816, ip 11888 pattern: ip 8017
  invitations.invitee_email rule=none known: email 585 pattern: email 1450
  invitations.invitee_name rule=none known: name 585 pattern: -
  invitations.message rule=none known: name 2035 pattern: -
  orders.contact_email rule=none known: email 40191 pattern: -
  orders.shipping_name rule=none known: name 36939 pattern: -
  orders.shipping_address rule=none known: phone 31371, address 36939 pattern: phone 3252, address 3252
  support_tickets.body rule=none known: email 1324, name 1231, phone 1162, address 588 pattern: email 687, phone 622
  users.search rule=review known: - pattern: email 20000, phone 220
  warning: public.audit_log has 21888 rows, the parent has 1888 (+20000)
  review: customer_summary.email: 20000 rows hold PII-shaped values (review: materialized view of users.email)
  review: users.search: 20000 rows hold PII-shaped values (review: generated from users.full_name and users.email)
FAIL: 16 problems

These are the rows that still hold a user's original PII:

Column	Rows	Rows with a user's original PII	What
`audit_log.old_row`	21,888	21,888	every kind; 20,000 rows were written by the masking run
`orders.contact_email`	40,191	40,191	the email, copied at checkout
`orders.shipping_name`	40,191	36,939	the name
`orders.shipping_address`	40,191	36,939	address and phone, inside `jsonb`
`events.payload`	59,738	34,689	emails, names, IPs, inside `jsonb`
`support_tickets.body`	6,558	2,486	free text
`invitations.message`	2,035	2,035	the inviter's name
`audit_log.new_row`	21,888	1,888	history from before the branch
`invitations.invitee_email`, `invitee_name`	2,035	585	invitees who are users

Counted as distinct values, this is how much is still readable on the branch:

Kind	Readable anywhere	Readable outside `audit_log`
Email	20,000 of 20,000	18,776 (93.9%)
Full name	17,396 of 17,396	16,024 (92.1%)
Phone	16,967 of 16,967	13,639 (80.4%)
Street address	20,000 of 20,000	15,907 (79.5%)
IP address	20,000 of 20,000	9,208 (46.0%)

The scan also flagged PII of people who are not users, such as 1,450 invitee emails and 8,017 login IPs that are not signup IPs. The prediction matched 21 of the 26 columns, and each difference points to something the masking run did. Two of them are good news: the generated search column and the materialized view customer_summary held no original value, because Postgres computes a generated column again on every UPDATE and Neon refreshes materialized views after the run. The audit trigger caused the others.

The masking run fires your triggers

The prediction for audit_log was 1,888 rows. The branch had 21,888. Static masking updates every row of each table that has a rule, so a row-level AFTER UPDATE trigger fires once for each masked row. Our trigger is a common audit pattern:

CREATE FUNCTION audit_users() RETURNS trigger LANGUAGE plpgsql AS $$
BEGIN
  INSERT INTO audit_log (table_name, row_id, action, old_row, new_row)
  VALUES ('users', OLD.id, TG_OP, to_jsonb(OLD), CASE WHEN TG_OP = 'UPDATE' THEN to_jsonb(NEW) END);
  RETURN NULL;
END $$;

So the masking wrote 20,000 new audit rows, and each old_row holds one user's original email, name, phone, address and IP. A rule on users alone made the branch hold a second full copy of users.

There is no clean way to stop this on the branch:

The branch refuses connections before the run starts. With start_anonymization: false our client was refused too, so you cannot run ALTER TABLE ... DISABLE TRIGGER first.
The trigger cannot skip the masking run by role. All 21,888 audit rows on the masked branch record neondb_owner, the role the app uses too.

What worked was a rule on the audit table itself (old_row and new_row masked with NULL). In our runs Neon masked users first and audit_log after it, so the masking also cleared the rows that its own trigger had written. We put the audit rules first in one rules file (v2-audit-first), and the order did not change. We did not find this order documented, so do not depend on it. The 20,000 extra rows stay on the branch with NULL in both columns, which is why the scanner reports tables whose row count changed.

Round 2: mask the copies, edit the free text

The second rule set (v2-careful) adds a rule for every structured copy the first scan found: the order contact email and shipping name, the shipping address rebuilt with jsonb_build_object, the invitee email and name, and the two audit columns. For the ticket bodies it keeps the text and cuts out emails and phone numbers with the same patterns the scanner uses:

pg_catalog.regexp_replace(
  pg_catalog.regexp_replace(body, '<email pattern>', '[email]', 'g'),
  '<phone pattern>', '[phone]', 'g')

Three columns still failed:

Column	Rule	Rows with a user's original PII
`events.payload`	none	34,689 (emails 19,931, names 12,816, IPs 11,888)
`invitations.message`	none	2,035 (names)
`support_tickets.body`	`regexp_replace`	1,819 (names 1,231, addresses 588)

The regexp_replace rule did its job. No email or phone number was left in any ticket body, and the JavaScript prediction matched the scan exactly. But regular expressions find shapes, and a name has no shape. "Contact Maeve Ashquist on [phone] before 5pm" is still personal data.

The events table has no rule because we could not write one that keeps the payload. The rule lab (03-rule-lab.mjs) tried 17 expressions, each on its own anonymized branch:

Expression	Result
`anon.dummy_ipv4()::inet`, `CAST(... AS inet)`	rejected with 400: not a valid label
`CASE WHEN id % 2 = 0 THEN 'even' ELSE 'odd' END`	rejected with 400
`pg_catalog.jsonb_set(doc, '{email}', ...)`	rejected with 400
`pg_catalog.regexp_replace(doc, ...)` on `jsonb`	the run failed: no `regexp_replace(jsonb, ...)`
`pg_catalog.jsonb_build_object('email', anon.dummy_safe_email(), ...)`	masked: the whole document is rebuilt
`masking_value: '{}'`	masked

The rule parser accepts function calls from the anon schema and 36 pg_catalog functions that the extension labels as trusted, such as concat, regexp_replace, md5 and jsonb_build_object. jsonb_set is not one of them, and without casts a rule cannot turn jsonb into text and back. So in our tests a JSON column had two options: rebuild the whole document from fake values, which loses its shape when the shape changes from row to row (our events have four), or replace it with a constant.

Round 3: replace what you cannot edit

The third rule set (v3-strict) replaces what round 2 could only edit: ticket bodies become anon.lorem_ipsum(words := 20), invitation messages become anon.lorem_ipsum(words := 10), and event payloads become {}. It also takes IP addresses and phone numbers from ranges that no real value can be in (more on that below). That passed:

scan the round 3 template

$ node scripts/scan.mjs --target template-v3 --known main --evidence --out results/scan-template-v3.json
scanned 26 columns in 7 tables of template-v3 (41.7 s)
  customer_summary.email rule=review known: - pattern: email 20000, phone 212
  users.search rule=review known: - pattern: email 20000, phone 207
  warning: public.audit_log has 21888 rows, the parent has 1888 (+20000)
  review: customer_summary.email: 20000 rows hold PII-shaped values (review: materialized view of users.email)
  review: users.search: 20000 rows hold PII-shaped values (review: generated from users.full_name and users.email)
PASS: no PII from the source columns and no PII-shaped values found outside replaced and review columns (2 to review)

It took 37 seconds to mask six tables. This is how much of the original PII each round left readable, counted as distinct values:

Original PII still readable on the masked branch

	round 1	round 2	round 3
Email	100%	65.2%	0%
Full name	100%	59.5%	0%
Phone	100%	0%	0%
Street address	100%	2.9%	0%
IP address	100%	46%	0%

Distinct original values of the 20,000 users found anywhere on each template, as a share of all distinct values of that kind (20,000 emails, 17,396 full names, 16,967 phones, 20,000 addresses, 20,000 IPs). Round 1: 6 rules on users. Round 2: 14 rules, regexp_replace on ticket bodies. Round 3: 16 rules, free text and JSON replaced.

The pass has a price. The support inbox in a preview shows lorem ipsum, and the events table has no payloads, so code that reads payloads has nothing to test against. If those features need realistic data, you can generate synthetic tickets and events for the template, or keep the free text and give the branch to fewer people. The regex rule is not a middle ground, because it cannot find names.

The gate, and preview branches from the template

The workflow we tested is: create the anonymized branch, wait for the run, scan it, and only then make preview branches from it.

main production data
anonymized branch masking rules from git
wait state: anonymized
scan.mjs known values + patterns

Outcomes:

pass: make preview branches
fail: delete the branch

The wait matters if you use Neon's GitHub Action. neondatabase/create-branch-action v6 accepts masking_rules and returns the connection string right away, before the masking run ends. A job that connects next gets Access to this endpoint is temporarily disabled, and after our failed runs the branch stayed closed, so a job that only retries waits until it times out. scan.mjs --wait polls anonymized_status first and fails if the run did not finish. The repository has an example nightly workflow with these steps. We ran the steps through the scripts, not in GitHub Actions.

Two more things to know about the template branch:

Do not give it an expiry. The API refused to create a child from an anonymized branch with expires_at: Branches with an expiration date cannot have child branches. Set expires_at on the previews instead.
Check the data before you make previews. A scan report is only good for the data it read. 04-previews.mjs refuses to branch from the template unless its report is a pass, the branch has no database and no data that the scan did not read, and the data still has the scanned fingerprint. It then checks each preview against the same fingerprint.

From the passing template, three previews were ready in 2.3 to 2.5 seconds each, and each one matched the scanned fingerprint. To see the check refuse, the script makes a copy of the template, scans it, and writes one support ticket with a real user's email into it, the way someone might paste a production row into a shared branch:

previews behind the gate

$ node scripts/04-previews.mjs --template template-v3
gate: template-v3 passed its scan and still has the scanned data (0ff0ef425216dfe8)
preview-pr-101: ready in 2.4 s, checked after 6.1 s, same data as scanned: true
preview-pr-102: ready in 2.3 s, checked after 6.0 s, same data as scanned: true
preview-pr-103: ready in 2.5 s, checked after 6.1 s, same data as scanned: true
reset br-restless-paper-b9i5hgnz to the template in 3.9 s; same data as scanned again: true
stale gate: after the scan open; after one INSERT refused: data changed since the scan: public.support_tickets (rows 6558 -> 6559)

So treat the template as read-only once it passes: hand out preview branches, never the template's own connection string, and scan again after any change to it.

Then we broke it on purpose. 07-break.mjs runs a migration on a branch of main that adds a newsletter_signups table, a copy of 9,956 users' emails and names. The round 3 rules do not know that table, so we made a new template with them and scanned it:

scan after the migration

$ node scripts/scan.mjs --target template-drift --known main-drift --out results/scan-template-drift.json
scanned 29 columns in 8 tables of template-drift (39.1 s)
  customer_summary.email rule=review known: - pattern: email 20000, phone 213
  newsletter_signups.email rule=none known: email 9956 pattern: -
  newsletter_signups.full_name rule=none known: name 9956 pattern: -
  users.search rule=review known: - pattern: email 20000, phone 207
  warning: public.audit_log has 21888 rows, the parent has 1888 (+20000)
  review: customer_summary.email: 20000 rows hold PII-shaped values (review: materialized view of users.email)
  review: users.search: 20000 rows hold PII-shaped values (review: generated from users.full_name and users.email)
FAIL: 2 problems

This is why the scan reads all the text and JSON columns, and not only the columns in the rules file. Rules describe the schema as it was when someone wrote them. The scan checks the schema as it is now. The new first_name column was not flagged, because a first name alone is not a full name.

What your tests will notice

Unique columns. Fake data generators repeat themselves. We generated each email function many times and counted the repeats:

Function	Repeats in 1,000	Repeats in 20,000	Repeats in 100,000
`anon.dummy_free_email()`	0	180	4,504
`anon.dummy_safe_email()`	57	11,967	90,980
Random Unique Email (`concat(anon.dummy_uuidv4(), '@example.com')`)	0	0	0
`anon.fake_email()`	378	19,001	99,001

On the unique users.email column, masking runs with dummy_free_email() and dummy_safe_email() both failed with duplicate key value violates unique constraint "users_email_key". In the final rule lab run, dummy_free_email() also failed on a 200-row table with a unique column, so a small table is no guarantee. The unmasked data stayed unreachable, but a nightly template job fails.

Where fake emails point. dummy_free_email() writes addresses at yahoo.com, hotmail.com and gmail.com, so a fake address can belong to a real person. If a preview app sends password resets or receipts, use a function that writes example.com addresses, or switch email off in previews.

Stability. Two templates made from the same parent with the round 1 rules shared no masked email, name, phone, address or IP, so a test that expects "user 42 is called X" fails on the next template. Values derived from the primary key do not move: with pg_catalog.concat('user-', id, '@example.com') and anon.pseudo_first_name(id), both templates matched in 20,000 of 20,000 rows. Only use such rules where the input is not PII itself, because anyone with the function can compute the output again.

Foreign keys. The docs say foreign key columns cannot be masked directly, and that the Console shows an alert instead. Through the API, a rule on orders.user_id was accepted and the run completed. Afterwards, all 40,191 orders belonged to a different user than on the parent, so joins between orders and users no longer match the parent.

Generators that can produce real values. With the first gate, 2 masked IP addresses from anon.dummy_ipv4() and 1 phone number from anon.dummy_phone_number() were equal to real values of other users. The round 3 rules take IP addresses from the 198.51.100.0/24 documentation range and give phone numbers the area code 000, which no real number has. Names are harder, because anon.dummy_name() returns real names.

anon.detect() suggests PII columns from their names. On our schema it missed full_name, street_address, signup_ip and every copy. It is a start for a rules file, not a check.

How long masking takes

The masking run rewrites every row of every table that has a rule. 06-timing.mjs masked three columns of one table at three sizes, three runs each, with the compute fixed at 1 CU:

Rows	Table size	Masking run (median, min to max)	API call to usable branch (median)
10,000	2.3 MB	3 s (3 to 4)	13.2 s
100,000	22.1 MB	6 s (5 to 10)	15.6 s
1,000,000	226.6 MB	22 s (17 to 32)	33.8 s

The time grew much less than the row count: 100 times the rows took about 7 times as long. It also varied between runs, from 17 to 32 seconds at 1,000,000 rows, and three runs per size cannot show why. Neon reports the run's start and end in whole seconds. The time to a usable branch also includes creating the branch and starting its compute. For our 47.6 MB database it was 20.3 seconds for round 1 and 47.6 seconds for round 3, which rewrote six tables. We did not measure storage.

What we could not conclude

One schema and data we wrote. The scanner found everything we planted, in the formats we chose. Real free text has misspellings, nicknames and names in other scripts, so its recall on real data will be lower, and we cannot say by how much.
Names of people who are not users are invisible. In our data that left 5,985 names the scanner could not see by design: gift recipients, colleagues, accountants.
The table order is observed, not documented. With a different schema Neon may mask the audit table first, and then the trigger's copies would stay unmasked.
Small sizes and one region. One project in aws-eu-central-1, PostgreSQL 18, tables up to 1,000,000 rows, three runs per size.
Static masking only. A Neon blog post says dynamic masking is planned. We tested what is available now.

Summary

Anonymized branches masked every column we named. The leaks were all in our schema: copies made at checkout, free text, JSON payloads, a trigger, a new table from a migration. A rules file cannot list what nobody knows is there, so treat the masked branch the way you treat a build: test it before anyone gets a connection string.

What to do:

Write rules for the obvious columns, then scan the masked branch to find the copies.
Mask audit and history tables in the same rule set, and check row counts after the run.
Replace JSON columns that hold PII. We found no rule that edits JSON in place.
Decide what to do with free text. regexp_replace removes emails and phone numbers, not names.
Use Random Unique Email (or a value derived from the primary key) for unique email columns, and keep fake emails on example.com.
Gate on the scan, wait for the masking run before you connect, and make previews only from a template that passed. Do not give the template an expiry.

Run it against your own project:

git clone https://github.com/The-DevOps-Daily/neon-anonymized-branches
cd neon-anonymized-branches && cp .env.example .env # add a Neon API key
npm install && npm run setup && npm run seed
node scripts/02-template.mjs --rules rules/v1-obvious.json --name template-v1
npm run scan -- --target template-v1 --known main

Originally published at devops-daily.com.