DEV Community: Kalislav Smirnov

A /24 Subnet Fits 80 Pods on EKS. Your Nodes Will Be Bored.

Kalislav Smirnov — Wed, 21 Oct 2026 07:01:49 +0000

An m5.xlarge running 10 pods holds 30 IP addresses from your subnet.

That's with the defaults: the Amazon VPC CNI in secondary-IP mode and WARM_ENI_TARGET=1. A /24 has 251 usable addresses, so it's full at eight such nodes, 80 pods. If those are small 250m / 512Mi pods, the eight nodes have 62% of their CPU and 31% of their memory requested. They're bored, and you still can't schedule pod number 81. Most teams find out from a pod stuck in ContainerCreating:

Warning  FailedCreatePodSandBox  kubelet
  failed to assign an IP address to container

I went through the AWS docs, the VPC CNI source and every public IP-exhaustion story I could find, and ran the numbers for all 1,328 instance types with published ENI limits. Here's what eats your subnet, and the three ways out.

The warm ENI nobody asked for

With the default settings every pod gets a real VPC address from one of the node's ENIs (elastic network interfaces). The CNI keeps one spare ENI fully loaded (WARM_ENI_TARGET=1) so new pods start fast. An ENI on an m5.xlarge carries 15 addresses, so the node holds:

pods on node   ENIs in use + warm   IPs held   IPs per pod
10             1 + 1                30         3.0
20             2 + 1                45         2.3
58 (max)       4                    60         1.0

Why would anyone want that? Because attaching a new ENI can take up to 10 seconds, and the CNI docs say plainly that this is "the main reason why we have WARM_ENI_TARGET=1 as the default setting." Fast pod starts, paid for in addresses.

Small, half-empty nodes are the worst case. At a quarter of max pods, a t3.small holds 4 IPs per running pod, a t3.medium or t3.large 3, an m5.large 2.86, and the whole xlarge class (m5, m6i, c5, r5) 2.14. Across the fleet it gets silly: for 926 of 1,328 instance types, a node running a single pod holds 60 or more addresses, and for 604 types one full node plus its warm ENI is already bigger than a /24.

What it looks like when it happens to someone big

The best public write-up I found is Neon's: AWS CNI lessons from a Production Outage (June 2025, with a postmortem). They're not a small shop with a /24. They had three /20 subnets, 12,273 usable addresses, in us-east-1, starting around 500 pods a minute.

When the pod count grew from roughly 5,000 to 8,000, only 71% of the addresses were assigned to pods, but 99% were allocated to nodes. One node held about 450 IPs for 200 pods. In their words, "we never free IP addresses back to the subnet." Pod starts stalled for a total of 5.5 hours over two days.

The fix that sounds obvious made it worse first. With WARM_IP_TARGET=1, every pod deletion put an IP into a 30-second cooldown that still counted as "warm," so new pods waited. They ended up setting all three warm targets to 0.

It isn't a one-off:

Wealthsimple watched one EKS subnet go from 80% to 100% in two weeks during a migration (November 2024). Short term, more subnets. Long term, custom networking.
Adevinta got paged because DaemonSets on a small, mostly unused cluster failed with "failed to assign an IP address to container" (2024), and later paused their own migration because addresses ran low.
A VPC CNI issue from 2020 shows that event repeated 23,123 times over 21 hours on m5 nodes.

The common thread: plenty of CPU, no addresses.

Max pods is an ENI limit, not a CPU limit

With secondary IPs:

max pods = ENIs × (IPs per ENI − 1) + 2

m5.large    3 × (10 − 1) + 2 = 29
t3.medium   3 × (6 − 1)  + 2 = 17
m5.xlarge   4 × (15 − 1) + 2 = 58

The "+ 2" is aws-node and kube-proxy, which run on the host network and need no pod IP. These numbers match AWS's own eni-max-pods.txt for every one of the 1,302 types that appear in both lists.

Past that number pods stay Pending, however much CPU is free. And since EKS reserves 11 MiB of memory per possible pod, the same number also decides how much memory the node gives you. More on that here:

Kalislav Smirnov

Oct 11

Your Kubernetes node is smaller than you think: 3 reservation traps in EKS, GKE and AKS

#kubernetes #devops #aws #cloud

3 min read

Here's a twist I didn't expect: for most clusters, max pods is not the limit that hits first. With 250m / 512Mi pods, only 132 of 1,264 non-GPU instance types run out of ENI slots before they run out of CPU or memory. An m5.large fits 6 such pods by resources and 26 by ENIs. Max pods only bites with tiny pods: at 100m / 128Mi it's the limit on 620 types. The thing that runs out first is usually the subnet.

And a confession. While checking the article, I found that my own EKS calculator got max pods wrong for 62 big instance types, the ones with several network cards, like c6in.32xlarge. AWS only counts the ENIs of network card 0, so the right answer is 394 pods, not the 786 my code showed. It's fixed now. The AWS script got the same fix in 2024, so at least I was wrong in good company.

Way out 1: fewer warm IPs

The cheapest fix is a config change on the aws-node DaemonSet: replace the warm ENI with WARM_IP_TARGET and MINIMUM_IP_TARGET. The node keeps a few spare addresses instead of a whole ENI's worth.

How many pods a subnet fits, for m5.xlarge nodes:

                      default (warm ENI)      WARM_IP_TARGET=2 +       prefix delegation
                                              MINIMUM_IP_TARGET=pods
/24, 10 pods/node      8 nodes    80 pods      22 nodes   220 pods      7 nodes    70 pods
/24, 20 pods/node      5 nodes   100 pods      11 nodes   220 pods      5 nodes   100 pods
/22, 20 pods/node     22 nodes   440 pods      46 nodes   920 pods     20 nodes   400 pods
/20, 10 pods/node    136 nodes 1,360 pods     371 nodes 3,710 pods    123 nodes 1,230 pods

Same subnet, 2.7× the pods, one environment variable.

The price is more EC2 API calls (the CNI docs warn about throttling with low targets) and slower starts during bursts, because addresses are attached on demand. Neon's cooldown story above is what that looks like at 500 pods a minute. There's also a long-running issue where nodes report Ready before their IPs exist when these targets are set. Tune it, then test a burst before you trust it.

Way out 2: prefix delegation (on dense nodes)

Prefix delegation puts a /28 (16 addresses) into each ENI slot, so an m5.large can run 110 pods instead of 29. The cap is 110 for nodes with up to 30 vCPUs and 250 above that, and managed node groups enforce it. Attaching a prefix takes under a second instead of up to 10 for an ENI.

But the node now takes addresses in blocks of 16, plus one warm block. At 20 pods an m5.large holds 49 IPs with prefixes and 30 without. On sparse nodes, prefixes cost more addresses, not fewer, which is why the prefix column in the table above never wins on a /24. AWS's own guidance is honest about it: "If you can afford (in the worst case) up to 15 unused IPs per node, we strongly recommend using the newer prefix assignment."

It also needs contiguous free /28 blocks. A fragmented subnet fails with InsufficientCidrBlocks even when plenty of single addresses are free. CloudKeeper wrote up exactly that: a /25 with prefix delegation, the console showing free IPs, and zero intact /28 blocks left.

A subnet CIDR reservation for prefixes avoids that. Set it up before the subnet fragments, not after.

Way out 3: custom networking

The usual way out at scale: add a secondary VPC CIDR (AWS recommends 100.64.0.0/10) and put pods there with custom networking. The node's primary ENI no longer hosts pods, so max pods drops: an m5.large goes from 29 to 20. Wealthsimple hit exactly that number in their migration: "large instance types are limited to 20 pods, whereas without custom networking we could run 29."

You trade a few pods per node for a pod address space that's no longer shared with everything else in the VPC. If your subnets are already tight, it's the only fix that doesn't depend on how bursty you are.

The checklist I'd run today

Count allocated, not assigned. The VPC console's "available IPs" for your node subnets is the real number. Pods-per-node math hides the warm pool.
Look at your smallest, emptiest nodes. They hold the most addresses per pod.
Change WARM_ENI_TARGET deliberately. Either keep it and size subnets for it, or switch to WARM_IP_TARGET / MINIMUM_IP_TARGET and test a burst.
Prefix delegation for dense nodes only, with a CIDR reservation.
Plan custom networking before the migration, not during it. eksctl already creates /19 subnets by default for a reason.

Do the math before the subnet does

I built a calculator for this in NodeSize. Pick an instance type and CNI mode, enter nodes and pods per node, and it shows max pods, IPs per node including the warm pool, and whether your subnet fits: EKS max pods and subnet IP calculator. It covers the ENI limits of 1,328 instance types, with the formulas and AWS sources on the page (and now the right answer for multi-card instances).

Your turn

When your subnet ran dry, what did you do: bigger CIDR, prefixes, warm-IP tuning, or custom networking?

I Built a Kubernetes CVE Bot. Its First Alert Was About Eclipse Che.

Kalislav Smirnov — Wed, 14 Oct 2026 07:01:07 +0000

I wanted exactly one thing: a message on my phone when a new CVE hits a version I actually run.

So I built a Telegram bot. It pulls the official Kubernetes CVE feed, GitHub advisories, NVD and CISA's "known exploited" list, merges them and posts to a channel. Simple, clean, exactly one job.

Then I opened the channel to admire my work. The very first post was a CVE in Eclipse Che. Eclipse Che is a browser IDE. A browser IDE is not what I asked for.

Eclipse Che runs on Kubernetes. It is not Kubernetes.

Eclipse Che is a perfectly nice browser IDE. It runs on Kubernetes. So do a lot of things. My bot did not appreciate the difference.

NVD's keyword search for "kubernetes" happily returns anything that mentions the word somewhere, in a reference link, a product list, a footnote. My bot, raised on exact string matching, took every one of those as a yes. Out of its first six posts, three went to Eclipse Che, the HyperShift operator and Apache Camel Karavan.

You know the two lumberjacks. One yells "we're cutting down the wrong forest!" and the other yells back "who cares, look at the progress!". My channel was making excellent progress.

The fix was boring. Components I watch by name get through from medium severity up; everything else from the wider ecosystem needs CVSS 7+ and has to talk about Kubernetes in its own first sentence. Posts that stop qualifying get deleted. (The channel shrank by half. Very professional.)

A CVE feed tells you everything. That's the problem.

Even with clean data, a feed has a structural problem. It tells you what was published. You want to know what hits you.

Remember IngressNightmare? Four CVEs in ingress-nginx (CVE-2025-1097, CVE-2025-1098, CVE-2025-24514 and CVE-2025-1974), unauthenticated remote code execution. Wiz found over 6,500 clusters with the vulnerable admission controller exposed and estimated that about 43% of cloud environments were vulnerable.

And what did most of us do that week? Not read the advisory. We opened the Hacker News comments to find out whether it was bad, then went to check which ingress-nginx version we had, then went looking for the chart value to bump. The advisory was open in another tab the whole time, unread, like a gym membership.

Three questions, every time: am I on an affected version, what do I upgrade to, and what's the exact command. A feed answers none of them. It wasn't built to.

Fine. The bot asks kubectl instead.

The new feature is a cluster snapshot. You send the bot /cluster, it gives you one command to copy:

kubectl version -o json
kubectl get nodes -o jsonpath='{range .items[*]}{.status.nodeInfo.kubeletVersion} {.status.nodeInfo.containerRuntimeVersion}{"\n"}{end}'
kubectl get pods -A -o jsonpath='{range .items[*]}{range .spec.containers[*]}{.image}{"\n"}{end}{end}' | sort -u

That prints versions and image names. No secrets, no env vars, no manifests. You paste the output into the chat (or send it as a .txt, Telegram cuts long messages), and the bot works out what you run from the images: the control plane, kubelet, containerd, ingress-nginx, CoreDNS, Cilium, Istio, Argo CD, cert-manager and friends.

It keeps only component and version pairs. Your private registry and your payments-api image are read once and forgotten. /forget deletes the rest. I don't want your cluster. I have enough trouble with mine.

A boring EKS cluster: 9 components, 0 of them fine

To test it I wrote a snapshot shaped like a very ordinary EKS cluster: Kubernetes 1.29, containerd 1.7, Argo CD 2.10, cert-manager 1.13, the community ingress-nginx, Flux, and Traefik on latest because someone was in a hurry. A made-up cluster, not anyone's real one. Just the kind we all have somewhere.

Here's a trimmed version of what the bot replied, run against its live data:

🛰 Cluster snapshot · Amazon EKS · 9 components
🔴 affected: 7 · ⏳ support: 5 · ✅ fine: 0 · ❔ unknown: 2

Argo CD v2.10.4: 14 CVE
  🔴 CVE-2025-55190: Argo CD's Project API Token Exposes Repository Credentials
→ upgrade to v3.2.12
helm upgrade  argo/argo-cd -n  --reuse-values --version 

ingress-nginx v1.10.1: 12 CVE
  🔴 CVE-2025-1974: ingress-nginx admission controller RCE escalation
→ upgrade to v1.13.8

⏳ Support
• Kubernetes 1.29 (EKS): support ended 2026-03-23, no more security fixes.
• ingress-nginx: retired in March 2026, no more releases or security fixes.

❔ Version unknown
• Traefik: the image tag has no version (latest or digest), check it by hand

Seven of nine components affected, five out of support, and zero "you're fine". The two it couldn't read, it says so instead of guessing. I'd rather get "I don't know" than a green tick that lies.

That ingress-nginx line is the one I'd frame. Kubernetes retired Ingress NGINX in March 2026. One of the most widely used ingress controllers now has open CVEs and no future patches. If your cluster still runs it, the report says so in plain words, which is more than the cluster will ever do.

A CVE list doesn't get an upgrade scheduled. $365 a month does.

Then I tried snapshots on EKS 1.31, 1.32 and 1.33, and every one came back with this:

• Kubernetes 1.32 (EKS): standard support ended 2026-03-23, now in extended support
  until 2027-03-23. That's +$365/month per cluster (+$0.50/h at AWS prices).

That's straight from AWS pricing: $0.10 per cluster-hour in standard support, $0.60 in extended. Fifty cents an hour doesn't sound like anything until you multiply it by 730 hours and by the number of clusters nobody upgraded.

Security people get a CVE list. Whoever pays the AWS bill gets a number. Turns out the number is what gets an upgrade on the calendar. (I'd love to say the CVEs did it. Nobody has ever rescheduled a sprint over a CVSS score.)

It's not Trivy (and it once flagged a 2023 bug on Kubernetes 1.34)

It's not Trivy. Trivy and trivy-operator scan every package inside your images; this bot looks at the versions of the platform components you run. Different question: "which of my cluster's building blocks are affected" rather than "which library inside my app image". If you run Trivy, keep running it.

It also can't see what kubectl can't: runc, for example, so it tells you to run runc --version on a node. Honest beats clever.

And version ranges are harder than they look. During development my checker announced that Kubernetes v1.34 was vulnerable to CVE-2023-5528, a 2023 Windows bug, because one source described the affected range as ">= v1.8.0" with no upper end. A 2023 bug flagged on a 2025 release. (It was me. It's always the regex.) Now the fixed versions override open-ended ranges, and the bot has stopped time-travelling.

For context, the biggest free option in Telegram is @cveNotify with about 19.5k subscribers. It posts every CVE in the world, unfiltered. That's a different tool for a different person, probably one with more patience than me.

Try it on your cluster

The bot is @cveAlertive_bot. Snapshot reports are free and unlimited, you can save one cluster for free, and you get a weekly digest that says either what hit you or "nothing touched your versions this week" (the second message is the one you actually want).

The paid part is the channel, 199 Telegram Stars a month: personal alerts within about half an hour when a new CVE hits a saved cluster's versions, a 🔥 follow-up when it lands in CISA's exploited list, and up to five clusters.

If you liked counting money you didn't know you were spending, the previous post was about the same thing for node sizes:

Kalislav Smirnov

Oct 11

Your Kubernetes node is smaller than you think: 3 reservation traps in EKS, GKE and AKS

#kubernetes #devops #aws #cloud

3 min read

Your turn

How do you find out today that a new CVE hits a version you actually run, and how long does it take you to be sure? A scanner in CI, a Slack channel nobody reads, the cloud provider's bulletin, or the Hacker News thread? If your honest answer is "Trivy already does this", say so, I want to hear that case too. And tell me what a report like this should say that mine doesn't.

Your Kubernetes node is smaller than you think: 3 reservation traps in EKS, GKE and AKS

Kalislav Smirnov — Sun, 11 Oct 2026 11:02:33 +0000

You buy a 16 GiB node. Kubernetes lets your pods use 11.9 GiB of it. The other 4 GiB go to kubelet reservations you never configured — and the managed services changed the rules recently, so most blog posts and calculators still show the old numbers.
I went through the EKS nodeadm source and the current GKE and AKS docs and computed allocatable CPU and memory for every instance type. Three things surprised me.

1. On EKS, prefix delegation quietly eats a gigabyte

EKS reserves memory per possible pod: 11 MiB × maxPods + 255 MiB. With the default VPC CNI, maxPods is the ENI limit — 17 on a t3.medium. Turn on prefix delegation (to fit more pods) and maxPods jumps to 110, so the reservation jumps too:

t3.medium, default CNI: 442 MiB reserved → 87% of RAM usable- t3.medium, prefix delegation: 1465 MiB reserved → 62% of RAM usable- m5.large: 92% → 81%On small nodes you pay a gigabyte for pod slots you will never fill, because the 4 GiB box runs out of memory long before pod 110. Prefix delegation pays off on big nodes (m6i.4xlarge goes from 95.5% to 97.6%), not small ones. ## 2. GKE 1.37 gave you up to 7 GiB per node back Until 1.36, GKE reserved a regressive share of memory: 25% of the first 4 GiB, 20% of the next 4, 10% of the next 8, 6% up to 128 GiB. From 1.37 on Container-Optimized OS it is min(old value, 15 MiB × maxPods + 500 MiB) — about 2.1 GiB with 110 pods, whatever the node size:
n2-standard-16 (64 GiB): 5.5 GiB reserved → 2.1 GiB- n2-standard-32 (128 GiB): 9.3 GiB reserved → 2.1 GiB, +7.2 GiB for podsIf you sized node pools on the old numbers, a node pool upgrade alone may let you drop nodes. The CPU reservation is also capped at 1 vCPU now. (Small print: shared-core e2-medium still reserves 1060m of its 2 vCPUs — 940m left.) ## 3. AKS with Azure CNI Overlay reserves for 250 pods by default Since 1.29 AKS reserves min(20 MB × maxPods + 50 MB, 25% of RAM). Azure CNI Overlay defaults to 250 max pods, so on a 16 GiB D4s v5 you hit the 25% cap:
D4s v5, overlay default (250 pods): 4 GiB reserved → 74% usable- D4s v5, maxPods 110: 2.2 GiB reserved → 86% usableThat is 1.8 GiB per node for one flag at node pool creation. You can’t change maxPods on an existing pool — you add a new pool and migrate. ## What I do now
Set maxPods to what the node can realistically run, not the maximum the CNI allows.- Size requests against kubectl describe node → Allocatable, minus DaemonSets — never against the instance spec.- Re-check after control-plane upgrades: the formula can change under you (GKE 1.37 did).I put the math for all ~2,900 AWS, GCP and Azure instance types into a free calculator: enter your pod requests and replica count, and it ranks node types by monthly cost after these reservations — nodesize.kalik8s.com. Each instance type has its own page, e.g. t3.medium on EKS. Sources for every formula are on the formulas page; if a number disagrees with your cluster, tell me and I’ll fix it. Which reservation bit you? I’m curious whether anyone measured the GKE 1.37 change on a real cluster yet. AI-assisted: numbers computed by a script I wrote with an AI coding assistant, checked against the sources above.

What My AI Broke This Week #1: Nothing. It Was Me, Three Times

Kalislav Smirnov — Thu, 08 Oct 2026 14:31:03 +0000

This week the agent broke nothing. I broke three things, and it spent the week politely proving it.

The setup: one mini-PC with Proxmox in the corner, a pile of side projects, Claude Code as the pair.

DNS is down (DNS was fine)

Saturday night, half past midnight. Nothing at home resolves, so I type the obvious: "something's wrong with DNS on the server, can you look?"

Three minutes later, the verdict. DNS was fine. One name didn't open, and it was the new monitoring dashboard. The reverse proxy had 11 hosts and that one wasn't among them, so TLS died with unrecognized name. The setup plan had a step 5, "add proxy host in the UI". Someone skipped it. (Me.)

The agent didn't add it either. It had no login to the proxy and refused to hand-edit the config, because the proxy rewrites that file from its own database. Correct, and slightly annoying, which is the right amount.

Then the real outage, the five minutes when nothing resolved. The logs:

04:31     Tailscale on the server changed its external addresses
04:35:09  51 DNS connections from the Mac dropped in the same second
          AdGuard: 4 days up, 0 restarts, 1 upstream timeout in 24 h

My Mac had "Use Tailscale DNS" switched on. So every lookup left the laptop, went through the tunnel to a relay in another country, came back through the subnet router, and landed on an AdGuard box three metres from my chair. When the relay hiccuped, DNS went with it. Every other device at home talked to AdGuard directly and never noticed.

What I'd tell another engineer: before you blame the DNS server, draw the path your query actually takes. Mine had a layover.

Out of memory (we had 16 GB)

Monday evening. "The server has almost no RAM left. Find what's eating it and what we can stop."

The agent stopped nothing. The host showed 25 of 31 GB used, with memory pressure at zero and 5.5 GB free. The number was inflated by config, and the config was mine.

VM            allocated   held on host   guest actually uses
side-project  8 GB        8.3 GB         1.7 GB
render box    6 GB        6.2 GB         0.5 GB   (idle since 2 Oct)
ZFS ARC       up to 3.1   3.0 GB

Every VM had balloon: 0. A guest fills its page cache once, after a render or a build, and never hands the memory back, so the host charges the full allocation forever. That 6 GB box renders video for a side project and had last done anything three days earlier.

The fix took five minutes: smaller allocations with ballooning on, ARC capped at 2 GB, a 7.8 GB zram swap. The host went from 25 GB used to 14, with 16 available. Then I said the words: "better commit and apply."

The plan wanted to create a VM that was already running. The side-project VM wasn't in Terraform state at all, so a full apply would have built a second copy on top of the live one, imported five LXC containers, rewritten the tunnel config and added four DNS records on the way. The agent added an import block, applied with -target on the two VMs it could account for, and got "No changes" on the next plan. The rest of the drift is still there, waiting for a calmer evening.

What I'd tell another engineer: state is a story the repo tells itself. Read the plan like it's lying to you.

It's the network (it was the uptime)

Some Claude Code sessions worked. Others died with ECONNRESET on every retry. I asked the working one to debug the broken one, which is a sentence I didn't expect to write.

The dead sessions had screenshots in them. Lots. One had 28 images, about 17 MB of base64, and Claude Code resends all of it on every turn. The healthy chats carried 2 to 9 images, around 3 MB. A curl test: 1 MB fine, 20 MB fine, 8 MB died mid-stream with an HTTP/2 error, 35 MB rejected with a 413. The network was fine. Multi-megabyte uploads sometimes weren't.

I asked whether we could report this to Anthropic and maybe get paid. The agent explained, kindly, that the bounty covers security bugs and this was a networking inconvenience. Then it found an open issue with the same picture (46 images), drafted a comment with our numbers, and posted it from my account after I said "go".

Last week I told everyone to write up their bugs instead of their feelings. Here I am doing it in a GitHub thread at nine on a Sunday morning, so at least I'm consistent.

Kalislav Smirnov

Oct 8

The job market is brutal. Your complaint thread won't fix it.

#discuss #career #writing #devops

4 min read

By lunchtime a bigger session died, half a million tokens of context, and /compact died with it, because compaction sends the same giant request. Another issue, another thread of Mac users, and what fixed it for them was a reboot. One had 52 days of uptime; a restart brought back all eight of his dead sessions, one of them at 826k tokens.

The agent then checked mine. 102 days. I had been quietly proud of that number.

What I'd tell another engineer: an agent can prove the network innocent ten different ways. It still won't tell you to turn it off and on again, because you taught it not to be rude.

Your turn

What did your AI pair spend this week proving wasn't its fault, and was it right?

The job market is brutal. Your complaint thread won't fix it.

Kalislav Smirnov — Thu, 08 Oct 2026 13:55:08 +0000

Four of this week's front-page posts are about the job search. One calls the market disgusting. I'm in the same market, applying for DevOps and platform roles, so this isn't shouted from the shore.

The market really is bad

22% fewer software development postings on Indeed in the US than in February 2020 (index at 78 on 2 October, FRED).
61% of job seekers ghosted after an interview; 18 to 22% of jobs on Greenhouse's own platform classified as ghost jobs in any quarter (Greenhouse, 2024).
37% of job seekers say they apply to more jobs than ever and hear back less; 73% of HR professionals say under half of applications meet the listed criteria (LinkedIn, January 2025).
22% of candidates use bots to apply automatically (Greenhouse, 2025); Greenhouse's own explanation for the ghosting is employers drowning in AI applications.
One dev.to author sent 59 applications in 14 days. 51 got no answer.

Nobody who has seen these numbers gets to say "try harder". The market is unfair, and good engineers got laid off for reasons unrelated to their work.

The post doesn't help you either

I pulled this week's dev.to posts under career and jobsearch and kept the 43 with "job", "interview", "market", "rejected" or "ghost" in the title.

This week on dev.to	Posts	Zero reactions	10+ reactions
Job-search posts	43	31	2
`devops`, `kubernetes`, `sre`, `terraform` posts	292	180	23

So yes, most of everything gets ignored, mine included. The difference is what happens after. An ignored Terraform post still answers someone's search in six months and still sits under your name when a recruiter looks. An ignored rant does neither. And the two job-search posts that did break ten reactions weren't rants. (The "disgusting" post is tagged jobmarket, so it isn't in the table; it had 23 reactions the morning after. Fair enough. That's still sympathy from people who can't hire you.)

And look who's in the thread. Other job seekers. The hiring manager isn't, and if she wanders in she learns that you're frustrated, which she assumed.

What a stranger sees

Picture a recruiter with your name in a search box and four minutes before the next call. She finds three posts about ghosting and nothing you built. Next tab.

Same recruiter, same four minutes, and the first result is a README that explains one decision. She reads it. Maybe she writes. That's the gap between "unread" and "read", the only gap a post of yours can close.

It doesn't have to be impressive

It has to be specific. My home lab is one mini-PC running Proxmox, which is nothing. The README line I'd point a reader at says that changing a container's network interface through the Terraform provider restarts the container even though the plan shows an in-place update. One sentence, and it says more about how I work than a cover letter.

An incident write-up with a timeline. A comment that politely corrects someone's YAML. A write-up from March still gets found in October; a rant from March is gone by Tuesday.

One more Greenhouse number: 28% of candidates admit to using AI to generate fake work samples. A repo with a commit history and a README that explains why beats a polished sample because it's hard to fake.

Where this stops working

No time: skip the homelab. One write-up of a problem from your last job, names filed off, thirty minutes. Less than the forms nobody reads.

Burned out: rest, and ignore everyone telling you to build in public, me included. This is for people who still have the energy for a thousand-word complaint.

Visa, location, a role that doesn't exist in your city: proof of work gets you read. It doesn't get you past a wall the hiring side can't move either.

The ask

Keep writing. Change the ratio. For every post about the search, two about the work: the bug, the migration, the thing that broke on Friday. If the market is still like this in a year, you'll have a year of visible work instead of a year of visible frustration. Only one of those is something a hiring manager can act on.

Tell me I'm wrong in the comments. Or better: drop the one write-up you'd want a recruiter to find first. I'll read every one.

Sources

FRED, "Software Development Job Postings on Indeed in the United States" (IHLIDXUSTPSOFTDEVE), index Feb 1 2020 = 100, observation 2 October 2026. https://fred.stlouisfed.org/series/IHLIDXUSTPSOFTDEVE
Greenhouse, "Ghosting, ghost jobs and bots: Candidates reveal their top challenges in the Greenhouse 2024 State of Job Hunting report", 10 December 2024. https://www.greenhouse.com/blog/greenhouse-2024-state-of-job-hunting-report
Greenhouse, "Fighting bias, bots and burnout: Global job market trends from the Greenhouse 2025 Workforce & Hiring Report". https://www.greenhouse.com/blog/greenhouse-2025-workforce-hiring-report
LinkedIn, "More Applicants But Harder To Hire: Majority Of Job Applications Do Not Meet Listed Criteria, Finds LinkedIn", 15 January 2025. https://news.linkedin.com/2025/talent-campaign-b2b-press-release
"I Applied to 59 Tech Jobs in 14 Days. Here's What Actually Happened", dev.to, 1 October 2026. https://dev.to/mikachu/i-applied-to-59-tech-jobs-in-14-days-heres-what-actually-happened-46lp
dev.to API, /api/articles?top=7&tag=..., pulled 8 October 2026; counts in the table are from that pull.
My home lab repository, README note on Terraform and container network interfaces. https://github.com/kalikys/homelab

A denylist let 46 of 75 prompt injections through. Capabilities let 3.

Kalislav Smirnov — Sat, 03 Oct 2026 16:40:44 +0000

Two documents appeared on 8 September. An arXiv paper measures what happens when a coding agent reads a poisoned repository under four permission models. A Google threat intelligence report describes malware that, inside GitHub Actions, reads the runner's OIDC token out of memory and publishes packages with valid SLSA Build Level 3 attestations. Together they say something uncomfortable about how most teams give agents permissions today, which is by writing sentences.

My position: in a pipeline, an agent's permissions are whatever the runner job holds. A rule in CLAUDE.md that says "do not push to main" is a request to a reader, and the attacker can write to that file too. If the agent job holds id-token: write, the agent can publish. If it holds actions: write, it can delete the logs that would show it did. Permission has to be a property of the job, fixed before the agent reads a single untrusted byte. This is commentary on the two documents linked below; I have not reproduced either.

What the Google report shows

Most of GTIG's "From Prompting to Autonomy" is about attackers using agents. The part that belongs on an operations desk is the section on UNC6780, which has been compromising PyPI, npm and Docker Hub packages since March 2026, and its credential stealer DUSTMAKER.

The malware checks where it is: "DUSTMAKER samples contain functionality to detect when it is running in a continuous integration and continuous delivery (CI/CD) environment. If confirmed, it extracts OIDC tokens from the process memory of GitHub Actions runners." With those tokens it "authorizes itself as a trusted publisher and publishes compromised versions of packages with valid, cryptographically signed SLSA Build 3 attestations", and such packages "will pass AI coding agent automated trust checks."

It hides where agents look: it "drops or modifies malicious files into hidden project workspace directories for AI coding assistants and integrated development environments (IDEs) (.claude/, .vscode/, .cursor/, etc.)" and uses them "to instruct the AI assistant to run arbitrary commands or scripts". In Actions it "creates malicious pipeline tasks disguised under AI-themed names such as 'Copilot Setup'" and "issues automated API calls to delete the workflow execution logs."

Last week I wrote that signature verification has to live in the harness. This report is the other half: a signature is evidence about who held the token, and when the token came out of your own runner the attestation is valid and tells you nothing.

The same report describes an actor who ran a credential-harvesting campaign in under six hours from "an AI coding chatbot, a prompt, and a set of agent instructions", with "preconfigured markdown instruction sets as operational playbooks." Their agents are configured the way ours are.

What it does not show

I could not find a count of repositories hit by DUSTMAKER or what the stolen tokens were scoped to when used. "Thousands of third-party credentials" is a magnitude, and the victim is unnamed. So I cannot tell you how likely this is to hit a given pipeline, only that it works.

What the paper shows

Bouras, Dai and Mechtaev, "Authority Is Not a String" (8 September 2026, accepted at LMPL '26) built a harness they call CapScope for the Pi coding agent. Before the agent reads any repository content, a preflight step looks only at the trusted user request and the file tree and sets a ceiling of typed capabilities, read, write and exec, each scoped to a path or argument prefix. The capabilities live in a host-side store the model cannot see or edit. Every tool call is checked against the store of the agent that proposed it, compound commands segment by segment. Sub-agents get a subset: the test runner gets exec, the patcher gets write, and a write proposed by the runner is refused even when it would be useful.

They ran five Python repair tasks with injections in README, AGENTS.md, skill files, source comments and tool output. 300 runs. Under ambient authority the injected effect executed in 47 of 75 runs. Under a static denylist of the familiar hazards (.env, .ssh, recursive delete, sudo, curl) it executed in 46 of 75. A policy written for the specific task, shared by all agents: 33 of 75. CapScope: 3 of 75. Repairs still completed in 68 of 75 runs, against 72 with no protection at all.

The denylist number is the one I keep coming back to. That is the model most teams actually run, a list of things the agent must not do, written once and applied everywhere. It stopped one more attack than nothing, because the injected target was an ordinary source path no deny pattern matches.

What the paper does not show

It is small: five tasks, one language, one agent, no CI/CD setting. Mean wall-clock time was 316 seconds per run against 145 without the harness, five runs hitting a 900-second timeout because the model kept proposing the refused call. The authors draw the boundary themselves: "A permitted read followed by a permitted write can still move data, and an injection can misuse authority deliberately granted to its reader. Those cases require complementary information-flow controls or sandboxing." Running pytest still imports untrusted project code. The 3 of 75 is a result about authority, and says nothing about confidentiality.

Whether this matters in two years

I think the capability model wins, as a product default rather than something teams build. GitHub's own agentic workflows architecture already describes it: "the agent job runs with minimal read-only permissions, while write operations are deferred to separate jobs", with outputs buffered as artifacts and applied by a job holding issues: write or contents: write, and egress through a proxy with a domain allowlist. That is CapScope's split with CI job boundaries doing the enforcement. In two years I expect every hosted agent runner to look like this, and instruction files to remain as a convenience for the agent and nothing more.

The DUSTMAKER case will not go away, because it does not need the agent to misbehave. It needs a runner that holds a publishing token while executing repository code, which was a pipeline design problem before agents arrived.

Monday

On a secure-development platform I ran Gitleaks, Trivy and SCA in every pipeline through shared GitLab CI templates. The template is where an agent job's permissions belong too, because the platform team owns it and the repository the agent is about to read does not.

Give the token to the job that needs it. GitHub's documentation says to set id-token: write inside the single job that fetches the token rather than at workflow level; GitLab's id_tokens is a job keyword. The agent job should not have it. Publishing happens after review, in a different job, on a different runner.

Separate reading from writing. If the agent runs with contents: read and emits a patch as an artifact, a later job with write permission applies it. Whatever the agent is told, it cannot push, publish or approve.

Put .claude/, .cursor/, .vscode/, AGENTS.md and CLAUDE.md under CODEOWNERS and review them like pipeline YAML, because that is what they are now. In CI, load them from the protected branch or not at all.

Alert on log deletion. It needs write access to Actions; the agent's token should not have it, and a deletion in the audit log should page someone.

Use ephemeral runners. Google's July mitigation guidance asks for "ephemeral runners for build pipelines that are purged immediately after completing a single task" and for federated credentials that "expire in a matter of minutes". A token read from memory is worth less when the process that held it is already gone.

What would change my mind

A replication of CapScope on a real CI workflow with a publishing step, showing the preflight ceiling is either too wide to matter or too narrow to let work through. Or incident data showing job-scoped, short-lived OIDC tokens abused at a rate close to long-lived PATs, which would mean the job boundary is not the control I think it is.

What does the agent job in your pipeline hold today, and if the answer is "the same as the rest of the workflow", what is stopping it from publishing?

Sources

Google Threat Intelligence Group. "From Prompting to Autonomy: The Evolution of Adversarial AI." 8 September 2026. https://cloud.google.com/blog/topics/threat-intelligence/from-prompting-to-autonomy-the-evolution-of-adversarial-ai
Dimitrios Stamatios Bouras, Yihan Dai, Sergey Mechtaev. "Authority Is Not a String: A Capability-Scoped Harness for Prompt-Injection-Resistant Coding Agents." arXiv:2609.08371, 8 September 2026. https://arxiv.org/abs/2609.08371
GitHub Agentic Workflows, architecture page. https://github.github.com/gh-aw/introduction/architecture/
Google Threat Intelligence Group. "Mitigation guidance for supply chain compromise." 31 July 2026. https://cloud.google.com/blog/topics/threat-intelligence/mitigation-guidance-for-supply-chain-compromise
GitHub Docs, OpenID Connect reference (id-token: write per job). https://docs.github.com/en/actions/reference/security/oidc
GitHub Docs, permissions required for fine-grained tokens (delete workflow run logs: Actions, write). https://docs.github.com/en/rest/authentication/permissions-required-for-fine-grained-personal-access-tokens
GitLab Docs, CI/CD YAML id_tokens. https://docs.gitlab.com/ci/yaml/#id_tokens

An SBOM is a tool output, not a fact about your software

Kalislav Smirnov — Sat, 03 Oct 2026 16:40:20 +0000

Two papers appeared on arXiv in September that describe, from opposite ends, a pipeline control I have shipped many times. The first, from Inria and ANSSI, ran three SBOM generators over the same 3,326 repositories and found that two of them miss more than half of the dependencies in JavaScript projects. The second ran 1,920 trials of AI coding assistants installing software and found that they opened an SBOM, signature or attestation before installing in 0.5% of trials and verified one in none.

My position: an SBOM generated in CI is evidence about the tool you ran, at the version you ran it, under the flags you set. It is weak evidence about the software. That still leaves it useful, but the thing to put under version control and defend in an audit is the generation recipe. The component count on its own means little.

This is commentary on those two papers, linked where used; I have not reproduced them.

What the Inria paper shows

[Prado, Zendra, Boinot and Barais](https://dub.sh/2DBS8sB (17 September 2026, accepted at SCORED 2026) took 2,050 JavaScript and 1,276 Rust repositories with over 1,000 GitHub stars, regenerated a fresh lockfile for each, and used that lockfile as ground truth. They ran Syft 1.38.2, Trivy 0.68.2 and cdxgen 12.0.0, all producing CycloneDX 1.6, and compared.

With all lockfiles exposed, Syft reported 43.78% of the lockfile packages on JavaScript and Trivy 32.71%. On Rust, Syft reached 100% and Trivy 83.85%. cdxgen was at 98.72% and 99.34%. With no lockfile at all, Syft and Trivy produced an empty SBOM, and the paper is specific that this happens "with no error or warning shown to the operator."

The useful part is why. Most of the JavaScript gap is devDependencies. Syft excludes them by design, but only for some package managers: in one npm project it reported 2249 of 2249 production packages and 0 of 4930 dev packages, while in a Yarn and a pnpm project it reported both at 100%. Trivy excludes them by default too, but the exclusion stops at the first level, so a transitive dependency of a dev package can show up attached to the root as if it were a top-level production dependency. The authors call Syft's behaviour a design choice applied inconsistently and Trivy's an implementation error. cdxgen's small residual gap was a hard-coded list that skips any directory named examples, along with docs, tests and .github.

The same package also comes out with a different identifier depending on the tool. An npm alias is reported under the alias by cdxgen and Trivy and under the real scoped name by Syft. pnpm peer-dependency suffixes end up in the version string for cdxgen and Syft. One git-pinned dependency came back as a commit hash from one tool, a tarball URL from the second and a semantic version from the third. Vulnerability matching works on these identifiers, so the same code scanned with two tools can produce two different vulnerability lists.

The Supplier element, one of NTIA's minimum elements since 2021, is populated by none of the three tools on either ecosystem. Licenses: 75.8% for cdxgen, 60.2% for Syft, 0% for Trivy on JavaScript, and 0% for all three on Rust. Hashes come almost only from cdxgen, CPEs almost only from Syft.

What it does not show

The sample is popular GitHub repositories, and the authors say they did not measure how the 1,000-star threshold affects coverage. The results are tied to three specific versions; the authors note that a 2026 study using Trivy 0.66.0 and Syft 1.33.0 got empty SBOMs for package-lock.json, and a few releases later the same tools produce partial ones. Trivy 0.75.0 and Syft 1.54.0 both shipped on 1 October, so some of these numbers may already be stale. The field-completeness measurement checks whether a field is present, not whether its content is right. And the paper covers JavaScript and Rust only; I would not assume Java or Python look better.

What the second paper adds

[Pengyin Shan's pre-registered audit](https://dub.sh/vp0XYGh (7 September 2026) asks the other question: when the thing installing software is an AI assistant, does it read any of this? Six research software projects, nine variants each (no signal, a valid SBOM, a valid signature, a valid attestation, two with wrong-issuer material, one with every signal, one with contradictory metadata), three models, two harnesses, every trial in a network-less container with file access and commands logged, protocol deposited with a DOI before the first trial.

In 9 of 1,920 trials the assistant opened a provenance file before installing. In 0 of 2,114 trials, including a frontier-model supplement, did any assistant run a verification command. Signal presence had no measurable effect (p = 0.50). The model that cost $1.00 per trial verified nothing in 50 trials; the only one that opened signals more than once cost about $0.10 per trial and did it in 9 of 360. The author's conclusion is that verification "has to be built into the program that runs the assistant."

The limits are real: six projects, research software, a container with no package index so most installs failed at dependency resolution, and material signed by the author's own key rather than the upstream's. It measures the decision to install, not a successful install. Still, zero verification commands across six models and two harnesses is hard to explain away with sample size.

Where this goes

The CRA's main obligations apply from 11 December 2027, with reporting obligations from 11 September 2026, [per the Commission's own page](https://dub.sh/iIYkCB9 Most manufacturers will discharge that with one of these three tools, and the Inria authors put it bluntly: tool choice "is no longer just an engineering decision, it is a compliance-exposure question."

I think SBOMs matter in two years as an inventory you can defend, and I doubt they will matter as a trust signal anyone checks at install time. The second paper shows the install-time consumer is already often an agent, and the agent does not look. The first shows that a human who looks cannot tell from the document whether it is complete. The fix for the first is standards work: the authors ask CycloneDX and SPDX to define which dependency scopes are in, how provenance is recorded, and what the root of the graph is, which will take years. The fix for the second is a harness that runs cosign verify or slsa-verifier before pip install.

Monday

I have run security checks (Gitleaks, Trivy, SCA) in every pipeline of a secure development platform through shared GitLab CI templates, and I have delivered signed packages into an air-gapped bank, where the receiving side cannot regenerate anything and works with what is in the bundle. This is what I would change this week.

Pin the generator and write its version into the SBOM metadata. The paper's results changed between minor releases of the same tool. If an auditor asks why a component is missing, "Trivy 0.68.2 with default flags on a Yarn project" is an answer. "We run Trivy" is not.

Fail the job on an empty SBOM. Syft and Trivy return zero components and exit 0 when there is no lockfile. A jq '(.components // []) | length' with a threshold is five lines in a shared template and catches the case the paper calls more severe than under-reporting.

Decide the dev-dependency question on purpose. Trivy's --include-dev-deps exists; the default excludes them, and the exclusion leaks. For a deployment SBOM you may want them out. For a view of your build's supply chain you want them in, since the paper notes development dependencies have themselves been targets of supply-chain attacks. Write the choice down.

Commit the lockfile, or generate the SBOM from the built artifact instead of the source tree. Without a resolved lockfile the tools are either guessing or silent.

Run two generators once, on your five most important repositories, and diff the component sets. An afternoon of that tells you which scope and identifier rules bite your stack, before you promise anyone a complete inventory.

For air-gapped delivery, put the SBOM inside the signed bundle together with the generator name, version and command line, and have the receiving side verify the bundle signature. The SBOM records what you believe is in there; the signature records who is accountable if that belief is wrong.

If an AI agent sits anywhere in your dependency flow, put the verification step in the pipeline that runs it, because the model will not do it on its own.

What would change my mind

A benchmark that pins these edge cases, which the Inria authors say they are building, with Syft and Trivy above 95% on JavaScript. A CycloneDX or SPDX release that defines dependency scope as a standard attribute, so "complete" means one thing across tools. Or a re-run of the agent study in which a production harness verifies attestations by default and leaves a log.

Sources

Alan Prado, Olivier Zendra, Philippe Boinot, Olivier Barais. "Mind the Gap: How SBOM Specification Ambiguities Lead to Divergent Software Bills of Materials. An Empirical Tool Study." arXiv:2609.19920, 17 September 2026. https://dub.sh/ca1sWHN
Pengyin Shan. "Do AI Coding Assistants Check Before They Install? A Pre-Registered Demand-Side Audit of Trust Signals in the Research Software Supply Chain." arXiv:2609.07754, 7 September 2026. https://dub.sh/QX6af62
European Commission, "Cyber Resilience Act" policy page (entry into force and application dates). https://dub.sh/mkcsQtY
Trivy documentation, Node.js coverage (--include-dev-deps). https://dub.sh/qeU8iuX
Release pages checked 2 October 2026: https://dub.sh/274rekh and https://dub.sh/bSvz3pX

An SBOM is a tool output, not a fact about your software

Kalislav Smirnov — Fri, 02 Oct 2026 15:06:25 +0000

This is commentary on those two papers, linked where used; I have not reproduced them.

What the Inria paper shows

Prado, Zendra, Boinot and Barais (17 September 2026, accepted at SCORED 2026) took 2,050 JavaScript and 1,276 Rust repositories with over 1,000 GitHub stars, regenerated a fresh lockfile for each, and used that lockfile as ground truth. They ran Syft 1.38.2, Trivy 0.68.2 and cdxgen 12.0.0, all producing CycloneDX 1.6, and compared.

What it does not show

What the second paper adds

Pengyin Shan's pre-registered audit (7 September 2026) asks the other question: when the thing installing software is an AI assistant, does it read any of this? Six research software projects, nine variants each (no signal, a valid SBOM, a valid signature, a valid attestation, two with wrong-issuer material, one with every signal, one with contradictory metadata), three models, two harnesses, every trial in a network-less container with file access and commands logged, protocol deposited with a DOI before the first trial.

Where this goes

The CRA's main obligations apply from 11 December 2027, with reporting obligations from 11 September 2026, per the Commission's own page. Most manufacturers will discharge that with one of these three tools, and the Inria authors put it bluntly: tool choice "is no longer just an engineering decision, it is a compliance-exposure question."

Monday

Commit the lockfile, or generate the SBOM from the built artifact instead of the source tree. Without a resolved lockfile the tools are either guessing or silent.

If an AI agent sits anywhere in your dependency flow, put the verification step in the pipeline that runs it, because the model will not do it on its own.

What would change my mind

Sources

Alan Prado, Olivier Zendra, Philippe Boinot, Olivier Barais. "Mind the Gap: How SBOM Specification Ambiguities Lead to Divergent Software Bills of Materials. An Empirical Tool Study." arXiv:2609.19920, 17 September 2026. https://arxiv.org/abs/2609.19920
Pengyin Shan. "Do AI Coding Assistants Check Before They Install? A Pre-Registered Demand-Side Audit of Trust Signals in the Research Software Supply Chain." arXiv:2609.07754, 7 September 2026. https://arxiv.org/abs/2609.07754
European Commission, "Cyber Resilience Act" policy page (entry into force and application dates). https://digital-strategy.ec.europa.eu/en/policies/cyber-resilience-act
Trivy documentation, Node.js coverage (--include-dev-deps). https://trivy.dev/latest/docs/coverage/language/nodejs/
Release pages checked 2 October 2026: https://github.com/aquasecurity/trivy/releases and https://github.com/anchore/syft/releases